Profile    Mohammed Shiroz Status   Loading  
Logo
Share This
Back to blog
Filter by:
Tags
//Article title

Queues and Message Brokers Explained: Redis, RabbitMQ and SQS Without the Jargon

About Post

A user uploads a signed contract. Your controller saves it, generates a PDF, emails three people, pushes a notification to the mobile app and updates a report. The user stares at a spinner for eight seconds, gives up, and taps the button again.

Nothing in that list was slow on its own. The mistake was doing all of it while the user waited. That's the problem queues solve, and once you understand them properly, a lot of "mysterious" production behaviour starts to make sense.

Why queue anything at all?

A queue is a to-do list that sits between two parts of your system. One side writes tasks onto it, the other side picks them up and does the work later, usually within milliseconds or seconds.

Think of a busy restaurant. The waiter doesn't walk into the kitchen and cook your meal. They write the order on a ticket, pin it to the rail, and go back to the tables. The kitchen works through the tickets at its own pace. If ten orders arrive at once, the rail gets longer, but nobody at the tables is kept waiting at the door.

That gives you three things:

  • Fast responses. The request only has to save the important bit and drop a ticket on the rail.
  • Absorbed spikes. A burst of traffic becomes a longer queue, not a crashed server.
  • Retries for free. If the email provider is down for a minute, the job fails, waits, and tries again. The user never sees it.

Producers, consumers and the broker in the middle

The vocabulary is simpler than it sounds:

  • The producer puts a message on the queue. In Laravel, that's your code calling SendContractEmail::dispatch($contract).
  • The consumer (or worker) takes messages off and processes them. In Laravel, that's php artisan queue:work running under Supervisor or systemd.
  • The broker is the thing that stores the messages in between: Redis, RabbitMQ, Amazon SQS, or even your database.

The producer and the consumer never talk to each other directly. That's the whole point. You can restart workers, add more of them, or deploy new code, and the messages wait patiently on the broker.

The part people skip: at-least-once delivery

Here's the thing that bites people in production. Almost every queue you'll use promises at-least-once delivery. Not exactly once. At least once.

Why? Picture a worker that picks up a job, sends the email, and then crashes before it can tell the broker "done". The broker never got the acknowledgement, so after a timeout it assumes the worker died and hands the message to someone else. The email goes out twice.

No broker can fully fix that, because the crash happened in your code, between the side effect and the acknowledgement. So the fix has to live in your code too: make your jobs idempotent. Running the job twice should have the same effect as running it once.

public function handle(): void
{
    // Already done? Then this is a duplicate delivery. Skip it.
    if ($this->invoice->receipt_sent_at !== null) {
        return;
    }

    Mail::to($this->invoice->tenant)->send(new RentReceipt($this->invoice));

    $this->invoice->update(['receipt_sent_at' => now()]);
}

This is simplified (there's still a tiny window between sending and saving), but it turns "duplicate on every retry" into "duplicate only if the worker dies at exactly the wrong moment". For payments, use a unique constraint or an idempotency key so the database itself refuses the second attempt.

One Laravel-specific trap: the retry_after value in config/queue.php must be longer than your longest job's timeout. If a job takes 120 seconds and retry_after is 90, the broker decides the job is lost and gives it to a second worker while the first one is still busy. Now you have two workers doing the same job, and no crash was involved at all.

Dead letter queues: where bad messages go

Some messages will never succeed. The record was deleted, the payload is malformed, or the code has a bug. Without a limit, they'd be retried forever, clogging the queue and filling your logs.

A dead letter queue (DLQ) is a separate queue where a message goes after it has failed a set number of times. It stops the retry loop and keeps the message so a human can look at it, fix the cause and replay it.

  • In SQS, you attach a redrive policy with a maxReceiveCount.
  • In RabbitMQ, you configure a dead letter exchange on the queue.
  • In Laravel, the failed_jobs table plays this role. Set $tries and $backoff on the job, and use php artisan queue:failed and queue:retry to inspect and replay.

The rule: a DLQ nobody looks at is just a slower way of losing data. Alert on it. A failed job count that's growing is one of the most useful signals you can have.

Redis, RabbitMQ or SQS?

All three move messages from A to B. They differ in what they're good at and what they cost you to run.

RedisRabbitMQAmazon SQS
What it isIn-memory data store that also does queuesA dedicated message broker (AMQP)A fully managed queue service
Best atSimple, fast job queues, especially if you already run Redis for cacheRouting: one message to many queues, topic patterns, prioritiesZero maintenance, scales without you thinking about it
You manageThe Redis server, memory and persistence settingsThe broker, clustering, upgradesAlmost nothing, but you're tied to AWS
Watch out forMemory limits; persistence settings decide what survives a restartMore concepts to learn (exchanges, bindings, acks)Visibility timeouts; standard queues can deliver out of order
Laravel supportBuilt in, plus Horizon for monitoringVia a community packageBuilt in

How I'd choose

For a typical Laravel app, my default is simple:

  1. Start with the database driver if you have low volume and want zero new infrastructure. It's slower, but it's honest and easy to debug.
  2. Move to Redis when you want speed and visibility. Laravel Horizon gives you a dashboard, metrics and failed-job views with almost no setup.
  3. Pick SQS if you're already on AWS and don't want to babysit another server. It's a very good fit for spiky workloads.
  4. Reach for RabbitMQ when the problem is routing: several services that each need their own copy of an event, or messages that go to different consumers based on type. That's where it really shines.

Kafka deserves a mention too, but it's a different animal: a distributed log you can replay, not a to-do list. If you need event streaming and history, look there. If you need "send this email in the background", you don't.

If you remember one thing

  • Queue anything the user doesn't need to wait for.
  • Assume every message can arrive twice, and write jobs that don't care.
  • Keep retry_after longer than your slowest job.
  • Give failures somewhere to go, and watch that place.
  • Choose the broker for the problem: Redis for simple speed, SQS for no ops, RabbitMQ for routing.

The Laravel queues documentation is worth a full read once; most of the settings above are covered there.

What's the first job you moved to a queue in your app, and what made you finally do it?

Comments (0)
Leave your review

Thanks for your valuable comments. Your comments has been updated and appreciate your getting in touch...

01. About Shiroz

Mohammed Shiroz

Hi, I'm Mohammed Shiroz, a software engineer and AI enthusiast from Sri Lanka who turns ideas into intelligent, real-world solutions. With over 9 years of hands-on experience, I currently lead real estate ERP development at Kate Group, a...

03.My Projects

04. Categories

Ready To order Your Project ?

Get in Touch
Close