A message queue is one of the most important building blocks in distributed system design.
At a basic level:
Producer
|
v
Message Queue
|
v
ConsumerThe producer creates a message.
The queue temporarily stores the message.
The consumer processes it.
This simple mechanism solves several major system-design problems:
Message queues are especially important when one service should not have to wait for another service to finish expensive work.
Consider an application:
User
|
v
Order API
|
v
Payment Service
|
v
Email ServiceSuppose the email service takes 2 seconds.
If the Order API waits for it:
User
|
v
Order API
|
v
Payment
|
v
Email
|
v
ResponseThe user may wait unnecessarily.
Instead:
User
|
v
Order API
|
+----→ Database
|
+----→ Queue
|
v
Email WorkerThe API can return after the important transaction is accepted.
The email can be sent asynchronously.
A → B → responseA waits for B.
Example:
Browser → API → DatabaseA → Queue
|
v
BA does not need to wait for B to finish.
Example:
Order API
|
v
Queue
|
v
Notification WorkerSuppose a user uploads a video.
The upload API should not wait for transcoding.
Bad architecture:
Upload API
|
v
Upload file
|
v
Transcode video
|
v
Generate thumbnails
|
v
Return responseBetter:
Upload API
|
v
Store video
|
v
Queue
|
v
Video WorkerThe user gets a fast response while processing continues in the background.
Important terms:
| Term | Meaning |
|---|---|
| Producer | Creates messages |
| Consumer | Processes messages |
| Queue | Stores messages waiting for processing |
| Message | Unit of work/data |
| Broker | System that manages messages |
| Worker | Process that consumes and executes work |
| Acknowledgement | Confirmation that processing completed |
| Retry | Attempt to process failed work again |
| Dead-letter queue | Stores messages that repeatedly fail |
| Visibility timeout | Temporary period during which a received message is hidden |
| Partition | Ordered subset of messages in some queue systems |
| Consumer group | Set of consumers sharing work |
A producer sends a message.
Example:
{
"type": "SEND_EMAIL",
"userId": "123",
"template": "welcome"
}The producer might be:
A consumer receives messages and performs work.
Example:
Queue
|
+--> Worker 1
+--> Worker 2
+--> Worker 3The workers may process:
SEND_EMAIL
GENERATE_THUMBNAIL
PROCESS_PAYMENT
GENERATE_REPORTOne of the most important mental models:
A queue absorbs temporary differences between production rate and processing rate.
Suppose:
Incoming work = 10,000 messages/minute
Processing = 8,000 messages/minuteThe queue absorbs the difference:
10,000 in
|
v
+----------------+
| QUEUE |
+----------------+
|
v
8,000 processedQueue depth grows.
If the traffic later drops:
Incoming = 5,000/minute
Processing = 8,000/minuteWorkers can drain the backlog.
Without a queue:
Service A ─────→ Service BWith a queue:
Service A ─────→ Queue ─────→ Service BThe queue introduces an intermediate durability and coordination layer.
This provides:
But also introduces:
Queues are not automatically better.
Use them when asynchronous decoupling provides value.
A database stores structured application state.
A queue represents work or messages that need processing.
Database:
Users
Orders
Products
PaymentsQueue:
Send email
Generate invoice
Resize image
Process payment eventA database can sometimes implement a work queue, but a dedicated queue system usually provides better semantics and operational behavior for high-volume asynchronous workloads.
A production message often contains:
{
"messageId": "msg_123",
"type": "IMAGE_RESIZE",
"version": 1,
"createdAt": "2026-10-01T10:00:00Z",
"traceId": "trace_456",
"attempt": 1,
"payload": {
"mediaId": "media_789"
}
}Useful fields include:
Every message should normally have a unique identifier.
Example:
msg_123456Why?
Because duplicates can happen.
A consumer can store:
processed_message_idor use an idempotency key.
Then:
same message arrives again
|
v
already processed?
|
YES
|
v
skip duplicate workAn acknowledgement tells the queue:
The consumer has successfully processed this message.
Conceptually:
Queue
|
| deliver
v
Worker
|
| process
v
Success
|
| ACK
v
Queue removes/completes messageThe important point is:
Do not acknowledge work before the required processing has actually succeeded.
Suppose:
Queue → WorkerThe worker crashes before finishing.
If the message has already been permanently removed, the work may be lost.
A safer model is:
Queue
|
v
Worker
|
+---- crash
|
v
message becomes available againThis is one reason acknowledgement semantics matter.
Some queue systems use a visibility timeout.
Flow:
Message visible
|
v
Consumer receives
|
v
Message temporarily hidden
|
+---- success → delete/ack
|
+---- crash → timeout expires → visible againExample:
Visibility timeout = 60 secondsIf the worker crashes, the message may become available after approximately that period.
If the timeout is too short:
worker processing = 5 minutes
timeout = 1 minuteThe message may reappear while the first worker is still processing it.
Now two workers may process the same message.
If the timeout is too long:
worker crashes
timeout = 1 hourThe retry is delayed.
Therefore timeout selection should consider:
normal processing time
+
worst-case expected processing timeSome systems allow the worker to extend the visibility timeout.
Three important concepts:
At-most-once
At-least-once
Exactly-onceUnderstanding these is essential for interviews.
The message is delivered zero or one time.
Message
|
+---- deliveredA message can be lost.
Advantage:
no duplicate processingDisadvantage:
possible message lossUseful when occasional loss is acceptable.
Example:
non-critical metricsThe message is guaranteed to be delivered eventually under the system's assumptions, but it may be delivered more than once.
Message
|
+--> Worker
|
+--> Worker againThis is common in distributed systems.
Therefore consumers should be idempotent.
The goal is:
message processed exactly onceThis is much harder than it sounds.
Network failures can create ambiguity.
For example:
Worker processes payment
|
v
Payment succeeds
|
X
ACK lost
|
v
Queue retries messageThe queue does not know whether the payment succeeded.
So the same message may be delivered again.
A useful distinction:
exactly-once message deliveryis different from:
exactly-once business effectA system may receive a message twice but still ensure the business operation happens once.
Example:
Message ID = msg_123
First attempt:
payment recorded
Second attempt:
msg_123 already processed
→ skipThis is often called effectively-once processing.
An operation is idempotent if repeating it produces the same intended final result.
Example:
SET user.status = "active"Running it twice produces:
activeRunning it multiple times does not change the intended final state.
Non-idempotent example:
balance = balance + 100Running twice adds 200.
Therefore financial operations often require explicit idempotency controls.
Example:
idempotency_key = payment_123Database:
payment_123 → SUCCESSIf the same request/message arrives again:
Does payment_123 exist?
|
YES
|
v
Return existing resultThis prevents duplicate business effects.
Failures are inevitable.
A worker might fail because:
Retrying may solve transient failures.
Not every error should be retried.
Retryable:
HTTP 429
HTTP 503
network timeout
temporary database failureUsually not retryable:
invalid input
unsupported file type
authorization failure
malformed messageRetrying a permanent error wastes resources.
Instead of retrying immediately:
retry
retry
retry
retryuse increasing delays:
1 second
2 seconds
4 seconds
8 seconds
16 secondsFormula conceptually:
delay = base × 2^attemptIn production, cap the maximum delay.
If thousands of workers retry at exactly:
10:00:00they may overload the recovering dependency again.
This creates a retry storm.
Add randomness:
base delay + random jitterExample:
worker A → 4.2s
worker B → 4.8s
worker C → 5.1sNow retries are spread over time.
A message that repeatedly fails should not retry forever.
Example:
Main Queue
|
v
Worker
|
+---- failure
|
v
Retry
|
v
Retry
|
v
Retry
|
v
Dead-Letter QueueThe DLQ allows engineers to inspect problematic messages.
A poison message is a message that repeatedly causes consumer failure.
Example:
{
"type": "CREATE_INVOICE",
"amount": "INVALID"
}If the worker cannot process it:
retry foreverwould be harmful.
Instead:
limited retries
→ DLQA DLQ should not simply become a garbage dump.
Monitor:
DLQ message count
age of oldest DLQ message
failure reasonsEngineers can:
Sometimes message order matters.
Example:
Account created
Account updated
Account deletedCorrect order:
CREATE
UPDATE
DELETEIncorrect order:
DELETE
CREATE
UPDATEcould create incorrect state.
A single queue can theoretically provide a global order:
1
2
3
4
5But global ordering can limit scalability.
If only one consumer can safely process messages sequentially:
throughput is limitedTherefore large systems often seek a weaker form of ordering.
Suppose messages belong to users:
user A
user B
user CWe can preserve order within each user:
User A:
1 → 2 → 3
User B:
1 → 2 → 3
User C:
1 → 2 → 3But process different users concurrently.
This is a powerful scaling pattern.
Many distributed messaging systems use partitions.
Example:
Topic
|
+---- Partition 0
+---- Partition 1
+---- Partition 2
+---- Partition 3Messages can be distributed using a key.
For example:
partition = hash(userId) % NThen all messages for a particular user can go to the same partition.
This can preserve per-key ordering.
A consumer group allows multiple consumers to share work.
Example:
Queue / Topic
|
+---- Consumer Group
|
+---- Worker 1
+---- Worker 2
+---- Worker 3Each message is assigned to one consumer within the group.
This allows horizontal scaling.
Suppose:
4 partitionsand:
2 consumersA possible assignment:
Consumer 1 → P0, P1
Consumer 2 → P2, P3Add two more consumers:
Consumer 1 → P0
Consumer 2 → P1
Consumer 3 → P2
Consumer 4 → P3If there are only 4 partitions, adding a fifth consumer may not increase parallelism within that group.
Backpressure means slowing producers or controlling work when consumers cannot keep up.
Suppose:
Producer = 100,000 messages/sec
Consumer = 20,000 messages/secThe backlog can grow rapidly.
Possible strategies:
Suppose queue depth increases continuously:
100
500
2,000
10,000
50,000This may indicate insufficient processing capacity.
Autoscaling can consider:
queue depth
+
message age
+
worker utilizationrather than CPU alone.
Queue depth alone can be misleading.
Example:
Queue depth = 10,000If workers process very quickly, this might be acceptable.
A better signal can be:
age of oldest unprocessed messageExample:
oldest message = 2 secondsversus:
oldest message = 30 minutesThe second indicates serious processing delay.
Some workloads have different importance levels.
Example:
HIGH
MEDIUM
LOWA payment notification may have higher priority than an analytics report.
Conceptually:
High Priority Queue
|
v
Workers
Low Priority Queue
|
v
WorkersPriority systems must be designed carefully because low-priority work can starve.
Sometimes work should happen later.
Examples:
send reminder in 24 hours
retry payment in 30 minutes
expire reservation in 10 minutesA delayed-message mechanism can schedule delivery for the future.
Conceptually:
Message
|
v
Delay 10 minutes
|
v
Queue
|
v
WorkerA queue is not always a scheduler.
For recurring jobs:
Every day at 9 AMa scheduler may create a message:
Scheduler
|
v
Queue
|
v
WorkerThis separation is useful.
The scheduler decides:
whenThe queue handles:
delivery and bufferingThe worker performs:
workSuppose 100,000 messages arrive.
Processing one at a time may be inefficient.
Workers may batch messages:
100 messages
|
v
one database operationBatching can improve throughput.
But larger batches can increase:
Therefore batch size is a trade-off.
Do not put huge payloads directly into the queue when avoidable.
Bad:
10 GB video inside messageBetter:
{
"mediaId": "123",
"objectKey": "uploads/123/video.mp4"
}The queue contains a reference.
The worker retrieves the actual object from storage.
This is especially important for:
Messages may need to remain available for a certain period.
Example:
retention = 7 daysRetention helps with:
But longer retention increases storage requirements.
Some event-streaming systems support replaying historical messages.
Example:
Messages from yesterday
|
v
New consumer
|
v
Reprocess historyUseful for:
Replay requires consumers to handle historical data correctly.
One event may need to trigger multiple independent consumers.
Example:
+--> Email Service
|
Order Event -----+--> Analytics
|
+--> Inventory
|
+--> NotificationA single queue where only one consumer receives the message would not provide this behavior.
Instead, use a pub/sub or topic-style architecture.
Usually:
one message
→ one worker in a consumer groupGood for:
work distributionUsually:
one event
→ multiple subscribersGood for:
event distributionA messaging platform may support both models.
Order processing:
Orders
|
v
Queue
|
+--> Worker 1
+--> Worker 2
+--> Worker 3Each order should be processed by one worker.
This is a work queue.
Order created:
Order Service
|
v
OrderCreated event
|
+----→ Analytics
+----→ Email
+----→ Inventory
+----→ RecommendationMultiple services independently react.
This is closer to event-driven pub/sub architecture.
What happens if the queue itself fails?
Production systems may use:
The queue becomes infrastructure that itself requires high availability.
A durable queue should survive failures according to its durability guarantees.
Possible architecture:
Producer
|
v
Broker A
|
+---- replicate ----→ Broker B
|
+---- replicate ----→ Broker CThe exact durability semantics depend on the messaging technology.
The producer may need confirmation that the broker accepted the message.
Conceptually:
Producer
|
| send
v
Broker
|
| ACK
v
ProducerWithout an acknowledgement, the producer may not know whether the message was accepted.
This creates another distributed-systems ambiguity:
Did the broker receive my message?Suppose:
Producer → BrokerBroker stores the message.
Before the producer receives the acknowledgement:
network failureThe producer retries.
Now:
same logical message
sent twiceTherefore producer-side idempotency or deduplication may also be needed.
Reliable systems often require idempotency at multiple layers:
Producer
|
v
Broker
|
v
Consumer
|
v
Database / External APIA duplicate can appear at several boundaries.
Design for duplicate safety across the business operation, not just the queue.
A common problem:
Update database
+
publish messageSuppose the database update succeeds but publishing fails.
Now the system has inconsistent state.
Example:
Order inserted ✓
Message published ✗The email service never learns about the order.
Write the business record and an outbox event in the same database transaction.
Transaction
|
+---- Order
|
+---- Outbox EventBoth succeed or fail together.
Then a separate publisher reads the outbox:
Outbox
|
v
Publisher
|
v
Message BrokerThis improves reliability.
Application
|
v
Database Transaction
|
+---- Order
|
+---- Outbox Event
|
v
Outbox Publisher
|
v
Queue
|
v
ConsumerThe publisher itself must handle duplicates safely.
The inbox pattern addresses duplicate message processing.
Consumer stores the message ID:
Inbox
----------------
message_id
processed_atProcessing flow:
Message
|
v
Check Inbox
|
+---- already exists → skip
|
+---- new → process
|
v
record IDThe exact transaction design matters.
Some systems provide transactional features, but do not assume that a distributed transaction automatically solves every reliability problem.
The key design question is:
What must happen atomically?Often the business goal is more practical:
database state
+
message publication
+
idempotent consumptionrather than a global transaction across every service.
There is usually a trade-off:
strong ordering
↓
less parallelismversus:
more parallelism
↓
weaker ordering guaranteesA good system identifies the smallest scope where ordering is actually required.
For example:
per accountinstead of:
entire platformSuppose messages are partitioned by:
customerIdIf one customer generates 80% of all traffic:
Customer A → Partition 3Partition 3 becomes hot.
Possible strategies:
Partition-key selection is therefore a system-design decision.
For SaaS systems, one tenant can dominate queue traffic.
Example:
Tenant A → 90%
Tenant B → 5%
Tenant C → 5%Possible solutions:
This prevents noisy-neighbor problems.
Messages may contain sensitive information.
Protect:
Avoid putting sensitive payloads into messages when a secure object reference is sufficient.
Important queue metrics:
queue depth
oldest message age
publish rate
consume rate
processing latency
ack latency
retry count
failure rate
DLQ count
consumer lag
broker health
partition skewTracing should connect:
HTTP request
|
v
message
|
v
worker
|
v
database/APIUse a correlation or trace ID.
For partitioned/event-streaming systems, consumer lag is a critical metric.
Conceptually:
latest message position
-
consumer processed position
=
lagLarge and growing lag indicates consumers are falling behind.
A simple scaling strategy:
if queue depth high:
add workers
if queue depth low:
reduce workersA more robust strategy considers:
queue depth
message age
processing latency
worker utilization
incoming rateScaling too aggressively can increase costs or overload downstream services.
Suppose:
Queue has 1 million messagesand you suddenly start:
500 workersThe downstream database may collapse.
Therefore queue scaling must consider downstream capacity.
Sometimes you deliberately limit consumers:
Queue → 20 workers → Databaseeven if more workers are available.
This is another form of backpressure.
A useful architecture:
Traffic Spike
|
v
Queue
|
v
Controlled ProcessingFor example:
Normal:
1,000 requests/sec
Spike:
20,000 requests/secThe queue can absorb asynchronous work while workers process at a safe rate.
However, the queue cannot magically make an overloaded system healthy.
If incoming work stays above processing capacity indefinitely:
queue grows foreverBasic model:
incoming rate = λ
processing rate per worker = μ
workers = N
total capacity ≈ N × μYou need:
N × μ > λwith sufficient headroom.
Example:
λ = 1,000 msg/sec
μ = 50 msg/sec/workerMinimum theoretical workers:
1000 / 50 = 20In production, you would usually provision additional capacity for:
Total processing time can be thought of as:
queue wait time
+
processing time
+
downstream latencyExample:
queue wait = 2 sec
processing = 1 sec
database = 500 ms
total ≈ 3.5 secA service can have fast workers but still have poor end-to-end latency if queue backlog is large.
For jobs taking minutes or hours:
Queue
|
v
Worker
|
+---- progress
|
+---- checkpoint
|
+---- retryThe job should be designed so that a worker failure does not necessarily restart all work.
Checkpointing can help.
Example:
Video processing:
segment 1 ✓
segment 2 ✓
segment 3 ✓
segment 4 ✗Retry only the failed unit if the architecture supports it.
Messages often live longer than the code that produced them.
Example:
Version 1:
{
"userId": "123"
}Version 2:
{
"userId": "123",
"region": "IN"
}Consumers should handle compatible schema changes.
Include:
schema versionwhen useful.
Avoid breaking all old consumers unexpectedly.
Suppose a new producer sends a field that an old consumer cannot understand.
Possible outcomes:
consumer failure
→ retry
→ retry
→ retry
→ DLQTherefore schema compatibility should be tested before deployment.
Some messages become useless after a certain time.
Example:
OTP verification jobAfter 10 minutes, processing it may have no value.
A message TTL can prevent stale work from consuming resources.
Examples:
password-reset reminder → short TTL
analytics batch → longer TTLDo not confuse message expiration with business data deletion.
A message can expire because processing is no longer useful.
The underlying database record may still need to remain.
For example:
expired notification jobdoes not necessarily mean:
delete notification recordQueues are useful when:
Do not add a queue just because it sounds scalable.
A queue may add unnecessary complexity when:
Start from requirements.
Application
|
v
Email Queue
|
+--> Worker 1
+--> Worker 2
+--> Worker 3
|
v
Email ProviderWorker responsibilities:
Order API
|
v
Order DB
|
v
Outbox
|
v
Message Broker
|
+----→ Inventory
|
+----→ Payment
|
+----→ Notification
|
+----→ AnalyticsThis architecture demonstrates:
From Chapter 18:
Object Storage
|
v
Media Queue
|
+----→ Scanner
+----→ Image Worker
+----→ Video Worker
+----→ Metadata WorkerQueues allow each processing pipeline to scale independently.
Notification API
|
v
Notification Queue
|
+----→ Email Worker
+----→ SMS Worker
+----→ Push WorkerEach channel can have different:
Treating a queue as a database.
Assuming messages are processed exactly once.
Acknowledging before successful processing.
Retrying permanent errors forever.
No dead-letter queue.
No idempotency.
Putting huge payloads into messages.
Ignoring ordering requirements.
Scaling consumers without considering downstream capacity.
Monitoring queue depth but ignoring message age.
Ignoring poison messages.
Publishing an event without solving database/message consistency.
When asked:
Design a queue-based system.
Start by asking:
What is the workload?
How many messages/sec?
Average message size?
Maximum message size?
Ordering required?
How long can messages wait?
Can messages be duplicated?
Can messages be lost?
How long should messages be retained?
How many consumers?
Need replay?
Need delayed delivery?
What happens on failure?Then design:
Producer
↓
Broker / Queue
↓
Consumers
↓
Database / External ServiceThen discuss:
delivery guarantees
retry
DLQ
idempotency
ordering
scaling
backpressure
observability
failure recoveryA concise answer:
A queue decouples producers from consumers, absorbs temporary traffic spikes, enables asynchronous processing, supports retries, and allows consumers to scale independently.
At-least-once:
message may be processed multiple timesExactly-once:
intended effect occurs onceIn practice, robust systems often combine:
at-least-once delivery
+
idempotent processing
+
deduplicationto achieve effectively-once business behavior.
A dead-letter queue stores messages that cannot be successfully processed after a configured number of attempts, allowing the main queue to continue while failed messages are investigated separately.
A strong answer is:
First identify the smallest scope that actually requires ordering. Instead of enforcing global ordering, partition messages by a business key such as account ID so messages for the same key remain ordered while unrelated keys can be processed concurrently.
Use:
unique message ID
+
idempotency key
+
deduplication record
+
transactional business operationFor example:
message_id = msg_123
if processed(msg_123):
return
process()
record(msg_123)The actual implementation should make the check and state change safe against concurrent processing.
A strong answer:
message is not permanently acknowledged
|
v
message becomes available again
|
v
another consumer retriesThen discuss:
Build an asynchronous job-processing system.
Create:
POST /jobs
GET /jobs/:id
GET /jobs/:id/statusArchitecture:
API
|
v
Database
|
v
Queue
|
+---- Worker 1
+---- Worker 2
+---- Worker 3Implement:
Job
----------------
id
type
payload
status
attempt_count
created_at
started_at
completed_at
last_errorPossible statuses:
QUEUED
PROCESSING
COMPLETED
FAILED
DEAD_LETTER
CANCELLEDAfter the basic system works, add:
priority
delayed jobs
rate limiting
worker autoscaling
consumer groups
partitioning
batch processing
trace IDs
job cancellation
job timeout
dead-letter replayThis turns the project into a realistic distributed job-processing platform.
Remember:
Producer
|
v
Message
|
v
Queue / Broker
|
v
Consumer
|
v
Business OperationFor production:
Producer
|
v
Durable Broker
|
+---- retry
+---- acknowledgement
+---- ordering
+---- retention
+---- DLQ
|
v
Idempotent Consumer
|
v
Database / ServiceThe most important lessons are:
The core production principle is:
Assume failure.
Assume retries.
Assume duplicates.
Design the business operation to remain correct.You should now be able to explain:
In the next chapter we will go deeper into distributed event streaming and Kafka-style architectures.
We will learn:
Mark it as complete to track your mastery.