You are the engineer on call. Two tickets have come in against the order pipeline — an SQS FIFO queue feeding a Lambda consumer — plus one theory from the platform team.
Your job is triage and a minimal fix — not a rewrite.
OPS-882 — opened by Ama (Support)
Customers are getting their order confirmation email twice.
On a normal quiet day it is maybe one order in fifty. Nobody would have noticed except one of them replied to complain.
On a bad day it is not one in fifty. Tuesday morning we sent just over 400 confirmation emails for 51 orders. Tuesday was also the morning the payments provider was flapping, if that is relevant.
OPS-884 — opened by Dev (Ops)
When one order jams, everything stops. Not that customer's orders — everything, for every customer, in every region, until somebody goes into the console and purges the message by hand. We did it twice last week.
Second thing, and I do not understand it: our dead-letter queue has never had a single message in it. Not one, since we deployed in March. Orders definitely fail. Where are they going?
Comment on OPS-884 from Marcus (Platform), yesterday
This is a concurrency ceiling, nothing more. We have
reservedConcurrency: 2on the consumer, which is absurd for a queue this size. Take it to 50 and both of these tickets go away — the backlog will clear faster than a jam can build up. I can raise the PR this afternoon.
serverless.yml (the parts that matter):
service: order-pipeline
provider:
name: aws
runtime: nodejs22.x
region: eu-west-1
functions:
consumeOrders:
handler: src/consumer.handler
timeout: 120
memorySize: 1024
reservedConcurrency: 2
events:
- sqs:
arn: !GetAtt OrdersQueue.Arn
batchSize: 10
resources:
Resources:
OrdersQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: orders.fifo
FifoQueue: true
ContentBasedDeduplication: false
VisibilityTimeout: 30
MessageRetentionPeriod: 345600
OrdersDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: orders-dlq.fifo
FifoQueue: true
MessageRetentionPeriod: 1209600
RedrivePolicy:
deadLetterTargetArn: !GetAtt OrdersQueue.Arn
maxReceiveCount: 5
src/producer.ts:
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";
import { randomUUID } from "node:crypto";
import type { Order } from "./types";
const sqs = new SQSClient({});
const QUEUE_URL = process.env.ORDERS_QUEUE_URL!;
/**
* Publish an order for downstream processing. FIFO, because finance need
* an order's events to arrive in the order they happened.
*/
export async function publishOrder(order: Order): Promise<void> {
await sqs.send(
new SendMessageCommand({
QueueUrl: QUEUE_URL,
MessageBody: JSON.stringify(order),
MessageGroupId: "orders",
MessageDeduplicationId: randomUUID(),
})
);
}
src/consumer.ts:
import type { SQSEvent, SQSRecord } from "aws-lambda";
import { chargeCard } from "./payments";
import { reserveStock } from "./inventory";
import { sendConfirmationEmail } from "./email";
import { logger } from "./logger";
import type { Order } from "./types";
/**
* One order, end to end. Typically 400ms–1.5s. The payment provider's p99 is a
* lot worse than that — several seconds — and reserveStock retries internally.
*/
async function processOrder(order: Order): Promise<void> {
await chargeCard(order);
await reserveStock(order);
await sendConfirmationEmail(order);
logger.info("order processed", { orderId: order.id });
}
export async function handler(event: SQSEvent): Promise<void> {
logger.info("batch received", { count: event.Records.length });
for (const record of event.Records as SQSRecord[]) {
const order = JSON.parse(record.body) as Order;
await processOrder(order);
}
logger.info("batch complete", { count: event.Records.length });
}
A table, one row per reported symptom. There are three — treat the empty dead-letter queue as its own row.
| # | Symptom | Verdict | Cause |
|---|---|---|---|
| 1 | Duplicate confirmation emails | ||
| 2 | One bad order stops the whole pipeline | ||
| 3 | Dead-letter queue is always empty |
Verdict is exactly one of REAL or NOT A BUG. Cause is one line. If a symptom has more than one independent cause in this configuration, say so and name each — one of these does, and a single cause is a partial answer.
One paragraph on Marcus's concurrency proposal. Would raising reservedConcurrency to 50 change any of the three symptoms? Be specific about the mechanism, not the intent.
For each symptom you marked REAL:
Do not give me the whole file back. Minimal changes, reviewable in a diff.
At most two further defects nobody has reported, each one you can quote a line for.
If there is nothing else worth raising, write exactly Nothing else.
Every claim should be checkable against the files above or against documented AWS behaviour you can state precisely. If a fix depends on a specific documented rule about how these services interact, give the rule and the number — I am going to look it up. A confident diagnosis of a problem this configuration does not have costs you more than missing one it does.
| # | Symptom | Verdict | Cause |
|---|---|---|---|
| 1 | Duplicate confirmation emails | REAL | (a) VisibilityTimeout: 30 is shorter than the documented SQS FIFO redrive threshold of 30 seconds, so a message that takes >30s to process is redelivered as a duplicate before it can be counted toward maxReceiveCount (AWS docs: FIFO queues redrive messages that remain in the queue for more than 30 seconds after a visibility timeout); (b) the consumer processes all 10 records in a batch sequentially with no per-record error handling, so a single failing chargeCard/reserveStock/sendConfirmationEmail throws and the entire batch is returned to the queue, redelivered, and re-processed — including the already-succeeded records in that batch. |
| 2 | One bad order stops the whole pipeline | REAL | The handler loops over event.Records and awaits processOrder for each; any thrown error (e.g. a flapping payment provider) propagates out of the handler, causing Lambda to report a batch failure and return all 10 messages to the queue. Because the queue is FIFO with MessageGroupId: "orders" (a single group), messages are processed strictly in order within that group, so a poison message blocks every subsequent message in the same group until it is manually purged. |
| 3 | Dead-letter queue is always empty | REAL | Two independent causes: (a) the RedrivePolicy in serverless.yml has deadLetterTargetArn: !GetAtt OrdersQueue.Arn — it points the DLQ policy back at the source queue itself instead of at OrdersDLQ, so failed messages are never routed to the DLQ; (b) the FIFO redrive mechanism (visibility timeout > 30s) redelivers messages as duplicates rather than counting them toward maxReceiveCount, so even if the DLQ ARN were correct, messages that exceed the 30-second visibility window would be redelivered instead of dead-lettered. |
Marcus's proposal to raise reservedConcurrency to 50 would not fix any of the three symptoms. Concurrency controls how many Lambda instances can run in parallel, but the FIFO queue with a single MessageGroupId: "orders" enforces strict ordering within that group — only one message from a given group can be processed at a time regardless of how many concurrent Lambdas are available (AWS docs: FIFO queues support up to 300 messages per second per queue, but only one message per MessageGroupId is processed concurrently). So raising concurrency does nothing for the ordering bottleneck. The duplicate-email and DLQ-empty symptoms are caused by the visibility timeout / redrive configuration and the broken RedrivePolicy ARN, neither of which is affected by concurrency. The "one bad order stops everything" symptom is caused by the single MessageGroupId and the lack of per-record error handling, not by a concurrency ceiling — more Lambdas still can't skip past a poison message in a single ordered group.
Symptom 1 & 2 (batch failure / sequential processing / no per-record error handling):
src/consumer.ts:
for (const record of event.Records as SQSRecord[]) {
const order = JSON.parse(record.body) as Order;
await processOrder(order);
}
Replace with:
for (const record of event.Records as SQSRecord[]) {
try {
const order = JSON.parse(record.body) as Order;
await processOrder(order);
} catch (err) {
logger.error("order failed", { orderId: JSON.parse(record.body).id, error: err });
throw err; // fail the batch so SQS can retry / redrive this record individually
}
}
This ensures a failing record throws and fails the batch, allowing SQS to retry that specific record up to maxReceiveCount rather than silently re-processing the entire batch (which causes duplicates for the already-succeeded records). Note: this alone does not fix the FIFO ordering block — see Anything else for the group ID issue.
Symptom 3 (DLQ never receives messages — broken RedrivePolicy ARN):
serverless.yml:
RedrivePolicy:
deadLetterTargetArn: !GetAtt OrdersQueue.Arn
maxReceiveCount: 5
Replace with:
RedrivePolicy:
deadLetterTargetArn: !GetAtt OrdersDLQ.Arn
maxReceiveCount: 5
This points the dead-letter target at the actual DLQ resource instead of back at the source queue, so messages that exceed maxReceiveCount are routed to OrdersDLQ.
Symptom 1 (duplicate emails from visibility timeout / FIFO redrive):
serverless.yml:
VisibilityTimeout: 30
Replace with:
VisibilityTimeout: 60
AWS FIFO queues have a documented behaviour: if a message's visibility timeout is set to 30 seconds or less, and the message is not deleted within that window, SQS redelivers it as a duplicate rather than counting it toward maxReceiveCount (the FIFO redrive mechanism requires a visibility timeout greater than 30 seconds to trigger the dead-letter counter). Raising to 60s ensures the redrive counter works as intended and reduces the window where a slow payment call causes a duplicate delivery.
All orders share a single MessageGroupId, so FIFO ordering blocks the entire queue on any one message. In src/producer.ts:
MessageGroupId: "orders",
Replace with something that scopes the group to the individual order, e.g. MessageGroupId: order.id. AWS FIFO queues process messages in order only within a MessageGroupId; with a single group "orders", every message in the queue is strictly ordered, so one poison message blocks every other customer's order. Using a per-order group ID allows independent orders to be processed in parallel while still preserving per-order event ordering.
ContentBasedDeduplication: false with a client-generated randomUUID() deduplication ID means SQS cannot detect genuine retries of the same order. In serverless.yml:
ContentBasedDeduplication: false
If the producer retries a SendMessageCommand (e.g. on a network timeout), it generates a new UUID each time, so SQS treats each retry as a new message rather than deduplicating. Setting ContentBasedDeduplication: true (and removing the explicit MessageDeduplicationId from the producer) lets SQS hash the message body and deduplicate identical retries automatically. Alternatively, derive MessageDeduplicationId from a stable order identifier instead of randomUUID().