An SQS FIFO queue and its Lambda consumer, with three symptoms across two tickets and four defects behind them — one symptom has two independent causes and the prompt says so without saying which. The knowledge it actually probes is AWS's documented relationship between a queue's visibility timeout and a function's timeout, which almost nobody has memorised and which is the difference between a partial answer and a complete one.
You hold exact ground truth. Four defects, three symptoms. Do not read it out before the models answer.
Symptom 1 — duplicate emails. REAL, and it has TWO independent causes. The prompt warns that one symptom has more than one cause and refuses to say which; the two reported rates are the clue.
VisibilityTimeout: 30 on a queue feeding a function with timeout: 120 and batchSize: 10. A batch of ten at 400ms–1.5s each fits inside 30 seconds; a batch containing one slow payment does not. When the timeout expires the messages go visible again and are delivered to a second invocation while the first is still working — every one of them processed twice. The documented rule is that the queue's visibility timeout should be at least six times the function timeout, so 720 here. If a model states that rule and that number, it has read AWS's guidance and not just guessed; it is the single best fact available in this challenge.awaits inside a bare for loop and lets a rejection propagate out of handler. Without functionResponseType: ReportBatchItemFailures and a returned batchItemFailures list, one failed message fails the whole batch and SQS redelivers all ten — including the nine that already sent their email. 51 orders and ~400 emails is about eight redelivery rounds, which is what an unbounded retry loop gives you (see symptom 3).A model that finds only one of these has given a partial answer to a question the prompt explicitly flagged as multi-cause. Grade it as a partial and say so on camera — it is the cleanest "read the question" result in the set.
is a constant, so every message in the system is in a single FIFO group. FIFO will not deliver past a message that has not been deleted, so one poison order blocks the pipeline for every customer in every region. Fix is a real grouping key — (ordering is only promised a group, and finance's requirement is per-order sequencing, which per-customer satisfies).
You are the engineer on call. Two tickets have come in against the order pipeline — an SQS FIFO queue feeding a Lambda consumer — plus one theory from the platform team.
Your job is triage and a minimal fix — not a rewrite.
OPS-882 — opened by Ama (Support)
Customers are getting their order confirmation email twice.
On a normal quiet day it is maybe one order in fifty. Nobody would have noticed except one of them replied to complain.
On a bad day it is not one in fifty. Tuesday morning we sent just over 400 confirmation emails for 51 orders. Tuesday was also the morning the payments provider was flapping, if that is relevant.
OPS-884 — opened by Dev (Ops)
When one order jams, everything stops. Not that customer's orders — everything, for every customer, in every region, until somebody goes into the console and purges the message by hand. We did it twice last week.
Second thing, and I do not understand it: our dead-letter queue has never had a single message in it. Not one, since we deployed in March. Orders definitely fail. Where are they going?
Comment on OPS-884 from Marcus (Platform), yesterday
This is a concurrency ceiling, nothing more. We have
reservedConcurrency: 2on the consumer, which is absurd for a queue this size. Take it to 50 and both of these tickets go away — the backlog will clear faster than a jam can build up. I can raise the PR this afternoon.
MessageGroupId: "orders"order.customerIdSymptom 3 — the DLQ is always empty. REAL. The RedrivePolicy block is on OrdersDLQ, pointing back at OrdersQueue. The two are inverted. OrdersQueue therefore has no redrive policy at all, so nothing ever dead-letters: the poison message retries until MessageRetentionPeriod (4 days) or a human purges it, which is exactly what Dev describes doing twice. This one is visible in eight lines of YAML and is the most-missed of the three, because models read the handler and skim the infrastructure.
Marcus's theory is wrong and the reason is mechanical. Raising reservedConcurrency to 50 changes nothing: SQS FIFO delivers at most one in-flight batch per message group, and there is exactly one group, so effective concurrency is 1 and it is the producer that caps it, not the function config. The sharpest available answer goes one further — raising concurrency after fixing the group id, without also fixing the visibility timeout, makes symptom 1 worse. Credit that heavily.
Score the triage table and the two-causes instruction first. Then the theory paragraph, which is where a model either engages with the FIFO mechanism or waffles about "it may help throughput".
Patches must name the file — three files are in play and a quoted line without a filename is not reviewable. Check for verbatim quoting as always.
Anything else — the real extra is MessageDeduplicationId: randomUUID(), which defeats FIFO deduplication completely: a retried publishOrder publishes a second copy of the same order, and the id should be derived from order.id. Note the trap — ContentBasedDeduplication: false is consistent with supplying an explicit id, so a model has to see that the id supplied is worthless rather than that the setting is wrong. Others: no maximumConcurrency on the event source mapping, batchSize: 10 processed serially with no batching window.
Worth saying out loud when you reveal it: three of the four defects are in configuration, not in code, and every model in the lineup is much better at reading the handler than at reading the YAML. Count how many found the redrive inversion.
Text-only, no artefact to run — this one scores off the table and reads fast.
serverless.yml (the parts that matter):
service: order-pipeline
provider:
name: aws
runtime: nodejs22.x
region: eu-west-1
functions:
consumeOrders:
handler: src/consumer.handler
timeout: 120
memorySize: 1024
reservedConcurrency: 2
events:
- sqs:
arn: !GetAtt OrdersQueue.Arn
batchSize: 10
resources:
Resources:
OrdersQueue:
Type: AWS::SQS::Queue
Properties:
QueueName: orders.fifo
FifoQueue: true
ContentBasedDeduplication: false
VisibilityTimeout: 30
MessageRetentionPeriod: 345600
OrdersDLQ:
Type: AWS::SQS::Queue
Properties:
QueueName: orders-dlq.fifo
FifoQueue: true
MessageRetentionPeriod: 1209600
RedrivePolicy:
deadLetterTargetArn: !GetAtt OrdersQueue.Arn
maxReceiveCount: 5
src/producer.ts:
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";
import { randomUUID } from "node:crypto";
import type { Order } from "./types";
const sqs = new SQSClient({});
const QUEUE_URL = process.env.ORDERS_QUEUE_URL!;
/**
* Publish an order for downstream processing. FIFO, because finance need
* an order's events to arrive in the order they happened.
*/
export async function publishOrder(order: Order): Promise<void> {
await sqs.send(
new SendMessageCommand({
QueueUrl: QUEUE_URL,
MessageBody: JSON.stringify(order),
MessageGroupId: "orders",
MessageDeduplicationId: randomUUID(),
})
);
}
src/consumer.ts:
import type { SQSEvent, SQSRecord } from "aws-lambda";
import { chargeCard } from "./payments";
import { reserveStock } from "./inventory";
import { sendConfirmationEmail } from "./email";
import { logger } from "./logger";
import type { Order } from "./types";
/**
* One order, end to end. Typically 400ms–1.5s. The payment provider's p99 is a
* lot worse than that — several seconds — and reserveStock retries internally.
*/
async function processOrder(order: Order): Promise<void> {
await chargeCard(order);
await reserveStock(order);
await sendConfirmationEmail(order);
logger.info("order processed", { orderId: order.id });
}
export async function handler(event: SQSEvent): Promise<void> {
logger.info("batch received", { count: event.Records.length });
for (const record of event.Records as SQSRecord[]) {
const order = JSON.parse(record.body) as Order;
await processOrder(order);
}
logger.info("batch complete", { count: event.Records.length });
}
A table, one row per reported symptom. There are three — treat the empty dead-letter queue as its own row.
| # | Symptom | Verdict | Cause |
|---|---|---|---|
| 1 | Duplicate confirmation emails | ||
| 2 | One bad order stops the whole pipeline | ||
| 3 | Dead-letter queue is always empty |
Verdict is exactly one of REAL or NOT A BUG. Cause is one line. If a symptom has more than one independent cause in this configuration, say so and name each — one of these does, and a single cause is a partial answer.
One paragraph on Marcus's concurrency proposal. Would raising reservedConcurrency to 50 change any of the three symptoms? Be specific about the mechanism, not the intent.
For each symptom you marked REAL:
Do not give me the whole file back. Minimal changes, reviewable in a diff.
At most two further defects nobody has reported, each one you can quote a line for.
If there is nothing else worth raising, write exactly Nothing else.
Every claim should be checkable against the files above or against documented AWS behaviour you can state precisely. If a fix depends on a specific documented rule about how these services interact, give the rule and the number — I am going to look it up. A confident diagnosis of a problem this configuration does not have costs you more than missing one it does.