The queue that stopped

You are the engineer on call. Two tickets have come in against the order pipeline — an SQS FIFO queue feeding a Lambda consumer — plus one theory from the platform team.

Your job is triage and a minimal fix — not a rewrite.

The tickets

OPS-882 — opened by Ama (Support)

Customers are getting their order confirmation email twice.

On a normal quiet day it is maybe one order in fifty. Nobody would have noticed except one of them replied to complain.

On a bad day it is not one in fifty. Tuesday morning we sent just over 400 confirmation emails for 51 orders. Tuesday was also the morning the payments provider was flapping, if that is relevant.

OPS-884 — opened by Dev (Ops)

When one order jams, everything stops. Not that customer's orders — everything, for every customer, in every region, until somebody goes into the console and purges the message by hand. We did it twice last week.

Second thing, and I do not understand it: our dead-letter queue has never had a single message in it. Not one, since we deployed in March. Orders definitely fail. Where are they going?

Comment on OPS-884 from Marcus (Platform), yesterday

This is a concurrency ceiling, nothing more. We have reservedConcurrency: 2 on the consumer, which is absurd for a queue this size. Take it to 50 and both of these tickets go away — the backlog will clear faster than a jam can build up. I can raise the PR this afternoon.

The code

serverless.yml (the parts that matter):

service: order-pipeline

provider:
  name: aws
  runtime: nodejs22.x
  region: eu-west-1

functions:
  consumeOrders:
    handler: src/consumer.handler
    timeout: 120
    memorySize: 1024
    reservedConcurrency: 2
    events:
      - sqs:
          arn: !GetAtt OrdersQueue.Arn
          batchSize: 10

resources:
  Resources:
    OrdersQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: orders.fifo
        FifoQueue: true
        ContentBasedDeduplication: false
        VisibilityTimeout: 30
        MessageRetentionPeriod: 345600

    OrdersDLQ:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: orders-dlq.fifo
        FifoQueue: true
        MessageRetentionPeriod: 1209600
        RedrivePolicy:
          deadLetterTargetArn: !GetAtt OrdersQueue.Arn
          maxReceiveCount: 5

src/producer.ts:

import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";
import { randomUUID } from "node:crypto";
import type { Order } from "./types";

const sqs = new SQSClient({});
const QUEUE_URL = process.env.ORDERS_QUEUE_URL!;

/**
 * Publish an order for downstream processing. FIFO, because finance need
 * an order's events to arrive in the order they happened.
 */
export async function publishOrder(order: Order): Promise<void> {
  await sqs.send(
    new SendMessageCommand({
      QueueUrl: QUEUE_URL,
      MessageBody: JSON.stringify(order),
      MessageGroupId: "orders",
      MessageDeduplicationId: randomUUID(),
    })
  );
}

src/consumer.ts:

import type { SQSEvent, SQSRecord } from "aws-lambda";
import { chargeCard } from "./payments";
import { reserveStock } from "./inventory";
import { sendConfirmationEmail } from "./email";
import { logger } from "./logger";
import type { Order } from "./types";

/**
 * One order, end to end. Typically 400ms–1.5s. The payment provider's p99 is a
 * lot worse than that — several seconds — and reserveStock retries internally.
 */
async function processOrder(order: Order): Promise<void> {
  await chargeCard(order);
  await reserveStock(order);
  await sendConfirmationEmail(order);
  logger.info("order processed", { orderId: order.id });
}

export async function handler(event: SQSEvent): Promise<void> {
  logger.info("batch received", { count: event.Records.length });

  for (const record of event.Records as SQSRecord[]) {
    const order = JSON.parse(record.body) as Order;
    await processOrder(order);
  }

  logger.info("batch complete", { count: event.Records.length });
}

Answer in exactly these sections

Triage

A table, one row per reported symptom. There are three — treat the empty dead-letter queue as its own row.

Verdict is exactly one of REAL or NOT A BUG. Cause is one line. If a symptom has more than one independent cause in this configuration, say so and name each — one of these does, and a single cause is a partial answer.

The theory on the ticket

One paragraph on Marcus's concurrency proposal. Would raising reservedConcurrency to 50 change any of the three symptoms? Be specific about the mechanism, not the intent.

The patches

For each symptom you marked REAL:

• Quote the exact line or lines that are wrong, verbatim from the files above, naming which file.
• Give the exact replacement.
• One line on why that fixes it.

Do not give me the whole file back. Minimal changes, reviewable in a diff.

Anything else

At most two further defects nobody has reported, each one you can quote a line for.

If there is nothing else worth raising, write exactly Nothing else.

The standard I'm holding you to

Every claim should be checkable against the files above or against documented AWS behaviour you can state precisely. If a fix depends on a specific documented rule about how these services interact, give the rule and the number — I am going to look it up. A confident diagnosis of a problem this configuration does not have costs you more than missing one it does.