The queue that stopped

Aug 14, 2026 · YouTube

serverless.yml (the parts that matter):

service: order-pipeline

provider:
  name: aws
  runtime: nodejs22.x
  region: eu-west-1

functions:
  consumeOrders:
    handler: src/consumer.handler
    timeout: 120
    memorySize: 1024
    reservedConcurrency: 2
    events:
      - sqs:
          arn: !GetAtt OrdersQueue.Arn
          batchSize: 10

resources:
  Resources:
    OrdersQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: orders.fifo
        FifoQueue: true
        ContentBasedDeduplication: false
        VisibilityTimeout: 30
        MessageRetentionPeriod: 345600

    OrdersDLQ:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: orders-dlq.fifo
        FifoQueue: true
        MessageRetentionPeriod: 1209600
        RedrivePolicy:
          deadLetterTargetArn: !GetAtt OrdersQueue.Arn
          maxReceiveCount: 5

src/producer.ts:

import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";
import { randomUUID } from "node:crypto";
import type { Order } from "./types";

const sqs = new SQSClient({});
const QUEUE_URL = process.env.ORDERS_QUEUE_URL!;

/**
 * Publish an order for downstream processing. FIFO, because finance need
 * an order's events to arrive in the order they happened.
 */
export async function publishOrder(order: Order): Promise<void> {
  await sqs.send(
    new SendMessageCommand({
      QueueUrl: QUEUE_URL,
      MessageBody: JSON.stringify(order),
      MessageGroupId: "orders",
      MessageDeduplicationId: randomUUID(),
    })
  );
}

src/consumer.ts:

import type { SQSEvent, SQSRecord } from "aws-lambda";
import { chargeCard } from "./payments";
import { reserveStock } from "./inventory";
import { sendConfirmationEmail } from "./email";
import { logger } from "./logger";
import type { Order } from "./types";

/**
 * One order, end to end. Typically 400ms–1.5s. The payment provider's p99 is a
 * lot worse than that — several seconds — and reserveStock retries internally.
 */
async function processOrder(order: Order): Promise<void> {
  await chargeCard(order);
  await reserveStock(order);
  await sendConfirmationEmail(order);
  logger.info("order processed", { orderId: order.id });
}

export async function handler(event: SQSEvent): Promise<void> {
  logger.info("batch received", { count: event.Records.length });

  for (const record of event.Records as SQSRecord[]) {
    const order = JSON.parse(record.body) as Order;
    await processOrder(order);
  }

  logger.info("batch complete", { count: event.Records.length });
}

Answer in exactly these sections

Triage

A table, one row per reported symptom. There are three — treat the empty dead-letter queue as its own row.

Verdict is exactly one of REAL or NOT A BUG. Cause is one line. If a symptom has more than one independent cause in this configuration, say so and name each — one of these does, and a single cause is a partial answer.

The theory on the ticket

One paragraph on Marcus's concurrency proposal. Would raising reservedConcurrency to 50 change any of the three symptoms? Be specific about the mechanism, not the intent.

The patches

For each symptom you marked REAL:

• Quote the exact line or lines that are wrong, verbatim from the files above, naming which file.
• Give the exact replacement.
• One line on why that fixes it.

Do not give me the whole file back. Minimal changes, reviewable in a diff.

Anything else

At most two further defects nobody has reported, each one you can quote a line for.

If there is nothing else worth raising, write exactly Nothing else.

The standard I'm holding you to

Every claim should be checkable against the files above or against documented AWS behaviour you can state precisely. If a fix depends on a specific documented rule about how these services interact, give the rule and the number — I am going to look it up. A confident diagnosis of a problem this configuration does not have costs you more than missing one it does.