Run Detail

Triage

The theory on the ticket

Raising reservedConcurrency to 50 would not fix any of the three symptoms. FIFO queues permit only one active consumer for a given MessageGroupId, and every message currently has MessageGroupId: "orders", so this queue has one effective processing lane regardless of whether Lambda reserves 2 or 50 concurrent executions. It does not change the 30-second visibility timeout, Lambda’s all-or-nothing batch acknowledgement behaviour, the missing source-queue DLQ policy, or the constant message group. At most it would help after the producer is changed to use multiple group IDs; it could not make the current jam clear.

The patches

1. Duplicate confirmation emails

serverless.yml — wrong line:

        VisibilityTimeout: 30

Replacement:

        VisibilityTimeout: 720

AWS Lambda’s documented SQS rule is a visibility timeout of at least six times the function timeout plus the maximum batching window; here that is 6 × 120 + 0 = 720 seconds. A 30-second timeout permits a still-running invocation to lose visibility and have its message delivered again.

---

serverless.yml — wrong line:

          batchSize: 10

Replacement:

          batchSize: 10
          functionResponseType: ReportBatchItemFailures

This opts the SQS event-source mapping into Lambda partial batch response handling; without it, one thrown record causes Lambda to retry every record in the batch.

---

src/consumer.ts — wrong import line:

import type { SQSEvent, SQSRecord } from "aws-lambda";

Replacement:

import type { SQSEvent, SQSRecord, SQSBatchResponse } from "aws-lambda";

---

src/consumer.ts — wrong handler:

export async function handler(event: SQSEvent): Promise<void> {
  logger.info("batch received", { count: event.Records.length });

  for (const record of event.Records as SQSRecord[]) {
    const order = JSON.parse(record.body) as Order;
    await processOrder(order);
  }

  logger.info("batch complete", { count: event.Records.length });
}

Replacement:

export async function handler(event: SQSEvent): Promise<SQSBatchResponse> {
  logger.info("batch received", { count: event.Records.length });

  const batchItemFailures: { itemIdentifier: string }[] = [];

  for (let index = 0; index < event.Records.length; index += 1) {
    const record = event.Records[index] as SQSRecord;

    try {
      const order = JSON.parse(record.body) as Order;
      await processOrder(order);
    } catch (error) {
      logger.error("order processing failed", {
        messageId: record.messageId,
        error,
      });

      // For FIFO, fail this record and all later records in this batch so
      // their group ordering is preserved.
      batchItemFailures.push(
        ...event.Records.slice(index).map(({ messageId }) => ({
          itemIdentifier: messageId,
        }))
      );
      break;
    }
  }

  logger.info("batch complete", {
    count: event.Records.length,
    failures: batchItemFailures.length,
  });

  return { batchItemFailures };
}

With ReportBatchItemFailures, omitted records are acknowledged and deleted; for FIFO, AWS documents that processing must stop at the first failure and both the failed and unprocessed records must be returned as failures to preserve ordering.

---

src/consumer.ts — wrong line for a durable duplicate-email guarantee:

  await sendConfirmationEmail(order);

Required replacement contract:

  await sendConfirmationEmail(order, {
    idempotencyKey: `order-confirmation:${order.id}`,
  });

The email implementation/provider must durably honour that key and send at most one confirmation for it. SQS-to-Lambda processing is at-least-once, so the visibility and partial-batch fixes remove the configured duplicate paths but cannot guarantee exactly-once external email delivery after a process crash or timeout between sending the email and acknowledging the SQS record. The src/email.ts implementation is not provided, so its corresponding exact API change cannot be quoted here.

2. One bad order stops the whole pipeline

src/producer.ts — wrong line:

      MessageGroupId: "orders",

Replacement:

      MessageGroupId: order.id,

FIFO ordering is per MessageGroupId, not per queue. Using the order ID keeps events for one order ordered while allowing unrelated orders to use separate FIFO groups and proceed independently.

---

serverless.yml — wrong OrdersQueue property:

        MessageRetentionPeriod: 345600

Replacement:

        MessageRetentionPeriod: 345600
        RedrivePolicy:
          deadLetterTargetArn: !GetAtt OrdersDLQ.Arn
          maxReceiveCount: 5

The redrive policy belongs on the source queue. After the configured receive threshold, SQS moves the poison message to the DLQ, allowing later messages in that FIFO group to proceed.

3. Dead-letter queue is always empty

serverless.yml — wrong lines under OrdersDLQ:

        RedrivePolicy:
          deadLetterTargetArn: !GetAtt OrdersQueue.Arn
          maxReceiveCount: 5

Replacement:

Delete those lines; the RedrivePolicy added to OrdersQueue in patch 2 is the replacement.

SQS only redrives messages from a queue which itself has a RedrivePolicy; configuring a policy on the DLQ does not make the source queue send failures there.

Anything else

1. FIFO producer deduplication is ineffective for repeated application-level publishes. src/producer.ts contains:

   MessageDeduplicationId: randomUUID(),
   

   FIFO deduplication compares deduplication IDs during SQS’s five-minute deduplication interval. A newly generated UUID means a retried or duplicated application publish is always treated as a new message. Use a stable, unique business event ID for MessageDeduplicationId; do not use order.id unless there is exactly one publishable event per order.