MCP Server SQS — async job queuing, dead-letter queues, batch processing

Client setup and queue selection

The SQS client is initialized once and reused across all tool calls. Credentials resolve from the standard chain: IAM role (ECS task role, Lambda execution role, EC2 instance profile), ~/.aws/credentials for local development, or AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables.

import { SQSClient, SendMessageCommand, ReceiveMessageCommand,
  DeleteMessageCommand, SendMessageBatchCommand } from "@aws-sdk/client-sqs";

// Create once at module level — the client manages connection pooling internally
const sqs = new SQSClient({ region: process.env.AWS_REGION ?? "us-east-1" });

const QUEUE_URL = process.env.SQS_QUEUE_URL!;
// Queue URL format: https://sqs.{region}.amazonaws.com/{account-id}/{queue-name}
// For FIFO queues: name must end in .fifo
// const FIFO_QUEUE_URL = process.env.SQS_FIFO_QUEUE_URL!; // ends in .fifo

Most MCP tool async patterns use Standard queues because tool invocations are independent — two calls to the same tool do not depend on each other's order. Use FIFO only when an agent workflow has a strict sequence that must be preserved (rare).

Enqueue pattern: submit-and-return-ID tool

The common MCP pattern is a "submit" tool that enqueues work and returns a job ID, combined with a "poll" tool that checks job status. This keeps MCP tool call latency under 200ms even for jobs that take minutes.

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { z } from "zod";
import { randomUUID } from "crypto";

const server = new McpServer({ name: "async-worker", version: "1.0.0" });

server.tool(
  "submit_analysis",
  "Enqueue a data analysis job and return a job ID for polling",
  {
    dataset_url: z.string().url().describe("S3 or HTTPS URL of the dataset"),
    options: z.object({
      max_rows: z.number().int().positive().optional(),
      output_format: z.enum(["json", "csv", "parquet"]).default("json"),
    }).optional(),
  },
  async ({ dataset_url, options }) => {
    const jobId = randomUUID();
    const payload = { jobId, dataset_url, options, submitted_at: new Date().toISOString() };

    await sqs.send(new SendMessageCommand({
      QueueUrl: QUEUE_URL,
      MessageBody: JSON.stringify(payload),
      // Optional: message attributes for filtering in the worker
      MessageAttributes: {
        jobType: { DataType: "String", StringValue: "analysis" },
      },
      // For FIFO queues, these two fields are required:
      // MessageGroupId: "analysis-jobs",  // groups related messages
      // MessageDeduplicationId: jobId,    // prevents duplicate processing within 5 min
    }));

    return {
      content: [{ type: "text", text: JSON.stringify({ jobId, status: "queued" }) }],
    };
  }
);

Visibility timeout: the most critical SQS setting

When a worker receives a message, SQS hides it from other consumers for the visibility timeout period. If the worker does not delete the message before the timeout expires, SQS makes the message visible again and another worker picks it up — causing double processing.

// Rule: visibility timeout = 2× expected job duration
// If your analysis job takes up to 5 minutes, set VisibilityTimeout to 600 seconds (10 min)

// Queue creation with correct visibility timeout (CloudFormation / CDK equivalent):
// AWS::SQS::Queue:
//   VisibilityTimeout: 600  # seconds
//   MessageRetentionPeriod: 86400  # 24 hours — default is 4 days
//   ReceiveMessageWaitTimeSeconds: 20  # enables long polling

// For jobs with variable duration: extend visibility timeout during processing
import { ChangeMessageVisibilityCommand } from "@aws-sdk/client-sqs";

async function processWithHeartbeat(receiptHandle: string, jobFn: () => Promise) {
  const EXTENSION_SECONDS = 60;
  const interval = setInterval(async () => {
    try {
      await sqs.send(new ChangeMessageVisibilityCommand({
        QueueUrl: QUEUE_URL,
        ReceiptHandle: receiptHandle,
        VisibilityTimeout: EXTENSION_SECONDS,
      }));
    } catch { /* message may have been deleted — ignore */ }
  }, (EXTENSION_SECONDS / 2) * 1000);

  try {
    await jobFn();
  } finally {
    clearInterval(interval);
  }
}

The receipt handle from ReceiveMessageCommand is what you use for visibility extension and deletion — it is different from the MessageId and is valid only for the current consumer. Do not share receipt handles across processes.

Long polling and the worker loop

SQS supports two polling modes: short polling (returns immediately, even if the queue is empty) and long polling (WaitTimeSeconds: 1–20, waits up to N seconds for a message before returning). Long polling with WaitTimeSeconds: 20 reduces empty receive cost by ~95% and is the correct default for all MCP background workers.

import { DeleteMessageCommand } from "@aws-sdk/client-sqs";

async function runWorkerLoop(signal: AbortSignal) {
  while (!signal.aborted) {
    const response = await sqs.send(new ReceiveMessageCommand({
      QueueUrl: QUEUE_URL,
      MaxNumberOfMessages: 10,   // process up to 10 messages per receive call
      WaitTimeSeconds: 20,       // long poll — ALWAYS set this
      MessageAttributeNames: ["All"],
      AttributeNames: ["ApproximateReceiveCount"],
    }));

    if (!response.Messages?.length) continue;  // empty queue — loop again

    await Promise.allSettled(
      response.Messages.map(async (msg) => {
        const receiveCount = parseInt(msg.Attributes?.ApproximateReceiveCount ?? "1");
        try {
          const payload = JSON.parse(msg.Body!);
          await processWithHeartbeat(msg.ReceiptHandle!, () => processJob(payload));

          // Delete ONLY after successful processing
          await sqs.send(new DeleteMessageCommand({
            QueueUrl: QUEUE_URL,
            ReceiptHandle: msg.ReceiptHandle!,
          }));
        } catch (err) {
          // Do NOT delete — SQS will make it visible again after VisibilityTimeout
          // After maxReceiveCount failures, it moves to the DLQ
          console.error({ jobId: JSON.parse(msg.Body!).jobId, receiveCount, err });
        }
      })
    );
  }
}

// Start worker loop when MCP server initialises
const abortController = new AbortController();
runWorkerLoop(abortController.signal);  // runs in background

Dead-letter queue configuration

Without a DLQ, a malformed message that always throws during processing loops forever: it becomes visible, a worker picks it up, it fails, it becomes visible again. maxReceiveCount on the source queue's redrive policy controls how many times a message is attempted before being moved to the DLQ.

// AWS CDK example: source queue with DLQ redrive policy
import * as sqs from "aws-cdk-lib/aws-sqs";
import * as cdk from "aws-cdk-lib";

const dlq = new sqs.Queue(this, "AnalysisDLQ", {
  queueName: "analysis-jobs-dlq",
  retentionPeriod: cdk.Duration.days(14),  // keep failed messages for 2 weeks
});

const queue = new sqs.Queue(this, "AnalysisQueue", {
  queueName: "analysis-jobs",
  visibilityTimeout: cdk.Duration.seconds(600),
  receiveMessageWaitTime: cdk.Duration.seconds(20),
  deadLetterQueue: {
    queue: dlq,
    maxReceiveCount: 3,  // move to DLQ after 3 failed processing attempts
  },
});

// In your MCP server: poll the DLQ to surface failed jobs to the agent
server.tool(
  "list_failed_jobs",
  "Return up to 10 jobs that failed processing and landed in the dead-letter queue",
  {},
  async () => {
    const response = await sqs.send(new ReceiveMessageCommand({
      QueueUrl: process.env.SQS_DLQ_URL!,
      MaxNumberOfMessages: 10,
      WaitTimeSeconds: 1,
    }));
    const failed = (response.Messages ?? []).map(m => JSON.parse(m.Body!));
    return { content: [{ type: "text", text: JSON.stringify(failed) }] };
  }
);

Batch send and batch delete

SQS charges per API request, not per message. SendMessageBatch sends up to 10 messages in one API call. DeleteMessageBatch deletes up to 10 in one call. For high-throughput MCP tools that enqueue many jobs per invocation, batching reduces cost and latency.

// Batch enqueue — up to 10 messages per SendMessageBatch call
async function batchEnqueue(jobs: Array<{ id: string; payload: object }>) {
  // Split into chunks of 10
  const chunks = [];
  for (let i = 0; i < jobs.length; i += 10) {
    chunks.push(jobs.slice(i, i + 10));
  }

  for (const chunk of chunks) {
    const result = await sqs.send(new SendMessageBatchCommand({
      QueueUrl: QUEUE_URL,
      Entries: chunk.map(job => ({
        Id: job.id,               // must be unique within the batch (1-80 chars, alphanumeric + _ -)
        MessageBody: JSON.stringify(job.payload),
      })),
    }));

    // Check for partial failures — SendMessageBatch does NOT throw on per-message failure
    if (result.Failed?.length) {
      console.error("Batch send partial failure", result.Failed);
      // Retry the failed entries or surface to DLQ logic
    }
  }
}

// Batch delete in the worker loop after processing a batch
import { DeleteMessageBatchCommand } from "@aws-sdk/client-sqs";

async function batchDelete(receiptHandles: string[]) {
  const chunks = [];
  for (let i = 0; i < receiptHandles.length; i += 10) {
    chunks.push(receiptHandles.slice(i, i + 10));
  }
  for (const chunk of chunks) {
    await sqs.send(new DeleteMessageBatchCommand({
      QueueUrl: QUEUE_URL,
      Entries: chunk.map((rh, i) => ({ Id: String(i), ReceiptHandle: rh })),
    }));
  }
}

IAM policy for MCP server SQS access

Prefer IAM roles over access keys. For ECS Fargate, attach a task role. For Lambda, use the execution role. For EC2, use an instance profile. The minimum permissions for a worker that sends, receives, and deletes:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:SendMessage",
        "sqs:SendMessageBatch",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:DeleteMessageBatch",
        "sqs:ChangeMessageVisibility",
        "sqs:GetQueueAttributes"
      ],
      "Resource": [
        "arn:aws:sqs:us-east-1:123456789012:analysis-jobs",
        "arn:aws:sqs:us-east-1:123456789012:analysis-jobs-dlq"
      ]
    }
  ]
}
// Never give sqs:* — it allows queue deletion and policy changes

For local development, use aws configure sso or AWS Vault rather than long-lived access keys. The SDK credential chain checks IAM roles before environment variables, so a developer with an appropriate IAM identity attached does not need to set AWS_ACCESS_KEY_ID at all.

Common failure modes