MCP Server ECS — Fargate deployment, task roles, health checks, rolling updates

Task definition: the two IAM roles

Every ECS task definition references two distinct IAM roles. Confusing them is the #1 ECS mistake for MCP server developers new to the platform.

// AWS CDK: ECS Fargate task definition with both roles
import * as ecs from "aws-cdk-lib/aws-ecs";
import * as iam from "aws-cdk-lib/aws-iam";
import * as logs from "aws-cdk-lib/aws-logs";

// Execution role: only the ECS agent uses this
const executionRole = new iam.Role(this, "ExecutionRole", {
  assumedBy: new iam.ServicePrincipal("ecs-tasks.amazonaws.com"),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName("service-role/AmazonECSTaskExecutionRolePolicy"),
    // ^^ includes ecr:GetAuthorizationToken, ecr:BatchGetImage, logs:CreateLogStream, logs:PutLogEvents
  ],
});

// Task role: your MCP server's application code uses this for AWS API calls
const taskRole = new iam.Role(this, "TaskRole", {
  assumedBy: new iam.ServicePrincipal("ecs-tasks.amazonaws.com"),
});
taskRole.addToPolicy(new iam.PolicyStatement({
  actions: ["sqs:SendMessage", "sqs:ReceiveMessage", "sqs:DeleteMessage", "s3:GetObject"],
  resources: ["*"],  // narrow to specific ARNs in production
}));

const logGroup = new logs.LogGroup(this, "LogGroup", {
  logGroupName: "/ecs/mcp-server",
  retention: logs.RetentionDays.ONE_MONTH,
});

const taskDefinition = new ecs.FargateTaskDefinition(this, "TaskDef", {
  cpu: 512,        // 0.5 vCPU
  memoryLimitMiB: 1024,
  executionRole,
  taskRole,
});

taskDefinition.addContainer("McpServer", {
  image: ecs.ContainerImage.fromEcrRepository(ecrRepo, "latest"),
  portMappings: [{ containerPort: 3000 }],
  logging: ecs.LogDrivers.awsLogs({
    streamPrefix: "ecs",
    logGroup,
  }),
});

Secrets injection: SSM and Secrets Manager

Never put secrets in the environment array of a task definition — they appear as plaintext in CloudFormation templates and the ECS console. Use the secrets array instead: ECS retrieves the value from SSM Parameter Store or Secrets Manager at task start and injects it as an environment variable, and the execution role (not task role) needs permission to read the secret.

// CDK: inject secrets from SSM and Secrets Manager
import * as ssm from "aws-cdk-lib/aws-ssm";
import * as secretsmanager from "aws-cdk-lib/aws-secretsmanager";

const dbPasswordSecret = secretsmanager.Secret.fromSecretNameV2(this, "DbPassword",
  "mcp-server/db-password");
const apiKeyParam = ssm.StringParameter.fromStringParameterName(this, "ApiKey",
  "/mcp-server/api-key");

// Grant execution role permission to read these
dbPasswordSecret.grantRead(executionRole);
apiKeyParam.grantRead(executionRole);

taskDefinition.addContainer("McpServer", {
  image: ecs.ContainerImage.fromEcrRepository(ecrRepo, "latest"),
  environment: {
    NODE_ENV: "production",     // non-secret config — fine in environment
    PORT: "3000",
    AWS_REGION: "us-east-1",
  },
  secrets: {
    // These become environment variables in the container (DB_PASSWORD, API_KEY)
    // Values fetched at task start; NOT refreshed during container lifetime
    DB_PASSWORD: ecs.Secret.fromSecretsManager(dbPasswordSecret),
    API_KEY: ecs.Secret.fromSsmParameter(apiKeyParam),
  },
});

Secrets are fetched at task start and are not refreshed while the container runs. Rotating a secret requires redeploying (stopping and starting tasks). If a secret is rotated, running tasks continue using the old value until replaced. For short-rotation secrets, prefer fetching from SDK inside the application at call time rather than at startup.

Health check: ALB vs container health check

ECS has two independent health check mechanisms: the ALB target group health check (HTTP request to the container, decides whether to route traffic) and the container health check in the task definition (runs a command inside the container, decides whether to restart it). They serve different purposes and must both be configured correctly.

// Health endpoint in your MCP HTTP server (Express example)
// Must return 200 in < ALB timeout (typically 5 seconds)
app.get("/health", (_req, res) => {
  // Keep this fast — no database queries, no external calls
  // The ALB will mark the task unhealthy if this takes > timeoutSeconds
  res.status(200).json({
    status: "ok",
    uptime: process.uptime(),
    version: process.env.SERVICE_VERSION,
  });
});

// CDK: ALB target group with health check
import * as elbv2 from "aws-cdk-lib/aws-elasticloadbalancingv2";
import * as cdk from "aws-cdk-lib";

const targetGroup = new elbv2.ApplicationTargetGroup(this, "TargetGroup", {
  vpc,
  port: 3000,
  protocol: elbv2.ApplicationProtocol.HTTP,
  targetType: elbv2.TargetType.IP,
  healthCheck: {
    path: "/health",           // MUST be a dedicated fast endpoint — never "/"
    healthyThresholdCount: 2,  // 2 consecutive 200s = healthy
    unhealthyThresholdCount: 3,// 3 consecutive non-200s = unhealthy → task replaced
    interval: cdk.Duration.seconds(30),
    timeout: cdk.Duration.seconds(5),  // health check must respond in 5s
    healthyHttpCodes: "200",
  },
});

// CDK: container health check (separate from ALB health check)
taskDefinition.addContainer("McpServer", {
  // ...
  healthCheck: {
    command: ["CMD-SHELL", "curl -f http://localhost:3000/health || exit 1"],
    interval: cdk.Duration.seconds(30),
    timeout: cdk.Duration.seconds(5),
    retries: 3,
    startPeriod: cdk.Duration.seconds(60),  // grace period for slow startup
  },
});

Rolling deployment: zero-downtime updates

ECS rolling deployments start new tasks, wait until they are healthy (both ALB and container health checks pass), then stop old tasks. The minimumHealthyPercent and maximumPercent service settings control how many tasks run during the transition.

// CDK: ECS service with rolling deployment configuration
const service = new ecs.FargateService(this, "McpService", {
  cluster,
  taskDefinition,
  desiredCount: 2,
  assignPublicIp: false,  // private subnet; ALB routes traffic inbound
  deploymentController: {
    type: ecs.DeploymentControllerType.ECS,  // rolling; not CODE_DEPLOY (blue/green)
  },
  circuitBreaker: {
    rollback: true,  // auto-rollback if deployment fails health checks
  },
  minHealthyPercent: 100,  // never go below 2 healthy tasks during deployment
  maxHealthyPercent: 200,  // allow up to 4 tasks during transition (2 old + 2 new)
});

service.attachToApplicationTargetGroup(targetGroup);

// Deployment flow with these settings on a 2-task service:
// 1. Start 2 new tasks (total = 4, max 200% of desired = 4)
// 2. Wait until both new tasks pass health checks
// 3. Stop 2 old tasks (total = 2, min 100% of desired = 2)
// No request is lost — ALB drains connections before stopping old tasks

Connection draining: when ECS stops a task, it first deregisters the target from the ALB and waits for the deregistration delay (default 300 seconds) before stopping the container. This allows in-flight requests to complete. For MCP servers with long-running tool calls, increase the deregistration delay or add a SIGTERM handler that waits for active requests before exiting.

SIGTERM handler for graceful shutdown

// Handle SIGTERM (sent by ECS when stopping the task)
// Give in-flight MCP tool calls up to 25 seconds to complete
// (ECS force-kills with SIGKILL after 30 seconds)

import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StreamableHTTPServerTransport } from "@modelcontextprotocol/sdk/server/streamableHttp.js";
import { createServer } from "http";

const server = new McpServer({ name: "mcp-server", version: "1.0.0" });
const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined });
const httpServer = createServer(transport.requestHandler);

process.on("SIGTERM", async () => {
  console.log(JSON.stringify({ level: "INFO", event: "shutdown.started" }));

  // Stop accepting new connections
  httpServer.close(() => {
    console.log(JSON.stringify({ level: "INFO", event: "shutdown.http_server_closed" }));
  });

  // Allow up to 25 seconds for in-flight requests to complete
  // ECS sends SIGKILL after 30 seconds (stopTimeout: 30000 in task definition)
  setTimeout(() => {
    console.log(JSON.stringify({ level: "INFO", event: "shutdown.timeout" }));
    process.exit(0);
  }, 25_000);
});

await server.connect(transport);
httpServer.listen(3000);

Common failure modes