AWS DevOps Pipeline for MCP Servers: ECR, CDK, CodeBuild, Parameter Store, and Secrets Manager — Image Registry, Infrastructure as Code, CI Builds, and Configuration Management

Pattern 1 — The ECR IAM surface: GetAuthorizationToken, two-role model, and registry hardening

Amazon ECR is a regional container image registry. Before any image can be pulled or pushed, Docker must authenticate to the ECR registry with a short-lived token. That authentication step — calling ECR's GetAuthorizationToken API — is the source of the most common ECR permission failure because it has a different resource scope requirement from every other ECR API call.

The GetAuthorizationToken resource scope requirement

ecr:GetAuthorizationToken returns a base64-encoded Docker credential for the entire registry (all repositories in the account and region). Because it is an account-level API — there is no specific repository ARN in the request — IAM will not evaluate it against a specific repository ARN. If you write the IAM policy with "Resource": "arn:aws:ecr:us-east-1:123456789:repository/mcp-server", the call silently fails even though the policy looks correct. The permission must be "Resource": "*".

All other ECR permissions — ecr:BatchGetImage, ecr:GetDownloadUrlForLayer, ecr:BatchCheckLayerAvailability, ecr:InitiateLayerUpload, ecr:UploadLayerPart, ecr:CompleteLayerUpload, ecr:PutImage — can and should be scoped to the specific repository ARN. The result is a two-statement IAM policy pattern:

// CDK: minimal ECR IAM for ECS execution role (pull only)
executionRole.addToPolicy(new iam.PolicyStatement({
  actions: ["ecr:GetAuthorizationToken"],
  resources: ["*"],               // account-level call — must be wildcard
}));
executionRole.addToPolicy(new iam.PolicyStatement({
  actions: [
    "ecr:BatchCheckLayerAvailability",
    "ecr:GetDownloadUrlForLayer",
    "ecr:BatchGetImage",
  ],
  resources: [repository.repositoryArn],  // scoped to the specific repo
}));

// CDK: CI push role (used by CodeBuild, not ECS)
ciRole.addToPolicy(new iam.PolicyStatement({
  actions: ["ecr:GetAuthorizationToken"],
  resources: ["*"],
}));
ciRole.addToPolicy(new iam.PolicyStatement({
  actions: [
    "ecr:InitiateLayerUpload",
    "ecr:UploadLayerPart",
    "ecr:CompleteLayerUpload",
    "ecr:PutImage",
    "ecr:BatchCheckLayerAvailability",
  ],
  resources: [repository.repositoryArn],
}));

Note that ECS execution roles and CI roles need different ECR permission sets. The execution role only ever pulls — it never pushes. The CI role (attached to the CodeBuild service role) only ever pushes — it never needs to be granted to ECS tasks. Keeping them separate means a compromise of the ECS runtime environment cannot be used to overwrite images in the registry.

In CDK, attaching AmazonECSTaskExecutionRolePolicy to the execution role is easier than writing these grants manually — AWS's managed policy already contains the correct GetAuthorizationToken wildcard plus the pull permissions. For the CodeBuild project, CDK's repository.grantPullPush(project.role!) handles the push-side grants.

The ECR auth token 12-hour expiration

The token returned by GetAuthorizationToken expires after 12 hours. This is not a problem for ECS tasks — ECS calls GetAuthorizationToken fresh at each task start. It is a problem for long-running CI pipelines or build agents that cache the Docker login credential: if the same Docker credential is reused across sessions or across a 12-hour window, the pull will fail with "Your authorization token has expired." The fix is to re-authenticate at the start of every CI job, not once per runner boot:

# buildspec.yml pre_build phase — re-authenticate every build
aws ecr get-login-password --region $AWS_DEFAULT_REGION \
  | docker login --username AWS --password-stdin \
    $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com

Registry hardening: tag immutability and lifecycle policies

Two ECR repository settings should be enabled on every production MCP server repository from the start, not added later as an afterthought.

Tag immutability prevents overwriting an existing image tag. Once :v1.2.3 is pushed, a second push with that tag returns ImageAlreadyExistsException. This is the correct behavior for production — it prevents silent overwrites and makes the image history auditable. The tradeoff is that workflows that push :latest on every build must be updated to use a unique tag per build (a git SHA works perfectly, as discussed in Pattern 3). Do not enable tag immutability without also updating the CI pipeline to use git SHA tags.

Lifecycle policies cap storage costs. Without a lifecycle policy, every image push accumulates in ECR at $0.10 per GB per month. The default pattern that works for most MCP server repositories:

// CDK: ECR repository with lifecycle policy and tag immutability
const repository = new ecr.Repository(this, "McpRepo", {
  repositoryName: "mcp-server",
  imageTagMutability: ecr.TagMutability.IMMUTABLE,
  imageScanOnPush: true,
  removalPolicy: cdk.RemovalPolicy.RETAIN,
  lifecycleRules: [
    {
      rulePriority: 1,
      description: "Keep last 10 release images",
      tagStatus: ecr.TagStatus.TAGGED,
      tagPrefixList: ["v", "release-"],
      maxImageCount: 10,
    },
    {
      rulePriority: 2,
      description: "Delete untagged images after 1 day",
      tagStatus: ecr.TagStatus.UNTAGGED,
      maxImageAge: cdk.Duration.days(1),
    },
    {
      rulePriority: 3,
      description: "Hard cap at 50 images total",
      tagStatus: ecr.TagStatus.ANY,
      maxImageCount: 50,
    },
  ],
});

The critical rule is the untagged image cleanup. When tag immutability is off and CI pushes to :latest on every build, the old image becomes untagged but remains in the registry. Rule 2 deletes these within a day. Without it, 5 builds per day × 500 MB per image × $0.10/GB-month = $7.50/month per active repository — and it compounds over time.

For vulnerability scanning: imageScanOnPush: true enables ECR BasicScanning (free, uses Clair, covers OS packages only). Enable Amazon Inspector EnhancedScanning at the registry level for Node.js dependency scanning — it covers package.json transitive dependencies that BasicScanning misses, at $0.11/image/month.

Pattern 2 — CDK as the glue: bootstrap, RemovalPolicy, cdk diff, and context

AWS CDK v2 is the practical choice for MCP server infrastructure because it models the IAM grant relationships between services as TypeScript method calls — repository.grantPullPush(buildProject.role!), queue.grantConsumeMessages(taskDef.taskRole), dbSecret.grantRead(taskDef.taskRole) — rather than requiring you to construct ARN strings and enumerate action lists manually. The construct library knows which permissions each service needs and generates the minimum IAM policy statements.

Bootstrap: the mandatory first step

CDK requires a bootstrap stack in every account+region combination before any CDK deployment can succeed. The bootstrap stack (CDKToolkit) deploys an S3 bucket for CloudFormation templates and Lambda/Docker assets, an ECR repository for CDK Docker assets, and several IAM roles used by the CDK CLI. Without it, cdk deploy fails with: "This stack uses assets, so the toolkit stack must be deployed to the environment first."

# Bootstrap a single account+region (one-time)
npx cdk bootstrap aws://123456789012/us-east-1

# For multi-account pipelines: trust the tools account
npx cdk bootstrap aws://PROD_ACCOUNT/us-east-1 \
  --trust TOOLS_ACCOUNT \
  --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess

# Re-running bootstrap on an already-bootstrapped account is safe
# It upgrades the bootstrap stack without touching deployed stacks

RemovalPolicy: the silent default that deletes production data

CDK's RemovalPolicy controls what happens to a resource when the CDK stack is destroyed. The safest default would be to retain everything. The actual defaults are: stateful resources (DynamoDB tables, S3 buckets, RDS instances, ECR repositories, Secrets Manager secrets) default to RemovalPolicy.RETAIN; stateless resources (Lambda functions, ECS services, IAM roles) default to RemovalPolicy.DESTROY.

The failure mode is not the default for stateful resources — RETAIN is correct for production. The failure mode is not reading the default and assuming the behavior. Two anti-patterns:

• Over-DESTROY in dev: leaving default RETAIN on a dev stack that gets frequently torn down leaves orphaned ECR repositories, S3 buckets, and Secrets Manager secrets on every cdk destroy. Explicitly set removalPolicy: cdk.RemovalPolicy.DESTROY on dev stacks so teardowns are clean.
• Silent RETAIN in prod: if a production stack is refactored and a resource is removed from the CDK code, CloudFormation will retain the old resource (because RETAIN is the default) but remove it from the stack's managed resources. The resource continues to exist and accrue costs, invisible to CDK and to cdk destroy. Audit orphaned resources periodically.

// Explicit RemovalPolicy — never rely on the default
const dbSecret = new secretsmanager.Secret(this, "DbSecret", {
  secretName: "/mcp-server/prod/db",
  removalPolicy: cdk.RemovalPolicy.RETAIN,   // explicit, not default
});

const repository = new ecr.Repository(this, "McpRepo", {
  removalPolicy: cdk.RemovalPolicy.RETAIN,   // explicit, not default
});

// Development stack: make teardowns clean
// (set in a separate dev stack or via environment-based condition)
const devTable = new dynamodb.Table(this, "DevSessions", {
  removalPolicy: cdk.RemovalPolicy.DESTROY,  // OK for dev
  billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
});

cdk diff: the mandatory pre-deploy gate

cdk diff shows the CloudFormation change set before cdk deploy applies it. Run it on every CI deploy that touches a stateful stack. The output distinguishes Add, Modify, and Remove operations on each resource — skipping it is how production databases get replaced.

The specific scenario to watch for: if a CDK refactor changes the logical ID of a stateful resource (renaming a Construct ID changes the CloudFormation resource's logical ID), CloudFormation will delete the old resource and create a new one. cdk diff will show this as a Modify (replacement). The symptom in the CloudFormation change set is "Replacement": "True" for resources that should never be replaced. If you see that, stop and investigate before deploying.

# CI workflow: require diff before deploy
npx cdk diff McpServerStack 2>&1 | tee /tmp/cdk-diff.txt

# Fail CI if any resource is being replaced unexpectedly
# (adjust the grep for your expected change patterns)
if grep -q "Replacement: true" /tmp/cdk-diff.txt; then
  echo "ERROR: CDK diff shows resource replacement — review before deploying"
  exit 1
fi

npx cdk deploy McpServerStack --require-approval never

cdk.context.json: commit it

During cdk synth, CDK makes live AWS API calls to look up environment-specific values it needs to synthesize correctly — VPC IDs, availability zone names, AMI IDs. It caches these in cdk.context.json. If that file is not committed to the repository, every CI run either fails (if the CI role doesn't have read access to those APIs) or produces a slightly different synthesized template (if the AZ order changes between runs).

Commit cdk.context.json. Delete individual entries from it only when you intentionally want CDK to re-query the live environment (for example, after replacing the VPC). Never .gitignore it.

The complete CDK stack: ECR + ECS + SQS + Secrets Manager in one TypeScript file

The CDK grant methods wire IAM permissions between constructs without manual ARN construction. The L2 construct pattern for a complete MCP server stack:

import * as cdk from "aws-cdk-lib";
import * as ec2 from "aws-cdk-lib/aws-ec2";
import * as ecs from "aws-cdk-lib/aws-ecs";
import * as ecr from "aws-cdk-lib/aws-ecr";
import * as elbv2 from "aws-cdk-lib/aws-elasticloadbalancingv2";
import * as sqs from "aws-cdk-lib/aws-sqs";
import * as secretsmanager from "aws-cdk-lib/aws-secretsmanager";
import * as logs from "aws-cdk-lib/aws-logs";
import { Construct } from "constructs";

export class McpServerStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const vpc = new ec2.Vpc(this, "Vpc", { maxAzs: 2, natGateways: 1 });

    // ECR: tag immutability + lifecycle policy (see Pattern 1)
    const repository = new ecr.Repository(this, "McpRepo", {
      repositoryName: "mcp-server",
      imageTagMutability: ecr.TagMutability.IMMUTABLE,
      imageScanOnPush: true,
      removalPolicy: cdk.RemovalPolicy.RETAIN,
      lifecycleRules: [
        { tagStatus: ecr.TagStatus.UNTAGGED, maxImageAge: cdk.Duration.days(1) },
        { tagStatus: ecr.TagStatus.ANY, maxImageCount: 50 },
      ],
    });

    // SQS: DLQ + visibility timeout set to max job duration
    const dlq = new sqs.Queue(this, "Dlq", {
      retentionPeriod: cdk.Duration.days(14),
    });
    const queue = new sqs.Queue(this, "Queue", {
      visibilityTimeout: cdk.Duration.seconds(300),
      deadLetterQueue: { queue: dlq, maxReceiveCount: 3 },
    });

    // Secrets Manager: auto-generated password, explicit RETAIN
    const dbSecret = new secretsmanager.Secret(this, "DbSecret", {
      secretName: "/mcp-server/prod/db",
      generateSecretString: {
        secretStringTemplate: JSON.stringify({ username: "mcp_user" }),
        generateStringKey: "password",
        excludePunctuation: true,
        passwordLength: 32,
      },
      removalPolicy: cdk.RemovalPolicy.RETAIN,
    });

    const cluster = new ecs.Cluster(this, "Cluster", { vpc });

    const taskDef = new ecs.FargateTaskDefinition(this, "TaskDef", {
      cpu: 512,
      memoryLimitMiB: 1024,
    });

    // fromEcrRepository automatically grants execution role ECR pull permissions
    taskDef.addContainer("app", {
      image: ecs.ContainerImage.fromEcrRepository(repository, process.env.IMAGE_TAG ?? "latest"),
      portMappings: [{ containerPort: 3000 }],
      logging: ecs.LogDrivers.awsLogs({
        streamPrefix: "mcp-server",
        logRetention: logs.RetentionDays.ONE_MONTH,
      }),
      secrets: {
        // ecs.Secret.fromSecretsManager grants execution role read permission automatically
        DB_PASSWORD: ecs.Secret.fromSecretsManager(dbSecret, "password"),
      },
      environment: { SQS_QUEUE_URL: queue.queueUrl, NODE_ENV: "production" },
      healthCheck: {
        command: ["CMD-SHELL", "curl -f http://localhost:3000/health || exit 1"],
        interval: cdk.Duration.seconds(30),
        timeout: cdk.Duration.seconds(5),
        retries: 3,
        startPeriod: cdk.Duration.seconds(60),
      },
    });

    // Grant methods handle IAM policy statements — no ARN strings needed
    queue.grantSendMessages(taskDef.taskRole);
    queue.grantConsumeMessages(taskDef.taskRole);

    const alb = new elbv2.ApplicationLoadBalancer(this, "Alb", {
      vpc, internetFacing: true,
    });
    const service = new ecs.FargateService(this, "Service", {
      cluster, taskDefinition: taskDef, desiredCount: 2,
      circuitBreaker: { rollback: true },
    });

    alb.addListener("Http", { port: 80, open: true })
       .addTargets("McpServer", {
         port: 3000,
         protocol: elbv2.ApplicationProtocol.HTTP,
         targets: [service],
         healthCheck: { path: "/health", interval: cdk.Duration.seconds(30) },
       });
  }
}

Pattern 3 — CodeBuild CI pipeline: privileged mode, git SHA tags, post_build gating, and layer caching

AWS CodeBuild is the managed build service that runs the Docker image build, executes tests, pushes the tagged image to ECR, and writes the image definition artifact consumed by a downstream ECS deploy stage. Three configuration decisions cause the majority of CodeBuild failures for MCP server teams.

Privileged mode: the Docker prerequisite

CodeBuild build environments are containers. Running Docker inside a container requires access to the Docker daemon socket (/var/run/docker.sock), which is disabled by default in CodeBuild for security. Without privileged: true on the build environment, every docker build command fails with "Cannot connect to the Docker daemon at unix:///var/run/docker.sock".

// CDK: CodeBuild project for MCP server Docker builds
const buildProject = new codebuild.Project(this, "McpBuild", {
  projectName: "mcp-server-build",
  source: codebuild.Source.gitHub({
    owner: "my-org",
    repo: "mcp-server",
    webhook: true,
    webhookFilters: [
      codebuild.FilterGroup.inEventOf(codebuild.EventAction.PUSH)
        .andBranchIs("main"),
    ],
  }),
  environment: {
    buildImage: codebuild.LinuxBuildImage.STANDARD_7_0,
    computeType: codebuild.ComputeType.MEDIUM,  // 7 GB — needed for multi-platform builds
    privileged: true,   // REQUIRED for Docker
  },
  cache: codebuild.Cache.local(codebuild.LocalCacheMode.DOCKER_LAYER),
  buildSpec: codebuild.BuildSpec.fromSourceFilename("buildspec.yml"),
  timeout: cdk.Duration.minutes(30),
  logging: {
    cloudWatch: {
      logGroup: new logs.LogGroup(this, "BuildLogs", {
        logGroupName: "/codebuild/mcp-server",
        retention: logs.RetentionDays.ONE_WEEK,
        removalPolicy: cdk.RemovalPolicy.DESTROY,
      }),
    },
  },
});

// Grant CodeBuild service role push permission to ECR
repository.grantPullPush(buildProject.role!);

Compute type selection: SMALL (3 GB, 2 vCPU) is sufficient for single-platform linux/amd64 builds. Use MEDIUM (7 GB) or LARGE (15 GB) for multi-platform builds (linux/amd64,linux/arm64) — QEMU emulation for the ARM target is memory-intensive and frequently OOM-kills on SMALL.

Git SHA image tags: the traceability requirement

Every image pushed to ECR needs a tag that ties it to a specific commit. The CodeBuild environment variable CODEBUILD_RESOLVED_SOURCE_VERSION contains the full git SHA of the source commit that triggered the build. Use the first 8 characters as the ECR image tag. This makes every image traceable to a commit, every rollback possible (just update the ECS task definition to point to a known-good SHA), and every audit trail complete.

Never push only :latest. If you push only :latest: rollbacks require a new push (you cannot roll back to a previous :latest); cdk deploy sees no change in the task definition JSON and does not update the ECS service; tag immutability (which you should enable) blocks the second push of the same tag.

post_build gating: the push guard

In CodeBuild's buildspec phase model, a failure in the build phase does not always prevent post_build from running. The post_build phase runs unless the entire build is cancelled — a failed docker build or failed test run in build can still trigger the ECR push in post_build. Check the environment variable $CODEBUILD_BUILD_SUCCEEDING at the start of every post_build phase to gate the push.

# buildspec.yml — complete MCP server Docker build pipeline
version: 0.2

env:
  variables:
    AWS_DEFAULT_REGION: us-east-1
    ECR_REPO_NAME: mcp-server
  parameter-store:
    AWS_ACCOUNT_ID: /codebuild/aws-account-id  # avoid hardcoding in source

phases:
  pre_build:
    commands:
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION |
          docker login --username AWS --password-stdin
          $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
      - IMAGE_TAG=$CODEBUILD_RESOLVED_SOURCE_VERSION
      - SHORT_SHA=${IMAGE_TAG:0:8}
      - ECR_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$ECR_REPO_NAME
      - docker pull $ECR_URI:cache || true  # pull cache layer (ignore failure if not found)

  build:
    commands:
      - |
        docker build \
          --cache-from $ECR_URI:cache \
          --build-arg BUILDKIT_INLINE_CACHE=1 \
          --tag $ECR_URI:$SHORT_SHA \
          --tag $ECR_URI:cache \
          .
      - docker run --rm $ECR_URI:$SHORT_SHA npm test

  post_build:
    commands:
      # Gate: post_build runs even when build phase fails
      - |
        if [ "$CODEBUILD_BUILD_SUCCEEDING" != "1" ]; then
          echo "Build failed — skipping ECR push"
          exit 0
        fi
      - docker push $ECR_URI:$SHORT_SHA
      - docker push $ECR_URI:cache
      - printf '{"name":"mcp-server","imageUri":"%s"}' "$ECR_URI:$SHORT_SHA" > imagedefinitions.json

artifacts:
  files:
    - imagedefinitions.json

Dockerfile layer ordering for maximum cache reuse

Docker builds layers sequentially. Each layer is cached until any file that the layer depends on changes. The layer ordering that maximizes cache reuse for a Node.js MCP server: copy only package.json and the lockfile first (these rarely change), install dependencies (cache this expensive step until the lockfile changes), then copy the source code (changes every commit) and build.

# Optimized multi-stage Dockerfile for MCP server
FROM node:22-alpine AS deps
WORKDIR /app
COPY package.json pnpm-lock.yaml ./        # Layer 1: rarely changes
RUN corepack enable && pnpm install --frozen-lockfile  # Layer 2: cached until lockfile changes

FROM node:22-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .                                   # Layer 3: changes every commit
RUN pnpm build                             # Layer 4: cached until source changes

FROM node:22-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder /app/dist ./dist
COPY --from=deps /app/node_modules ./node_modules
EXPOSE 3000
HEALTHCHECK --interval=30s --timeout=5s CMD wget -qO- http://localhost:3000/health || exit 1
CMD ["node", "dist/index.js"]

The wrong ordering — COPY . . before RUN pnpm install — invalidates the npm install cache on every source change, turning a 5-second cached install into a 90-second full install on every commit.

For layer caching strategy across builds: the --cache-from $ECR_URI:cache pattern (pull the previous build's cache tag, use it as the layer cache source for the new build, push the new build as the new cache tag) outperforms CodeBuild's S3 cache for Docker layers when the cache hit rate is high. S3 cache is better for npm install steps on large node_modules trees — but benchmark it: on a 500 MB node_modules tree, the S3 upload/download can take 60-90 seconds, which sometimes exceeds the cost of a fresh pnpm install.

Pattern 4 — The Parameter Store vs Secrets Manager configuration split

MCP servers have two categories of configuration that look similar but have fundamentally different lifecycle requirements. Getting the split wrong leads to either security exposure (credentials in Parameter Store without rotation) or unnecessary API cost (calling GetSecretValue per-request).

What goes in SSM Parameter Store

SSM Parameter Store is the right store for non-secret, environment-specific configuration: feature flags, queue URLs, log levels, service endpoint URLs, connection strings with usernames but without embedded passwords. These values:

• Don't rotate — they change only when infrastructure changes (a new queue is provisioned, a service endpoint is updated)
• Can be loaded at startup and cached for the lifetime of the process — they don't need per-request freshness
• Can be organized hierarchically (/mcp-server/prod/QUEUE_URL, /mcp-server/prod/LOG_LEVEL) and loaded in a single paginated GetParametersByPath call

The IAM trap: when ECS injects parameters from SSM via the task definition secrets array, the ECS agent calls ssm:GetParameters (plural batch API), not ssm:GetParameter (singular). Granting only the singular action to the execution role leaves the task failing to start with "AccessDeniedException" logged on the ECS console. The fix is always to grant ssm:GetParameters (plural) to the execution role:

// CDK: execution role for ECS task with SSM parameter injection
executionRole.addToPolicy(new iam.PolicyStatement({
  actions: ["ssm:GetParameters"],  // PLURAL — ECS uses the batch API
  resources: [
    `arn:aws:ssm:${this.region}:${this.account}:parameter/mcp-server/prod/*`,
  ],
}));

// If using SecureString parameters, also grant kms:Decrypt
executionRole.addToPolicy(new iam.PolicyStatement({
  actions: ["kms:Decrypt"],
  resources: [
    `arn:aws:kms:${this.region}:${this.account}:alias/aws/ssm`,
  ],
}));

For MCP server application code that loads config at startup rather than via ECS injection, use GetParametersByPath with pagination:

import { SSMClient, GetParametersByPathCommand } from "@aws-sdk/client-ssm";

const ssm = new SSMClient({ region: process.env.AWS_REGION ?? "us-east-1" });

async function loadConfig(path: string): Promise> {
  const params: Record = {};
  let nextToken: string | undefined;

  // GetParametersByPath returns max 10 per call — paginate until NextToken is undefined
  do {
    const response = await ssm.send(new GetParametersByPathCommand({
      Path: path,
      Recursive: true,
      WithDecryption: true,
      MaxResults: 10,
      NextToken: nextToken,
    }));

    for (const param of response.Parameters ?? []) {
      if (!param.Name || !param.Value) continue;
      const key = param.Name.replace(`${path}/`, "");
      params[key] = param.Value;
    }
    nextToken = response.NextToken;
  } while (nextToken);

  return params;
}

// Load once at startup — cache for the process lifetime
// For long-running processes: refresh on a 5-minute timer for TTL-sensitive values
const config = await loadConfig("/mcp-server/prod");
console.log(JSON.stringify({ level: "INFO", event: "config.loaded", keyCount: Object.keys(config).length }));

What goes in Secrets Manager

AWS Secrets Manager is the right store for credentials that rotate: database passwords, OAuth client secrets, API keys, JWT signing keys. These values:

• Rotate on a schedule — Secrets Manager triggers a Lambda rotation function that creates a new credential, applies it to the target system, tests it, then promotes it to AWSCURRENT
• Have a three-stage lifecycle: AWSPENDING (new version being tested), AWSCURRENT (active version), AWSPREVIOUS (prior version, kept briefly for in-flight requests using old credentials)
• Cost more to call: GetSecretValue at $0.05 per 10,000 API calls — at 1,000 requests per second this is $12,960/month if called per-request; cache with a 1-hour TTL in-process

The ECS secret injection refresh problem: secrets injected via the task definition secrets array are fetched once by the ECS agent at task start and injected as environment variables. They are not refreshed while the task runs. After a rotation event promotes the new password to AWSCURRENT, running ECS tasks still hold the old password. If the rotation Lambda's finishSecret step keeps AWSPREVIOUS valid briefly (standard rotation behavior), there is a window during which both passwords work — which is why rotation should be followed by a force-new-deployment to replace tasks with fresh environment:

# After Secrets Manager rotation completes: force new ECS deployment
aws ecs update-service \
  --cluster mcp-server-cluster \
  --service mcp-server-service \
  --force-new-deployment

Alternatively, switch from ECS injection to runtime GetSecretValue with an in-process cache. This adds the secretsmanager:GetSecretValue permission to the task role (not the execution role) and requires a cache implementation in the application code:

import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";

const client = new SecretsManagerClient({ region: process.env.AWS_REGION });
let cachedSecret: { value: string; fetchedAt: number } | null = null;
const CACHE_TTL_MS = 60 * 60 * 1000;  // 1 hour — balances freshness vs API cost

async function getDbCredentials(): Promise<{ username: string; password: string; host: string }> {
  const now = Date.now();
  if (cachedSecret && now - cachedSecret.fetchedAt < CACHE_TTL_MS) {
    return JSON.parse(cachedSecret.value);
  }
  const response = await client.send(new GetSecretValueCommand({
    SecretId: "/mcp-server/prod/db",
    // VersionStage defaults to AWSCURRENT — always the active value
  }));
  if (!response.SecretString) throw new Error("Secret has no string value");
  cachedSecret = { value: response.SecretString, fetchedAt: now };
  return JSON.parse(response.SecretString);
}

The rotation Lambda: four required steps, all idempotent

Secrets Manager rotation calls the rotation Lambda four times in sequence: createSecret (create a new secret version in AWSPENDING), setSecret (apply the new credential to the target system — the database), testSecret (verify the new credential works), finishSecret (promote AWSPENDING to AWSCURRENT). Each step must be idempotent because Secrets Manager may retry on transient failures.

The most common rotation failure: the Lambda is in a VPC (because the database is in the VPC) but has no route to the Secrets Manager API. The Lambda can create the AWSPENDING version (using the Secrets Manager API via the VPC's internet gateway), but setSecret or testSecret times out trying to reach the same API from a private subnet with no NAT gateway and no VPC endpoint. The fix: add a Secrets Manager VPC interface endpoint to the VPC, or attach the rotation Lambda to a subnet with NAT gateway access.

The configuration split in one table

How the four patterns connect: a full deployment cycle

A production MCP server deployment cycle using all five services looks like this:

1. Developer pushes to main: GitHub webhook triggers a CodeBuild build (Pattern 3)
2. CodeBuild pre_build: authenticates to ECR with GetAuthorizationToken (Pattern 1), pulls the :cache layer
3. CodeBuild build: builds the Docker image with --cache-from $ECR_URI:cache, runs tests inside the container
4. CodeBuild post_build: checks $CODEBUILD_BUILD_SUCCEEDING (Pattern 3), pushes :$SHORT_SHA and :cache to ECR (Pattern 1 — lifecycle policy keeps only 10 tagged + deletes untagged within 1 day)
5. CDK deploy: cdk diff shows the change — only the image tag in the task definition changes (Pattern 2); cdk deploy creates a new ECS task definition revision pointing to the new SHA
6. ECS task start: ECS agent pulls the new image (Pattern 1 — execution role has GetAuthorizationToken + pull permissions), injects SSM parameters (Pattern 4 — GetParameters plural on execution role) and Secrets Manager credentials (Pattern 4 — GetSecretValue on execution role)
7. Application startup: MCP server loads remaining config via GetParametersByPath (Pattern 4), connects to the database using the injected credentials, opens the SQS long-poll loop
8. Secrets rotation (nightly): Secrets Manager triggers the rotation Lambda; finishSecret promotes the new password to AWSCURRENT; a post-rotation automation triggers aws ecs update-service --force-new-deployment (Pattern 4) so the new tasks get the updated credential

The prior article in this series covers the runtime side of this stack — SQS for async tool execution, SNS and EventBridge for event routing, and CloudWatch for observability — which is what runs after these DevOps patterns have delivered the image to ECS and the tasks are running. This article covers the delivery pipeline; that one covers the service itself.

Common failure mode reference

What to read next

This article covered the DevOps pipeline that gets your MCP server image built, stored, and deployed. The related guides cover each service in depth:

• ECR guide — full coverage of IAM permission categories, cross-account pull, vulnerability scanning, lifecycle policy options
• CDK guide — complete McpServerStack with CDK L2 constructs, cross-stack references, context file handling
• CodeBuild guide — full buildspec.yml, three layer caching strategies compared, compute type selection
• Parameter Store guide — Standard vs Advanced tier selection, SecureString with KMS, per-path IAM scoping
• Secrets Manager guide — full four-step rotation Lambda skeleton, cross-account resource policy, 7-day recovery window
• AWS Core Services guide — the runtime side: SQS async execution, SNS/EventBridge event routing, CloudWatch observability

If you run MCP servers in production, AliveMCP pings every public MCP endpoint every 60 seconds and alerts you before your users notice an outage.