Chaos Engineering for MCP Servers — fault injection, resilience testing, and blast radius control

Why chaos engineering matters for MCP servers

An MCP server that has never failed in production has never had its recovery path tested. The monitoring alerts might be misconfigured. The runbook might reference a Slack channel that was archived. The on-call rotation might have the wrong engineer. The process manager might be configured to restart the server but failing silently due to a permissions issue. You discover these problems only when a real outage occurs at 3 AM.

Chaos engineering surfaces these gaps during business hours, in a controlled experiment with a defined blast radius. The Netflix Simian Army popularized this discipline; the principles apply equally to a single MCP server as to a thousand-service microservice mesh. At minimum, run the following three experiments before considering a production MCP server "battle-tested":

1. Process kill. Kill the MCP server process and verify that AliveMCP detects the outage within two probe cycles (≤ 2 minutes), the correct alert channel receives the notification, and the process manager restarts the server within your target recovery time.
2. Dependency failure. Block the server's access to its primary upstream dependency (database, external API) and verify the server degrades gracefully rather than crash-looping, and that your health endpoint returns 503 with a meaningful reason field.
3. Latency injection. Add 2–5 seconds of artificial latency to the server's tool call responses and verify that AliveMCP's P95 alert fires before the latency reaches user-visible impact levels.

The steady-state hypothesis

Every chaos experiment starts with a steady-state hypothesis: a measurable claim about what "normal" looks like, which you verify before and after the experiment to confirm the system returned to its baseline.

For an MCP server, a useful steady-state hypothesis has two components:

// Steady-state definition for a production MCP server
const STEADY_STATE = {
  synthetic_probe: {
    consecutive_failures: 0,       // no recent probe failures
    p95_latency_ms: { max: 500 },  // tools/list completes in under 500ms at P95
  },
  process: {
    uptime_seconds: { min: 3600 }, // running for at least one hour without restart
  },
  health_endpoint: {
    status_code: 200,               // /health returns 200
  },
};

// Measure steady state before and after the experiment
async function measureSteadyState(serverUrl, healthUrl) {
  const probeResult = await probeMcpServer(serverUrl, [], 8000);
  const healthResponse = await fetch(healthUrl);

  return {
    probe_ok: probeResult.ok,
    probe_latency: probeResult.total_ms,
    health_status: healthResponse.status,
    at: new Date().toISOString(),
  };
}

If the system is not in steady state before the experiment, do not run the chaos injection — you would be adding a controlled failure on top of an existing unknown failure, making it impossible to attribute what you observe to your experiment versus the pre-existing condition.

Experiment 1: Process kill

The simplest and highest-value chaos experiment. Kill the MCP server process and measure how long it takes for: (a) AliveMCP to detect the failure, (b) the process manager to restart the process, and (c) the synthetic probe to confirm recovery.

#!/bin/bash
# chaos-kill.sh — kill the MCP server and record recovery metrics
# Run with: bash chaos-kill.sh --server-pid $(pgrep -f "node server.js") --duration 120

SERVER_PID=$1
MONITOR_URL="https://your-alivemcp-dashboard.com/api/servers/production-mcp/status"
HEALTH_URL="https://mcp.yourapp.com/health"

echo "[$(date)] Steady state before kill:"
curl -s "$HEALTH_URL" | jq .

echo "[$(date)] Injecting failure: kill -SIGTERM $SERVER_PID"
kill -SIGTERM "$SERVER_PID"
KILL_TIME=$(date +%s)

# Poll until server is unreachable
until ! curl -sf "$HEALTH_URL" > /dev/null 2>&1; do
  sleep 1
done
OUTAGE_DETECTED=$(date +%s)
echo "[$(date)] Server unreachable after $((OUTAGE_DETECTED - KILL_TIME))s"

# Poll until server recovers
until curl -sf "$HEALTH_URL" > /dev/null 2>&1; do
  sleep 5
done
RECOVERY_TIME=$(date +%s)
echo "[$(date)] Server recovered after $((RECOVERY_TIME - KILL_TIME))s total"

echo "[$(date)] Steady state after recovery:"
curl -s "$HEALTH_URL" | jq .

What to verify during a process kill experiment:

• AliveMCP fires an alert within 2 minutes (2 consecutive probe failures at 60s interval)
• The alert includes failure_reason: connection_refused (not a generic "server down" message)
• The alert reaches the correct channel (Slack, PagerDuty, or OpsGenie)
• The process manager (PM2, systemd, Docker restart policy) restarts the process automatically
• AliveMCP fires a recovery notification after the server comes back online
• Total time from kill to recovery is within your stated RTO

Experiment 2: Network latency injection

Use the Linux tc (traffic control) tool to add artificial latency to outbound network traffic from the MCP server. This simulates a slow upstream dependency, a congested network path, or a database under load — without actually taking the dependency down.

#!/bin/bash
# Inject 2000ms latency on the MCP server's outbound port 5432 (PostgreSQL)
# Requires root / sudo

INTERFACE="eth0"
LATENCY_MS=2000
TARGET_PORT=5432

# Add queueing discipline
tc qdisc add dev "$INTERFACE" root handle 1: prio

# Add latency for traffic to target port
tc qdisc add dev "$INTERFACE" parent 1:3 handle 30: netem delay "${LATENCY_MS}ms"
tc filter add dev "$INTERFACE" protocol ip parent 1: prio 3 \
  u32 match ip dport "$TARGET_PORT" 0xffff flowid 1:3

echo "Latency injected: +${LATENCY_MS}ms on port $TARGET_PORT"
echo "Run 'tc qdisc del dev $INTERFACE root' to remove"

For environments where tc is not available (managed containers, macOS dev), use fault-injection middleware in the MCP server itself. A delay middleware that activates via an environment variable gives you the same control without requiring root access:

// Fault injection middleware — activated by CHAOS_DELAY_MS environment variable
function createFaultInjectionMiddleware(opts = {}) {
  const delayMs = parseInt(process.env.CHAOS_DELAY_MS || '0', 10);
  const errorRate = parseFloat(process.env.CHAOS_ERROR_RATE || '0');

  return async function faultInjection(toolName, args, next) {
    // Artificial latency
    if (delayMs > 0) {
      await new Promise(resolve => setTimeout(resolve, delayMs));
    }

    // Artificial error injection
    if (errorRate > 0 && Math.random() < errorRate) {
      throw new Error(`chaos_injected_error: ${toolName}`);
    }

    return next(toolName, args);
  };
}

During latency injection, verify that AliveMCP's P95 latency alert fires before the latency becomes user-visible. If your P95 alert threshold is set to 2× baseline (e.g., alert when P95 exceeds 1000ms on a server with 200ms baseline), injecting 2000ms of upstream latency should trigger the alert within 1–2 probe cycles.

Experiment 3: Upstream dependency failure

Block the MCP server's access to its primary upstream dependency using iptables rules. This simulates a database crash, an external API outage, or a DNS failure without modifying the application code.

#!/bin/bash
# Block outbound connections to PostgreSQL (port 5432)
# Simulates database outage from the MCP server's perspective

TARGET_PORT=5432
RESTORE_AFTER=60  # seconds

echo "Blocking port $TARGET_PORT for ${RESTORE_AFTER}s..."
iptables -A OUTPUT -p tcp --dport "$TARGET_PORT" -j REJECT

sleep "$RESTORE_AFTER"

echo "Restoring connectivity..."
iptables -D OUTPUT -p tcp --dport "$TARGET_PORT" -j REJECT

echo "Done. Check server health:"
curl -s https://mcp.yourapp.com/health | jq .

What a well-implemented MCP server should do during a dependency outage:

The most common failure mode discovered during this experiment: the MCP server's database connection pool throws an error once the connection is established, but the server does not propagate this error to the /health endpoint. The protocol probe continues to pass, AliveMCP sees no failure, and users receive degraded tool responses without any alert firing. Fix: add a dependency health check to the /health endpoint before the chaos experiment surfaces this gap in production.

Experiment 4: Connection pool exhaustion

Simulate high concurrency by opening many simultaneous MCP sessions and verifying the server handles pool saturation gracefully rather than returning wrong results silently.

// load-spike.ts — open N simultaneous MCP connections to exhaust connection pool
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { SSEClientTransport } from '@modelcontextprotocol/sdk/client/sse.js';

async function openSession(serverUrl: string, sessionId: number) {
  const client = new Client({ name: `chaos-session-${sessionId}`, version: '1.0' }, {});
  const transport = new SSEClientTransport(new URL(serverUrl));

  await client.connect(transport);

  // Hold the session open and make periodic tool calls
  while (true) {
    const start = Date.now();
    try {
      await client.callTool({ name: 'search_documents', arguments: { query: 'chaos test' } });
      console.log(`Session ${sessionId}: ok, ${Date.now() - start}ms`);
    } catch (err) {
      console.log(`Session ${sessionId}: error — ${err.message}`);
    }
    await new Promise(r => setTimeout(r, 5000));
  }
}

// Open 20 simultaneous sessions (adjust based on your pool size)
const NUM_SESSIONS = 20;
const SERVER_URL = process.env.MCP_SERVER_URL!;

await Promise.all(
  Array.from({ length: NUM_SESSIONS }, (_, i) => openSession(SERVER_URL, i))
);

Expected observations at pool saturation: individual tool call latency rises (P95 spike visible in AliveMCP), new session connection attempts queue or reject, the /ready probe starts returning 503 when pool.waitingCount exceeds threshold. What you do not want to see: empty results returned silently without error, or the server accepting connections indefinitely until the host OOMs.

Blast radius management

Chaos experiments should have defined blast radii — the maximum impact the experiment can cause before you abort. For MCP servers, blast radius has two dimensions: who is affected (all users vs a subset) and how severe the impact is (degraded quality vs complete outage).

Run experiments during low-traffic windows (business hours when teams are available, but not peak usage). Never run more than one chaos experiment simultaneously on the same server — overlapping experiments make it impossible to attribute observations to the correct cause.

Using AliveMCP to validate chaos experiments

AliveMCP serves two roles during a chaos experiment. First, it validates that your monitoring is working: if you kill the server and AliveMCP does not alert within 2 probe cycles, your monitoring has a gap. Second, it provides objective recovery timing: the timestamp when AliveMCP's probe first fails and the timestamp when it first passes after recovery give you the precise MTTD (mean time to detect) and MTTR (mean time to recover) for the experiment.

Before each experiment, verify AliveMCP is actively probing. After each experiment, review the AliveMCP incident log to confirm:

• The correct failure_reason was recorded (process kill → connection_refused; latency injection → timeout; dependency failure with 503 health check → custom health check failure)
• The alert was delivered to the correct channel within the expected window
• The recovery notification was fired after the server returned to steady state
• No false positives were generated during probe execution

If AliveMCP did not alert during an experiment where the server was genuinely down, treat this as a P1 finding — your monitoring is broken — and fix it before running any other experiments or shipping new features.

Frequently asked questions

Further reading

• Synthetic Monitoring for MCP Servers — external probes and uptime detection
• Smoke Testing MCP Servers — post-deploy validation and deployment gates
• MCP Server Incident Response — runbooks and on-call procedures
• MCP Server Health Checks — liveness, readiness, and custom probes
• MCP Server Circuit Breaker — handling upstream dependency failures
• MCP Server Reliability — SLOs, error budgets, and resilience patterns