MCP Server Regression Testing — catching performance regressions, schema drift, and behavioral changes

Three types of MCP server regressions

Performance regressions

A deploy increases tool call latency — P95 rises from 200ms to 800ms without any code change that was expected to affect performance. Common causes: an N+1 database query introduced by a new feature, a dependency update that changed memory behavior, a configuration change that disabled connection pooling, or a database index that was accidentally dropped during a migration.

Performance regressions are insidious because they don't cause errors. The tool call succeeds. The server passes health checks. Users experience slower responses. Without baseline comparison, the degradation is invisible until it's severe enough to cause timeouts.

Schema regressions

A deploy changes the structure of a tool's response in a way that breaks clients that depend on the previous structure. An AI agent calling search_documents parses the response expecting a results array with objects containing text and source fields. After a deploy, the field is renamed to content and url. The agent's parsing code fails silently or throws a runtime error.

Schema regressions are most dangerous when the change is unintentional — a refactoring that renamed a field inside a tool handler without updating the response mapping. They can remain undetected for hours until an agent's parsing logic encounters the changed field name.

Behavioral regressions

A deploy changes what a tool returns for a given input — the response structure is unchanged, but the content is different. A get_user_profile tool that previously returned the user's full name now returns only the first name because a data normalization step was added. The schema is unchanged; the behavior is wrong.

Behavioral regressions are the hardest to catch without golden fixtures because they require knowing what the "correct" output looks like for a specific input.

Performance regression detection

Establishing a baseline

A baseline is the P50 and P95 latency measured over a stable window — at least 24 hours of normal operation with no deployments or incidents. Record the baseline for the critical path: the protocol handshake (initialize + tools/list), your most frequently called tool, and your most latency-sensitive tool.

// baseline-capture.ts — run after a stable 24-hour window with no incidents
async function captureLatencyBaseline(serverUrl: string, toolName: string, iterations = 100) {
  const client = await connectMcpClient(serverUrl);
  const latencies: number[] = [];

  for (let i = 0; i < iterations; i++) {
    const start = Date.now();
    await client.callTool({ name: toolName, arguments: BASELINE_INPUT });
    latencies.push(Date.now() - start);
    await sleep(500);  // pace to avoid load testing the server
  }

  latencies.sort((a, b) => a - b);
  const p50 = latencies[Math.floor(latencies.length * 0.5)];
  const p95 = latencies[Math.floor(latencies.length * 0.95)];
  const p99 = latencies[Math.floor(latencies.length * 0.99)];

  const baseline = { tool: toolName, p50, p95, p99, captured_at: new Date().toISOString(), n: iterations };

  // Save baseline to a versioned file in your repository
  await fs.writeFile('baselines/latency.json', JSON.stringify(baseline, null, 2));
  console.log('Baseline captured:', baseline);
  return baseline;
}

Regression detection at deploy time

After a deployment, run a latency measurement against the new version and compare against the committed baseline. In CI, fail the deployment if P95 exceeds the regression threshold.

// latency-regression-check.ts — run as a post-deploy CI step
async function checkLatencyRegression(serverUrl: string, baselinePath: string) {
  const baseline = JSON.parse(await fs.readFile(baselinePath, 'utf8'));
  const REGRESSION_THRESHOLD = 1.5;  // Alert at 1.5× baseline P95

  const client = await connectMcpClient(serverUrl);
  const latencies: number[] = [];

  // Run 20 iterations (quick regression check, not full baseline)
  for (let i = 0; i < 20; i++) {
    const start = Date.now();
    await client.callTool({ name: baseline.tool, arguments: BASELINE_INPUT });
    latencies.push(Date.now() - start);
    await sleep(200);
  }

  latencies.sort((a, b) => a - b);
  const newP95 = latencies[Math.floor(latencies.length * 0.95)];
  const regressionRatio = newP95 / baseline.p95;

  console.log(`P95 comparison: new=${newP95}ms baseline=${baseline.p95}ms ratio=${regressionRatio.toFixed(2)}×`);

  if (regressionRatio > REGRESSION_THRESHOLD) {
    console.error(`REGRESSION DETECTED: P95 is ${regressionRatio.toFixed(2)}× baseline (threshold: ${REGRESSION_THRESHOLD}×)`);
    process.exit(1);  // Fail the CI step
  }

  console.log('No latency regression detected.');
}

Continuous regression detection with AliveMCP

The deploy-time regression check catches obvious regressions immediately. But some regressions are slow burns — memory leaks that increase latency over hours, database table growth that makes queries gradually slower, cache eviction patterns that only manifest under sustained load. AliveMCP's continuous P95 tracking catches these by recording probe latency on every 60-second cycle and alerting when the sustained P95 exceeds a threshold above the recent rolling average.

Configure your AliveMCP alert threshold based on your baseline: if your healthy P95 is 200ms, set the alert at 500ms (2.5× baseline). AliveMCP fires the alert when the threshold is sustained for multiple consecutive probes — not on a single spike, which filters out transient network blips.

Schema regression detection

The MCP tools/list response includes an inputSchema for each tool — the JSON Schema defining what the tool accepts. This schema is the contract between the server and its clients. Snapshot it in CI and diff across versions to catch breaking changes.

// schema-snapshot.ts — run in CI pre-merge and store output as an artifact
import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { SSEClientTransport } from '@modelcontextprotocol/sdk/client/sse.js';

async function captureSchemaSnapshot(serverUrl: string, outputPath: string) {
  const transport = new SSEClientTransport(new URL(serverUrl));
  const client = new Client({ name: 'schema-capture', version: '1.0' }, {});
  await client.connect(transport);

  const { tools } = await client.listTools();
  await client.close();

  // Stable JSON representation for diffing
  const snapshot = tools
    .sort((a, b) => a.name.localeCompare(b.name))
    .map(tool => ({
      name: tool.name,
      description: tool.description,
      inputSchema: tool.inputSchema,
    }));

  await fs.writeFile(outputPath, JSON.stringify(snapshot, null, 2));
  return snapshot;
}

// In CI: diff current snapshot against main branch snapshot
async function detectSchemaRegression(currentPath: string, baselinePath: string) {
  const current = JSON.parse(await fs.readFile(currentPath, 'utf8'));
  const baseline = JSON.parse(await fs.readFile(baselinePath, 'utf8'));

  const currentByName = Object.fromEntries(current.map(t => [t.name, t]));
  const baselineByName = Object.fromEntries(baseline.map(t => [t.name, t]));

  const regressions: string[] = [];

  for (const [name, baseTool] of Object.entries(baselineByName)) {
    if (!currentByName[name]) {
      regressions.push(`BREAKING: tool '${name}' removed`);
      continue;
    }
    const diff = schemaBreakingChanges(baseTool.inputSchema, currentByName[name].inputSchema);
    regressions.push(...diff.map(d => `BREAKING (${name}): ${d}`));
  }

  return regressions;
}

The key distinction: not all schema changes are regressions. Additions (new optional fields, new optional parameters) are backward-compatible. Removals and type changes are breaking. The schemaBreakingChanges function should flag: removed required fields, type changes on existing fields, removed optional fields that clients likely depend on, and renamed tools.

Behavioral regression detection with golden fixtures

Golden fixtures (also called snapshot tests) capture the expected output for a known input and fail the test when the output changes. For MCP servers, they verify behavioral consistency across versions.

// golden-fixtures/search-documents.json — committed in repository
{
  "tool": "search_documents",
  "input": { "query": "MCP server connection refused", "top_k": 3 },
  "expectations": {
    "structure": {
      "total_results": { "type": "number", "min": 1 },
      "results": {
        "type": "array",
        "minLength": 1,
        "items": {
          "text": { "type": "string", "minLength": 10 },
          "source": { "type": "string" },
          "score": { "type": "number", "min": 0, "max": 1 }
        }
      }
    },
    "content_includes": ["connection_refused", "MCP"]
  }
}

// golden-fixture-test.ts
async function runGoldenFixtures(serverUrl: string, fixtureDir: string) {
  const fixtures = await loadFixtures(fixtureDir);
  const client = await connectMcpClient(serverUrl);
  const failures: string[] = [];

  for (const fixture of fixtures) {
    const result = await client.callTool({
      name: fixture.tool,
      arguments: fixture.input,
    });

    let parsed: unknown;
    try {
      parsed = JSON.parse(result.content[0].text);
    } catch {
      failures.push(`${fixture.tool}: response not parseable as JSON`);
      continue;
    }

    const violations = assertExpectations(parsed, fixture.expectations);
    failures.push(...violations.map(v => `${fixture.tool}: ${v}`));
  }

  return { ok: failures.length === 0, failures };
}

Golden fixtures should be checked into the repository and updated deliberately — only when an intentional behavioral change is expected. When a golden fixture test fails after a deploy, the failure prompts a decision: is this the expected new behavior (update the fixture and document the change) or an unintended regression (roll back or fix)?

Keep golden fixture inputs stable over time. If your search tool indexes a live database, the results for a given query will change as the database contents change. Use a fixed test corpus for golden fixture testing — a small, stable set of documents that you control and that do not change with production data updates. Seed the test corpus in a staging environment and run golden fixture tests there, not against production.

Canary version comparison

For high-risk deployments, run the new version in parallel with the current production version and compare responses. This is regression testing in near-production conditions, not against a static fixture.

// canary-comparison.ts — run after deploying canary alongside production
async function compareCanaryToProduction(
  productionUrl: string,
  canaryUrl: string,
  testInputs: ToolCallInput[],
) {
  const productionClient = await connectMcpClient(productionUrl);
  const canaryClient = await connectMcpClient(canaryUrl);

  const comparisons: ComparisonResult[] = [];

  for (const input of testInputs) {
    const [prodResult, canaryResult] = await Promise.all([
      timedToolCall(productionClient, input),
      timedToolCall(canaryClient, input),
    ]);

    comparisons.push({
      tool: input.name,
      latencyDiff: canaryResult.ms - prodResult.ms,
      schemaMatch: schemasMatch(prodResult.parsed, canaryResult.parsed),
      contentSimilarity: computeSimilarity(prodResult.parsed, canaryResult.parsed),
    });
  }

  const regressions = comparisons.filter(c =>
    c.latencyDiff > 500 ||           // Canary is 500ms slower
    !c.schemaMatch ||                 // Schema changed
    c.contentSimilarity < 0.85        // Content differs significantly
  );

  return { ok: regressions.length === 0, comparisons, regressions };
}

Canary comparison is most useful for detecting behavioral regressions that golden fixtures miss — cases where the output changed for a real-world input that isn't in your fixture set. The tradeoff: canary comparison requires running two versions simultaneously, which adds infrastructure cost, and the "content similarity" comparison requires defining what counts as "meaningfully different" for your specific tools.

AliveMCP as the production regression watchdog

CI-based regression tests run at deploy time. AliveMCP provides ongoing regression detection by tracking P95 latency continuously across the server's lifetime. The two signals complement each other for detecting performance regressions:

• CI latency regression check: catches regressions that are immediately apparent after a deploy (an N+1 query, a missing index). Runs 20 probe calls within 5 minutes of deployment.
• AliveMCP P95 tracking: catches regressions that emerge gradually over hours (memory leak, table growth, cache eviction). Runs every 60 seconds permanently. Alerts when sustained P95 exceeds threshold.

Review the AliveMCP P95 trend line after every deployment. A healthy deployment shows a stable P95 in the post-deploy window. A regression shows a step change upward that correlates precisely with the deploy timestamp — making it easy to attribute the regression to the deployment rather than an external factor.

Frequently asked questions

Further reading

• Synthetic Monitoring for MCP Servers — continuous external probing
• Smoke Testing MCP Servers — post-deploy validation gates
• Four Golden Signals for MCP Servers — latency, traffic, errors, saturation
• MCP Server Schema Evolution — backward-compatible tool changes
• MCP Server Versioning — protocol versioning and client compatibility
• MCP Server Canary Deployment — staged rollouts and version comparison