MCP Server X-Ray Sampling Rules — reservoir, fixed-rate, centralized rules

How sampling works: reservoir + fixed rate

X-Ray applies sampling rules in priority order (lowest number wins). For each incoming request, the SDK evaluates rules top-to-bottom until one matches. Within the matching rule, the decision is: sample this request if either (a) the reservoir for this second has not been exhausted, or (b) a random draw is below the fixed rate.

// Sampling decision pseudocode (SDK internals)
function shouldSample(request: IncomingRequest, rules: SamplingRule[]): boolean {
  for (const rule of sortByPriority(rules)) {
    if (!ruleMatches(rule, request)) continue;

    // Reservoir: a token bucket that refills to rule.ReservoirSize each second
    // Shared per-service-instance via centralized targets, or per-process if local
    if (consumeReservoirToken(rule)) return true;  // always sample within reservoir

    // Fixed rate: probabilistic fallback after reservoir is exhausted
    return Math.random() < rule.FixedRate;
  }
  return false; // no rule matched — should not happen if Default rule exists
}

// At 500 req/sec with ReservoirSize=5 and FixedRate=0.05:
// - First 5 requests in each second: always sampled (reservoir)
// - Remaining 495 requests: 5% sampled = ~25 more
// - Total per second: ~30 traces — roughly 6% sampling rate
// X-Ray pricing: $5 per 1M traces recorded — 30 traces/sec = ~$13/month

Custom sampling rule: CDK and CloudFormation

Define custom rules in CDK or CloudFormation to version-control your sampling configuration. Rules take effect across all service instances within ~10 seconds of creation (centralized sampling target refresh interval).

import * as xray from "@aws-cdk/aws-xray";  // or aws-cdk-lib/aws-xray

// CDK: custom sampling rule for the MCP server
new xray.CfnSamplingRule(this, "McpServerSamplingRule", {
  samplingRule: {
    ruleName: "mcp-server-production",
    priority: 10,           // lower = higher priority; Default rule is 10000
    reservoirSize: 10,      // sample 10 requests/sec per host, always
    fixedRate: 0.02,        // sample 2% of requests beyond the reservoir
    serviceName: "mcp-server",  // matches AWSXRay.openSegment("mcp-server") name
    serviceType: "*",       // "AWS::ECS::Container" to restrict to ECS only
    host: "*",
    httpMethod: "*",
    urlPath: "/mcp*",       // only match MCP endpoints, not /health or /metrics
    resourceArn: "*",
    version: 1,
  },
});

// A separate high-rate rule for /mcp/sse connection establishment specifically
// — you may want 100% sampling of connection events for session debugging
new xray.CfnSamplingRule(this, "McpSseConnectionRule", {
  samplingRule: {
    ruleName: "mcp-sse-connections",
    priority: 5,            // higher priority than the general rule
    reservoirSize: 100,     // always sample up to 100 SSE connections/sec
    fixedRate: 0.5,         // sample 50% beyond that
    serviceName: "mcp-server",
    serviceType: "*",
    host: "*",
    httpMethod: "GET",
    urlPath: "/mcp/sse",    // only SSE connection requests
    resourceArn: "*",
    version: 1,
  },
});

Equivalent CloudFormation (if not using CDK):

# CloudFormation
McpServerSamplingRule:
  Type: AWS::XRay::SamplingRule
  Properties:
    SamplingRule:
      RuleName: mcp-server-production
      Priority: 10
      ReservoirSize: 10
      FixedRate: 0.02
      ServiceName: mcp-server
      ServiceType: "*"
      Host: "*"
      HTTPMethod: "*"
      URLPath: "/mcp*"
      ResourceARN: "*"
      Version: 1

Centralized vs local sampling

Centralized sampling (the default when the task role has the required permissions) fetches sampling rules from the X-Ray API and synchronizes quota allocations across all running instances every 10 seconds. Local sampling falls back to a hard-coded local rules file.

// Centralized sampling (recommended): requires these IAM permissions on the task role:
// - xray:GetSamplingRules
// - xray:GetSamplingTargets
// - xray:GetSamplingStatisticSummaries
// No code changes needed — the SDK uses centralized by default.

// Verify centralized is active: check daemon logs for "Fetch sampling rules" lines.
// If you see "Failed to get sampling rules" → falling back to local → reservoir is
// per-process rather than distributed across the fleet.

// Local sampling (fallback): create sampling-rules.json and pass path to SDK
import AWSXRay from "aws-xray-sdk-node";

// Used when centralized is unavailable (local dev, offline testing)
AWSXRay.middleware.setSamplingRules({
  version: 2,
  rules: [
    {
      description: "MCP SSE connections",
      host: "*",
      http_method: "GET",
      url_path: "/mcp/sse",
      fixed_target: 10,   // equivalent to ReservoirSize
      rate: 0.5,          // fixed rate
    },
    {
      description: "MCP POST endpoints",
      host: "*",
      http_method: "POST",
      url_path: "/mcp*",
      fixed_target: 5,
      rate: 0.02,
    },
  ],
  default: {
    fixed_target: 1,
    rate: 0.05,
  },
});

Sampling rule matching for MCP endpoints

MCP servers expose different URL paths depending on transport: /mcp or /mcp/sse for legacy SSE, /mcp (POST) for Streamable HTTP. Configure separate rules if you want different sampling rates for connections vs health checks.

// URL path patterns supported in X-Ray sampling rules:
// - Exact match:  "/mcp"
// - Prefix match: "/mcp*"    (matches /mcp, /mcp/sse, /mcp/messages)
// - Wildcard:     "*/mcp/*"  (matches any path containing /mcp/)
// X-Ray does NOT support regex in URLPath — use wildcard * only.

// Recommended rule set for a typical MCP server:
//
// Priority 5:  GET /mcp/sse       → ReservoirSize=50,  FixedRate=1.0  (dev) or 0.3 (prod)
//              Rationale: SSE connection events are sparse and highly diagnostic.
//              100% sample in dev; 30% in prod gives good session visibility.
//
// Priority 10: POST /mcp*         → ReservoirSize=10,  FixedRate=0.02
//              Rationale: Streamable HTTP tool calls can be high-frequency.
//              2% + reservoir keeps costs low while preserving error visibility.
//
// Priority 100: GET /health*      → ReservoirSize=0,   FixedRate=0.0
//              Rationale: Health check probes (ECS target group, ALB) generate
//              constant traffic. Zero sampling prevents them from exhausting
//              the reservoir and crowding out real tool call traces.
//
// Priority 10000: Default         → ReservoirSize=1,   FixedRate=0.05
//              The built-in Default rule — catches anything unmatched.

Cost model: estimating X-Ray spend

// AWS X-Ray pricing (us-east-1, 2026):
// - Traces recorded: $5.00 per million (first 100K free per month)
// - Traces retrieved/scanned: $0.50 per million
// - Insights: $35 per group per month if enabled

// Cost estimation for a production MCP server:
// Assumptions: 200 req/sec, ReservoirSize=10, FixedRate=0.05, 3 ECS tasks
//
// Per task per second:
//   Reservoir: min(200, 10) = 10 always-sampled
//   Post-reservoir: (200 - 10) * 0.05 = 9.5 ≈ 9 sampled
//   Total per task: ~19 traces/sec
//
// Fleet total (3 tasks): 19 * 3 = 57 traces/sec
//   Per day: 57 * 86400 = ~4.9M traces
//   Per month: ~150M traces
//   Monthly X-Ray cost: (150M - 0.1M free) / 1M * $5 = ~$750/month
//
// At FixedRate=0.01 instead:
//   Post-reservoir: 190 * 0.01 = 1.9 ≈ 2 per task per sec
//   Fleet: (10 + 2) * 3 = 36 traces/sec
//   Monthly: 93M traces → ~$465/month
//
// Key lever: reduce FixedRate first, then ReservoirSize.
// ReservoirSize=0 + FixedRate=0.01 = pure probabilistic, ~$32/month.

Common failure modes