AWS KMS CloudTrail Monitoring for MCP Servers — Key Usage Alerts, Anomaly Detection

Management events vs data events

CloudTrail splits KMS API calls into two event categories with different default logging behaviors and costs:

# Enable KMS data event logging on an existing trail (adds Decrypt, GenerateDataKey, etc.)
# Option 1: log ALL KMS data events in this region (simplest, most complete audit)
aws cloudtrail put-event-selectors \
  --trail-name mcp-audit-trail \
  --event-selectors '[{
    "ReadWriteType": "All",
    "IncludeManagementEvents": true,
    "DataResources": [{
      "Type": "AWS::KMS::Key",
      "Values": ["arn:aws:kms:us-east-1:123456789012:key/abc12345-1234-1234-1234-abcdef123456"]
    }]
  }]'

# Option 2: log data events for ALL KMS keys in this account/region
aws cloudtrail put-event-selectors \
  --trail-name mcp-audit-trail \
  --event-selectors '[{
    "ReadWriteType": "All",
    "IncludeManagementEvents": true,
    "DataResources": [{
      "Type": "AWS::KMS::Key",
      "Values": ["arn:aws:kms:us-east-1:123456789012:key/"]
    }]
  }]'
# Note: trailing slash on the key ARN prefix means "all keys in this account+region"

# Verify data events are now being logged
aws cloudtrail get-event-selectors --trail-name mcp-audit-trail \
  --query 'EventSelectors[].DataResources'

The $0.10 per 100,000 events cost for KMS data events adds up when your MCP server processes thousands of requests per day — each request with an envelope decrypt call = one event. At 1M Decrypt calls per day, KMS data event logging costs $1/day ($30/month). Whether that cost is justified depends on your compliance requirements and breach risk; for regulated environments it is non-negotiable.

CloudWatch alarms for key access anomalies

KMS emits CloudWatch metrics under the AWS/KMS namespace. The most actionable metric is NumberOfRequestsForKeyMaterial (total API calls for a key) but for anomaly detection the key signal is in CloudTrail-based metrics. CloudWatch Logs metric filters on the CloudTrail log group let you create alarms on specific event patterns.

# Create a CloudWatch Logs metric filter that fires on KMS AccessDeniedException
# (indicates a credential compromise attempting to use a key they shouldn't have)
aws logs put-metric-filter \
  --log-group-name "CloudTrail/AllRegions" \
  --filter-name "KMSAccessDenied" \
  --filter-pattern '{ ($.eventSource = "kms.amazonaws.com") && ($.errorCode = "AccessDeniedException") }' \
  --metric-transformations '[{
    "metricName": "KMSAccessDenied",
    "metricNamespace": "MCP/SecurityAlerts",
    "metricValue": "1",
    "defaultValue": 0
  }]'

# Create an alarm that fires after 5 AccessDenied errors in 5 minutes
aws cloudwatch put-metric-alarm \
  --alarm-name "KMS-AccessDenied-Alert" \
  --alarm-description "KMS AccessDeniedException — possible credential compromise or key policy change" \
  --metric-name "KMSAccessDenied" \
  --namespace "MCP/SecurityAlerts" \
  --statistic Sum \
  --period 300 \
  --threshold 5 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-security-alerts \
  --treat-missing-data notBreaching

# Alarm on unusual Decrypt volume (potential bulk exfiltration)
# Baseline: calculate P99 of hourly Decrypt calls in last 30 days, then set threshold at 3x P99
aws cloudwatch put-metric-alarm \
  --alarm-name "KMS-Decrypt-Volume-Spike" \
  --alarm-description "Decrypt call volume 3x above baseline — possible bulk key exfiltration" \
  --metric-name "NumberOfRequestsForKeyMaterial" \
  --namespace "AWS/KMS" \
  --dimensions Name=KeyId,Value=abc12345-1234-1234-1234-abcdef123456 \
  --statistic Sum \
  --period 3600 \
  --threshold 50000 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:mcp-security-alerts

CloudWatch Logs Insights queries for KMS audit

When CloudTrail delivers KMS events to a CloudWatch Logs log group, you can run Logs Insights queries to answer forensic questions without writing custom tooling.

# Query: which principals called Decrypt on this key in the past 24 hours?
# Run in CloudWatch Logs Insights on your CloudTrail log group
fields @timestamp, userIdentity.arn, userIdentity.type, sourceIPAddress,
       requestParameters.encryptionContext, responseElements.keyId
| filter eventSource = "kms.amazonaws.com"
    and eventName = "Decrypt"
    and requestParameters.keyId like "abc12345"
| stats count(*) as decryptCount by userIdentity.arn, sourceIPAddress
| sort decryptCount desc
| limit 20

# Query: all Decrypt calls in the past 7 days grouped by hour (volume trending)
fields @timestamp
| filter eventSource = "kms.amazonaws.com"
    and eventName in ["Decrypt", "GenerateDataKey"]
| stats count(*) as callCount by bin(1h)
| sort @timestamp asc

# Query: failed KMS calls (AccessDeniedException) — who is being denied?
fields @timestamp, userIdentity.arn, eventName, errorCode, errorMessage
| filter eventSource = "kms.amazonaws.com"
    and ispresent(errorCode)
| stats count(*) as errorCount by userIdentity.arn, eventName, errorCode
| sort errorCount desc
| limit 30

# Query: key management events (policy changes, key deletion) — change audit
fields @timestamp, userIdentity.arn, eventName, requestParameters
| filter eventSource = "kms.amazonaws.com"
    and eventName in [
      "PutKeyPolicy", "ScheduleKeyDeletion", "CancelKeyDeletion",
      "DisableKey", "CreateGrant", "RevokeGrant", "DeleteAlias",
      "EnableKeyRotation", "DisableKeyRotation"
    ]
| sort @timestamp desc
| limit 50

# Query: cross-account Decrypt calls (consumer account != key owner account)
fields @timestamp, userIdentity.accountId, userIdentity.arn, eventName
| filter eventSource = "kms.amazonaws.com"
    and eventName = "Decrypt"
    and userIdentity.accountId != "123456789012"
| stats count(*) as crossAccountDecrypts by userIdentity.accountId, userIdentity.arn
| sort crossAccountDecrypts desc

EventBridge rules for key management events

CloudWatch Logs queries are retrospective. For real-time alerting on critical key management operations use EventBridge rules that match CloudTrail events — these fire within seconds of the API call.

# EventBridge rule: alert immediately on any ScheduleKeyDeletion call
# (key deletion cannot be instantly cancelled by automation — you need immediate human review)
aws events put-rule \
  --name "KMS-Key-Deletion-Alert" \
  --description "Fire immediately when any KMS key is scheduled for deletion" \
  --event-pattern '{
    "source": ["aws.kms"],
    "detail-type": ["AWS API Call via CloudTrail"],
    "detail": {
      "eventSource": ["kms.amazonaws.com"],
      "eventName": ["ScheduleKeyDeletion"]
    }
  }' \
  --state ENABLED

aws events put-targets \
  --rule "KMS-Key-Deletion-Alert" \
  --targets '[{
    "Id": "SnsAlert",
    "Arn": "arn:aws:sns:us-east-1:123456789012:mcp-security-alerts",
    "InputTransformer": {
      "InputPathsMap": {
        "keyId": "$.detail.requestParameters.keyId",
        "principal": "$.detail.userIdentity.arn",
        "time": "$.time",
        "pendingDays": "$.detail.requestParameters.pendingWindowInDays"
      },
      "InputTemplate": "\"KMS KEY DELETION SCHEDULED: Key <keyId> scheduled by <principal> at <time> with <pendingDays> day waiting period. Cancel with: aws kms cancel-key-deletion --key-id <keyId>\""
    }
  }]'

# EventBridge rule: alert on key policy changes
aws events put-rule \
  --name "KMS-Policy-Change-Alert" \
  --event-pattern '{
    "source": ["aws.kms"],
    "detail-type": ["AWS API Call via CloudTrail"],
    "detail": {
      "eventSource": ["kms.amazonaws.com"],
      "eventName": ["PutKeyPolicy", "DisableKey", "RevokeGrant"]
    }
  }' \
  --state ENABLED

The 7–30 day deletion waiting period exists precisely to give EventBridge alerts time to reach the right people and trigger a cancel-key-deletion call. Configure SNS to route the deletion alert to PagerDuty or on-call — the moment it hits email-only routing is when someone is on vacation and the window closes.

Using CloudTrail Lake for long-term KMS audit

CloudTrail Lake stores events in an AWS-managed data store and supports SQL queries against historical event data — useful for compliance audits that need to answer "which principals decrypted data for tenant X over the last 90 days."

# Query CloudTrail Lake for KMS Decrypt events in the past 90 days
aws cloudtrail-data start-query \
  --query-statement "
    SELECT
      eventTime,
      userIdentity.arn AS principal,
      sourceIPAddress,
      requestParameters.encryptionContext AS context,
      errorCode
    FROM <event-data-store-arn>
    WHERE eventSource = 'kms.amazonaws.com'
      AND eventName = 'Decrypt'
      AND eventTime > DATE_ADD('day', -90, NOW())
      AND requestParameters.keyId LIKE '%abc12345%'
    ORDER BY eventTime DESC
    LIMIT 1000
  "
# Returns a queryId; retrieve results with:
aws cloudtrail-data get-query-results --query-id <query-id>

Failure modes reference