MCP Server CloudTrail S3 Delivery — log file validation, Athena, lifecycle rules

S3 delivery prefix structure

CloudTrail delivers log files and digest files to separate prefixes within the same bucket. Understanding the layout is essential for Athena table configuration:

s3://mcp-cloudtrail-audit-123456789012/
├── AWSLogs/
│   └── 123456789012/                                  # account ID
│       ├── CloudTrail/                                # LOG FILES — point Athena here
│       │   └── us-east-1/
│       │       └── 2026/
│       │           └── 09/
│       │               └── 19/
│       │                   └── 123456789012_CloudTrail_us-east-1_20260919T1420Z_AbCdEf12.json.gz
│       ├── CloudTrail-Digest/                         # DIGEST FILES — DO NOT include in Athena
│       │   └── us-east-1/
│       │       └── 2026/
│       │           └── 09/
│       │               └── 19/
│       │                   └── 123456789012_CloudTrail-Digest_us-east-1_20260919T1420Z_abc123.json.gz
│       └── CloudTrail-Insight/                        # INSIGHTS EVENTS (if Insights enabled)
│           └── us-east-1/
│               └── 2026/09/19/
│                   └── 123456789012_CloudTrail-Insight_us-east-1_20260919T1430Z_xyz789.json.gz

The Athena LOCATION must point to the CloudTrail/ prefix — not the account ID prefix, not the bucket root. If you point Athena at AWSLogs/123456789012/, the CloudTrailSerde will attempt to parse digest files and fail silently, returning empty results or error counts in query output.

Complete S3 bucket policy for CloudTrail delivery

CloudTrail requires two bucket policy statements to function. Missing either one causes delivery failure:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CloudTrailAclCheck",
      "Effect": "Allow",
      "Principal": { "Service": "cloudtrail.amazonaws.com" },
      "Action": "s3:GetBucketAcl",
      "Resource": "arn:aws:s3:::mcp-cloudtrail-audit-123456789012",
      "Condition": {
        "StringEquals": {
          "AWS:SourceArn": "arn:aws:cloudtrail:us-east-1:123456789012:trail/mcp-audit-trail"
        }
      }
    },
    {
      "Sid": "CloudTrailWrite",
      "Effect": "Allow",
      "Principal": { "Service": "cloudtrail.amazonaws.com" },
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::mcp-cloudtrail-audit-123456789012/AWSLogs/123456789012/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": "bucket-owner-full-control",
          "AWS:SourceArn": "arn:aws:cloudtrail:us-east-1:123456789012:trail/mcp-audit-trail"
        }
      }
    },
    {
      "Sid": "DenyDeleteByAnyone",
      "Effect": "Deny",
      "Principal": "*",
      "Action": ["s3:DeleteObject", "s3:DeleteObjectVersion"],
      "Resource": "arn:aws:s3:::mcp-cloudtrail-audit-123456789012/AWSLogs/*"
    },
    {
      "Sid": "DenyNonTLS",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::mcp-cloudtrail-audit-123456789012",
        "arn:aws:s3:::mcp-cloudtrail-audit-123456789012/*"
      ],
      "Condition": { "Bool": { "aws:SecureTransport": "false" } }
    }
  ]
}

For multi-account or AWS Organizations trails that deliver logs from many accounts into a single bucket, replace the account-specific resource ARN AWSLogs/123456789012/* with AWSLogs/* or enumerate each account ID explicitly. Using AWSLogs/* is simpler for org-level trails but allows any account in the org to write to any prefix — fine if you trust the org boundary.

Log file validation: SHA-256 digest chain

When EnableLogFileValidation: true is set on the trail, CloudTrail generates an hourly digest file for each region. The digest file is a JSON document listing all log files delivered in that hour, including their SHA-256 hash and the S3 key. Each digest file also contains the hash of the previous digest file, forming a tamper-evident chain. If any log file is modified or deleted, validation fails.

# Validate log file integrity for a specific date range
# Requires aws cloudtrail validate-logs CLI command
# Must be run from a principal with s3:GetObject on the audit bucket

aws cloudtrail validate-logs \
  --trail-arn arn:aws:cloudtrail:us-east-1:123456789012:trail/mcp-audit-trail \
  --start-time 2026-09-01T00:00:00Z \
  --end-time 2026-09-19T23:59:59Z \
  --verbose

# Expected output for clean logs:
# Validating log files for trail arn:aws:cloudtrail:us-east-1:...
# 2026-09-19T14:00:00Z - 2026-09-19T15:00:00Z: VALID
# 2026-09-19T15:00:00Z - 2026-09-19T16:00:00Z: VALID
# ...
# Results requested for 2026-09-01T00:00:00Z to 2026-09-19T23:59:59Z
# 456 log file(s) valid, 0 log file(s) INVALID.

# If a log file was deleted or modified:
# 2026-09-19T14:00:00Z - 2026-09-19T15:00:00Z: INVALID
# Log file s3://.../.json.gz: Expected hash abc123, actual hash def456

The validation command reads digest files from S3 and reconstructs the hash chain from the start time. It does not need access to the CloudTrail API — only S3 GetObject on the audit bucket. For compliance audits, run validation monthly and store the output. Note: if you deliberately delete old log files via lifecycle rules, validation will fail for those periods. The validate-logs command accepts a --s3-bucket override to point at the delivery bucket even if the trail is in a different account.

A common mistake: enabling S3 Object Lock WORM mode on the audit bucket in Compliance mode (not Governance mode) prevents lifecycle rule transitions and expirations — the bucket will retain all objects until the Object Lock retention period expires, regardless of lifecycle rules. Use Governance mode for audit buckets where you want lifecycle management but still need immutability protection from operators.

Athena table setup with partition projection

Partition projection avoids the need to run MSCK REPAIR TABLE every day when new partitions (date prefixes) arrive. Configure it once at table creation:

CREATE EXTERNAL TABLE cloudtrail_logs (
  eventVersion STRING,
  userIdentity STRUCT<
    type: STRING,
    principalId: STRING,
    arn: STRING,
    accountId: STRING,
    invokedBy: STRING,
    accessKeyId: STRING,
    userName: STRING,
    sessionContext: STRUCT<
      attributes: STRUCT,
      sessionIssuer: STRUCT
    >
  >,
  eventTime STRING,
  eventSource STRING,
  eventName STRING,
  awsRegion STRING,
  sourceIPAddress STRING,
  userAgent STRING,
  errorCode STRING,
  errorMessage STRING,
  requestParameters STRING,
  responseElements STRING,
  additionalEventData STRING,
  requestId STRING,
  eventId STRING,
  resources ARRAY>,
  eventType STRING,
  apiVersion STRING,
  readOnly STRING,
  recipientAccountId STRING,
  serviceEventDetails STRING,
  sharedEventID STRING,
  vpcEndpointId STRING
)
COMMENT "CloudTrail Logs"
PARTITIONED BY (
  account STRING,
  region STRING,
  year STRING,
  month STRING,
  day STRING
)
ROW FORMAT SERDE "com.amazon.emr.hive.serde.CloudTrailSerde"
STORED AS INPUTFORMAT "com.amazon.emr.cloudtrail.CloudTrailInputFormat"
OUTPUTFORMAT "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
LOCATION "s3://mcp-cloudtrail-audit-123456789012/AWSLogs/"
TBLPROPERTIES (
  "projection.enabled" = "true",
  "projection.account.type" = "enum",
  "projection.account.values" = "123456789012",       -- add all accounts for multi-account
  "projection.region.type" = "enum",
  "projection.region.values" = "us-east-1,us-west-2,eu-west-1",
  "projection.year.type" = "integer",
  "projection.year.range" = "2026,2030",
  "projection.month.type" = "integer",
  "projection.month.range" = "1,12",
  "projection.month.digits" = "2",
  "projection.day.type" = "integer",
  "projection.day.range" = "1,31",
  "projection.day.digits" = "2",
  "storage.location.template" =
    "s3://mcp-cloudtrail-audit-123456789012/AWSLogs/${account}/CloudTrail/${region}/${year}/${month}/${day}"
);

Note the storage template ends at the CloudTrail/ subdirectory — this excludes the CloudTrail-Digest/ and CloudTrail-Insight/ directories from being parsed, preventing serialization errors. If you want to query Insights events, create a separate Athena table with a storage template pointing to CloudTrail-Insight/.

Lifecycle rules for cost management

CloudTrail logs grow continuously. Without lifecycle rules, S3 costs accumulate indefinitely. A typical MCP server with management events only generates roughly 1-5 GB/month of compressed log data; with data events it can be 50-500 GB/month. Design lifecycle transitions based on your compliance retention requirements:

import * as s3 from "aws-cdk-lib/aws-s3";
import * as cdk from "aws-cdk-lib";

const auditBucket = new s3.Bucket(this, "AuditLogs", {
  bucketName: `mcp-cloudtrail-audit-${this.account}`,
  versioned: true,
  encryption: s3.BucketEncryption.S3_MANAGED,
  blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,
  lifecycleRules: [
    {
      id: "log-retention",
      prefix: "AWSLogs/",
      enabled: true,
      transitions: [
        // Active audit window: stay in S3 Standard (fast Athena queries)
        // After 90 days: move to S3-IA (60% storage cost reduction)
        {
          storageClass: s3.StorageClass.INFREQUENT_ACCESS,
          transitionAfter: cdk.Duration.days(90),
        },
        // After 1 year: move to Glacier Instant Retrieval (68% cost reduction vs S3-IA)
        // Athena can query Glacier Instant Retrieval without restore!
        {
          storageClass: s3.StorageClass.GLACIER_INSTANT_RETRIEVAL,
          transitionAfter: cdk.Duration.days(365),
        },
        // After 3 years: move to Glacier Deep Archive (lowest cost — restore takes 12h)
        {
          storageClass: s3.StorageClass.DEEP_ARCHIVE,
          transitionAfter: cdk.Duration.days(1095),
        },
      ],
      // Expire after 7 years (2,555 days) — common compliance requirement
      expiration: cdk.Duration.days(2555),
    },
    {
      id: "digest-retention",
      prefix: "AWSLogs/",
      // Digest files are tiny (< 10KB each) — keep them longer than logs
      // to enable validation after log expiry for compliance proof
      enabled: true,
      expiration: cdk.Duration.days(3650), // 10 years
    },
  ],
});

Glacier Instant Retrieval and Athena: Athena can query objects in S3 Glacier Instant Retrieval without a restore step — the retrieval time is milliseconds, and Athena treats it transparently as S3-IA for query purposes. This makes it the optimal tier for 1-3 year old audit logs that are rarely queried but must remain queryable. Do NOT use S3 Glacier Flexible Retrieval or Deep Archive for logs you need to query without advance notice — those require restore jobs (hours to days).

CloudWatch Logs delivery as a real-time alternative

CloudTrail can deliver events to CloudWatch Logs in near-real-time (typically within 5 minutes versus 5-15 minutes for S3). Enable CloudWatch Logs delivery in addition to S3 delivery for use cases requiring faster alert latency:

import * as logs from "aws-cdk-lib/aws-logs";
import * as iam from "aws-cdk-lib/aws-iam";

const trailLogGroup = new logs.LogGroup(this, "TrailLogGroup", {
  logGroupName: "/aws/cloudtrail/mcp-audit",
  retention: logs.RetentionDays.THREE_MONTHS, // Keep 90 days in CWL (expensive at scale)
});

// CloudTrail needs a dedicated IAM role to write to CloudWatch Logs
const cwlRole = new iam.Role(this, "CloudTrailCwlRole", {
  assumedBy: new iam.ServicePrincipal("cloudtrail.amazonaws.com"),
  inlinePolicies: {
    CloudWatchLogsWrite: new iam.PolicyDocument({
      statements: [
        new iam.PolicyStatement({
          actions: ["logs:CreateLogStream", "logs:PutLogEvents"],
          resources: [`${trailLogGroup.logGroupArn}:log-stream:*`],
        }),
      ],
    }),
  },
});

// CloudTrail L1 construct to add CWL delivery
const cfn = trail.node.defaultChild as cfnTrail.CfnTrail;
cfn.cloudWatchLogsLogGroupArn = trailLogGroup.logGroupArn;
cfn.cloudWatchLogsRoleArn = cwlRole.roleArn;

// Metric filter: count AccessDenied errors by MCP role
new logs.MetricFilter(this, "AccessDeniedFilter", {
  logGroup: trailLogGroup,
  filterPattern: logs.FilterPattern.literal(
    '{ $.errorCode = "AccessDenied" && $.userIdentity.sessionContext.sessionIssuer.arn = "*mcp-server*" }'
  ),
  metricNamespace: "MCP/Audit",
  metricName: "AccessDeniedCount",
  metricValue: "1",
  defaultValue: 0,
});

Keep CloudWatch Logs retention short (90 days or less) — CloudWatch Logs storage costs $0.03/GB/month, versus $0.023/GB/month for S3 Standard. For long-term retention, rely on the S3 delivery with lifecycle rules, and use CloudWatch Logs only for the real-time metric filter and alerting use case.

Failure modes