AWS MediaConvert S3 Workflow — Lambda-Triggered Video Processing and EventBridge Completion Notifications

S3 event notification to Lambda trigger

Configure the S3 input bucket to emit ObjectCreated events when new video files arrive. Filter by prefix (e.g., uploads/) and suffix (e.g., .mp4, .mov, .mkv) to avoid triggering on unrelated objects. Lambda must have a resource policy allowing S3 to invoke it, and S3 must have permission to call Lambda — this is set up via the Lambda resource policy, not IAM role policies.

# Add Lambda resource policy to allow S3 to invoke it
aws lambda add-permission \
  --function-name mediaconvert-job-creator \
  --statement-id s3-trigger \
  --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::my-video-input \
  --source-account 123456789012

# Configure S3 bucket notification
aws s3api put-bucket-notification-configuration \
  --bucket my-video-input \
  --notification-configuration '{
    "LambdaFunctionConfigurations": [{
      "Id": "video-upload-trigger",
      "LambdaFunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:mediaconvert-job-creator",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            { "Name": "prefix", "Value": "uploads/" },
            { "Name": "suffix", "Value": ".mp4" }
          ]
        }
      }
    }]
  }'

# For multiple suffixes, use EventBridge S3 notifications instead:
# Enable EventBridge on the bucket, then write an EventBridge rule
# with pattern matching on detail.object.key using suffix conditions

S3 event notification suffix filters only support a single suffix per configuration — you cannot filter for both .mp4 and .mov in one rule. The workaround is to add separate LambdaFunctionConfigurations entries for each suffix, all pointing to the same Lambda ARN. Alternatively, enable EventBridge integration on the S3 bucket and write an EventBridge rule with content-based filtering — EventBridge supports suffix matching via suffix comparison operator in event patterns, and a single rule can match multiple video formats.

Lambda function: creating the MediaConvert job

The Lambda that creates MediaConvert jobs needs: the account-specific MediaConvert endpoint (cache it between invocations), the IAM role ARN for MediaConvert, and a way to derive the output path from the input key. The idempotency guard uses DynamoDB conditional writes — insert a record keyed on the S3 object ETag before calling MediaConvert; if the insert fails (item already exists), the upload has already been processed and Lambda should return without creating a second job.

import boto3
import os
import json

# Cache endpoint between Lambda invocations (survives warm starts)
_mc_endpoint = None

def get_endpoint(region):
    global _mc_endpoint
    if _mc_endpoint is None:
        client = boto3.client("mediaconvert", region_name=region)
        resp = client.describe_endpoints()
        _mc_endpoint = resp["Endpoints"][0]["Url"]
    return _mc_endpoint

def lambda_handler(event, context):
    region = os.environ["AWS_REGION"]
    role_arn = os.environ["MEDIACONVERT_ROLE_ARN"]
    template_name = os.environ["JOB_TEMPLATE_NAME"]
    output_bucket = os.environ["OUTPUT_BUCKET"]

    ddb = boto3.client("dynamodb")

    for record in event["Records"]:
        bucket = record["s3"]["bucket"]["name"]
        key = record["s3"]["object"]["key"]
        etag = record["s3"]["object"]["eTag"].strip('"')

        # Idempotency guard: fail fast if this ETag was already processed
        try:
            ddb.put_item(
                TableName=os.environ["JOBS_TABLE"],
                Item={
                    "etag": {"S": etag},
                    "input_key": {"S": key},
                    "status": {"S": "SUBMITTED"}
                },
                ConditionExpression="attribute_not_exists(etag)"
            )
        except ddb.exceptions.ConditionalCheckFailedException:
            print(f"Already processed ETag {etag}, skipping")
            continue

        # Derive output prefix from input key:
        # uploads/user-123/video.mp4 -> processed/user-123/video/
        input_name = key.rsplit("/", 1)[-1].rsplit(".", 1)[0]
        user_prefix = key.split("/")[1] if "/" in key else "unknown"
        output_prefix = f"processed/{user_prefix}/{input_name}/"

        endpoint = get_endpoint(region)
        mc = boto3.client(
            "mediaconvert",
            region_name=region,
            endpoint_url=endpoint
        )

        job = mc.create_job(
            Role=role_arn,
            JobTemplate=template_name,
            Settings={
                "Inputs": [{
                    "FileInput": f"s3://{bucket}/{key}",
                    "AudioSelectors": {
                        "Audio Selector 1": {"DefaultSelection": "DEFAULT"}
                    },
                    "VideoSelector": {}
                }],
                "OutputGroups": [{
                    "OutputGroupSettings": {
                        "FileGroupSettings": {
                            "Destination": f"s3://{output_bucket}/{output_prefix}"
                        }
                    }
                }]
            }
        )

        job_id = job["Job"]["Id"]

        # Update DynamoDB with job ID for later correlation
        ddb.update_item(
            TableName=os.environ["JOBS_TABLE"],
            Key={"etag": {"S": etag}},
            UpdateExpression="SET job_id = :jid, input_key = :key",
            ExpressionAttributeValues={
                ":jid": {"S": job_id},
                ":key": {"S": key}
            }
        )

        print(f"Created MediaConvert job {job_id} for s3://{bucket}/{key}")

    return {"statusCode": 200}

The Lambda execution role needs mediaconvert:CreateJob, mediaconvert:DescribeEndpoints, iam:PassRole (scoped to the MediaConvert role ARN), and dynamodb:PutItem / dynamodb:UpdateItem. The iam:PassRole is required because creating a MediaConvert job passes the MediaConvert IAM role — without it, Lambda receives AccessDeniedException at job creation.

EventBridge completion handler

MediaConvert emits job state change events to EventBridge automatically — you do not need to configure anything on the MediaConvert side. Create an EventBridge rule matching source: aws.mediaconvert and detail.status: COMPLETE to trigger a second Lambda that updates your database, notifies the user, and invalidates any CloudFront cache entries for the output files.

# Create EventBridge rule for MediaConvert job completion
aws events put-rule \
  --name "mediaconvert-job-complete" \
  --event-pattern '{
    "source": ["aws.mediaconvert"],
    "detail-type": ["MediaConvert Job State Change"],
    "detail": {
      "status": ["COMPLETE"]
    }
  }' \
  --state ENABLED

# Add the completion Lambda as the rule target
aws events put-targets \
  --rule "mediaconvert-job-complete" \
  --targets '[{
    "Id": "completion-handler",
    "Arn": "arn:aws:lambda:us-east-1:123456789012:function:mediaconvert-completion-handler"
  }]'

# The EventBridge event detail for a COMPLETE job looks like:
# {
#   "version": "0",
#   "id": "...",
#   "source": "aws.mediaconvert",
#   "detail-type": "MediaConvert Job State Change",
#   "detail": {
#     "timestamp": 1696300000000,
#     "accountId": "123456789012",
#     "queue": "arn:aws:mediaconvert:...:queues/Default",
#     "jobId": "1696300000000-abc123",
#     "status": "COMPLETE",
#     "userMetadata": {},
#     "outputGroupDetails": [{
#       "outputDetails": [{
#         "outputFilePaths": ["s3://my-video-output/processed/user-123/video/_1080p.mp4"],
#         "durationInMs": 120000,
#         "videoDetails": { "widthInPx": 1920, "heightInPx": 1080 }
#       }],
#       "type": "FILE_GROUP"
#     }]
#   }
# }

The outputGroupDetails array in the completion event contains the S3 paths of every output file produced — use these to update your database with the playback URLs rather than reconstructing paths from the input key. If a job produces multiple output groups (e.g., MP4 file group + HLS group), outputGroupDetails has one entry per group. For PROGRESSING events, the event also contains jobProgress.jobPercentComplete — useful if you want to stream progress to users via SSE or WebSocket.

Handling job errors and retries

MediaConvert job errors fall into two categories: recoverable (transient S3 access issues, capacity exhaustion) and unrecoverable (corrupt input file, unsupported codec). Create a separate EventBridge rule matching detail.status: ERROR to route failures to a DLQ or alert Lambda. Include the detail.errorMessage in your alert — MediaConvert provides specific error codes and descriptions that indicate whether a retry is appropriate.

# EventBridge rule for job errors
aws events put-rule \
  --name "mediaconvert-job-error" \
  --event-pattern '{
    "source": ["aws.mediaconvert"],
    "detail-type": ["MediaConvert Job State Change"],
    "detail": {
      "status": ["ERROR"]
    }
  }' \
  --state ENABLED

# Error event detail contains:
# {
#   "detail": {
#     "status": "ERROR",
#     "jobId": "1696300000000-abc123",
#     "errorMessage": "Job encountered an error. The input file is not valid.",
#     "errorCode": 1040
#   }
# }

# Common error codes:
# 1040 - Invalid input file (corrupt or unsupported format — do not retry)
# 3450 - S3 access denied on input (fix IAM — then retry)
# 3451 - S3 access denied on output (fix IAM — then retry)
# 4000+ - Transcoding errors (check input format compatibility)

# Retry logic in completion Lambda:
def handle_error(event):
    error_code = event["detail"].get("errorCode", 0)
    job_id = event["detail"]["jobId"]

    # Transient errors — retry the job
    if error_code in (3450, 3451, 3500):
        print(f"Transient error {error_code} on job {job_id} — scheduling retry")
        # Look up input S3 key from DynamoDB by job_id and resubmit
        return resubmit_job(job_id)

    # Permanent errors — notify and do not retry
    print(f"Permanent error {error_code} on job {job_id}: {event['detail']['errorMessage']}")
    notify_user(job_id, error_code)

MediaConvert does not retry jobs automatically — every retry requires a new job creation with a new job ID. Build retry logic in your completion Lambda by looking up the original input key from DynamoDB (using the failed job ID as the lookup key) and submitting a new job. Include a retry count in DynamoDB to cap retries at 2-3 attempts and avoid infinite retry loops on permanently corrupt inputs.

Failure modes reference