MCP Server EMR Serverless — CreateApplication, StartJobRun, pre-initialized capacity

Application and job run state machines

An EMR Serverless application and each job run have independent state machines. The application must reach STARTED before you can submit job runs.

Creating and starting an application

An application is a reusable container for job runs. Create it once per workload type (one for Spark ETL jobs, one for Hive queries, etc.) and reuse it across many job run submissions.

import {
  EMRServerlessClient,
  CreateApplicationCommand,
  StartApplicationCommand,
  GetApplicationCommand,
  type ApplicationState,
} from '@aws-sdk/client-emr-serverless';

const emrServerless = new EMRServerlessClient({
  region: process.env.AWS_REGION ?? 'us-east-1',
});

export async function createSparkApplication(params: {
  name: string;
  releaseLabel?: string;
}): Promise<string> {
  const response = await emrServerless.send(new CreateApplicationCommand({
    name: params.name,
    type: 'SPARK',
    releaseLabel: params.releaseLabel ?? 'emr-7.2.0',
    // Worker sizing — override defaults here
    workerTypeSpecifications: {
      Driver: {
        workerConfiguration: {
          cpu: '4 vCPU',
          memory: '16 GB',
          disk: '20 GB',
        },
      },
      Executor: {
        workerConfiguration: {
          cpu: '4 vCPU',
          memory: '16 GB',
          disk: '20 GB',
        },
      },
    },
    // Auto-stop idle application after 15 minutes (saves cost between job runs)
    autoStopConfiguration: {
      enabled: true,
      idleTimeoutMinutes: 15,
    },
    // Auto-start when a job run is submitted (no need to call StartApplication explicitly)
    autoStartConfiguration: {
      enabled: true,
    },
    tags: { ManagedBy: 'alivemcp' },
  }));

  return response.applicationId!;
}

export async function startApplication(applicationId: string): Promise<void> {
  await emrServerless.send(new StartApplicationCommand({ applicationId }));

  // Wait for STARTED state
  const deadline = Date.now() + 5 * 60 * 1000;
  while (Date.now() < deadline) {
    const r = await emrServerless.send(new GetApplicationCommand({ applicationId }));
    const state = r.application?.state as ApplicationState;
    if (state === 'STARTED') return;
    if (state === 'TERMINATED') throw new Error('Application was terminated during start');
    await new Promise(res => setTimeout(res, 10_000));
  }
  throw new Error('Application did not start within 5 minutes');
}

When autoStartConfiguration.enabled is true (the default), submitting a job run on a STOPPED application automatically starts it before queueing the job. This means you do not need to call StartApplicationCommand explicitly for on-demand workloads. For latency-sensitive pipelines where the 60–120 second cold start is unacceptable, use pre-initialized capacity instead.

Submitting a job run with StartJobRunCommand

Job runs reference a Spark or Hive job via the jobDriver field. For Spark jobs, use jobDriver.sparkSubmit with an entryPoint pointing to your Python script or JAR on S3.

import {
  StartJobRunCommand,
  GetJobRunCommand,
  CancelJobRunCommand,
  type JobRunState,
} from '@aws-sdk/client-emr-serverless';

export async function submitSparkJob(params: {
  applicationId: string;
  executionRoleArn: string;
  name: string;
  entryPoint: string;         // s3://bucket/script.py or s3://bucket/app.jar
  entryPointArgs?: string[];
  sparkSubmitParams?: string; // e.g. '--conf spark.executor.memory=8g --num-executors 4'
  logBucket?: string;
  executionTimeoutMinutes?: number;
}): Promise<string> {
  const response = await emrServerless.send(new StartJobRunCommand({
    applicationId: params.applicationId,
    name: params.name,
    executionRoleArn: params.executionRoleArn,
    executionTimeoutMinutes: params.executionTimeoutMinutes ?? 60,
    jobDriver: {
      sparkSubmit: {
        entryPoint: params.entryPoint,
        entryPointArguments: params.entryPointArgs ?? [],
        sparkSubmitParameters: params.sparkSubmitParams ?? '',
      },
    },
    configurationOverrides: params.logBucket ? {
      monitoringConfiguration: {
        // Persist Spark logs to S3 for debugging
        s3MonitoringConfiguration: {
          logUri: `s3://${params.logBucket}/emr-serverless-logs/`,
        },
        // Also enable the managed log store (available in EMR console)
        managedPersistenceMonitoringConfiguration: {
          enabled: true,
        },
      },
    } : undefined,
    tags: { ManagedBy: 'alivemcp', JobName: params.name },
  }));

  return response.jobRunId!;
}

export async function waitForJobRun(
  applicationId: string,
  jobRunId: string,
  timeoutMs = 60 * 60 * 1000,  // 1h
): Promise<JobRunState> {
  const deadline = Date.now() + timeoutMs;
  const TERMINAL: JobRunState[] = ['SUCCESS', 'FAILED', 'CANCELLED'];

  while (Date.now() < deadline) {
    const r = await emrServerless.send(new GetJobRunCommand({
      applicationId,
      jobRunId,
    }));

    const jobRun = r.jobRun!;
    const state = jobRun.state as JobRunState;

    if (TERMINAL.includes(state)) {
      if (state !== 'SUCCESS') {
        throw new Error(
          `Job run ${jobRunId} ${state}: ${jobRun.stateDetails}`
        );
      }
      return state;
    }

    await new Promise(res => setTimeout(res, 15_000));
  }

  // Cancel the job before giving up
  await emrServerless.send(new CancelJobRunCommand({ applicationId, jobRunId }));
  throw new Error(`Job run ${jobRunId} timed out after ${timeoutMs}ms`);
}

Pre-initialized capacity for warm starts

By default, EMR Serverless workers start from scratch when a job run is submitted, taking 60–120 seconds for the first job. Pre-initialized capacity keeps a fixed number of warm workers running 24/7, eliminating this cold start. Use it when job latency matters more than idle cost.

import { CreateApplicationCommand } from '@aws-sdk/client-emr-serverless';

// Application with pre-initialized drivers and executors
export async function createWarmSparkApplication(name: string): Promise<string> {
  const response = await emrServerless.send(new CreateApplicationCommand({
    name,
    type: 'SPARK',
    releaseLabel: 'emr-7.2.0',
    // Keep 2 driver workers + 10 executor workers warm at all times
    initialCapacity: {
      Driver: {
        workerCount: 2,
        workerConfiguration: {
          cpu: '4 vCPU',
          memory: '16 GB',
          disk: '20 GB',
        },
      },
      Executor: {
        workerCount: 10,
        workerConfiguration: {
          cpu: '4 vCPU',
          memory: '16 GB',
          disk: '20 GB',
        },
      },
    },
    // Cap total concurrency to avoid runaway cost
    maximumCapacity: {
      cpu: '400 vCPU',
      memory: '3000 GB',
      disk: '20000 GB',
    },
    // Disable auto-stop so pre-init workers stay warm
    autoStopConfiguration: {
      enabled: false,
    },
    autoStartConfiguration: {
      enabled: true,
    },
  }));

  return response.applicationId!;
}

Pre-initialized capacity cost: EMR Serverless charges $0.052/vCPU-hour and $0.0057/GB-hour. 10 Executor workers at 4vCPU/16GB idle 24/7 ≈ $1.37/hr ($984/mo). Evaluate against the alternative of running a persistent EC2 cluster for the same workload.

Worker sizing guide

EMR Serverless enforces a fixed memory-to-CPU ratio of 4 GB per vCPU for workers. Valid configurations:

For most Spark ETL jobs the default 4 vCPU / 16 GB executor is a reasonable starting point. If you see OOM errors in driver logs (java.lang.OutOfMemoryError: GC overhead limit exceeded), increase the driver memory first (driver collects results). If you see executor OOM during shuffle or broadcast joins, increase executor memory. Disk size (minimum 20 GB, maximum 200 GB) is consumed by shuffle spill — increase it for wide aggregations on large datasets.

VPC connectivity

By default, EMR Serverless workers run in an AWS-managed VPC without access to your VPC. To access RDS, MSK, or VPC-private S3 endpoints from Spark jobs, configure VPC access on the application.

import { CreateApplicationCommand } from '@aws-sdk/client-emr-serverless';

export async function createVpcSparkApplication(params: {
  name: string;
  subnetIds: string[];
  securityGroupIds: string[];
}): Promise<string> {
  const response = await emrServerless.send(new CreateApplicationCommand({
    name: params.name,
    type: 'SPARK',
    releaseLabel: 'emr-7.2.0',
    networkConfiguration: {
      subnetIds: params.subnetIds,
      securityGroupIds: params.securityGroupIds,
    },
  }));

  return response.applicationId!;
}

When VPC is configured, all workers attach ENIs in the specified subnets. Ensure the subnets have NAT Gateway access (or VPC endpoints) for S3 — EMR Serverless workers still need to reach S3 to read/write job data, even with VPC networking enabled.

Cost model vs EMR on EC2

For a typical ETL job running 30 minutes using 4 drivers + 20 executors at 4vCPU/16GB each: EMR Serverless ≈ 24 workers × 4vCPU × 0.5h × $0.052 + 24 × 16GB × 0.5h × $0.0057 ≈ $2.50 per job run. Compare against running an m5.xlarge (4vCPU/16GB) EC2 cluster of equivalent size for 30 minutes at ~$0.192/hr × 20 × 0.5h ≈ $1.92 On-Demand or ~$0.60 Spot. For high-frequency jobs, EC2 Spot clusters will be cheaper; for infrequent bursts, EMR Serverless avoids idle charges.

Related pages

• EMR cluster management — RunJobFlow, AddJobFlowSteps, termination protection — EC2-backed EMR clusters
• EMR Spark job patterns — spark-submit, S3A, Glue metastore, dynamic allocation — Spark-specific configuration
• EMR Studio — CreateStudio, CreateEditor, SparkMagic notebooks — interactive development environment
• AWS Glue ETL jobs for MCP servers
• AWS Lake Formation for MCP servers