AWS Batch Job Lifecycle — SubmitJob, DescribeJobs, retryStrategy

Job state machine

Every Batch job transitions through states in this order. Terminal states are SUCCEEDED and FAILED:

A job in RUNNABLE state is waiting for capacity. If it stays in RUNNABLE for many minutes, check: (1) maxvCpus on the compute environment is too low for the requested resources, (2) the compute environment's status is not VALID, or (3) the requested instance types are not available in the subnets.

Polling job status

DescribeJobsCommand accepts up to 100 job IDs per call. The response jobs array contains full job metadata including status, statusReason, startedAt, stoppedAt, exitCode, and logStreamName for CloudWatch Logs.

import {
  BatchClient,
  DescribeJobsCommand,
  type JobDetail,
} from '@aws-sdk/client-batch';

const batch = new BatchClient({ region: 'us-east-1' });

export async function waitForJob(
  jobId: string,
  opts: { pollIntervalMs?: number; timeoutMs?: number } = {}
): Promise<JobDetail> {
  const pollInterval = opts.pollIntervalMs ?? 15_000;
  const timeout = opts.timeoutMs ?? 60 * 60 * 1000;  // 1 hour default
  const deadline = Date.now() + timeout;

  while (Date.now() < deadline) {
    const response = await batch.send(
      new DescribeJobsCommand({ jobs: [jobId] })
    );
    const job = response.jobs?.[0];
    if (!job) throw new Error(`Job ${jobId} not found`);

    if (job.status === 'SUCCEEDED') return job;
    if (job.status === 'FAILED') {
      throw new Error(
        `Job ${jobId} failed: ${job.statusReason ?? 'unknown reason'} ` +
        `(exit code ${job.attempts?.at(-1)?.container?.exitCode ?? 'N/A'})`
      );
    }

    await new Promise(r => setTimeout(r, pollInterval));
  }

  throw new Error(`Job ${jobId} timed out after ${timeout}ms`);
}

When a job fails, check job.attempts[last].container.reason and job.attempts[last].container.exitCode. The job.logStreamName field gives you the CloudWatch Logs stream name directly — use it to fetch logs with GetLogEventsCommand from @aws-sdk/client-cloudwatch-logs.

import {
  CloudWatchLogsClient,
  GetLogEventsCommand,
} from '@aws-sdk/client-cloudwatch-logs';

const logs = new CloudWatchLogsClient({ region: 'us-east-1' });

export async function getJobLogs(
  logStreamName: string,
  logGroupName = '/aws/batch/job'
): Promise<string[]> {
  const lines: string[] = [];
  let nextForwardToken: string | undefined;

  do {
    const response = await logs.send(new GetLogEventsCommand({
      logGroupName,
      logStreamName,
      startFromHead: true,
      nextToken: nextForwardToken,
    }));
    for (const event of response.events ?? []) {
      lines.push(event.message ?? '');
    }
    // GetLogEvents returns the same token when at the end
    if (nextForwardToken === response.nextForwardToken) break;
    nextForwardToken = response.nextForwardToken;
  } while (nextForwardToken);

  return lines;
}

Listing jobs by status

ListJobsCommand returns jobs for a specific queue filtered by status. It is paginated via nextToken. Use it to build dashboards or to find jobs to retry after a Spot interruption event.

import {
  ListJobsCommand,
  type JobStatus,
  type JobSummary,
} from '@aws-sdk/client-batch';

export async function listJobsByStatus(
  jobQueue: string,
  status: JobStatus,  // 'SUBMITTED' | 'PENDING' | 'RUNNABLE' | 'STARTING' | 'RUNNING' | 'SUCCEEDED' | 'FAILED'
  maxResults = 100,
): Promise<JobSummary[]> {
  const jobs: JobSummary[] = [];
  let nextToken: string | undefined;

  do {
    const response = await batch.send(new ListJobsCommand({
      jobQueue,
      jobStatus: status,
      maxResults: Math.min(maxResults - jobs.length, 100),
      nextToken,
    }));
    jobs.push(...(response.jobSummaryList ?? []));
    nextToken = response.nextToken;
  } while (nextToken && jobs.length < maxResults);

  return jobs;
}

// Example: find all currently running jobs in a queue
const running = await listJobsByStatus('ml-training-queue', 'RUNNING');
console.log(`${running.length} jobs running`);

ListJobsCommand without a jobStatus filter defaults to listing RUNNING jobs. To list jobs across all statuses, call ListJobsCommand once per status value or use DescribeJobsCommand with specific job IDs you've stored.

Cancelling and terminating jobs

AWS Batch distinguishes between cancellation (before a job runs) and termination (after it starts). Calling the wrong one silently succeeds but has no effect.

import {
  CancelJobCommand,
  TerminateJobCommand,
} from '@aws-sdk/client-batch';

// CancelJobCommand: only works for SUBMITTED, PENDING, or RUNNABLE jobs
// If the job has already moved to STARTING or RUNNING, this call succeeds
// but has NO effect — the job continues to run
export async function cancelQueuedJob(jobId: string, reason: string) {
  await batch.send(new CancelJobCommand({ jobId, reason }));
}

// TerminateJobCommand: works for STARTING or RUNNING jobs
// Sends SIGTERM to the container; if not exited within a grace period, SIGKILL
// Also works for RUNNABLE jobs (redundant with Cancel but harmless)
export async function terminateRunningJob(jobId: string, reason: string) {
  await batch.send(new TerminateJobCommand({ jobId, reason }));
}

// Safe stop: use whichever is appropriate based on current status
export async function stopJob(jobId: string, reason: string) {
  const response = await batch.send(new DescribeJobsCommand({ jobs: [jobId] }));
  const job = response.jobs?.[0];
  if (!job) return;

  const preRunStates = new Set(['SUBMITTED', 'PENDING', 'RUNNABLE']);
  const runningStates = new Set(['STARTING', 'RUNNING']);

  if (preRunStates.has(job.status!)) {
    await cancelQueuedJob(jobId, reason);
  } else if (runningStates.has(job.status!)) {
    await terminateRunningJob(jobId, reason);
  }
  // SUCCEEDED/FAILED: already terminal, no action needed
}

Retry strategy with evaluateOnExit

The default retry behavior retries on any non-zero exit code, up to attempts times. evaluateOnExit rules let you specify per-exit-code or per-reason behavior — either RETRY or EXIT (stop retrying immediately). Rules are evaluated in order; the first match wins.

import { type RetryStrategy } from '@aws-sdk/client-batch';

// Smart retry: retry on Spot interruption, fail fast on application errors
const retryStrategy: RetryStrategy = {
  attempts: 5,
  evaluateOnExit: [
    {
      // Spot interruption: statusReason contains this phrase
      onStatusReason: 'Host EC2 (*) was terminated',
      action: 'RETRY',
    },
    {
      // OOM kill: container killed by kernel, exit code 137
      onExitCode: '137',
      action: 'RETRY',
    },
    {
      // Explicit non-retryable exit codes from your application
      onExitCode: '1',
      action: 'EXIT',
    },
    {
      // Catch-all: retry on any other non-zero exit code
      onExitCode: '[2-9]|[1-9][0-9]+',
      action: 'RETRY',
    },
  ],
};

await batch.send(new RegisterJobDefinitionCommand({
  jobDefinitionName: 'ml-training',
  type: 'container',
  containerProperties: { /* ... */ },
  retryStrategy,
  platformCapabilities: ['FARGATE'],
}));

The onStatusReason, onReason, and onExitCode fields support glob-style wildcards (* matches any sequence of characters). onStatusReason matches the top-level job statusReason; onReason matches the container-level reason within attempts[N].container.reason.

Important: evaluateOnExit rules are only evaluated when a job attempt ends with a non-successful exit. The first rule whose criteria match determines the action — subsequent rules are skipped. If no rule matches, the default is RETRY up to attempts.

Timeout configuration

Use timeout.attemptDurationSeconds in RegisterJobDefinitionCommand to set a wall-clock timeout per attempt. When the timer expires, Batch sends SIGTERM, waits 30 seconds, then sends SIGKILL. The job transitions to FAILED with statusReason: Job attempt duration exceeded timeout.

await batch.send(new RegisterJobDefinitionCommand({
  jobDefinitionName: 'etl-job',
  type: 'container',
  containerProperties: { /* ... */ },
  // Terminate the job if it runs longer than 2 hours
  timeout: { attemptDurationSeconds: 7200 },
  retryStrategy: { attempts: 2 },
  platformCapabilities: ['EC2'],
}));

You can also set a per-submission timeout override in SubmitJobCommand:

await batch.send(new SubmitJobCommand({
  jobName:       'etl-run-2026-11-07',
  jobQueue:      'etl-queue',
  jobDefinition: 'etl-job:3',
  // Override the job definition timeout for this specific run
  timeout: { attemptDurationSeconds: 3600 },
}));

Related pages

• AWS Batch overview — CreateComputeEnvironment, CreateJobQueue, RegisterJobDefinition — set up the infrastructure before submitting jobs
• AWS Batch container properties — resourceRequirements, Fargate, logConfiguration — full container configuration reference
• AWS Batch array jobs and job dependencies — arrayProperties, dependsOn — fan-out patterns and sequential pipelines
• AWS Step Functions for MCP servers — orchestrate Batch jobs in a state machine
• AWS CloudWatch for MCP servers — query Batch job logs