AWS Batch Array Jobs and Job Dependencies — arrayProperties, dependsOn

Array jobs — fan-out pattern

An array job creates N child jobs from a single SubmitJobCommand. Each child is identical except for the AWS_BATCH_JOB_ARRAY_INDEX environment variable, which is an integer from 0 to N-1. Your container code uses this index to select its slice of work — a specific S3 key, a row range in a database, or a shard of a dataset.

import {
  BatchClient,
  SubmitJobCommand,
  DescribeJobsCommand,
  ListJobsCommand,
  type JobStatus,
} from '@aws-sdk/client-batch';

const batch = new BatchClient({ region: 'us-east-1' });

export async function submitArrayJob(params: {
  jobName: string;
  jobQueue: string;
  jobDefinition: string;
  size: number;             // number of child jobs (2–10,000)
  environment?: Record<string, string>;
}) {
  const response = await batch.send(new SubmitJobCommand({
    jobName:       params.jobName,
    jobQueue:      params.jobQueue,
    jobDefinition: params.jobDefinition,
    arrayProperties: { size: params.size },
    ...(params.environment ? {
      containerOverrides: {
        environment: Object.entries(params.environment).map(
          ([name, value]) => ({ name, value })
        ),
      },
    } : {}),
  }));

  // response.jobId is the PARENT job ID
  // Child job IDs are: {parentId}:0, {parentId}:1, ..., {parentId}:N-1
  return response.jobId!;
}

// Usage: process 500 files in parallel
const parentJobId = await submitArrayJob({
  jobName:       'daily-etl-2026-11-07',
  jobQueue:      'etl-queue',
  jobDefinition: 'file-processor:12',
  size:          500,
  environment: {
    // Your code reads AWS_BATCH_JOB_ARRAY_INDEX to pick the right file
    'S3_BUCKET': 'my-data-bucket',
    'S3_PREFIX': 'input/2026-11-07/',
  },
});

Inside the container, read the index to select your work slice:

// In your container code (Node.js example)
import { S3Client, GetObjectCommand } from '@aws-sdk/client-s3';

const index = parseInt(process.env.AWS_BATCH_JOB_ARRAY_INDEX ?? '0', 10);
const bucket = process.env.S3_BUCKET!;
const prefix = process.env.S3_PREFIX!;

// Pre-generate a manifest file listing all input keys; read the index-th entry
const s3 = new S3Client({ region: 'us-east-1' });
const manifestResponse = await s3.send(new GetObjectCommand({
  Bucket: bucket,
  Key:    `${prefix}manifest.json`,
}));
const keys: string[] = JSON.parse(await manifestResponse.Body!.transformToString());
const myKey = keys[index];  // this child processes only keys[index]

console.log(`Array index ${index}: processing s3://${bucket}/${myKey}`);

Tracking array job progress

Poll the parent job ID with DescribeJobsCommand to get aggregate progress. The response includes an arrayProperties.statusSummary map showing how many children are in each state.

export async function getArrayJobProgress(parentJobId: string) {
  const response = await batch.send(
    new DescribeJobsCommand({ jobs: [parentJobId] })
  );
  const job = response.jobs?.[0];
  if (!job) throw new Error(`Job ${parentJobId} not found`);

  const summary = job.arrayProperties?.statusSummary ?? {};
  const size    = job.arrayProperties?.size ?? 0;

  return {
    parentStatus: job.status,
    size,
    succeeded: summary['SUCCEEDED'] ?? 0,
    failed:    summary['FAILED']    ?? 0,
    running:   summary['RUNNING']   ?? 0,
    runnable:  summary['RUNNABLE']  ?? 0,
    pending:   summary['PENDING']   ?? 0,
    // Percentage complete
    pctComplete: size > 0
      ? Math.round(((summary['SUCCEEDED'] ?? 0) / size) * 100)
      : 0,
  };
}

// Poll until all children succeed or any fail
export async function waitForArrayJob(parentJobId: string): Promise<void> {
  while (true) {
    const progress = await getArrayJobProgress(parentJobId);
    console.log(
      `Array job ${parentJobId}: ${progress.pctComplete}% complete ` +
      `(${progress.succeeded}/${progress.size} succeeded, ${progress.failed} failed)`
    );

    if (progress.parentStatus === 'SUCCEEDED') return;
    if (progress.parentStatus === 'FAILED') {
      throw new Error(
        `Array job failed: ${progress.failed} of ${progress.size} children failed`
      );
    }

    await new Promise(r => setTimeout(r, 30_000));  // 30s poll interval for large arrays
  }
}

To inspect individual child jobs, use ListJobsCommand with arrayJobId: parentJobId. This returns the child job summaries filtered by optional jobStatus:

export async function listFailedChildren(
  parentJobId: string
): Promise<string[]> {
  const failed: string[] = [];
  let nextToken: string | undefined;

  do {
    const response = await batch.send(new ListJobsCommand({
      arrayJobId: parentJobId,
      jobStatus:  'FAILED',
      nextToken,
    }));
    for (const job of response.jobSummaryList ?? []) {
      failed.push(job.jobId!);
    }
    nextToken = response.nextToken;
  } while (nextToken);

  return failed;
}

Job dependencies — SEQUENTIAL and N_TO_N

The dependsOn array in SubmitJobCommand creates a dependency on a previously submitted job. The downstream job stays in PENDING state until the dependency resolves.

Two dependency types:

// Pattern 1: Aggregation after fan-out
// Step 1: fan-out processing (500 files in parallel)
const extractJobId = await submitArrayJob({
  jobName:       'extract',
  jobQueue:      'etl-queue',
  jobDefinition: 'extractor:5',
  size:          500,
});

// Step 2: aggregation job — starts only when ALL 500 extractors succeed
const aggregateJobId = (await batch.send(new SubmitJobCommand({
  jobName:       'aggregate',
  jobQueue:      'etl-queue',
  jobDefinition: 'aggregator:3',
  // No type = wait for all children of extractJobId to succeed
  dependsOn: [{ jobId: extractJobId }],
}))).jobId!;

// Pattern 2: N_TO_N — index-matched two-stage pipeline
// Stage 1: transform each of 200 files
const transformJobId = await submitArrayJob({
  jobName: 'transform', jobQueue: 'etl-queue',
  jobDefinition: 'transformer:2', size: 200,
});

// Stage 2: validate each transformed file — child K starts when transform child K succeeds
const validateJobId = (await batch.send(new SubmitJobCommand({
  jobName:       'validate',
  jobQueue:      'etl-queue',
  jobDefinition: 'validator:1',
  arrayProperties: { size: 200 },
  dependsOn: [{ jobId: transformJobId, type: 'N_TO_N' }],
}))).jobId!;

Multi-node parallel jobs

Multi-node parallel (MNP) jobs run a single logical job across multiple EC2 instances simultaneously, using a main node to coordinate. This is designed for tightly-coupled distributed workloads like MPI programs — unlike array jobs where each child is fully independent.

await batch.send(new RegisterJobDefinitionCommand({
  jobDefinitionName: 'distributed-training',
  type: 'multinode',
  nodeProperties: {
    numNodes: 4,
    mainNode: 0,  // which node index is the main node (0-indexed)
    nodeRangeProperties: [
      {
        targetNodes: '0',   // main node: different command or image if needed
        container: {
          image: '123456789012.dkr.ecr.us-east-1.amazonaws.com/training:latest',
          resourceRequirements: [
            { type: 'VCPU',   value: '4' },
            { type: 'MEMORY', value: '8192' },
          ],
          command: ['python', 'main_node.py'],
          jobRoleArn: 'arn:aws:iam::123456789012:role/batch-job-role',
        },
      },
      {
        targetNodes: '1:',  // nodes 1 through N-1 (all worker nodes)
        container: {
          image: '123456789012.dkr.ecr.us-east-1.amazonaws.com/training:latest',
          resourceRequirements: [
            { type: 'VCPU',   value: '4' },
            { type: 'MEMORY', value: '8192' },
          ],
          command: ['python', 'worker_node.py'],
          jobRoleArn: 'arn:aws:iam::123456789012:role/batch-job-role',
        },
      },
    ],
  },
  platformCapabilities: ['EC2'],  // MNP not supported on Fargate
}));

Batch injects these environment variables into each node: AWS_BATCH_JOB_NODE_INDEX (0 to numNodes-1), AWS_BATCH_JOB_MAIN_NODE_INDEX, AWS_BATCH_JOB_MAIN_NODE_PRIVATE_IPV4_ADDRESS (the main node's private IP for inter-node communication), and AWS_BATCH_JOB_NUM_NODES. Use these to bootstrap your MPI/Gloo/NCCL rendezvous.

Fair-share scheduling policies

Fair-share scheduling distributes compute capacity proportionally across multiple users or projects sharing a job queue. When capacity is constrained, the scheduler gives priority to whichever fair-share identifier has consumed less of its allocated share.

import {
  CreateSchedulingPolicyCommand,
  UpdateJobQueueCommand,
} from '@aws-sdk/client-batch';

// Create a fair-share policy for three teams
const policyResponse = await batch.send(new CreateSchedulingPolicyCommand({
  name: 'ml-team-fairshare',
  fairsharePolicy: {
    // After this many seconds, usage history decays by 50%
    // Lower value = more reactive to recent usage; 0 = no decay (pure proportional)
    shareDecaySeconds: 300,
    // Reserve this % of compute for jobs without a share identifier
    computeReservation: 0,
    shares: [
      { shareIdentifier: 'team-search',    weightFactor: 2.0 },  // 2x priority
      { shareIdentifier: 'team-recs',      weightFactor: 1.0 },  // baseline
      { shareIdentifier: 'team-platform',  weightFactor: 0.5 },  // half baseline
    ],
  },
}));

// Attach the scheduling policy to an existing job queue
await batch.send(new UpdateJobQueueCommand({
  jobQueue:            'ml-training-queue',
  schedulingPolicyArn: policyResponse.schedulingPolicyArn,
}));

When submitting jobs, set shareIdentifier to tag the job with a share:

await batch.send(new SubmitJobCommand({
  jobName:         'nightly-training-run',
  jobQueue:        'ml-training-queue',
  jobDefinition:   'model-trainer:8',
  shareIdentifier: 'team-search',  // must match a shareIdentifier in the policy
  schedulingPriorityOverride: 5,   // optional: within a share, higher = scheduled first (1–9999)
}));

Jobs without a shareIdentifier are treated as a single implicit share. schedulingPriorityOverride breaks ties within a share — higher numbers are scheduled first. The job queue's overall priority (set at creation) still governs between queues; fair-share only applies to jobs within the same queue competing for the same compute environment.

Related pages

• AWS Batch overview — CreateComputeEnvironment, CreateJobQueue, RegisterJobDefinition — set up the infrastructure before running array jobs
• AWS Batch job lifecycle — SubmitJob, DescribeJobs, retryStrategy — polling and error handling for individual jobs
• AWS Batch container properties — resourceRequirements, Fargate, logConfiguration — configure the containers inside each array child
• AWS Step Functions for MCP servers — orchestrate multi-stage Batch pipelines in a state machine
• AWS SQS for MCP servers — alternative fan-out using queues instead of array jobs