SageMaker JumpStart Foundation Models from MCP Server Tools

Deploying a JumpStart model for an MCP tool

JumpStart deployment takes 5-15 minutes depending on model size and instance type availability. Run this once during infrastructure setup — the resulting endpoint URL is used by every MCP tool call:

import boto3
import sagemaker
from sagemaker.jumpstart.model import JumpStartModel

session = sagemaker.Session(boto_session=boto3.Session(region_name="us-east-1"))
role_arn = "arn:aws:iam::123456789012:role/SageMakerJumpStartRole"

# Deploy Llama 3 8B Instruct — smaller variant suitable for conversational MCP tools
model = JumpStartModel(
    model_id="meta-textgeneration-llama-3-8b-instruct",
    model_version="*",    # latest available version
    role=role_arn,
    sagemaker_session=session,
    # Override environment variables for the TGI container
    env={
        "SAGEMAKER_MODEL_SERVER_TIMEOUT": "3600",
        "MAX_INPUT_LENGTH": "8192",
        "MAX_TOTAL_TOKENS": "10240",
    },
)

predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.12xlarge",    # 4× A10G GPUs, sufficient for 8B in 4-bit
    endpoint_name="mcp-llama3-8b-instruct",
    serializer=sagemaker.serializers.JSONSerializer(),
    deserializer=sagemaker.deserializers.JSONDeserializer(),
    # Volume size for model artifacts cache
    volume_size=512,  # GB, needed for large models
)

print(f"Endpoint deployed: {predictor.endpoint_name}")

# Minimum instance types by model family (as of 2026):
# meta-textgeneration-llama-3-8b-instruct:    ml.g5.2xlarge (8B, fp16) or ml.g5.xlarge (4-bit)
# meta-textgeneration-llama-3-70b-instruct:   ml.g5.48xlarge (70B, 4-bit)
# mistral-7b-instruct:                        ml.g5.2xlarge
# falcon-40b-instruct:                        ml.g5.12xlarge
# stable-diffusion-2-1:                       ml.g5.2xlarge
# cohere-gpt-medium:                          ml.g5.xlarge

model_version="*" always selects the latest version available in the JumpStart registry. For production endpoints, pin to a specific version (e.g., model_version="3.1.2") to prevent unexpected container or dependency changes during routine deployments. JumpStart versions are immutable — a new version means a new model artifact and possibly a different container image.

Invoking JumpStart models from MCP tool handlers

Payload format varies by model family. The MCP tool handler must format the request correctly for each backend:

import boto3
import json

sagemaker_runtime = boto3.client("sagemaker-runtime", region_name="us-east-1")

# Llama 3 Instruct format (TGI container — messages API)
def call_llama3(user_prompt: str, system_prompt: str = "") -> str:
    messages = []
    if system_prompt:
        messages.append({"role": "system", "content": system_prompt})
    messages.append({"role": "user", "content": user_prompt})

    payload = {
        "inputs": messages,
        "parameters": {
            "max_new_tokens": 512,
            "temperature": 0.7,
            "top_p": 0.9,
            "do_sample": True,
            "return_full_text": False,
        },
    }
    response = sagemaker_runtime.invoke_endpoint(
        EndpointName="mcp-llama3-8b-instruct",
        ContentType="application/json",
        Accept="application/json",
        Body=json.dumps(payload),
    )
    result = json.loads(response["Body"].read())
    return result[0]["generated_text"]

# Mistral Instruct format (different prompt template — [INST] markers)
def call_mistral(user_prompt: str, system_prompt: str = "") -> str:
    # Mistral uses the [INST] format, not the OpenAI messages API
    if system_prompt:
        prompt = f"[INST] {system_prompt}\n\n{user_prompt} [/INST]"
    else:
        prompt = f"[INST] {user_prompt} [/INST]"

    payload = {
        "inputs": prompt,
        "parameters": {
            "max_new_tokens": 512,
            "temperature": 0.7,
            "top_p": 0.95,
        },
    }
    response = sagemaker_runtime.invoke_endpoint(
        EndpointName="mcp-mistral-7b-instruct",
        ContentType="application/json",
        Accept="application/json",
        Body=json.dumps(payload),
    )
    result = json.loads(response["Body"].read())
    return result[0]["generated_text"]

# Stable Diffusion — binary image response
def generate_image(prompt: str, negative_prompt: str = "") -> bytes:
    payload = {
        "prompt": prompt,
        "negative_prompt": negative_prompt,
        "num_inference_steps": 30,
        "guidance_scale": 7.5,
        "width": 512,
        "height": 512,
    }
    response = sagemaker_runtime.invoke_endpoint(
        EndpointName="mcp-stable-diffusion-2-1",
        ContentType="application/json",
        Accept="image/png",
        Body=json.dumps(payload),
    )
    return response["Body"].read()  # PNG bytes

The return_full_text: False parameter for Llama 3 returns only the model's generated continuation, not the input prompt repeated back. Without it, the TGI container echoes the full input + output — the MCP tool handler would need to strip the prompt prefix before returning the result to the agent.

Inference components for multi-model GPU sharing

Inference Components allow multiple JumpStart models to share a single large GPU instance. Instead of paying for separate ml.g5.48xlarge instances for each model, you allocate GPU slices per component:

sagemaker_client = boto3.client("sagemaker")

# 1. Create endpoint with a large instance (no ProductionVariants model reference)
sagemaker_client.create_endpoint_config(
    EndpointConfigName="mcp-shared-gpu-config",
    ProductionVariants=[
        {
            "VariantName": "AllTraffic",
            "InstanceType": "ml.p4d.24xlarge",   # 8× A100 GPUs
            "InitialInstanceCount": 1,
            "RoutingConfig": {"RoutingStrategy": "LEAST_OUTSTANDING_REQUESTS"},
        }
    ],
    ExecutionRoleArn="arn:aws:iam::123456789012:role/SageMakerJumpStartRole",
)
sagemaker_client.create_endpoint(
    EndpointName="mcp-shared-gpu-endpoint",
    EndpointConfigName="mcp-shared-gpu-config",
)

# 2. Create inference components — each gets a GPU slice
for model_config in [
    {"name": "llama3-70b", "model_id": "meta-textgeneration-llama-3-70b-instruct",
     "gpu_count": 4, "memory_mb": 180000},
    {"name": "stable-diffusion-xl", "model_id": "stability-ai-stable-diffusion-xl",
     "gpu_count": 2, "memory_mb": 80000},
    {"name": "cohere-embed", "model_id": "cohere-gpt-medium",
     "gpu_count": 1, "memory_mb": 40000},
    {"name": "mistral-8x7b", "model_id": "mistral-mixtral-8x7b-instruct",
     "gpu_count": 1, "memory_mb": 60000},
]:
    sagemaker_client.create_inference_component(
        InferenceComponentName=f"mcp-{model_config['name']}",
        EndpointName="mcp-shared-gpu-endpoint",
        VariantName="AllTraffic",
        Specification={
            "ModelName": model_config["model_id"],
            "ComputeResourceRequirements": {
                "NumberOfAcceleratorDevicesRequired": model_config["gpu_count"],
                "MinMemoryRequiredInMb": model_config["memory_mb"],
            },
        },
        RuntimeConfig={"CopyCount": 1},  # 1 replica per component
    )

# 3. Invoke a specific inference component by name
response = sagemaker_runtime.invoke_endpoint(
    EndpointName="mcp-shared-gpu-endpoint",
    InferenceComponentName="mcp-llama3-70b",   # route to specific component
    ContentType="application/json",
    Accept="application/json",
    Body=json.dumps({"inputs": prompt, "parameters": {"max_new_tokens": 256}}),
)

Inference components are the preferred pattern when an MCP server needs to call multiple foundation models. Without inference components, each model requires a dedicated instance — four models on ml.g5.48xlarge would cost ~$50/hour. Inference components sharing one ml.p4d.24xlarge cost ~$30/hour total. Scale individual components independently by updating RuntimeConfig.CopyCount.

JumpStart vs Bedrock vs custom containers — choosing for MCP tools

Three options for self-hosted or managed model inference in an MCP server context, with different trade-offs:

• SageMaker JumpStart — model artifacts in your AWS account, full infrastructure control, VPC-isolated, no per-token pricing. Best for: regulated industries (HIPAA, FedRAMP) where data cannot leave your account; consistent cost at high volume (no per-token markup); fine-tuned model variants. Trade-off: higher operational overhead (endpoint lifecycle, scaling, patching); minimum instance costs even at idle; JumpStart model availability is 2-4 weeks behind public model releases.
• Amazon Bedrock — fully managed, pay-per-token, no idle costs, zero infrastructure management. Best for: variable traffic MCP tools; faster access to new model versions; cross-region inference for latency reduction. Trade-off: data processed by Bedrock (though not used for training per AWS terms); per-token cost is higher at volume; less control over model runtime configuration.
• Custom SageMaker container — bring your own Docker image + model weights from HuggingFace Hub or your own registry. Best for: models not available in JumpStart (new releases, proprietary checkpoints); custom serving frameworks (vLLM, llama.cpp); specific quantization formats. Trade-off: highest operational complexity; container maintenance and security patching is your responsibility.

# Comparison: same Llama 3 8B call, three deployment paths

# JumpStart (self-hosted)
response = sagemaker_runtime.invoke_endpoint(
    EndpointName="mcp-llama3-8b-jumpstart",
    ContentType="application/json",
    Body=json.dumps({"inputs": messages, "parameters": {"max_new_tokens": 256}}),
)

# Bedrock (managed)
bedrock_runtime = boto3.client("bedrock-runtime")
response = bedrock_runtime.invoke_model(
    modelId="meta.llama3-8b-instruct-v1:0",
    body=json.dumps({
        "prompt": formatted_prompt,
        "max_gen_len": 256,
        "temperature": 0.7,
    }),
)

# Custom vLLM container (self-hosted, OpenAI-compatible API)
response = sagemaker_runtime.invoke_endpoint(
    EndpointName="mcp-llama3-8b-vllm",
    ContentType="application/json",
    Body=json.dumps({
        "model": "meta-llama/Meta-Llama-3-8B-Instruct",
        "messages": messages,
        "max_tokens": 256,
    }),
)

For most MCP server use cases in 2026, Bedrock is the right starting point — zero idle cost, no endpoint management, and the newest models are available within weeks of release. Migrate to JumpStart when monthly Bedrock costs exceed the break-even point for always-on instances, or when data residency requirements mandate account-local inference.