Skip to main content

Bedrock Batches

Use Amazon Bedrock Batch Inference API through LiteLLM.

PropertyDetails
DescriptionAmazon Bedrock Batch Inference allows you to run inference on large datasets asynchronously
Provider DocAWS Bedrock Batch Inference ↗

Overview​

Use this to:

  • Run batch inference on large datasets with Bedrock models
  • Control batch model access by key/user/team (same as chat completion models)
  • Manage S3 storage for batch input/output files

(Proxy Admin) Usage​

Here's how to give developers access to your Bedrock Batch models.

1. Setup config.yaml​

  • Specify mode: batch for each model: Allows developers to know this is a batch model
  • Configure S3 bucket and AWS credentials for batch operations
litellm_config.yaml
model_list:
  - model_name: "bedrock-batch-claude"
    litellm_params:
      model: bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0
      #########################################################
      ########## batch specific params ########################
      s3_bucket_name: litellm-proxy
      s3_region_name: us-west-2
      s3_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      s3_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_batch_role_arn: arn:aws:iam::888602223428:role/service-role/AmazonBedrockExecutionRoleForAgents_BB9HNW6V4CV
    model_info: 
      mode: batch # 👈 SPECIFY MODE AS BATCH, to tell user this is a batch model

Required Parameters:

ParameterDescription
s3_bucket_nameS3 bucket for batch input/output files
s3_region_nameAWS region for S3 bucket
s3_access_key_idAWS access key for S3 bucket
s3_secret_access_keyAWS secret key for S3 bucket
aws_batch_role_arnIAM role ARN for Bedrock batch operations. Bedrock Batch APIs require an IAM role ARN to be set.
mode: batchIndicates to LiteLLM this is a batch model

2. Create Virtual Key​

create_virtual_key.sh
curl -L -X POST 'https://{PROXY_BASE_URL}/key/generate' \
-H 'Authorization: Bearer ${PROXY_API_KEY}' \
-H 'Content-Type: application/json' \
-d '{"models": ["bedrock-batch-claude"]}'

You can now use the virtual key to access the batch models (See Developer flow).

(Developer) Usage​

Here's how to create a LiteLLM managed file and execute Bedrock Batch CRUD operations with the file.

1. Create request.jsonl​

  • Check models available via /model_group/info
  • See all models with mode: batch
  • Set model in .jsonl to the model from /model_group/info
bedrock_batch_completions.jsonl
{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "bedrock-batch-claude", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello world!"}], "max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "bedrock-batch-claude", "messages": [{"role": "system", "content": "You are an unhelpful assistant."}, {"role": "user", "content": "Hello world!"}], "max_tokens": 1000}}

Expectation:

  • LiteLLM translates this to the bedrock deployment specific value (e.g. bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0)

2. Upload File​

Specify target_model_names: "<model-name>" to enable LiteLLM managed files and request validation.

model-name should be the same as the model-name in the request.jsonl

bedrock_batch.py
from openai import OpenAI

client = OpenAI(
    base_url="http://0.0.0.0:4000",
    api_key="sk-1234",
)

# Upload file
batch_input_file = client.files.create(
    file=open("./bedrock_batch_completions.jsonl", "rb"), # {"model": "bedrock-batch-claude"} <-> {"model": "bedrock/us.anthropic.claude-3-5-sonnet-20240620-v1:0"}
    purpose="batch",
    extra_body={"target_model_names": "bedrock-batch-claude"}
)
print(batch_input_file)

Where is the file written?:

The file is written to S3 bucket specified in your config and prepared for Bedrock batch inference.

3. Create the batch​

bedrock_batch.py
...
# Create batch
batch = client.batches.create( 
    input_file_id=batch_input_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"description": "Test batch job"},
)
print(batch)

FAQ​

Where are my files written?​

When a target_model_names is specified, the file is written to the S3 bucket configured in your Bedrock batch model configuration.

What models are supported?​

LiteLLM only supports Bedrock Anthropic Models for Batch API. If you want other bedrock models file an issue here.

Further Reading​