--- title: "Bedrock Batches" url: "/docs/providers/bedrock_batches" canonical_url: "https://docs.litellm.ai/docs/providers/bedrock_batches" type: "docs" last_updated: "2026-10-06" summary: "Use Amazon Bedrock Batch Inference API through LiteLLM." related: - "/docs/providers/bedrock_writer" - "/docs/providers/bedrock_realtime_with_audio" --- # Bedrock Batches > Index of all LiteLLM docs: https://docs.litellm.ai/llms.txt Use Amazon Bedrock Batch Inference API through LiteLLM. | Property | Details | |----------|---------| | Description | Amazon Bedrock Batch Inference allows you to run inference on large datasets asynchronously | | Provider Doc | [AWS Bedrock Batch Inference ↗](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html) | | Cost Tracking | ✅ Supported | ## Overview Use this to: - Run batch inference on large datasets with Bedrock models - Control batch model access by key/user/team (same as chat completion models) - Manage S3 storage for batch input/output files ## (Proxy Admin) Usage Here's how to give developers access to your Bedrock Batch models. ### 1. Setup config.yaml - Specify `mode: batch` for each model: Allows developers to know this is a batch model - Configure S3 bucket and AWS credentials for batch operations ```yaml showLineNumbers title="litellm_config.yaml" model_list: - model_name: "bedrock-batch-claude" litellm_params: model: bedrock/us.anthropic.claude-sonnet-5 ######################################################### ########## batch specific params ######################## s3_bucket_name: litellm-proxy s3_region_name: us-west-2 s3_access_key_id: os.environ/AWS_ACCESS_KEY_ID s3_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY aws_batch_role_arn: arn:aws:iam::888602223428:role/service-role/AmazonBedrockExecutionRoleForAgents_BB9HNW6V4CV # Optional: Custom KMS encryption key for the S3 input upload and the batch output # s3_encryption_key_id: arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012 # Optional: AWS account id that owns the S3 buckets, when they live in another account # s3_bucket_owner: "123456789012" model_info: mode: batch # 👈 SPECIFY MODE AS BATCH, to tell user this is a batch model ``` **Required Parameters:** | Parameter | Description | |-----------|-------------| | `s3_bucket_name` | S3 bucket for batch input/output files | | `s3_region_name` | AWS region for S3 bucket | | `s3_access_key_id` | AWS access key for S3 bucket | | `s3_secret_access_key` | AWS secret key for S3 bucket | | `aws_batch_role_arn` | IAM role ARN for Bedrock batch operations. Bedrock Batch APIs require an IAM role ARN to be set. | | `mode: batch` | Indicates to LiteLLM this is a batch model | **Optional Parameters:** | Parameter | Description | |-----------|-------------| | `s3_encryption_key_id` | Custom KMS encryption key ID for the batch input file LiteLLM uploads to S3 and for the batch output data. Requires `kms:GenerateDataKey` on that key for the credentials LiteLLM signs the upload with. If not specified, Bedrock uses AWS managed encryption keys. | | `s3_bucket_owner` | AWS account id that owns the batch input and output S3 buckets. Sent as `s3BucketOwner` on both the input and output data config of the Bedrock job. Set it when the buckets live in a different account than the one running the batch job, otherwise Bedrock validates bucket ownership against the job's account and the job fails. Also settable via the `AWS_S3_BUCKET_OWNER` env var. | ### 2. Create Virtual Key ```bash showLineNumbers title="create_virtual_key.sh" curl -L -X POST 'https://{PROXY_BASE_URL}/key/generate' \ -H 'Authorization: Bearer ${PROXY_API_KEY}' \ -H 'Content-Type: application/json' \ -d '{"models": ["bedrock-batch-claude"]}' ``` You can now use the virtual key to access the batch models (See Developer flow). ## (Developer) Usage Here's how to create a LiteLLM managed file and execute Bedrock Batch CRUD operations with the file. ### 1. Create request.jsonl - Check models available via `/model_group/info` - See all models with `mode: batch` - Set `model` in .jsonl to the model from `/model_group/info` ```json showLineNumbers title="bedrock_batch_completions.jsonl" {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "bedrock-batch-claude", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello world!"}], "max_tokens": 1000}} {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "bedrock-batch-claude", "messages": [{"role": "system", "content": "You are an unhelpful assistant."}, {"role": "user", "content": "Hello world!"}], "max_tokens": 1000}} ``` Expectation: - LiteLLM translates this to the bedrock deployment specific value (e.g. `bedrock/us.anthropic.claude-sonnet-5`) ### 2. Upload File Specify `target_model_names: ""` to enable LiteLLM managed files and request validation. model-name should be the same as the model-name in the request.jsonl **Python** ```python showLineNumbers title="bedrock_batch.py" from openai import OpenAI client = OpenAI( base_url="http://0.0.0.0:4000", api_key="sk-", ) # Upload file batch_input_file = client.files.create( file=open("./bedrock_batch_completions.jsonl", "rb"), # {"model": "bedrock-batch-claude"} <-> {"model": "bedrock/us.anthropic.claude-sonnet-5"} purpose="batch", extra_body={"target_model_names": "bedrock-batch-claude"} ) print(batch_input_file) ``` **Curl** ```bash showLineNumbers title="Upload File" curl http://localhost:4000/v1/files \ -H "Authorization: Bearer $LITELLM_API_KEY" \ -F purpose="batch" \ -F file="@bedrock_batch_completions.jsonl" \ -F extra_body='{"target_model_names": "bedrock-batch-claude"}' ``` **Where is the file written?**: The file is written to S3 bucket specified in your config and prepared for Bedrock batch inference. ### 3. Create the batch **Python** ```python showLineNumbers title="bedrock_batch.py" ... # Create batch batch = client.batches.create( input_file_id=batch_input_file.id, endpoint="/v1/chat/completions", completion_window="24h", metadata={"description": "Test batch job"}, ) print(batch) ``` **Curl** ```bash showLineNumbers title="Create Batch Request" curl http://localhost:4000/v1/batches \ -H "Authorization: Bearer $LITELLM_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "input_file_id": "file-abc123", "endpoint": "/v1/chat/completions", "completion_window": "24h", "metadata": {"description": "Test batch job"} }' ``` ### 4. Retrieve batch results Once the batch job is completed, download the results from S3: **Python** ```python showLineNumbers title="bedrock_batch.py" ... # Wait for batch completion (check status periodically) batch_status = client.batches.retrieve(batch_id=batch.id) if batch_status.status == "completed": # Download the output file result = client.files.content( file_id=batch_status.output_file_id, extra_headers={"custom-llm-provider": "bedrock"} ) # Save or process the results with open("batch_output.jsonl", "wb") as f: f.write(result.content) # Parse JSONL results for line in result.text.strip().split('\n'): record = json.loads(line) print(f"Record ID: {record['recordId']}") print(f"Output: {record.get('modelOutput', {})}") ``` **Curl** ```bash showLineNumbers title="Download Batch Results" # First retrieve batch to get output_file_id curl http://localhost:4000/v1/batches/batch_abc123 \ -H "Authorization: Bearer $LITELLM_API_KEY" # Then download the output file curl http://localhost:4000/v1/files/{output_file_id}/content \ -H "Authorization: Bearer $LITELLM_API_KEY" \ -H "custom-llm-provider: bedrock" \ -o batch_output.jsonl ``` **LiteLLM Direct** ```python showLineNumbers title="bedrock_batch.py" import litellm from litellm import file_content # Download using litellm directly (bypasses proxy managed files) result = file_content( file_id=batch_status.output_file_id, # Can be S3 URI or unified file ID custom_llm_provider="bedrock", aws_region_name="us-west-2", ) # Process results print(result.text) ``` **Output Format:** The batch output file is in JSONL format with each line containing: ```json { "recordId": "request-1", "modelInput": { "messages": [...], "max_tokens": 1000 }, "modelOutput": { "content": [...], "id": "msg_abc123", "model": "us.anthropic.claude-sonnet-5", "role": "assistant", "stop_reason": "end_turn", "usage": { "input_tokens": 15, "output_tokens": 10 } } } ``` ## FAQ ### Where are my files written? When a `target_model_names` is specified, the file is written to the S3 bucket configured in your Bedrock batch model configuration. ### What models are supported? Any Bedrock model that AWS lists for [batch inference](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference-supported.html) works, as long as LiteLLM can translate the OpenAI-format records in your input file into that model's request body. Today that covers Anthropic Claude models, Amazon Nova models (each record is written in the Converse request shape Nova expects), Amazon Titan Text Embeddings V2 for `/v1/embeddings` records, and the OpenAI-compatible Bedrock models such as `openai.gpt-oss-120b-1:0`, Qwen, and DeepSeek, whose records are passed through as OpenAI-style chat bodies. Chat records can use `/v1/chat/completions`, `/v1/completions`, or `/v1/responses` as their `url`; completions and responses records are converted to chat requests before upload. Titan Text Embeddings V2 is the only embedding model translated today. Use the model id AWS accepts for batch jobs in your region. Most current models only run batch jobs through a cross-region inference profile, so the id usually carries the `us.` (or `eu.`, `apac.`) prefix, for example `us.amazon.nova-lite-v1:0` rather than `amazon.nova-lite-v1:0`. Bedrock also rejects jobs with fewer than 100 records regardless of the model. A non-Anthropic batch model is configured the same way as the Claude example above: ```yaml showLineNumbers title="litellm_config.yaml" model_list: - model_name: "bedrock-batch-nova" litellm_params: model: bedrock/us.amazon.nova-lite-v1:0 s3_bucket_name: litellm-proxy s3_region_name: us-east-1 s3_access_key_id: os.environ/AWS_ACCESS_KEY_ID s3_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY aws_batch_role_arn: arn:aws:iam::123456789012:role/LiteLLMBedrockBatchRole model_info: mode: batch - model_name: "bedrock-batch-titan-embeddings" litellm_params: model: bedrock/amazon.titan-embed-text-v2:0 s3_bucket_name: litellm-proxy s3_region_name: us-east-1 s3_access_key_id: os.environ/AWS_ACCESS_KEY_ID s3_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY aws_batch_role_arn: arn:aws:iam::123456789012:role/LiteLLMBedrockBatchRole model_info: mode: batch ``` Records in the input file then reference the `model_name` (`{"model": "bedrock-batch-nova", ...}`), the upload sets `target_model_names` to that same name, and the batch is created with `endpoint` set to `/v1/chat/completions` for chat models or `/v1/embeddings` for the embeddings model. The IAM role in `aws_batch_role_arn` needs `bedrock:InvokeModel` on every model you run batch jobs with. If you need a Bedrock model LiteLLM does not translate yet, for example another embedding model, file an issue [here](https://github.com/BerriAI/litellm/issues/new/choose). ### How do I use a custom KMS encryption key? If your S3 bucket requires a custom KMS encryption key, you can specify it in your configuration using `s3_encryption_key_id`. This is useful for enterprise customers with specific encryption requirements. The key covers both objects LiteLLM touches: the batch input file it uploads to your bucket, and the batch output Bedrock writes back. The input upload is signed with `x-amz-server-side-encryption: aws:kms` and this key ARN, so the AWS identity LiteLLM uploads with needs `kms:GenerateDataKey` on the key. Grant that before setting the key, otherwise `POST /v1/files` fails with an S3 `AccessDenied` You can set the encryption key in 2 ways: 1. **In config.yaml** (recommended): ```yaml model_list: - model_name: "bedrock-batch-claude" litellm_params: model: bedrock/us.anthropic.claude-sonnet-5 s3_encryption_key_id: arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012 # ... other params ``` 2. **As an environment variable**: ```bash export AWS_S3_ENCRYPTION_KEY_ID=arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012 ``` ### How do I use S3 buckets owned by a different AWS account? Bedrock checks that the input and output buckets belong to the account named in `s3BucketOwner`, and when that field is missing it defaults to the account running the batch job. If your buckets live in another account the job fails validation with an S3 permission error even though the bucket policy grants access. Set `s3_bucket_owner` to the id of the account that owns the buckets and LiteLLM sends it on both the input and output data config Through the proxy, an `s3_bucket_owner` passed in the `/v1/batches` request body takes precedence over the deployment's `litellm_params` value, which in turn takes precedence over the `AWS_S3_BUCKET_OWNER` env var. This is the same order the router applies to every other deployment parameter such as `s3_bucket_name` or `s3_encryption_key_id`. When none is set the field is omitted and Bedrock keeps its default ```yaml model_list: - model_name: "bedrock-batch-claude" litellm_params: model: bedrock/us.anthropic.claude-sonnet-5 s3_bucket_name: shared-batch-bucket s3_bucket_owner: "123456789012" # ... other params ``` ```bash export AWS_S3_BUCKET_OWNER=123456789012 ``` ## Further Reading - [AWS Bedrock Batch Inference Documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/batch-inference.html) - [LiteLLM Managed Batches](../proxy/managed_batches) - [LiteLLM Authentication to Bedrock](https://docs.litellm.ai/docs/providers/bedrock#boto3---authentication) ## Related pages - [Bedrock - Writer Palmyra](https://docs.litellm.ai/docs/providers/bedrock_writer.md) - [Bedrock Realtime API](https://docs.litellm.ai/docs/providers/bedrock_realtime_with_audio.md)