Skip to main content

Vertex Batch APIs

Setup​

Configure the Vertex model in your config.yaml. gcs_bucket_name is the GCS bucket batch prediction files are stored in, a required param for vertexai to store files. The older bucket_name litellm param is still supported and is treated the same as gcs_bucket_name. If both are set on the same deployment, gcs_bucket_name wins

litellm-config.yaml
model_list:
  - model_name: vertex-batch
    litellm_params:
      model: vertex_ai/gemini-3.8-flash
      vertex_project: my-project
      vertex_location: us-central1
      vertex_credentials: /path/to/service_account.json
      gcs_bucket_name: my-batch-bucket # required param for vertexai to store files
    model_info:
      mode: batch

When gcs_bucket_name (or bucket_name) and vertex_credentials are not set on the model's litellm_params, LiteLLM falls back to these env vars. GCS_BATCH_BUCKET_NAME exists because the gcs_bucket logging callback already uses GCS_BUCKET_NAME as the bucket it writes LLM request logs to, so setting GCS_BUCKET_NAME to your batch bucket would also send request logs there. GCS_BATCH_BUCKET_NAME is checked first for batch files and, when it is unset, GCS_BUCKET_NAME is used

# GCS Bucket settings, used to store batch prediction files in
export GCS_BATCH_BUCKET_NAME="my-batch-bucket" # bucket for batch prediction files, takes precedence over GCS_BUCKET_NAME
export GCS_BUCKET_NAME="my-logging-bucket" # used for batch files only when GCS_BATCH_BUCKET_NAME is unset
export GCS_PATH_SERVICE_ACCOUNT="/path/to/service_account.json" # path to your service account json file

# Vertex /batch endpoint settings, used for LLM API requests
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service_account.json" # path to your service account json file
export VERTEXAI_LOCATION="us-central1" # can be any vertex location
export VERTEXAI_PROJECT="my-project"

Usage​

Follow this complete workflow: create JSONL file → upload file → create batch → retrieve batch status → get file content

1. Create a JSONL file of batch requests​

LiteLLM expects the file to follow the OpenAI batches files format.

Each body in the file should be an OpenAI API request.

Create a file called batch_requests.jsonl with your requests:

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-3.8-flash", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 10}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-3.8-flash", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 10}}

2. Upload the file​

Upload your JSONL file. For vertex_ai, the file will be stored in the deployment's gcs_bucket_name (or bucket_name), falling back to GCS_BATCH_BUCKET_NAME and then GCS_BUCKET_NAME.

upload_file.py
from openai import OpenAI

oai_client = OpenAI(
    api_key="sk-<your-litellm-api-key>",               # litellm proxy API key
    base_url="http://localhost:4000" # litellm proxy base url
)

file_obj = oai_client.files.create(
    file=open("batch_requests.jsonl", "rb"),
    purpose="batch",
    extra_headers={"custom-llm-provider": "vertex_ai"}
)

print(f"File uploaded with ID: {file_obj.id}")

Expected Response:

{
    "id": "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd",
    "bytes": 416,
    "created_at": 1758303684,
    "filename": "litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd",
    "object": "file",
    "purpose": "batch",
    "status": "uploaded",
    "expires_at": null,
    "status_details": null
}

3. Create a batch​

Create a batch job using the uploaded file ID.

create_batch.py
batch_input_file_id = file_obj.id # from step 2
create_batch_response = oai_client.batches.create(
    completion_window="24h",
    endpoint="/v1/chat/completions",
    input_file_id=batch_input_file_id, # e.g. "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd"
    extra_headers={"custom-llm-provider": "vertex_ai"}
)

print(f"Batch created with ID: {create_batch_response.id}")

Expected Response:

{
    "id": "7814463557919047680",
    "completion_window": "24hrs",
    "created_at": 1758328011,
    "endpoint": "",
    "input_file_id": "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd",
    "object": "batch",
    "status": "validating",
    "cancelled_at": null,
    "cancelling_at": null,
    "completed_at": null,
    "error_file_id": null,
    "errors": null,
    "expired_at": null,
    "expires_at": null,
    "failed_at": null,
    "finalizing_at": null,
    "in_progress_at": null,
    "metadata": null,
    "output_file_id": null,
    "request_counts": null,
    "usage": null
}

4. Retrieve batch status​

Check the status of your batch job. The batch will progress through states: validating → in_progress → completed. The output_file_id is null until the batch reaches completed.

retrieve_batch.py
retrieved_batch = oai_client.batches.retrieve(
    batch_id=create_batch_response.id, # Created batch id, e.g. 7814463557919047680
    extra_headers={"custom-llm-provider": "vertex_ai"}
)

print(f"Batch status: {retrieved_batch.status}")
if retrieved_batch.status == "completed":
    print(f"Output file: {retrieved_batch.output_file_id}")

Expected Response (when completed):

{
    "id": "7814463557919047680",
    "completion_window": "24hrs",
    "created_at": 1758328011,
    "endpoint": "",
    "input_file_id": "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd",
    "object": "batch",
    "status": "completed",
    "cancelled_at": null,
    "cancelling_at": null,
    "completed_at": null,
    "error_file_id": null,
    "errors": null,
    "expired_at": null,
    "expires_at": null,
    "failed_at": null,
    "finalizing_at": null,
    "in_progress_at": null,
    "metadata": null,
    "output_file_id": "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/prediction-model-2025-09-19T21:26:51.569037Z/predictions.jsonl",
    "request_counts": null,
    "usage": null
}

5. Get file content​

Once the batch is completed, retrieve the results using the output_file_id from the batch response.

Important: The output_file_id must be URL encoded when used in the request path.

get_file_content.py
import urllib.parse
import json

output_file_id = retrieved_batch.output_file_id
# URL encode the file ID
encoded_file_id = urllib.parse.quote_plus(output_file_id)

# Get file content
file_content = oai_client.files.content(
    file_id=encoded_file_id,
    extra_headers={"custom-llm-provider": "vertex_ai"}
)

# Process the results
for line in file_content.text.strip().split('\n'):
    result = json.loads(line)
    print(f"Request: {result['request']}")
    print(f"Response: {result['response']}")
    print("---")

Expected Response:

The response contains JSONL format with one result per line:

{"status":"","processed_time":"2025-09-19T21:29:47.352+00:00","request":{"contents":[{"parts":[{"text":"Hello world!"}],"role":"user"}],"generationConfig":{"max_output_tokens":10},"system_instruction":{"parts":[{"text":"You are a helpful assistant."}]}},"response":{"candidates":[{"avgLogprobs":-0.48079710006713866,"content":{"parts":[{"text":"Hello there! It's nice to meet you"}],"role":"model"},"finishReason":"MAX_TOKENS"}],"createTime":"2025-09-19T21:29:47.484619Z","modelVersion":"gemini-3.8-flash","responseId":"S8vNaIvKHdvshMIP_aOtuAg","usageMetadata":{"candidatesTokenCount":10,"candidatesTokensDetails":[{"modality":"TEXT","tokenCount":10}],"promptTokenCount":9,"promptTokensDetails":[{"modality":"TEXT","tokenCount":9}],"totalTokenCount":19,"trafficType":"ON_DEMAND"}}}
{"status":"","processed_time":"2025-09-19T21:29:47.358+00:00","request":{"contents":[{"parts":[{"text":"Hello world!"}],"role":"user"}],"generationConfig":{"max_output_tokens":10},"system_instruction":{"parts":[{"text":"You are an unhelpful assistant."}]}},"response":{"candidates":[{"avgLogprobs":-0.6168075137668185,"content":{"parts":[{"text":"I am unable to assist with this request."}],"role":"model"},"finishReason":"STOP"}],"createTime":"2025-09-19T21:29:47.470889Z","modelVersion":"gemini-3.8-flash","responseId":"S8vNaOneHISShMIP28nA8QQ","usageMetadata":{"candidatesTokenCount":9,"candidatesTokensDetails":[{"modality":"TEXT","tokenCount":9}],"promptTokenCount":9,"promptTokensDetails":[{"modality":"TEXT","tokenCount":9}],"totalTokenCount":18,"trafficType":"ON_DEMAND"}}}

Native Vertex JSONL passthrough​

By default LiteLLM translates each OpenAI-format line into a Vertex GenerateContent request when you upload the file, and converts the batch output back into OpenAI chat completion shape when you download it. Set passthrough=true on the upload to skip both: the file is stored in your GCS bucket byte for byte, and the batch output comes back in Vertex's native format, groundingMetadata included. Use it for Vertex features the translation does not cover, such as googleSearch grounding with excludeDomains, without handing your callers GCP credentials. Cost tracking is unchanged: LiteLLM reads usageMetadata from each native output row and bills it the way an online Gemini call is billed.

1. Create a native JSONL file​

Each line is a Vertex batch prediction request, a JSON object with a request key:

native_batch_requests.jsonl
{"request": {"contents": [{"role": "user", "parts": [{"text": "What is the tallest building in the world?"}]}], "tools": [{"googleSearch": {"excludeDomains": ["example.com"]}}]}}
{"request": {"contents": [{"role": "user", "parts": [{"text": "Who won the last FIFA World Cup?"}]}], "tools": [{"googleSearch": {"excludeDomains": ["example.com"]}}]}}

2. Upload it with passthrough=true​

Native rows carry no model, so name the Vertex deployment that will run the batch: target_model_names (managed files, needs a database) or the model query parameter (no database). LiteLLM stores the object under litellm-vertex-files/passthrough/ in that model's path, and the batch output lands beside it, which is how the download step knows to return it untouched.

config.yaml
model_list:
  - model_name: gemini-3.8-flash
    litellm_params:
      model: vertex_ai/gemini-3.8-flash
      vertex_project: my-project
      vertex_location: us-central1
      gcs_bucket_name: my-batch-bucket
upload_native_file.py
file_obj = oai_client.files.create(
    file=open("native_batch_requests.jsonl", "rb"),
    purpose="batch",
    extra_body={"target_model_names": "gemini-3.8-flash", "passthrough": True},
)

print(f"File uploaded with ID: {file_obj.id}")

Create the batch, poll it, and download the output exactly as in steps 3 to 5 above. The output is the raw Vertex predictions.jsonl, so each line carries request, status, and response, with candidates[].groundingMetadata on every row where grounding ran.

What passthrough checks​

  • purpose must be batch and the named model must be a vertex_ai deployment the key can use, otherwise the upload is a 400 naming the field.
  • Every line must be a JSON object with a request key. OpenAI-format lines are rejected with a 400 naming the line, and nothing is uploaded.
  • Batch guardrails only understand OpenAI-format rows, so a passthrough upload is refused with a 400 when the key, team, or request has pre-call guardrails configured.

Passthrough is per upload. The SDK-wide litellm.disable_vertex_batch_output_transformation flag still applies to every Vertex batch output but never touches the input side; passthrough covers both for the one file you set it on.

🚅
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails — built for production.
Learn more →