--- title: "Vertex Batch APIs" url: "/docs/providers/vertex_batch" canonical_url: "https://docs.litellm.ai/docs/providers/vertex_batch" type: "docs" last_updated: "2026-10-16" related: - "/docs/providers/vertex_transcription" - "/docs/providers/vertex_ocr" --- # Vertex Batch APIs > Index of all LiteLLM docs: https://docs.litellm.ai/llms.txt ## Setup Configure the Vertex model in your config.yaml. `gcs_bucket_name` is the GCS bucket batch prediction files are stored in, a required param for vertexai to store files. The older `bucket_name` litellm param is still supported and is treated the same as `gcs_bucket_name`. If both are set on the same deployment, `gcs_bucket_name` wins ```yaml showLineNumbers title="litellm-config.yaml" model_list: - model_name: vertex-batch litellm_params: model: vertex_ai/gemini-3.8-flash vertex_project: my-project vertex_location: us-central1 vertex_credentials: /path/to/service_account.json gcs_bucket_name: my-batch-bucket # required param for vertexai to store files model_info: mode: batch ``` When `gcs_bucket_name` (or `bucket_name`) and `vertex_credentials` are not set on the model's `litellm_params`, LiteLLM falls back to these env vars. `GCS_BATCH_BUCKET_NAME` exists because the [`gcs_bucket` logging callback](../observability/gcs_bucket_integration) already uses `GCS_BUCKET_NAME` as the bucket it writes LLM request logs to, so setting `GCS_BUCKET_NAME` to your batch bucket would also send request logs there. `GCS_BATCH_BUCKET_NAME` is checked first for batch files and, when it is unset, `GCS_BUCKET_NAME` is used ```bash # GCS Bucket settings, used to store batch prediction files in export GCS_BATCH_BUCKET_NAME="my-batch-bucket" # bucket for batch prediction files, takes precedence over GCS_BUCKET_NAME export GCS_BUCKET_NAME="my-logging-bucket" # used for batch files only when GCS_BATCH_BUCKET_NAME is unset export GCS_PATH_SERVICE_ACCOUNT="/path/to/service_account.json" # path to your service account json file # Vertex /batch endpoint settings, used for LLM API requests export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service_account.json" # path to your service account json file export VERTEXAI_LOCATION="us-central1" # can be any vertex location export VERTEXAI_PROJECT="my-project" ``` ### Usage Follow this complete workflow: create JSONL file → upload file → create batch → retrieve batch status → get file content #### 1. Create a JSONL file of batch requests LiteLLM expects the file to follow the **[OpenAI batches files format](https://platform.openai.com/docs/guides/batch)**. Each `body` in the file should be an **OpenAI API request**. Create a file called `batch_requests.jsonl` with your requests: ```jsonl {"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-3.8-flash", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 10}} {"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gemini-3.8-flash", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 10}} ``` #### 2. Upload the file Upload your JSONL file. For `vertex_ai`, the file will be stored in the deployment's `gcs_bucket_name` (or `bucket_name`), falling back to `GCS_BATCH_BUCKET_NAME` and then `GCS_BUCKET_NAME`. **Python** ```python showLineNumbers title="upload_file.py" from openai import OpenAI oai_client = OpenAI( api_key="sk-", # litellm proxy API key base_url="http://localhost:4000" # litellm proxy base url ) file_obj = oai_client.files.create( file=open("batch_requests.jsonl", "rb"), purpose="batch", extra_headers={"custom-llm-provider": "vertex_ai"} ) print(f"File uploaded with ID: {file_obj.id}") ``` **Curl** ```bash showLineNumbers title="Upload File" curl --request POST \ --url http://localhost:4000/v1/files \ --header 'Content-Type: multipart/form-data' \ --header 'custom-llm-provider: vertex_ai' \ --form purpose=batch \ --form file=@batch_requests.jsonl ``` **Expected Response:** ```json { "id": "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd", "bytes": 416, "created_at": 1758303684, "filename": "litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd", "object": "file", "purpose": "batch", "status": "uploaded", "expires_at": null, "status_details": null } ``` #### 3. Create a batch Create a batch job using the uploaded file ID. **Python** ```python showLineNumbers title="create_batch.py" batch_input_file_id = file_obj.id # from step 2 create_batch_response = oai_client.batches.create( completion_window="24h", endpoint="/v1/chat/completions", input_file_id=batch_input_file_id, # e.g. "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd" extra_headers={"custom-llm-provider": "vertex_ai"} ) print(f"Batch created with ID: {create_batch_response.id}") ``` **Curl** ```bash showLineNumbers title="Create Batch Request" curl --request POST \ --url http://localhost:4000/v1/batches \ --header 'Content-Type: application/json' \ --header 'custom-llm-provider: vertex_ai' \ --data '{ "input_file_id": "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd", "endpoint": "/v1/chat/completions", "completion_window": "24h" }' ``` **Expected Response:** ```json { "id": "7814463557919047680", "completion_window": "24hrs", "created_at": 1758328011, "endpoint": "", "input_file_id": "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd", "object": "batch", "status": "validating", "cancelled_at": null, "cancelling_at": null, "completed_at": null, "error_file_id": null, "errors": null, "expired_at": null, "expires_at": null, "failed_at": null, "finalizing_at": null, "in_progress_at": null, "metadata": null, "output_file_id": null, "request_counts": null, "usage": null } ``` #### 4. Retrieve batch status Check the status of your batch job. The batch will progress through states: `validating` → `in_progress` → `completed`. The `output_file_id` is `null` until the batch reaches `completed`. **Python** ```python showLineNumbers title="retrieve_batch.py" retrieved_batch = oai_client.batches.retrieve( batch_id=create_batch_response.id, # Created batch id, e.g. 7814463557919047680 extra_headers={"custom-llm-provider": "vertex_ai"} ) print(f"Batch status: {retrieved_batch.status}") if retrieved_batch.status == "completed": print(f"Output file: {retrieved_batch.output_file_id}") ``` **Curl** ```bash showLineNumbers title="Retrieve Batch Status" curl --request GET \ --url 'http://localhost:4000/batches/7814463557919047680?provider=vertex_ai' \ --header "Authorization: Bearer $LITELLM_API_KEY" ``` **Expected Response (when completed):** ```json { "id": "7814463557919047680", "completion_window": "24hrs", "created_at": 1758328011, "endpoint": "", "input_file_id": "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/abc123-def4-5678-9012-34567890abcd", "object": "batch", "status": "completed", "cancelled_at": null, "cancelling_at": null, "completed_at": null, "error_file_id": null, "errors": null, "expired_at": null, "expires_at": null, "failed_at": null, "finalizing_at": null, "in_progress_at": null, "metadata": null, "output_file_id": "gs://my-batch-bucket/litellm-vertex-files/publishers/google/models/gemini-3.8-flash/prediction-model-2025-09-19T21:26:51.569037Z/predictions.jsonl", "request_counts": null, "usage": null } ``` #### 5. Get file content Once the batch is completed, retrieve the results using the `output_file_id` from the batch response. **Important:** The `output_file_id` must be URL encoded when used in the request path. **Python** ```python showLineNumbers title="get_file_content.py" import urllib.parse import json output_file_id = retrieved_batch.output_file_id # URL encode the file ID encoded_file_id = urllib.parse.quote_plus(output_file_id) # Get file content file_content = oai_client.files.content( file_id=encoded_file_id, extra_headers={"custom-llm-provider": "vertex_ai"} ) # Process the results for line in file_content.text.strip().split('\n'): result = json.loads(line) print(f"Request: {result['request']}") print(f"Response: {result['response']}") print("---") ``` **Curl** ```bash showLineNumbers title="Get File Content" # Note: The file ID must be URL encoded curl --request GET \ --url 'http://localhost:4000/files/gs%253A%252F%252Fmy-batch-bucket%252Flitellm-vertex-files%252Fpublishers%252Fgoogle%252Fmodels%252Fgemini-2.5-flash-lite%252Fprediction-model-2025-09-19T21%253A26%253A51.569037Z%252Fpredictions.jsonl/content?provider=vertex_ai' \ --header "Authorization: Bearer $LITELLM_API_KEY" ``` **Expected Response:** The response contains JSONL format with one result per line: ```jsonl {"status":"","processed_time":"2025-09-19T21:29:47.352+00:00","request":{"contents":[{"parts":[{"text":"Hello world!"}],"role":"user"}],"generationConfig":{"max_output_tokens":10},"system_instruction":{"parts":[{"text":"You are a helpful assistant."}]}},"response":{"candidates":[{"avgLogprobs":-0.48079710006713866,"content":{"parts":[{"text":"Hello there! It's nice to meet you"}],"role":"model"},"finishReason":"MAX_TOKENS"}],"createTime":"2025-09-19T21:29:47.484619Z","modelVersion":"gemini-3.8-flash","responseId":"S8vNaIvKHdvshMIP_aOtuAg","usageMetadata":{"candidatesTokenCount":10,"candidatesTokensDetails":[{"modality":"TEXT","tokenCount":10}],"promptTokenCount":9,"promptTokensDetails":[{"modality":"TEXT","tokenCount":9}],"totalTokenCount":19,"trafficType":"ON_DEMAND"}}} {"status":"","processed_time":"2025-09-19T21:29:47.358+00:00","request":{"contents":[{"parts":[{"text":"Hello world!"}],"role":"user"}],"generationConfig":{"max_output_tokens":10},"system_instruction":{"parts":[{"text":"You are an unhelpful assistant."}]}},"response":{"candidates":[{"avgLogprobs":-0.6168075137668185,"content":{"parts":[{"text":"I am unable to assist with this request."}],"role":"model"},"finishReason":"STOP"}],"createTime":"2025-09-19T21:29:47.470889Z","modelVersion":"gemini-3.8-flash","responseId":"S8vNaOneHISShMIP28nA8QQ","usageMetadata":{"candidatesTokenCount":9,"candidatesTokensDetails":[{"modality":"TEXT","tokenCount":9}],"promptTokenCount":9,"promptTokensDetails":[{"modality":"TEXT","tokenCount":9}],"totalTokenCount":18,"trafficType":"ON_DEMAND"}}} ``` ### Native Vertex JSONL passthrough By default LiteLLM translates each OpenAI-format line into a Vertex `GenerateContent` request when you upload the file, and converts the batch output back into OpenAI chat completion shape when you download it. Set `passthrough=true` on the upload to skip both: the file is stored in your GCS bucket byte for byte, and the batch output comes back in Vertex's native format, `groundingMetadata` included. Use it for Vertex features the translation does not cover, such as `googleSearch` grounding with `excludeDomains`, without handing your callers GCP credentials. Cost tracking is unchanged: LiteLLM reads `usageMetadata` from each native output row and bills it the way an online Gemini call is billed. #### 1. Create a native JSONL file Each line is a Vertex batch prediction request, a JSON object with a `request` key: ```jsonl title="native_batch_requests.jsonl" {"request": {"contents": [{"role": "user", "parts": [{"text": "What is the tallest building in the world?"}]}], "tools": [{"googleSearch": {"excludeDomains": ["example.com"]}}]}} {"request": {"contents": [{"role": "user", "parts": [{"text": "Who won the last FIFA World Cup?"}]}], "tools": [{"googleSearch": {"excludeDomains": ["example.com"]}}]}} ``` #### 2. Upload it with `passthrough=true` Native rows carry no model, so name the Vertex deployment that will run the batch: `target_model_names` (managed files, needs a database) or the `model` query parameter (no database). LiteLLM stores the object under `litellm-vertex-files/passthrough/` in that model's path, and the batch output lands beside it, which is how the download step knows to return it untouched. ```yaml title="config.yaml" model_list: - model_name: gemini-3.8-flash litellm_params: model: vertex_ai/gemini-3.8-flash vertex_project: my-project vertex_location: us-central1 gcs_bucket_name: my-batch-bucket ``` **Python** ```python showLineNumbers title="upload_native_file.py" file_obj = oai_client.files.create( file=open("native_batch_requests.jsonl", "rb"), purpose="batch", extra_body={"target_model_names": "gemini-3.8-flash", "passthrough": True}, ) print(f"File uploaded with ID: {file_obj.id}") ``` **Curl** ```bash showLineNumbers title="Upload Native File" curl --request POST \ --url http://localhost:4000/v1/files \ --header "Authorization: Bearer $LITELLM_API_KEY" \ --form purpose=batch \ --form target_model_names=gemini-3.8-flash \ --form passthrough=true \ --form file=@native_batch_requests.jsonl ``` **LiteLLM SDK** ```python showLineNumbers title="upload_native_file_sdk.py" import litellm file_obj = litellm.create_file( file=open("native_batch_requests.jsonl", "rb"), purpose="batch", custom_llm_provider="vertex_ai", model="vertex_ai/gemini-3.8-flash", passthrough=True, ) ``` Create the batch, poll it, and download the output exactly as in steps 3 to 5 above. The output is the raw Vertex `predictions.jsonl`, so each line carries `request`, `status`, and `response`, with `candidates[].groundingMetadata` on every row where grounding ran. #### What passthrough checks - `purpose` must be `batch` and the named model must be a `vertex_ai` deployment the key can use, otherwise the upload is a 400 naming the field. - Every line must be a JSON object with a `request` key. OpenAI-format lines are rejected with a 400 naming the line, and nothing is uploaded. - Batch guardrails only understand OpenAI-format rows, so a passthrough upload is refused with a 400 when the key, team, or request has pre-call guardrails configured. Passthrough is per upload. The SDK-wide `litellm.disable_vertex_batch_output_transformation` flag still applies to every Vertex batch output but never touches the input side; `passthrough` covers both for the one file you set it on. ## Related pages - [Vertex AI Audio Transcription](https://docs.litellm.ai/docs/providers/vertex_transcription.md) - [Vertex AI OCR](https://docs.litellm.ai/docs/providers/vertex_ocr.md)