ZeroGPU
Available from v1.109.0.
Overview
| Property | Details |
|---|---|
| Description | ZeroGPU is the compute efficiency layer for AI inference. It runs high-volume tasks such as text generation, classification, extraction, moderation, embeddings and audio on specialized small and nano language models across an edge-powered network, through one OpenAI-compatible API |
| Provider Route on LiteLLM | zerogpu/ |
| Link to Provider Doc | ZeroGPU Documentation |
| Default Base URL | https://api.zerogpu.ai/v1 |
| Supported Operations | /chat/completions and /responses, plus /messages through LiteLLM's adapter |
LiteLLM's zerogpu/ route covers the ZeroGPU text generation models listed below. ZeroGPU is its own provider on LiteLLM rather than a generic OpenAI-compatible route, so its spend is priced from the zerogpu/ cost map entries and reported under provider zerogpu.
API Key
import os
os.environ["ZEROGPU_API_KEY"] = "your-api-key"
os.environ["ZEROGPU_API_BASE"] = "https://api.zerogpu.ai/v1" # optional override
Models
| Model | Context window | Input / 1M tokens | Output / 1M tokens | Cache read / 1M tokens |
|---|---|---|---|---|
zerogpu/deepseek-v4.1-flash | 1,048,576 | $0.30 | $1.20 | $0.003 |
zerogpu/glm-5.3-flash | 1,048,576 | $0.10 | $0.35 | $0.015 |
zerogpu/gpt-4.1-mini | 1,047,576 | $0.40 | $1.60 | $0.10 |
zerogpu/gpt-5.4-nano | 272,000 | $0.20 | $1.25 | $0.02 |
zerogpu/gpt-5.6-luna | 272,000 | $0.20 | $1.20 | $0.20 |
zerogpu/gpt-6-luna | 272,000 | $0.10 | $0.50 | $0.01 |
zerogpu/gpt-oss-120b | 128,000 | $0.15 | $0.60 | $0.03 |
zerogpu/LFM2.5-1.2B-Instruct | 32,768 | $0.02 | $0.05 | n/a |
zerogpu/LFM2.5-1.2B-Thinking | 32,768 | $0.02 | $0.05 | n/a |
zerogpu/llama-3.1-8b-instruct-fast | 131,072 | $0.15 | $0.28 | $0.025 |
zerogpu/llama-3.2-3b-instruct-fast | 131,072 | $0.05 | $0.30 | $0.025 |
zerogpu/qwen3-30b-a3b-fp8 | 32,768 | $0.10 | $0.45 | $0.05 |
zerogpu/qwen3.6-35b-a3b | 262,144 | $0.20 | $1.50 | $0.20 |
Pricing follows the ZeroGPU model catalog. deepseek-v4.1-flash, glm-5.3-flash, gpt-4.1-mini, gpt-5.4-nano, gpt-5.6-luna, gpt-6-luna, gpt-oss-120b, llama-3.1-8b-instruct-fast, qwen3-30b-a3b-fp8 and qwen3.6-35b-a3b support tools, tool_choice and a json_schema response_format. The reasoning models are deepseek-v4.1-flash, glm-5.3-flash, gpt-5.4-nano, gpt-5.6-luna, gpt-6-luna, gpt-oss-120b, qwen3-30b-a3b-fp8 and qwen3.6-35b-a3b.
Usage - LiteLLM Python SDK
Chat Completions
import os
from litellm import completion
os.environ["ZEROGPU_API_KEY"] = "your-api-key"
response = completion(
model="zerogpu/gpt-oss-120b",
messages=[{"role": "user", "content": "Write a python function that reverses a string"}],
)
print(response.choices[0].message.content)
Streaming
import os
from litellm import completion
os.environ["ZEROGPU_API_KEY"] = "your-api-key"
response = completion(
model="zerogpu/gpt-oss-120b",
messages=[{"role": "user", "content": "Explain a binary search in two sentences"}],
stream=True,
)
for chunk in response:
print(chunk)
Tool Calling
import os
from litellm import completion
os.environ["ZEROGPU_API_KEY"] = "your-api-key"
tools = [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string", "description": "City name"}},
"required": ["city"],
},
},
}
]
response = completion(
model="zerogpu/gpt-oss-120b",
messages=[{"role": "user", "content": "What is the weather in Paris?"}],
tools=tools,
tool_choice="auto",
)
print(response.choices[0].message.tool_calls)
Reasoning Effort
LiteLLM forwards reasoning_effort only to ZeroGPU models flagged supports_reasoning in the cost map, which are the reasoning models listed above. The rest, gpt-4.1-mini, llama-3.1-8b-instruct-fast, llama-3.2-3b-instruct-fast, LFM2.5-1.2B-Instruct and LFM2.5-1.2B-Thinking, reject it, so when the same call can reach them, pass drop_params=True and LiteLLM drops the parameter for them instead of rejecting the request.
import os
from litellm import completion
os.environ["ZEROGPU_API_KEY"] = "your-api-key"
response = completion(
model="zerogpu/gpt-5.6-luna",
messages=[{"role": "user", "content": "How many prime numbers are there below 100?"}],
reasoning_effort="high",
drop_params=True,
)
print(response.choices[0].message.content)
Usage - LiteLLM Proxy
Add ZeroGPU to your LiteLLM Proxy configuration:
model_list:
- model_name: gpt-oss-120b
litellm_params:
model: zerogpu/gpt-oss-120b
api_key: os.environ/ZEROGPU_API_KEY
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
Start the proxy:
export ZEROGPU_API_KEY="your-api-key"
export LITELLM_MASTER_KEY="sk-local-zerogpu"
litellm --config config.yaml --port 4000
# RUNNING on http://0.0.0.0:4000
- OpenAI SDK
- cURL
from openai import OpenAI
client = OpenAI(
base_url="http://localhost:4000",
api_key="sk-local-zerogpu",
)
response = client.chat.completions.create(
model="gpt-oss-120b",
messages=[{"role": "user", "content": "hello from litellm"}],
)
print(response.choices[0].message.content)
curl http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{
"model": "gpt-oss-120b",
"messages": [{"role": "user", "content": "hello from litellm"}]
}'
You can also add ZeroGPU from the Admin UI. Go to Models, then Add Model, pick ZeroGPU as the provider, choose one of the zerogpu/ models, and paste your key. The API Base field is optional and defaults to https://api.zerogpu.ai/v1.
Anthropic Messages and Responses Compatibility
LiteLLM translates Anthropic Messages-shaped requests into ZeroGPU chat completions, both through the SDK facade and the proxy's /v1/messages endpoint:
curl http://localhost:4000/v1/messages \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "anthropic-version: 2023-06-01" \
-d '{
"model": "gpt-oss-120b",
"max_tokens": 128,
"messages": [{"role": "user", "content": "hello from litellm"}]
}'
litellm.responses() and the proxy's /v1/responses endpoint call ZeroGPU's own /responses endpoint natively rather than going through the chat completions bridge, so no extra configuration is needed.
Cost Tracking
The zerogpu/ models are registered in LiteLLM's model cost map, so per-request spend is computed automatically, returned in the x-litellm-response-cost response header, and recorded in spend logs under provider zerogpu. When ZeroGPU reports cached prompt tokens in prompt_tokens_details.cached_tokens, LiteLLM bills those at the cache read rate from the table above and the rest of the prompt at the input rate.
print(f"Request cost: ${response._hidden_params['response_cost']}")
Custom Endpoints
Set ZEROGPU_API_BASE or pass api_base to point the zerogpu/ route at a different ZeroGPU endpoint. The route keeps the provider identity and pricing either way.
model_list:
- model_name: gpt-oss-120b
litellm_params:
model: zerogpu/gpt-oss-120b
api_base: https://your-zerogpu-endpoint/v1
api_key: os.environ/ZEROGPU_API_KEY
