Rust Migration
LiteLLM is moving to Rust
See how much of the move to Rust is built and what is next.
Migration tracker
Where we are today
24% of built, shown as a .
Deployment0%of 10 pts
- Gateway binary in the Docker image
- Forward unserved requests to Python
- Dispatch crate and gap table
- Per model group plans at config load
- Identify, dispatch and admit layers
Rust core100%of 8 pts
- Python bridge
- Python host calls
- HTTP client
- Streaming
- Async bridge
- Tracing
- Model catalog
- Config parsing
Messages translation69%of 7 pts
- Anthropic
- DeepSeek
- Vertex AI
- Bedrock
- Bedrock Mantle
- Azure AI
- MiniMax
- GitHub Copilot
- Tencent
- Eden AI
- OpenAI-compatible
Chat completions translation5%of 13 pts
- OpenAI
- Azure OpenAI
- Anthropic
- Bedrock
- Vertex AI
- Gemini
- Azure AI
- Mistral
- Cohere
- Groq
- xAI
- DeepSeek
- Together AI
- Fireworks AI
- OpenRouter
- Databricks
- watsonx
- Perplexity
- DeepInfra
- Cerebras
- NVIDIA NIM
- SageMaker
- Hugging Face
- Replicate
- AI21
- SambaNova
- Nebius
- Novita AI
- Hyperbolic
- Baseten
- Moonshot AI
- DashScope
- Volcengine
- MiniMax
- GitHub Copilot
- Snowflake
- Oracle OCI
- Vercel AI Gateway
- LM Studio
- Ollama
- vLLM
- OpenAI-compatible
Responses translation28%of 6.5 pts
- OpenAI
- Azure OpenAI
- Azure AI
- Bedrock
- Bedrock Mantle
- xAI
- OpenRouter
- Databricks
- Fireworks AI
- Perplexity
- GitHub Copilot
- ChatGPT
- vLLM
- Volcengine
- Eden AI
- Manus
- Sail
- LiteLLM Proxy
- OpenAI-compatible
Python-only request features2%of 38.5 pts
- Mock responses
- Proxy MCP tools
- Caller HTTP clients
- Request, stream and phase timeouts
- Agentic loop hooks
- TLS security level and curves
- modify_params rewrites
- Prompt cache breakpoint injection
- Anthropic workload identity tokens
- Thinking signature retry
- Cost in streamed usage
- Web search interception
- Messages interceptors
- Safeguards classifier
- drop_params
- Extra body fields
- Prompt management
- Model param gating
- Chat completions streaming
- Per-call model_list
- model_alias_map
- text_format parsing
- Responses through Chat completions
- Deployment and encoded item ids
- Responses input rewrites
- Provider affinity header
- Managed file and vector store ids
- Cache marks, tool search and tool schema repair
- Forwarded client headers
- Sync deployment hook
DB access0%of 8 pts
- Query registry and Python codegen
- Python runtime and PrismaExecutor
- Schema and codegen checks in CI
- Key, end user and spend queries
Auth and secrets66%of 13 pts
- AWS auth
- Azure auth
- Google Cloud auth
- Secret managers
- Virtual key lookup and cache
- Key cache invalidation across hosts
- Shared authorization crate
- JWT auth
Budgets and rate limits0%of 5 pts
- Shared rate limit scripts and key naming
- Budget checks
Spend tracking33%of 3 pts
- Cost calculation
- Spend writes through the shared Redis buffer
Router0%of 8 pts
- Shared routing strategy crate
- Cooldown and health state
- Retries and fallbacks
- Models stored in DB
Logging and callbacks27%of 12 pts
- Logging callbacks
- Shared logging payload
- Prometheus
- S3 and other exporters
- Hook pipeline contract
- Custom Python callbacks forwarded to Python
Response cache55%of 12 pts
- In-memory and Redis backends
- Python compatibility paths removed
- Cache built from config in the gateway
- Cache gaps in dispatch
- Cache hits in spend and logs
- Chat and Responses caching
- Other storage backends
- Cache management endpoints
Config0%of 2 pts
- Config stored in DB
The solid line shows released work. The dashed line shows estimated weekly goals toward the December 1 Messages milestone. How we measure progress
Earlier benchmark
Forwarding-only Rust beta results
The July 2026 AIGatewayBench report measured four gateways against the same mock server with no logging callbacks, spend tracking, or persistence enabled. These results isolate forwarding overhead; they do not measure the December 1 fast path with common proxy features.
Added latency at p99
- 0.7 msLiteLLM Rust
- 2.3 msPortkey
- 4.5 msBifrost
- 257.7 msLiteLLM Python v1
Logarithmic scale. A low value is better.
Maximum memory use
- 21.8 MBLiteLLM Rust
- 90.4 MBPortkey
- 199.1 MBBifrost
- 329.5 MBLiteLLM Python v1
A low value is better.
These numbers are the delay the gateway itself adds. Each gateway handled 5000 requests for each endpoint, and p99 is the slowest request in every 100. The AIGatewayBench report gives the full method, the raw data, and the limits of the test.
Engineering updates
How we are getting there
Watch the GitHub discussion for weekly updates directly from the developers, with much more detail than the blog posts.
FAQ
Frequently asked questions
What is planned for December 1?
Nate's October 26 update shifts the migration from porting translation, then routing, then the web server to moving a common request path to Rust end to end. The target is about 20 times less proxy latency, around 1.5 milliseconds, for these requests. Updated benchmarks with common proxy features are still to come.
The initial scope is
/v1/messageswith simple-shuffle routing, Anthropic models on Anthropic, Amazon Bedrock, or Vertex AI, and supported callbacks such as S3 logs and spend logging. Every enabled callback must be supported. Other endpoints, routing strategies, providers, and unsupported callbacks stay on Python until the fast path expands to cover them.This path needs Rust deployment, database-backed configuration, Redis access, routing, auth, response caching, and supported logging callbacks and spend tracking. We estimate that this work brings the broader migration to about 57% on the chart's existing scale.
December 1 is the target for this common configuration. Coverage will expand to more configurations and request types after this path. The latency target is an expected result; updated benchmarks will confirm performance.
How will requests reach Rust or Python?
An Axum server will sit in front of all requests and check early whether Rust can handle them. Supported requests will use the Rust fast path. Other requests will be forwarded to a Python worker on the same host. Rust implementations of the router and callbacks will be shared with Python through thin wrappers, and both servers will use the same Redis and Postgres protocols.
How is the percentage calculated?
This tracks released components across the broader migration. It does not measure the share of traffic served by Rust. The December 1 projection is a rough estimate of the catalog work needed for the common Messages fast path, including its shared infrastructure. The October 26 update replaces the earlier roadmap dates.
Every piece of work has a size in rough engineer-weeks, and a component's points are the sizes of its work. A piece earns 80% of its size once it is in a release but incomplete or not yet trusted, and all of it once it is done. Major APIs is 146 points in all.
Major APIs Points Today By Dec 1 Deployment 10 0 10 Rust core 8 8 8 Messages translation 7 4.8 5.4 Chat completions translation 13 0.6 0.6 Responses translation 6.5 1.8 1.8 Python-only request features 38.5 0.8 0.8 DB access 8 0 8 Auth and secrets 13 8.6 10.6 Budgets and rate limits 5 0 5 Spend tracking 3 1 3 Router 8 0 8 Logging and callbacks 12 3.2 9.8 Response cache 12 6.6 10.2 Config 2 0 2 Total 146 35.4 83.2 What do Partial and Done mean?
A level says how confident we are in the work itself, not whether anything uses it yet. Groundwork that is finished and trusted but waits for the gateway to deploy is already done.
Partial
In a release, but incomplete or not yet trusted, so it usually ships opt-in.
Done
Complete and trusted, whether or not it runs by default or anything uses it yet.