Rust Migration
LiteLLM is moving to Rust
See how much of the move to Rust is built and what is next.
Migration tracker
Where we are today
53% of built, shown as a .
Deployment54%of 10 pts
- Gateway binary in the Docker image
- Forward unserved requests to Python
- Dispatch crate and gap table
- Per model group plans at config load
- Identify, dispatch and admit layers
Rust core100%of 8 pts
- Python bridge
- Python host calls
- HTTP client
- Streaming
- Async bridge
- Tracing
- Model catalog
- Config parsing
Messages translation77%of 7 pts
- Anthropic
- DeepSeek
- Vertex AI
- Bedrock
- Bedrock Mantle
- Azure AI
- MiniMax
- GitHub Copilot
- Tencent
- Eden AI
- OpenAI-compatible
Chat completions translation5%of 13 pts
- OpenAI
- Azure OpenAI
- Anthropic
- Bedrock
- Vertex AI
- Gemini
- Azure AI
- Mistral
- Cohere
- Groq
- xAI
- DeepSeek
- Together AI
- Fireworks AI
- OpenRouter
- Databricks
- watsonx
- Perplexity
- DeepInfra
- Cerebras
- NVIDIA NIM
- SageMaker
- Hugging Face
- Replicate
- AI21
- SambaNova
- Nebius
- Novita AI
- Hyperbolic
- Baseten
- Moonshot AI
- DashScope
- Volcengine
- MiniMax
- GitHub Copilot
- Snowflake
- Oracle OCI
- Vercel AI Gateway
- LM Studio
- Ollama
- vLLM
- OpenAI-compatible
Responses translation28%of 6.5 pts
- OpenAI
- Azure OpenAI
- Azure AI
- Bedrock
- Bedrock Mantle
- xAI
- OpenRouter
- Databricks
- Fireworks AI
- Perplexity
- GitHub Copilot
- ChatGPT
- vLLM
- Volcengine
- Eden AI
- Manus
- Sail
- LiteLLM Proxy
- OpenAI-compatible
Python-only request features43%of 38.5 pts
- Mock responses
- Proxy MCP tools
- Caller HTTP clients
- Request, stream and phase timeouts
- Agentic loop hooks
- TLS security level and curves
- modify_params rewrites
- Prompt cache breakpoint injection
- Anthropic workload identity tokens
- Thinking signature retry
- Cost in streamed usage
- Web search interception
- Messages interceptors
- Safeguards classifier
- drop_params
- Extra body fields
- Prompt management
- Model param gating
- Chat completions streaming
- Per-call model_list
- model_alias_map
- text_format parsing
- Responses through Chat completions
- Deployment and encoded item ids
- Responses input rewrites
- Provider affinity header
- Managed file and vector store ids
- Cache marks, tool search and tool schema repair
- Forwarded client headers
- Sync deployment hook
DB access50%of 8 pts
- Query registry and Python codegen
- Python runtime and PrismaExecutor
- Schema and codegen checks in CI
- Key, end user and spend queries
Auth and secrets66%of 13 pts
- AWS auth
- Azure auth
- Google Cloud auth
- Secret managers
- Virtual key lookup and cache
- Key cache invalidation across hosts
- Shared authorization crate
- JWT auth
Budgets and rate limits80%of 5 pts
- Shared rate limit scripts and key naming
- Budget checks
Spend tracking87%of 3 pts
- Cost calculation
- Spend writes through the shared Redis buffer
Router88%of 8 pts
- Shared routing strategy crate
- Cooldown and health state
- Retries and fallbacks
- Models stored in DB
Logging and callbacks60%of 12 pts
- Logging callbacks
- Shared logging payload
- Prometheus
- S3 and other exporters
- Hook pipeline contract
- Custom Python callbacks forwarded to Python
Response cache55%of 12 pts
- In-memory and Redis backends
- Python compatibility paths removed
- Cache built from config in the gateway
- Cache gaps in dispatch
- Cache hits in spend and logs
- Chat and Responses caching
- Other storage backends
- Cache management endpoints
Config0%of 2 pts
- Config stored in DB
The solid line shows released work. The dashed line shows estimated weekly goals toward the December 1 Messages milestone. How we measure progress
Benchmarks
Rust with common proxy features
The November 2026 Better Benchmarks run tested each gateway at 50 requests per second with the features production gateways use: 1,000 virtual keys with budgets and rate limits on keys, users, teams, and team members, per-deployment limits, the dynamic rate limiter, spend logs, Prometheus metrics, S3 request logs, header hooks, and retries over 80 deployments in 20 model groups. Portkey's open-source gateway has none of these features, so it ran without them.
Median added latency
- LiteLLM Rust
- Bifrost
- Portkey
- LiteLLM Python
Linear scale in ms. A low value is better. LiteLLM Python runs past the top; its bar is cut.
Added latency at p99
- LiteLLM Rust
- Bifrost
- Portkey
- LiteLLM Python
Linear scale in ms. A low value is better. LiteLLM Python runs past the top; its bar is cut.
CPU used
- LiteLLM Rust
- Bifrost
- Portkey
- LiteLLM Python
Linear scale in cores. A low value is better. LiteLLM Python runs past the top; its bar is cut.
Peak memory
- LiteLLM Rust
- Portkey
- Bifrost
- LiteLLM Python
Linear scale in MiB. A low value is better. LiteLLM Python runs past the top; its bar is cut.
Engineering updates
How we are getting there
Watch the GitHub discussion for weekly updates directly from the developers, with much more detail than the blog posts.
· Blog post
Better Benchmarks· Blog post
Routing in Rust· Blog post
Rust Migration: /v1/messages end-to-end in Rust by Dec 1· Blog post
OCR uses Rust by default starting with v1.102.0-rc.1· Blog post
Benchmarking the LiteLLM Rust AI Gateway: Overhead, Memory, and Cost· Blog post
Migrating LiteLLM to Rust - Building the Fastest and Litest AI Gateway
FAQ
Frequently asked questions
What is planned for December 1?
Nate's October 26 update shifts the migration from porting translation, then routing, then the web server to moving a common request path to Rust end to end. The target is about 20 times less proxy latency, around 1.5 milliseconds, for these requests. Benchmarks with common proxy features measured 0.9 ms at the median and 1.8 ms at p99 at 50 requests per second.
The initial scope is
/v1/messageswith simple-shuffle routing, Anthropic models on Anthropic, Amazon Bedrock, or Vertex AI, and supported callbacks such as S3 logs and spend logging. Every enabled callback must be supported. Other endpoints, routing strategies, providers, and unsupported callbacks stay on Python until the fast path expands to cover them.This path needs Rust deployment, database-backed configuration, Redis access, routing, auth, response caching, and supported logging callbacks and spend tracking. We estimate that this work brings the broader migration to about 60% on the chart's existing scale.
December 1 is the target for this common configuration. Coverage will expand to more configurations and request types after this path. The benchmarked build is a feature branch; we expect comparable performance from the fast path.
How will requests reach Rust or Python?
An Axum server will sit in front of all requests and check early whether Rust can handle them. Supported requests will use the Rust fast path. Other requests will be forwarded to a Python worker on the same host. Rust implementations of the router and callbacks will be shared with Python through thin wrappers, and both servers will use the same Redis and Postgres protocols.
How is the percentage calculated?
This tracks released components across the broader migration. It does not measure the share of traffic served by Rust. The December 1 projection is a rough estimate of the catalog work needed for the common Messages fast path, including its shared infrastructure. The October 26 update replaces the earlier roadmap dates.
Every piece of work has a size in rough engineer-weeks, and a component's points are the sizes of its work. A piece earns 80% of its size once it is in a release but incomplete or not yet trusted, and all of it once it is done. Major APIs is 146 points in all.
Major APIs Points Today By Dec 1 Deployment 10 5.4 5.4 Rust core 8 8 8 Messages translation 7 5.4 5.4 Chat completions translation 13 0.6 0.6 Responses translation 6.5 1.8 1.8 Python-only request features 38.5 16.4 16.4 DB access 8 4 8 Auth and secrets 13 8.6 11.6 Budgets and rate limits 5 4 4 Spend tracking 3 2.6 3 Router 8 7 7 Logging and callbacks 12 7.2 7.2 Response cache 12 6.6 6.6 Config 2 0 2 Total 146 77.6 87 What do Partial and Done mean?
A level says how confident we are in the work itself, not whether anything uses it yet. Groundwork that is finished and trusted but waits for the gateway to deploy is already done.
Partial
In a release, but incomplete or not yet trusted, so it usually ships opt-in.
Done
Complete and trusted, whether or not it runs by default or anything uses it yet.