Skip to main content

Rust Migration

LiteLLM is moving to Rust

See how much of the move to Rust is built and what is next.

Migration tracker

Where we are today

53% of built, shown as a .

53 percent of Major APIs is built in v1.111.0-rc.2. DB + spend logs is planned to reach 56 percent. /messages e2e, simple is planned to reach 60 percent. Planned percentages are rough estimates of the broader migration. Horizontal distance represents elapsed time.OctNov0%50%100%1.102 rc1Sep 131.103 rc1Sep 201.104 rc1Sep 281.106 rc1Oct 111.107 rc1Oct 181.109 rc1Nov 453%1.111 rc2Nov 19 · latest56%DB + spend logsNov 21 target60%/messagese2e, simpleDec 1 target

The solid line shows released work. The dashed line shows estimated weekly goals toward the December 1 Messages milestone. How we measure progress

Benchmarks

Rust with common proxy features

The November 2026 Better Benchmarks run tested each gateway at 50 requests per second with the features production gateways use: 1,000 virtual keys with budgets and rate limits on keys, users, teams, and team members, per-deployment limits, the dynamic rate limiter, spend logs, Prometheus metrics, S3 request logs, header hooks, and retries over 80 deployments in 20 model groups. Portkey's open-source gateway has none of these features, so it ran without them.

Median added latency

0.9 ms1.2 ms2.1 msOff scale79.3 ms
  • LiteLLM Rust
  • Bifrost
  • Portkey
  • LiteLLM Python

Linear scale in ms. A low value is better. LiteLLM Python runs past the top; its bar is cut.

Added latency at p99

1.8 ms4.2 ms5.3 msOff scale1,587.8 ms
  • LiteLLM Rust
  • Bifrost
  • Portkey
  • LiteLLM Python

Linear scale in ms. A low value is better. LiteLLM Python runs past the top; its bar is cut.

CPU used

0.15 cores0.41 cores0.49 coresOff scale2.92 cores
  • LiteLLM Rust
  • Bifrost
  • Portkey
  • LiteLLM Python

Linear scale in cores. A low value is better. LiteLLM Python runs past the top; its bar is cut.

Peak memory

357 MiB609 MiB2,809 MiBOff scale9,553 MiB
  • LiteLLM Rust
  • Portkey
  • Bifrost
  • LiteLLM Python

Linear scale in MiB. A low value is better. LiteLLM Python runs past the top; its bar is cut.

Engineering updates

How we are getting there

Watch the GitHub discussion for weekly updates directly from the developers, with much more detail than the blog posts.

  1. · Blog post

    Better Benchmarks
  2. · Blog post

    Routing in Rust
  3. · Blog post

    Rust Migration: /v1/messages end-to-end in Rust by Dec 1
  4. · Blog post

    OCR uses Rust by default starting with v1.102.0-rc.1
  5. · Blog post

    Benchmarking the LiteLLM Rust AI Gateway: Overhead, Memory, and Cost
  6. · Blog post

    Migrating LiteLLM to Rust - Building the Fastest and Litest AI Gateway

FAQ

Frequently asked questions

  • What is planned for December 1?

    Nate's October 26 update shifts the migration from porting translation, then routing, then the web server to moving a common request path to Rust end to end. The target is about 20 times less proxy latency, around 1.5 milliseconds, for these requests. Benchmarks with common proxy features measured 0.9 ms at the median and 1.8 ms at p99 at 50 requests per second.

    The initial scope is /v1/messages with simple-shuffle routing, Anthropic models on Anthropic, Amazon Bedrock, or Vertex AI, and supported callbacks such as S3 logs and spend logging. Every enabled callback must be supported. Other endpoints, routing strategies, providers, and unsupported callbacks stay on Python until the fast path expands to cover them.

    This path needs Rust deployment, database-backed configuration, Redis access, routing, auth, response caching, and supported logging callbacks and spend tracking. We estimate that this work brings the broader migration to about 60% on the chart's existing scale.

    December 1 is the target for this common configuration. Coverage will expand to more configurations and request types after this path. The benchmarked build is a feature branch; we expect comparable performance from the fast path.

  • How will requests reach Rust or Python?

    An Axum server will sit in front of all requests and check early whether Rust can handle them. Supported requests will use the Rust fast path. Other requests will be forwarded to a Python worker on the same host. Rust implementations of the router and callbacks will be shared with Python through thin wrappers, and both servers will use the same Redis and Postgres protocols.

  • How is the percentage calculated?

    This tracks released components across the broader migration. It does not measure the share of traffic served by Rust. The December 1 projection is a rough estimate of the catalog work needed for the common Messages fast path, including its shared infrastructure. The October 26 update replaces the earlier roadmap dates.

    Every piece of work has a size in rough engineer-weeks, and a component's points are the sizes of its work. A piece earns 80% of its size once it is in a release but incomplete or not yet trusted, and all of it once it is done. Major APIs is 146 points in all.

    Major APIsPointsTodayBy Dec 1
    Deployment105.45.4
    Rust core888
    Messages translation75.45.4
    Chat completions translation130.60.6
    Responses translation6.51.81.8
    Python-only request features38.516.416.4
    DB access848
    Auth and secrets138.611.6
    Budgets and rate limits544
    Spend tracking32.63
    Router877
    Logging and callbacks127.27.2
    Response cache126.66.6
    Config202
    Total14677.687
  • What do Partial and Done mean?

    A level says how confident we are in the work itself, not whether anything uses it yet. Groundwork that is finished and trusted but waits for the gateway to deploy is already done.

    • Partial

      In a release, but incomplete or not yet trusted, so it usually ships opt-in.

    • Done

      Complete and trusted, whether or not it runs by default or anything uses it yet.