Skip to main content

Rust Migration

LiteLLM is moving to Rust

See what already runs on Rust and what is next.

Migration tracker

Where we are today

12% of migrated to Rust, shown as a .

12 percent of Major APIs is migrated in v1.106.0-rc.2. This week is planned to reach 18 percent. Next week is planned to reach 23 percent. The goal targets 100 percent by 2026-12-31. Horizontal distance represents elapsed time on the same scale for releases, plans, and the goal.OctNovDec0%50%100%1.102 rc1Sep 1312%1.106 rc2Oct 13 · latest18%This weekOct 1723%Next weekOct 24Major APIs: 100% by 2026-12-31100% goalGoalDec 31

Result

The result of the move to Rust

This is where the migration takes us. AIGatewayBench measured four gateways against the same mock server: the Rust gateway, Portkey, Bifrost, and the current Python proxy. The test measured the delay and the memory that each gateway adds.

Added latency at p99

  • 0.7 msLiteLLM Rust
  • 2.3 msPortkey
  • 4.5 msBifrost
  • 257.7 msLiteLLM Python v1

Logarithmic scale. A low value is better.

Maximum memory use

  • 21.8 MBLiteLLM Rust
  • 90.4 MBPortkey
  • 199.1 MBBifrost
  • 329.5 MBLiteLLM Python v1

A low value is better.

These numbers are the delay the gateway itself adds. Each gateway handled 5000 requests for each endpoint, and p99 is the slowest request in every 100. The AIGatewayBench report gives the full method, the raw data, and the limits of the test.

Engineering updates

How we are getting there

  1. · Blog post

    OCR uses Rust by default starting with v1.102.0-rc.1

  2. · Blog post

    Benchmarking the LiteLLM Rust AI Gateway: Overhead, Memory, and Cost

  3. · Blog post

    Migrating LiteLLM to Rust - Building the Fastest and Litest AI Gateway

FAQ

Frequently asked questions

  • How is the percentage calculated?

    Each area of work carries a share of 104 points for Major APIs, and within an area its providers or pieces split those points evenly. A piece earns 80% of its share once anyone can turn it on, even as an opt-in, and the last 20% once it runs on Rust by default or the Python path is gone. Shared groundwork, like cloud auth, counts in full once it is done. All APIs reweighs Long tail from 4 to 23 points, for 123 in all.

    Major APIsPointsTodayBy Oct 24
    Scaffold444
    OCR222
    Messages50.45
    Responses403.2
    Chat completions1000
    Adapters400
    Router600
    Gateway800
    Critical-path callbacks73.23.2
    Caching50.90.9
    Auth and secrets61.91.9
    Non-DB endpoints600
    Extension points800
    Database1500
    Observability callbacks600
    Perf and benchmarks202
    Test harness and CI202
    Long tail400
    Total10412.424.2
  • How does a feature move to Rust?

    Each feature starts on Python and moves through these stages one release at a time. For the two middle stages, the LITELLM_RUST environment variable flips the default for the whole process.

    • Opt-in

      Python runs by default. Turn Rust on to try it, with Python as the fallback.

      LITELLM_RUST=1Run on Rust

    • Rust default

      Rust runs by default, with Python as the fallback. Turn Rust off to stay on Python.

      LITELLM_RUST=0Stay on Python

    • Rust only

      Always runs on Rust. The Python path is gone and the switch has no effect.

    • Done

      Shared work like cloud auth has no rollout of its own. It is done once every Rust path can use it.