Skip to main content
Blog
openaitypesafedecisionsday 0 support

Decision models on LiteLLM: /v1/decisions and /v1/systemone

Published:
Kerry LuRust Engineer, LiteLLM

LiteLLM now serves decision models on POST /v1/decisions and POST /v1/systemone. You hand a decision model an input and a list of typed questions, and it answers each one with a probability: a yes/no, a pick from a fixed list, or a score on a scale. No prose comes back, so there is nothing to parse.

Why decision models matter​

Most "small" LLM calls in a production app are not generation at all. Should this ticket go to a human? Is this prompt a jailbreak? Which of our five models should handle this request? How frustrated is this customer? Today teams answer those with a chat model, a prompt that begs for JSON, a parser, and a retry loop for the times the model answers in a sentence anyway. That costs tokens, adds latency on every request, and still fails in ways you only find in logs.

A decision model fixes the shape of the problem. A predicate question returns a probability. A choice question returns the picked option with a probability per option. A score question returns a value on the scale you defined. The answer is a number you can threshold, log, and compare across models, and the model cannot drift into free text. Because these models skip generation, they are small and fast, which is what lets a decision sit in front of every request instead of only the expensive ones.

That is what makes them the natural building block for the gateway layer: routing a request to the right model, deciding whether a guardrail should block, triaging support traffic, ranking candidates before a bigger model sees them.

What LiteLLM adds​

  • One request shape, nine providers. OpenAI, TypeSafe Jev, Fireworks AI, Perplexity, Cloudflare Clef, Databricks (including ai_decide), Microsoft Foundry Decision-1, Strands Decider and self-hosted vLLM. LiteLLM translates between OpenAI's Decisions format and TypeSafe's System One format, so you can switch the model name and keep the request and the response shape.
  • The same controls as chat. Virtual keys, budgets, rate limits, load balancing and fallbacks apply, and every answer carries x-litellm-response-cost and lands in the spend logs next to your chat traffic.
  • Decisions as guardrails. A decision model can be the thing that decides whether a request is allowed, with the probability threshold under your control.
  • A playground to try it. The Admin UI has a Decisions tab that sends to either route and shows the probabilities back.

Setup, the request and response formats, the provider table and SDK usage are in the decisions docs.

Feedback​

Tell us which decision providers you want next in GitHub Discussions.