JevRank: decision models through LiteLLM’s rerank API
PREVIEW / OCTOBER 2026JevRank
Decision models
for reranking
Turn Perplexity and TypeSafe decision models into rerankers. Keep your query, your documents and your LiteLLM endpoint.
JevRank builds on the decision-model reranking research of Jakob Gruber at Swietelsky AG.
TypeSafe Jev
Perplexityquery, documents[]02 / TRANSFORMJevRankprobabilities → scores03 / OUTPUTRanked passagesindex, relevance_score
Benchmark results
In a SciFact reference benchmark, Perplexity Flash and Cohere Rerank 4 Pro both scored 0.775 nDCG@10. Their projected costs were $73 versus $2,500 per million queries, about 34× lower for Flash. SciFact · 300 queries · 20 candidates · October 2026. Reference results via OpenRouter. Flash costs $73 versus Cohere Pro’s $2,500 per million queries, about 34× lower. These are reference results, not our LiteLLM run. Costs exclude gateway overhead; Jev 1.14 is projected from a rounded per-1K cost. Small score gaps do not establish a quality win.Model Method and result origin Ranking quality nDCG@10 · higher is better Reranking cost USD / 1M queries · lower is better nDCG at 10: Projected reranking cost per million queries: nDCG at 10: Projected reranking cost per million queries: nDCG at 10: Projected reranking cost per million queries: 
nDCG at 10: Projected reranking cost per million queries:
Our SciFact benchmark
We ran 6,000 document decisions through LiteLLM with Jev 1.13.0. On the same BM25 candidates, nDCG@10 improved from 0.686 to 0.768, at an estimated $0.575 per 1,000 queries.
All 300 queries completed. We measured 0.50 s median latency and 1.15 s at p95, with eight calls in flight per query. Jev placed a relevant document first for 68% of queries, compared with 55% for BM25.
| Model Method and result origin | Ranking quality nDCG@10 · higher is better | Reranking cost USD / 1M queries · lower is better |
|---|---|---|
TypeSafe Jev 1.13.0 (JevRank)Per document LiteLLM run | nDCG at 10: Quality ↑0.768 | Projected reranking cost per million queries: Cost / 1M queries ↓≈$575 |
BM25Local retrieval LiteLLM run | nDCG at 10: Quality ↑0.686 | Projected reranking cost per million queries: Cost / 1M queries ↓No rerank calls |
SciFact · 300 queries · 20 BM25 candidates per query. Measured through LiteLLM. Jev’s cost is estimated from reported tokens and excludes gateway overhead. BM25 adds no reranking calls.
The reference run’s retrieval settings and candidate identity were not verified to match this run.
View exact chart data
| Model and method | Origin | nDCG@10 | Cost / 1M |
|---|---|---|---|
| Perplexity Decider Flash v1.2 (JevRank)Multiple choice | Reference | 0.775 | $73 |
| Perplexity Decider v1.2 (JevRank)Multiple choice | Reference | 0.777 | $146 |
| TypeSafe Jev 1.14 (JevRank)Per document | Reference | 0.776 | ≈$580 |
| Cohere Rerank 4 ProRerank API | Reference | 0.775 | $2,500 |
| TypeSafe Jev 1.13.0 (JevRank)Per document | LiteLLM run | 0.7676260618846087 | ≈$575.39426 |
| BM25Local retrieval | LiteLLM run | 0.6863002612489066 | No rerank calls |
Both axes start at zero. Lines connect labels to their actual data points. Reference and measured candidate protocols may differ. BM25 has no reranking calls; retrieval itself is not asserted to be free.
Reference results via OpenRouter; solid points are our LiteLLM run. Costs exclude gateway overhead. The 34× comparison applies only to reference Flash versus reference Cohere Pro; nearby scores do not establish a quality win.
Recorded examples Improvement, unchanged ranking and regression
Recorded examples
Explore three saved queries, including an improvement, an unchanged first result and a regression. The scores come from our benchmark. These controls make no model calls.
Localization of PIN1 in the roots of Arabidopsis does not require VPS9a
- 1Local, Efflux-Dependent Auxin Gradients as a Common Module for Plant Organ Formation
Plants, compared to animals, exhibit an amazing adaptability and plasticity in their development. This is largely dependent on the ability of plants to form new organs, such as lateral roots, leaves, and flowers during postembryonic…
Document 16557565 - 2A molecular framework for plant regeneration.
Plants and some animals have a profound capacity to regenerate organs from adult tissues. Molecular mechanisms for regeneration have, however, been largely unexplored. Here we investigate a local regeneration response in Arabidopsis roots.…
Document 38675228 - 3Generation of cell polarity in plants links endocytosis, auxin distribution and cell fate decisionsJudged relevant · SciFact
Dynamically polarized membrane proteins define different cell boundaries and have an important role in intercellular communication—a vital feature of multicellular development. Efflux carriers for the signalling molecule auxin from the PIN…
Document 4350400 - 4The rice FISH BONE gene encodes a tryptophan aminotransferase, which affects pleiotropic auxin-related processes.
Auxin is a fundamental plant hormone and its localization within organs plays pivotal roles in plant growth and development. Analysis of many Arabidopsis mutants that were defective in auxin biosynthesis revealed that the indole-3-pyruvic…
Document 19950357 - 5The WUSCHEL-related homeobox gene WOX11 is required to activate shoot-borne crown root development in rice.
In rice (Oryza sativa), the shoot-borne crown roots are the major root type and are initiated at lower stem nodes as part of normal plant development. However, the regulatory mechanism of crown root development is poorly understood. In…
Document 20402596
SciFact labels evidence relevance, including supporting and refuting evidence. The model’s scores can disagree with those judgments. Download these examples ↗
How JevRank works
Follow one recorded TypeSafe Jev request from input to ranking. We show three of the 20 passages from SciFact query 702.
Recorded query: Localization of PIN1 in the roots of Arabidopsis does not require VPS9a
- Input index 0: Local, Efflux-Dependent Auxin Gradients as a Common Module for Plant Organ Formation. Jev relevance score 0.14; output rank 2.
- Input index 1: A molecular framework for plant regeneration.. Jev relevance score 0.12; output rank 3.
- Input index 2: Generation of cell polarity in plants links endocytosis, auxin distribution and cell fate decisions. Jev relevance score 0.39; output rank 1.
Recorded · TypeSafe Jev 1.13.0 · Scroll to follow the passage
Send a query and documents
Call LiteLLM’s rerank API with your query and candidate passages. Keep their input order: you will use these indexes to find each document in the response.
Follow index 2, the passage about cell polarity. It starts third in this recorded request.
Localization of PIN1 in the roots of Arabidopsis does not require VPS9a
TypeSafe Jev 1.13.0Score each document
For TypeSafe Jev, LiteLLM asks the same relevance question for each passage. Jev returns a probability for each answer.
Jev assigned index 2 a score of 0.39, compared with 0.14 and 0.12 for the other two passages. These independent scores do not need to sum to one.
Localization of PIN1 in the roots of Arabidopsis does not require VPS9a
TypeSafe Jev 1.13.0Return the ranked indexes
LiteLLM sorts the scores from highest to lowest and returns relevance_score with the original index. You can retrieve the matching text from your input documents.
Index 2 now ranks first. The next two results are indexes 0 and 1. The response keeps each document’s identity after the order changes.
Localization of PIN1 in the roots of Arabidopsis does not require VPS9a
TypeSafe Jev 1.13.0Recorded example, no model calls. Scores describe the model’s assessment of relevance. They do not establish whether the query’s claim is true. Inspect the saved query and all 20 results ↗ A model can compare candidates in one choice question or assess each document against the query. Choice probabilities belong to the supplied candidate set; scores from separate batches do not share a scale. Provider errors and refusals fail the request. LiteLLM sends the full input and returns an error if it exceeds provider limits. Perplexity accepts 2 to 255 documents in one choice question. TypeSafe assesses documents in parallel; the rerank response reports their combined token usage.Score meaning and failures
Setup and usage
Use JevRank through your AI Gateway or call it from Python. These examples require the preview branch; no released version includes the adapter yet. For this preview, set LITELLM_LOCAL_MODEL_COST_MAP=True to load the branch’s model catalog.
Set PERPLEXITY_API_KEY in your gateway environment. Add this deployment, then send a rerank request with your gateway URL and LiteLLM key.
model_list:
- model_name: jevrank-flash
litellm_params:
model: perplexity/pplx-decider-flash-v1.2
api_key: os.environ/PERPLEXITY_API_KEY
Start the gateway with litellm --config config.yaml --port 4000.
curl "$LITELLM_BASE_URL/rerank" \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "jevrank-flash",
"query": "How long is the warranty?",
"documents": [
"Shipping takes three days.",
"Warranty coverage lasts two years."
],
"top_n": 1
}'
Set LITELLM_BASE_URL to your gateway URL and LITELLM_API_KEY to your gateway key.
The TypeSafe preset pins Jev 1.14; our measured run used Jev 1.13.0. Evaluate the version you deploy.
Benchmark methodology
LiteLLM run
We ran all 300 SciFact test queries with 20 fixed BM25 candidates per query through SDK commit 5aa0a8304c. All 6,000 judgments completed with returned version jev-1.13.0.
BM25s 0.3.13 used Lucene BM25, k1=1.5, b=0.75, English stop words and Snowball stemming over titles and full abstracts. Median rerank latency was 0.50 s; p95 was 1.15 s with eight calls in flight. Returned tokens imply $0.172618 total at the configured gateway rate, an estimate rather than an invoice.
Reference experiment
Jakob Gruber’s October 2026 reference experiment at Swietelsky AG used the same SciFact split and candidate count, with calls through OpenRouter. Retrieval settings and candidate identity may differ from our run. We could not access Perplexity or Cohere for our benchmark.
The 34× comparison belongs to reference Flash versus Cohere Pro: $73 versus $2,500 per million queries and 0.775 nDCG@10 for both. Its paired tests found no significant quality difference. Costs cover reranking and depend on model rates and input length.