Skip to main content

Evaluate agents

Unreleased API preview

agent_eval() and aagent_eval() are not available in a published LiteLLM release yet. The examples below preview the API. This guide needs a verified release version and installation instructions before publication.

Compare agent harnesses and models on your own tasks with litellm.agent_eval(). Define the work and its success check in an EvalTask, then choose a harness and model with an EvalTarget. Compare pass rate, cost and duration in the scorecard.

1. Configure your AI Gateway​

Use a LiteLLM AI Gateway virtual key that can call two model groups. This example names them model-a and model-b. Replace those names with your groups. Both targets use Tool Loop, which needs no CLI runtime.

In Bash or Zsh, set your gateway's root URL, without /v1, and enter your virtual key after running read. The terminal hides the input:

export LITELLM_PROXY_API_BASE=https://litellm.example.com
export LITELLM_PROXY_API_KEY
read -rs LITELLM_PROXY_API_KEY

Skip the read command if your secret manager has set LITELLM_PROXY_API_KEY. To compare CLI harnesses such as Claude Code or Codex, install their dependencies from the quickstart.

2. Define tasks and targets​

Save this complete example as evaluate_agents.py.

Choose a sandbox factory for each evaluation call. This example uses sandbox.local for trusted tasks that can run with your user's permissions. The runner passes a fresh temporary workspace to the factory for each attempt.

evaluate_agents.py
from pathlib import Path

import litellm
from litellm import EvalTarget, EvalTask, Harness, PredicateCheck, sandbox


tasks = [
EvalTask(
id="billing-ticket",
prompt=(
"Classify this ticket as billing, technical, or sales. "
"Reply with only the category.\n"
"Ticket: I was charged twice for my subscription."
),
check=PredicateCheck(
lambda result: result.text.strip().lower() == "billing"
),
),
EvalTask(
id="technical-ticket",
prompt=(
"Classify this ticket as billing, technical, or sales. "
"Reply with only the category.\n"
"Ticket: The app crashes whenever I upload a PDF."
),
check=PredicateCheck(
lambda result: result.text.strip().lower() == "technical"
),
),
]

targets = [
EvalTarget(Harness.TOOL_LOOP, "litellm_proxy/model-a", label="model-a"),
EvalTarget(Harness.TOOL_LOOP, "litellm_proxy/model-b", label="model-b"),
]

report = litellm.agent_eval(
tasks=tasks,
targets=targets,
sandbox=sandbox.local,
max_concurrency=2,
)

print(report.to_markdown())
Path("scorecard.json").write_text(report.to_json(), encoding="utf-8")

PredicateCheck receives the agent's Result and returns a boolean. Check the final answer, tool calls or another part of the result. Predicates can be sync or async.

Each target can use a different harness, model and harness-specific options. Use a label to identify each setup in the scorecard. Task IDs and resolved target labels must be unique.

3. Run the comparison​

From the directory containing evaluate_agents.py, run:

python evaluate_agents.py

Two tasks and two targets produce four runs. In the model-a and model-b summaries, confirm that each target has two attempts. A target that returns both expected categories and stops with done has 2/2 passed runs. Use scorecard.json for your own analysis.

Set trials=3 to repeat each task and target three times. This example then produces twelve runs.

A run passes only when the agent stops with stop_reason="done" and its check passes. A failed check or turn limit produces fail. Setup failures, runtime errors, timeouts and cancelled agent runs produce error rows while the remaining runs continue. The runner skips grading when the agent does not finish with done. Cancelling the suite stops execution and triggers cleanup.

4. Inspect the results​

Read the typed summaries or individual runs without parsing Markdown:

from litellm import EvalStatus

for summary in report.summaries:
print(summary.target, summary.pass_rate, summary.cost)

for run in report.runs:
if run.status is not EvalStatus.PASS:
print(run.task_id, run.target, run.trial, run.detail)

Add this code after the evaluation call in evaluate_agents.py.

Pass rate is passed runs divided by scheduled runs, including errors. Totals include known usage from unsuccessful runs. Check metrics_complete before treating a cost total as complete. A false value means some usage or cost is unavailable.

Mean duration covers workspace preparation, setup, agent execution, grading and cleanup. Compare it separately from total suite time, since runs can overlap.

Compare agent and judge costs in separate columns. JSON exports include result data and identifiers. They omit authentication configuration, Python callables and raw events.

The runner attaches agent_eval_suite, agent_eval_run, agent_eval_task, agent_eval_target and agent_eval_trial metadata to model requests. Pass metadata={"experiment": "support-routing"} to add your own label. Use these identifiers to correlate requests in existing gateway logs. Persisted spend logs require the gateway's existing database setup.

Async execution​

In an async function or notebook, use aagent_eval() with the same tasks and targets:

report = await litellm.aagent_eval(
tasks=tasks,
targets=targets,
sandbox=sandbox.local,
max_concurrency=2,
)

print(report.to_markdown())

Grade writing or research with a model​

Use JudgeCheck to grade against a rubric. Choose a judge model that supports JSON output. The judge receives the task prompt, answer and reported file changes, together with your rubric. It can grade file diffs but cannot read other workspace files.

from litellm import EvalTask, JudgeCheck

writing_task = EvalTask(
id="support-reply",
prompt=(
"Write a reply to a customer who was charged twice. "
"Explain that support will investigate without promising a refund."
),
check=JudgeCheck(
model="litellm_proxy/judge",
rubric=(
"The reply acknowledges the duplicate charge, gives a clear next "
"step and does not guarantee a refund."
),
timeout=30,
),
)

Add writing_task to tasks and allow your virtual key to call the judge model group. Budget for these calls in addition to the agent runs.

A valid negative verdict produces fail. A failed judge request or invalid verdict produces error.

Evaluate a task that changes files​

For a task that needs starting files, pass a project directory as workspace. Each attempt gets a copy. The agent edits that copy, and the runner excludes .git metadata to keep Git state separate from the source.

Use a real directory for workspace. The runner rejects symlinks in the copied project before creating a sandbox, including links within the project and broken links. It also rejects a symlink used as the workspace root.

Create calculator/calc.py with a bug:

calculator/calc.py
def add(a, b):
return a - b

Replace the tasks and targets in evaluate_agents.py with this example. Install the CLI runtimes first using the quickstart, and use model groups available to your key.

from pathlib import Path

from litellm import CommandCheck, EvalTarget, EvalTask, Harness

tasks = [
EvalTask(
id="fix-calculator",
prompt="Fix calc.py so add(a, b) returns the sum of a and b.",
workspace=Path(__file__).resolve().parent / "calculator",
check=CommandCheck(
[
"python",
"-c",
"from calc import add; "
"assert add(2, 3) == 5; "
"assert add(-2, 2) == 0",
],
timeout=30,
),
),
]

targets = [
EvalTarget(Harness.CLAUDE_CODE, "litellm_proxy/claude"),
EvalTarget(Harness.CODEX, "litellm_proxy/gpt"),
]

Use CommandCheck to run a grading command in the same sandbox after the agent finishes. Exit code zero passes. This example keeps assertions in the evaluation script, outside the copied project. Use a task's setup commands to install dependencies before the agent starts.

Tool Loop has no built-in file or shell tools. Use a coding harness for this example. Caller-supplied Python tools execute in the SDK process with its permissions.

Choose a sandbox and execution limits​

Both evaluation functions require an explicit sandbox factory. Pass sandbox=sandbox.local for trusted tasks that can run with your user's permissions. For CLI harnesses, use a container to isolate the runtime and its dependencies.

For container execution, pass a factory that receives each attempt's fresh workspace:

from litellm import sandbox

report = litellm.agent_eval(
tasks=tasks,
targets=targets,
sandbox=lambda workspace: sandbox.docker(
"my-eval-runtime:latest",
mounts={str(workspace): "/workspace"},
workdir="/workspace",
),
)

Replace my-eval-runtime:latest with an image you built containing the required CLI runtimes and project dependencies. Docker must be running. See Sandboxes for the sandbox contract.

OptionDefaultMeaning
sandboxRequiredFactory that receives each attempt's fresh workspace and returns a sandbox
trials1Independent attempts per task and target
max_concurrency1Concurrent attempts, including setup and grading
timeout300Seconds allowed for each agent execution
setup_timeout120Seconds allowed for the setup stage of each attempt
CommandCheck.timeout30Seconds allowed for the grading command
JudgeCheck.timeout30Seconds allowed for the judge request
max_turnsNoneExisting agent turn limit, which counts tool-call events rather than model calls

The runner closes its sandboxes and removes temporary workspaces after each attempt. Inspect file changes and check details in the scorecard, or export them to JSON.

Troubleshoot a run​

SymptomNext action
Importing agent_eval failsThis API is unreleased. Run these examples after a release includes it.
A CLI runtime is missingInstall that harness in the local environment or container image.
A workspace error mentions symbolic linksUse a real directory for workspace and replace links with the files or directories the task needs.
The gateway rejects a targetCheck its model group name and the virtual key's allowed models.
A run times outInspect its stop reason and check detail, then adjust the corresponding execution or grading timeout.
The agent finishes but the run failsInspect the grading result. A completed agent turn alone does not mean the task succeeded.

See the API reference for the typed evaluation inputs and outputs.