Skip to main content

Lens CLI

Evaluate models, harnesses, and custom agents on your own tasks.

Saved evaluation24 sample tasks × 12 configurations · 288 attempts
⏺ RecommendationClaude Agent SDK + gpt-5.4-mini24 / 24 passed · $0.0249 estimated total cost

▍ Results 12 / 12 rows

#HarnessModelPassFailErrEst. cost$/passP50
✻ Claude Agent SDK◎ gpt-5.4-mini100.0%00$0.0249$0.00101.7s
◎ OpenAI Agents◎ gpt-5.5100.0%00$0.0687$0.00292.3s
✻ Claude Agent SDK◎ gpt-5.5100.0%00$0.1260$0.00532.8s
⬡ Deep Agents◎ gpt-5.5100.0%00$0.3795$0.01582.4s
⬡ Deep Agents◎ gpt-5.4-mini95.8%10$0.0715$0.00311.5s
✻ Claude Agent SDK✻ claude-sonnet-5-595.8%10$0.0765$0.00331.8s
⬡ Deep Agents✻ claude-sonnet-5-595.8%10$0.2750$0.01201.5s
◎ OpenAI Agents◎ gpt-5.4-mini91.7%20$0.0088$0.00041.4s
◎ OpenAI Agents✻ claude-opus-5-591.7%20$0.1035$0.00471.4s
⬡ Deep Agents✻ claude-opus-5-591.7%20$0.5531$0.02511.4s
◎ OpenAI Agents✻ claude-sonnet-5-583.3%40$0.0517$0.00261.9s
✻ Claude Agent SDK✻ claude-opus-5-550.0%120$0.1778$0.01482.1s

▍ Cost vs quality

Select a saved configuration to inspect its pass rate and estimated cost. The cost axis is logarithmic.Pass rate70%80%90%100%$0.01$0.03$0.10$0.30$0.6011. OpenAI Agents, claude-sonnet-5-5: 83.3%, $0.0517 estimated total cost10. Deep Agents, claude-opus-5-5: 91.7%, $0.5531 estimated total cost9. OpenAI Agents, claude-opus-5-5: 91.7%, $0.1035 estimated total cost8. OpenAI Agents, gpt-5.4-mini: 91.7%, $0.0088 estimated total cost7. Deep Agents, claude-sonnet-5-5: 95.8%, $0.2750 estimated total cost6. Claude Agent SDK, claude-sonnet-5-5: 95.8%, $0.0765 estimated total cost5. Deep Agents, gpt-5.4-mini: 95.8%, $0.0715 estimated total cost4. Deep Agents, gpt-5.5: 100.0%, $0.3795 estimated total cost43. Claude Agent SDK, gpt-5.5: 100.0%, $0.1260 estimated total cost32. OpenAI Agents, gpt-5.5: 100.0%, $0.0687 estimated total cost21. Claude Agent SDK, gpt-5.4-mini: 100.0%, $0.0249 estimated total cost1Estimated total cost (USD, log)

Select a row or point to inspect the configuration.

✻ Claude Agent SDKgpt-5.4-mini100.0% pass · $0.0249 estimated cost · p50 1.7s
263 pass · 25 fail · 0 errorsSaved sample · full report below
Saved-report replay, revealed in rank order. Costs are token-based estimates.

Before, you had to write the eval loop, create temporary workspaces and collect the results in a spreadsheet. Then you changed the model or harness and repeated the work.

Which model and harness should you use for a given task? Lens CLI lets you run the same tasks through your agents and compare the results in one terminal report.

This is for developers building agents who want to evaluate models, harnesses, and their own agent code on the work their users need done.

A real Lens CLI report ranks twelve model and harness configurations on twenty-four sample legal-QA tasks.

A real saved run: 24 synthetic legal-QA tasks, three harnesses, four models. CLI output framed in Xnapper; model costs are estimates from token usage.

Choose your agents and tasks​

Choose representative tasks and what counts as a correct result. Compare built-in harnesses, such as a tool loop or Codex, alongside your own agent code. Keep the same tasks and grading across configurations.

The walkthrough supplies a complete configuration that connects your agents and records their answers, costs, and tool calls.

Already have datasets on your gateway? Open Lens > Datasets, select a dataset revision, and click Export JSONL. The walkthrough converts that snapshot into the CLI's task format while preserving the conversation and expected answers.

Run the CLI​

With the preview CLI installed and the walkthrough's files in your project directory, run:

lens run tasks.csv --config evals.py --trials 2

The CLI runs the task set against your configurations. Watch the progress, then read the completed report. For an exported gateway dataset, use the walkthrough's converted file:

lens run gateway-tasks.csv --config evals.py --trials 2

Read the report and choose your next change​

Start with the recommendation and results table. Compare pass rate and cost, then use the Cost vs quality chart to see the tradeoff across your model and harness choices.

The real Lens CLI chart compares estimated total cost with pass rate for the same twelve model and harness configurations.

Read up and left for configurations that pass more cases at lower estimated cost.

In this run, Claude Agent SDK with gpt-5.4-mini passed all 24 cases. The same model passed 22 through OpenAI Agents and 23 through Deep Agents. The result depends on these tasks and their grader, so evaluate your own workload before choosing a default.

The real Lens CLI report separates wrong answers, tool misuse and execution errors, with tool-call counts beneath them.

Inspect wrong answers, tool misuse, and errors before deciding what to change.

Use the failures to choose your next change, then run the same cases again.

Run through your Lens-enabled gateway, then open Lens > Traces to inspect captured model calls. Instrument your agent to include tool steps.

The Lens Traces UI shows captured agent runs with inputs, duration, costs, and findings.

Lens UI shown with sample data, separate from the CLI run above.