Skip to main content

Braintrust - Evals + Logging

Braintrust manages evaluations, logging, prompt playground, to data management for AI products.

Quick Startโ€‹

# pip install braintrust
import litellm
import os

# set env
os.environ["BRAINTRUST_API_KEY"] = ""
os.environ['OPENAI_API_KEY']=""

# set braintrust as a callback, litellm will send the data to braintrust
litellm.callbacks = ["braintrust"]

# openai call
response = litellm.completion(
model="gpt-3.5-turbo",
messages=[
{"role": "user", "content": "Hi ๐Ÿ‘‹ - i'm openai"}
]
)

OpenAI Proxy Usageโ€‹

  1. Add keys to env
BRAINTRUST_API_KEY=""
  1. Add braintrust to callbacks
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY

litellm_settings:
callbacks: ["braintrust"]
  1. Test it!
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer sk-1234' \
-D '{
"model": "groq-llama3",
"messages": [
{ "role": "system", "content": "Use your tools smartly"},
{ "role": "user", "content": "What time is it now? Use your tool"}
]
}'

Advanced - pass Project ID or nameโ€‹

It is recommended that you include the project_id or project_name to ensure your traces are being written out to the correct Braintrust project.

response = litellm.completion(
model="gpt-3.5-turbo",
messages=[
{"role": "user", "content": "Hi ๐Ÿ‘‹ - i'm openai"}
],
metadata={
"project_id": "1234",
# passing project_name will try to find a project with that name, or create one if it doesn't exist
# if both project_id and project_name are passed, project_id will be used
# "project_name": "my-special-project"
}
)

Note: Other metadata can be included here as well when using the SDK.

response = litellm.completion(
model="gpt-3.5-turbo",
messages=[
{"role": "user", "content": "Hi ๐Ÿ‘‹ - i'm openai"}
],
metadata={
"project_id": "1234",
"item1": "an item",
"item2": "another item"
}
)

Full API Specโ€‹

Here's everything you can pass in metadata for a braintrust request

braintrust_* - If you are adding metadata from proxy request headers, any metadata field starting with braintrust_ will be passed as metadata to the logging request. If you are using the SDK, just pass your metadata like normal (e.g., metadata={"project_name": "my-test-project", "item1": "an item", "item2": "another item"})

project_id - Set the project id for a braintrust call. Default is litellm.