Skip to main content

Prefix Block Hash IDs

Since v1.111.0

The opt-in prefix_block_hashes callback records which requests share a prompt prefix without storing that prompt in the hash metadata. Use these IDs with the input and output token counts already in spend logs to prepare Mooncake-style traces for production traffic replay, such as benchmarks with AIPerf or a trace conversion workflow for GuideLLM. This callback does not export a trace or replay traffic itself.

Each ID represents a block of tokens and all blocks before it. Requests with the same tokenized prefix share leading IDs and diverge after their inputs differ. The callback uses chained HMAC-SHA256 digests, truncated to 16 hexadecimal characters per ID, and includes the final partial block.

Enable the callback​

Set a dedicated, high-entropy secret in the proxy's process environment. For example, generate one before starting the proxy:

export LITELLM_PREFIX_BLOCK_HASHES_SECRET="$(openssl rand -hex 32)"

Keep this value in your secret manager and reuse it across workers and restarts when you need comparable IDs. It has no fallback to LITELLM_SALT_KEY or the master key. Changing the secret changes the IDs.

Add the callback to your existing proxy configuration and restart the proxy:

litellm_settings:
callbacks: ["prefix_block_hashes"]

Use a database-backed proxy with spend tracking enabled to retain the IDs in LiteLLM_SpendLogs. The callback does not enable prompt storage, disable other logging callbacks, or change your existing prompt-retention settings. Review those settings separately if prompts must not be retained.

The default block size is 512 tokens. To use a different size, set LITELLM_PREFIX_BLOCK_HASHES_BLOCK_SIZE before startup, then restart:

export LITELLM_PREFIX_BLOCK_HASHES_BLOCK_SIZE=64

The optional LITELLM_PREFIX_BLOCK_HASHES_MAX_CHARS setting limits the serialized prompt length before tokenization. Its default is 65,536 characters, or TOKEN_COUNTER_MAX_EXACT_CHARS if that is lower. It accepts positive integers up to TOKEN_COUNTER_MAX_EXACT_CHARS; it is a character budget, not a token limit. Set it before startup and restart the proxy:

export LITELLM_PREFIX_BLOCK_HASHES_MAX_CHARS=32768

Read the IDs from spend logs​

Both streaming and non-streaming chat requests write to metadata.spend_logs_metadata.prefix_block_hashes in the spend log row. Existing spend_logs_metadata entries are preserved. An illustrative row, with unrelated fields omitted, looks like this:

{
"startTime": "2026-11-17T00:00:00.000Z",
"prompt_tokens": 655,
"completion_tokens": 52,
"metadata": {
"spend_logs_metadata": {
"prefix_block_hashes": {
"block_size": 512,
"hash_ids": ["a1b2c3d4e5f60718", "b2c3d4e5f6071829"]
}
}
}
}

These are content-derived fingerprints, not a guarantee of anonymity. Readers can compare prefixes within the same secret and tokenization setup. Protect the secret and restrict access to the logs.

When hashing is skipped​

Hashing is optional and fails open. A missing or empty secret, a serialized prompt over the character budget, or a serialization or tokenizer error leaves the request unchanged. These conditions do not fail the model request; skip and error details are logged at debug level.

The callback also skips hashing when the logging path redacts messages, including global litellm_settings.turn_off_message_logging: true and applicable per-request redaction controls. For client-supplied turn_off_message_logging, the deployment must explicitly allow that field through litellm_params.configurable_clientside_auth_params, for example:

model_list:
- model_name: replay-model
litellm_params:
model: openai/gpt-6-luna
api_key: os.environ/OPENAI_API_KEY
configurable_clientside_auth_params: ["turn_off_message_logging"]

A client can then send "turn_off_message_logging": true in the request body. Without the required client-side permission, proxy authentication can reject the override before the callback runs. The callback's fail-open behavior does not bypass authentication. See message logging controls for the global setting and header controls.

Worker admission is bounded by TOKEN_COUNTER_MAX_CONCURRENT_COUNTS, which defaults to 4 per process. If all hash slots are occupied, the request skips hashing instead of queuing for a slot. Non-chat call types and requests without a non-empty messages list are also left unchanged. The supported call types are completion and acompletion.

Convert rows to a Mooncake trace​

AIPerf's Mooncake trace format uses timestamp for arrival time in milliseconds, input_length and output_length for token counts, and optional integer hash_ids. Convert the hexadecimal IDs to integers rather than passing the strings through.

The following script reads a JSON array of exported spend log rows from standard input and writes JSONL. It skips rows without IDs, handles metadata exported as either an object or a JSON string, and makes timestamps relative to the first retained request. Export rows from one model/tokenizer setup, secret, and block size for a comparable trace.

import json
import sys
from datetime import datetime

records = []
for row in json.load(sys.stdin):
metadata = row["metadata"]
if isinstance(metadata, str):
metadata = json.loads(metadata)
hashes = (metadata or {}).get("spend_logs_metadata", {}).get("prefix_block_hashes")
if hashes and hashes["hash_ids"]:
started = datetime.fromisoformat(row["startTime"].replace("Z", "+00:00"))
records.append((started, row, hashes))

records.sort(key=lambda record: record[0])
for started, row, hashes in records:
print(json.dumps({
"timestamp": (started - records[0][0]).total_seconds() * 1000,
"input_length": row["prompt_tokens"],
"output_length": row["completion_tokens"],
"hash_ids": [int(value, 16) for value in hashes["hash_ids"]],
}))

Save it as spend_to_mooncake.py and run python3 spend_to_mooncake.py < spend_rows.json > mooncake.jsonl. An empty input array or an export with no hash-bearing rows produces no trace entries. Use successful rows with recorded token usage. Match your replay tool's synthetic block configuration to the stored block_size; the four-field trace above does not carry it. Other tools, including GuideLLM, can require their own import or conversion step.

Tokenizer and replay limits​

The callback serializes tools first as sorted-key JSON, then each message's role and text, plus tool_calls, tool_call_id, and name when present. It keeps text parts from list content and ignores non-text parts such as images. Tokenization uses LiteLLM's model-based encode selection with its default-encoding fallback.

This is a deterministic representation, not the provider's exact chat template. Provider input token counts can differ from the serialized token count, so len(hash_ids) need not equal the provider's prompt_tokens divided into blocks. Use the spend row's actual token counts for trace lengths and treat prefix IDs as an approximation of sharing, not a prediction of cache hits. Replay synthesizes content rather than recovering original prompts. Keep the tokenizer, serialization, block size, and secret consistent when comparing IDs.

LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails, built for production.
Learn more →