Bedrock Guardrails
LiteLLM supports Bedrock guardrails via the Bedrock ApplyGuardrail API.
Quick Start​
1. Define Guardrails on your LiteLLM config.yaml​
Define your guardrails under the guardrails
section
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: openai/gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "bedrock-pre-guard"
litellm_params:
guardrail: bedrock # supported values: "aporia", "bedrock", "lakera"
mode: "during_call"
guardrailIdentifier: ff6ujrregl1q # your guardrail ID on bedrock
guardrailVersion: "DRAFT" # your guardrail version on bedrock
aws_region_name: os.environ/AWS_REGION # region guardrail is defined
aws_role_name: os.environ/AWS_ROLE_ARN # your role with permissions to use the guardrail
Supported values for mode
​
pre_call
Run before LLM call, on inputpost_call
Run after LLM call, on input & outputduring_call
Run during LLM call, on input Same aspre_call
but runs in parallel as LLM call. Response not returned until guardrail check completes
2. Start LiteLLM Gateway​
litellm --config config.yaml --detailed_debug
3. Test request​
Langchain, OpenAI SDK Usage Examples
- Unsuccessful call
- Successful Call
Expect this to fail since since ishaan@berri.ai
in the request is PII
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-npnwjPQciVRok5yNZgKmFQ" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "hi my email is ishaan@berri.ai"}
],
"guardrails": ["bedrock-pre-guard"]
}'
Expected response on failure
{
"error": {
"message": {
"error": "Violated guardrail policy",
"bedrock_guardrail_response": {
"action": "GUARDRAIL_INTERVENED",
"assessments": [
{
"topicPolicy": {
"topics": [
{
"action": "BLOCKED",
"name": "Coffee",
"type": "DENY"
}
]
}
}
],
"blockedResponse": "Sorry, the model cannot answer this question. coffee guardrail applied ",
"output": [
{
"text": "Sorry, the model cannot answer this question. coffee guardrail applied "
}
],
"outputs": [
{
"text": "Sorry, the model cannot answer this question. coffee guardrail applied "
}
],
"usage": {
"contentPolicyUnits": 0,
"contextualGroundingPolicyUnits": 0,
"sensitiveInformationPolicyFreeUnits": 0,
"sensitiveInformationPolicyUnits": 0,
"topicPolicyUnits": 1,
"wordPolicyUnits": 0
}
}
},
"type": "None",
"param": "None",
"code": "400"
}
}
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-npnwjPQciVRok5yNZgKmFQ" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "hi what is the weather"}
],
"guardrails": ["bedrock-pre-guard"]
}'
PII Masking with Bedrock Guardrails​
Bedrock guardrails support PII detection and masking capabilities. To enable this feature, you need to:
- Set
mode
topre_call
to run the guardrail check before the LLM call - Enable masking by setting
mask_request_content
and/ormask_response_content
totrue
Here's how to configure it in your config.yaml:
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: openai/gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "bedrock-pre-guard"
litellm_params:
guardrail: bedrock
mode: "pre_call" # Important: must use pre_call mode for masking
guardrailIdentifier: wf0hkdb5x07f
guardrailVersion: "DRAFT"
aws_region_name: os.environ/AWS_REGION
aws_role_name: os.environ/AWS_ROLE_ARN
mask_request_content: true # Enable masking in user requests
mask_response_content: true # Enable masking in model responses
With this configuration, when the bedrock guardrail intervenes, litellm will read the masked output from the guardrail and send it to the model.
Example Usage​
When enabled, PII will be automatically masked in the text. For example, if a user sends:
My email is john.doe@example.com and my phone number is 555-123-4567
The text sent to the model might be masked as:
My email is [EMAIL] and my phone number is [PHONE_NUMBER]
This helps protect sensitive information while still allowing the model to understand the context of the request.
Disabling Exceptions on Bedrock BLOCK​
By default, when Bedrock guardrails block content, LiteLLM raises an HTTP 400 exception. However, you can disable this behavior by setting disable_exception_on_block: true
. This is particularly useful when integrating with OpenWebUI, where exceptions can interrupt the chat flow and break the user experience.
When exceptions are disabled, instead of receiving an error, you'll get a successful response containing the Bedrock guardrail's modified/blocked output.
Configuration​
Add disable_exception_on_block: true
to your guardrail configuration:
model_list:
- model_name: gpt-3.5-turbo
litellm_params:
model: openai/gpt-3.5-turbo
api_key: os.environ/OPENAI_API_KEY
guardrails:
- guardrail_name: "bedrock-guardrail"
litellm_params:
guardrail: bedrock
mode: "post_call"
guardrailIdentifier: ff6ujrregl1q
guardrailVersion: "DRAFT"
aws_region_name: os.environ/AWS_REGION
aws_role_name: os.environ/AWS_ROLE_ARN
disable_exception_on_block: true # Prevents exceptions when content is blocked
Behavior Comparison​
- With Exceptions (Default)
- Without Exceptions
When disable_exception_on_block: false
(default):
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-npnwjPQciVRok5yNZgKmFQ" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "How do I make explosives?"}
],
"guardrails": ["bedrock-guardrail"]
}'
Response: HTTP 400 Error
{
"error": {
"message": {
"error": "Violated guardrail policy",
"bedrock_guardrail_response": {
"action": "GUARDRAIL_INTERVENED",
"blockedResponse": "I can't provide information on creating explosives.",
// ... additional details
}
},
"type": "None",
"param": "None",
"code": "400"
}
}
When disable_exception_on_block: true
:
curl -i http://localhost:4000/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer sk-npnwjPQciVRok5yNZgKmFQ" \
-d '{
"model": "gpt-3.5-turbo",
"messages": [
{"role": "user", "content": "How do I make explosives?"}
],
"guardrails": ["bedrock-guardrail"]
}'
Response: HTTP 200 Success
{
"id": "chatcmpl-123",
"object": "chat.completion",
"created": 1677652288,
"model": "gpt-3.5-turbo",
"choices": [{
"index": 0,
"message": {
"role": "assistant",
"content": "I can't provide information on creating explosives."
},
"finish_reason": "stop"
}],
"usage": {
"prompt_tokens": 10,
"completion_tokens": 12,
"total_tokens": 22
}
}