AWS Secret Manager

Enterprise feature

This feature requires a LiteLLM Enterprise license. Start a free 30-day trial or book a demo. See what Enterprise includes.

Store your proxy keys in AWS Secret Manager.

Feature	Support	Description
Reading Secrets	✅	Read secrets e.g OPENAI_API_KEY
Writing Secrets	✅	Store secrets e.g Virtual Keys

Proxy Usage

1. Save AWS Credentials in your environment

os.environ["AWS_ACCESS_KEY_ID"] = ""  # Access key
os.environ["AWS_SECRET_ACCESS_KEY"] = "" # Secret access key
os.environ["AWS_REGION_NAME"] = "" # us-east-1, us-east-2, us-west-1, us-west-2

2. Enable AWS Secret Manager in config.

Read Keys from AWS Secret Manager

general_settings:
  master_key: os.environ/litellm_master_key 
  key_management_system: "aws_secret_manager" # 👈 KEY CHANGE
  key_management_settings: 
    hosted_keys: ["litellm_master_key"] # 👈 Specify which env keys you stored on AWS 

Write Virtual Keys to AWS Secret Manager

This will only store virtual keys in AWS Secret Manager. No keys will be read from AWS Secret Manager.

general_settings:
  key_management_system: "aws_secret_manager" # 👈 KEY CHANGE
  key_management_settings: 
    store_virtual_keys: true # OPTIONAL. Defaults to False, when True will store virtual keys in secret manager
    prefix_for_stored_virtual_keys: "litellm/" # OPTIONAL. If set, this prefix will be used for stored virtual keys in the secret manager
    access_mode: "write_only" # Literal["read_only", "write_only", "read_and_write"]
    description: "litellm virtual key" # OPTIONAL, if set will set this as the description for all virtual keys
    tags: # OPTIONAL, if set will set this as the tags for all virtual keys
      Environment: "Prod"
      Owner: "AI Platform team"
    kms_key_id: "arn:aws:kms:us-east-1:123456789012:key/11111111-2222-3333-4444-555555555555" # OPTIONAL, customer-managed KMS key used to encrypt stored virtual keys

Read + Write Keys with AWS Secret Manager

general_settings:
  master_key: os.environ/litellm_master_key 
  key_management_system: "aws_secret_manager" # 👈 KEY CHANGE
  key_management_settings: 
    store_virtual_keys: true # OPTIONAL. Defaults to False, when True will store virtual keys in secret manager
    prefix_for_stored_virtual_keys: "litellm/" # OPTIONAL. If set, this prefix will be used for stored virtual keys in the secret manager
    access_mode: "read_and_write" # Literal["read_only", "write_only", "read_and_write"]
    hosted_keys: ["litellm_master_key"] # OPTIONAL. Specify which env keys you stored on AWS

3. Run proxy

litellm --config /path/to/config.yaml

Encrypt Virtual Keys with a Customer-Managed KMS Key

By default, secrets LiteLLM creates in AWS Secrets Manager are encrypted with the AWS-managed aws/secretsmanager key. Set kms_key_id to a KMS key ID, alias or ARN to encrypt them with your own customer-managed key (CMK) instead. The value is sent as KmsKeyId on every CreateSecret call, so it only applies to secrets created after the setting is in place. Existing secrets keep their current key.

general_settings:
  key_management_system: "aws_secret_manager"
  key_management_settings:
    store_virtual_keys: true
    aws_region_name: "us-east-1"
    kms_key_id: "arn:aws:kms:us-east-1:123456789012:key/11111111-2222-3333-4444-555555555555"

The IAM identity the proxy uses needs kms:GenerateDataKey and kms:Decrypt on that key in addition to its Secrets Manager permissions. You can confirm the key took effect with aws secretsmanager describe-secret --secret-id litellm/<key_alias>; the response's KmsKeyId should match the configured key.

Using K/V pairs in 1 AWS Secret

You can read multiple keys from a single AWS Secret using the primary_secret_name parameter:

general_settings:
  key_management_system: "aws_secret_manager"
  key_management_settings:
    hosted_keys: [
      "OPENAI_API_KEY_MODEL_1",
      "OPENAI_API_KEY_MODEL_2",
    ]
    primary_secret_name: "litellm_secrets" # 👈 Read multiple keys from one JSON secret

The primary_secret_name allows you to read multiple keys from a single AWS Secret as a JSON object. For example, the "litellm_secrets" would contain:

{
  "OPENAI_API_KEY_MODEL_1": "sk-key1...",
  "OPENAI_API_KEY_MODEL_2": "sk-key2..."
}

This reduces the number of AWS Secrets you need to manage.

IAM Role Assumption

Use IAM roles instead of static AWS credentials for better security.

Basic IAM Role

general_settings:
  key_management_system: "aws_secret_manager"
  key_management_settings:
    store_virtual_keys: true
    aws_region_name: "us-east-1"
    aws_role_name: "arn:aws:iam::123456789012:role/LiteLLMSecretManagerRole"
    aws_session_name: "litellm-session"

Cross-Account Access

general_settings:
  key_management_system: "aws_secret_manager"
  key_management_settings:
    store_virtual_keys: true
    aws_region_name: "us-east-1"
    aws_role_name: "arn:aws:iam::999999999999:role/CrossAccountRole"
    aws_external_id: "unique-external-id"

EKS with IRSA

general_settings:
  key_management_system: "aws_secret_manager"
  key_management_settings:
    store_virtual_keys: true
    aws_region_name: "us-east-1"
    aws_role_name: "arn:aws:iam::123456789012:role/LiteLLMServiceAccountRole"
    aws_web_identity_token: "os.environ/AWS_WEB_IDENTITY_TOKEN_FILE"

Configuration Parameters

Parameter	Description
aws_region_name	AWS region
aws_role_name	IAM role ARN to assume
aws_session_name	Session name (optional)
aws_external_id	External ID for cross-account
aws_profile_name	AWS profile from ~/.aws/credentials
aws_web_identity_token	OIDC token path for IRSA
aws_sts_endpoint	Custom STS endpoint for VPC
kms_key_id	Customer-managed KMS key (ID, alias or ARN) used to encrypt secrets LiteLLM creates