Skip to main content

Secret Manager

LiteLLM supports reading secrets from Azure Key Vault, Google Secret Manager

info

✨ This is an Enterprise Feature

Enterprise Pricing

Contact us here to get a free trial

Supported Secret Managers​

AWS Key Management V1​

tip

[BETA] AWS Key Management v2 is on the enterprise tier. Go here for docs

Use AWS KMS to storing a hashed copy of your Proxy Master Key in the environment. 

export LITELLM_MASTER_KEY="djZ9xjVaZ..." # 👈 ENCRYPTED KEY
export AWS_REGION_NAME="us-west-2"
general_settings:
  key_management_system: "aws_kms"
  key_management_settings:
    hosted_keys: ["LITELLM_MASTER_KEY"] # 👈 WHICH KEYS ARE STORED ON KMS

See Decryption Code

AWS Secret Manager​

Store your proxy keys in AWS Secret Manager.

Proxy Usage​

  1. Save AWS Credentials in your environment
os.environ["AWS_ACCESS_KEY_ID"] = ""  # Access key
os.environ["AWS_SECRET_ACCESS_KEY"] = "" # Secret access key
os.environ["AWS_REGION_NAME"] = "" # us-east-1, us-east-2, us-west-1, us-west-2
  1. Enable AWS Secret Manager in config. 
general_settings:
  master_key: os.environ/litellm_master_key 
  key_management_system: "aws_secret_manager" # 👈 KEY CHANGE
  key_management_settings: 
    hosted_keys: ["litellm_master_key"] # 👈 Specify which env keys you stored on AWS

  1. Run proxy
litellm --config /path/to/config.yaml

Azure Key Vault​

Usage with LiteLLM Proxy Server​

  1. Install Proxy dependencies 
pip install 'litellm[proxy]' 'litellm[extra_proxy]'
  1. Save Azure details in your environment
export["AZURE_CLIENT_ID"]="your-azure-app-client-id"
export["AZURE_CLIENT_SECRET"]="your-azure-app-client-secret"
export["AZURE_TENANT_ID"]="your-azure-tenant-id"
export["AZURE_KEY_VAULT_URI"]="your-azure-key-vault-uri"
  1. Add to proxy config.yaml 
model_list: 
    - model_name: "my-azure-models" # model alias 
        litellm_params:
            model: "azure/<your-deployment-name>"
            api_key: "os.environ/AZURE-API-KEY" # reads from key vault - get_secret("AZURE_API_KEY")
            api_base: "os.environ/AZURE-API-BASE" # reads from key vault - get_secret("AZURE_API_BASE")

general_settings:
  key_management_system: "azure_key_vault"

You can now test this by starting your proxy: 

litellm --config /path/to/config.yaml

Quick Test Proxy

Google Secret Manager​

Support for Google Secret Manager

  1. Save Google Secret Manager details in your environment
GOOGLE_SECRET_MANAGER_PROJECT_ID="your-project-id-on-gcp" # example: adroit-crow-413218

Optional Params

export GOOGLE_SECRET_MANAGER_REFRESH_INTERVAL = ""            # (int) defaults to 86400
export GOOGLE_SECRET_MANAGER_ALWAYS_READ_SECRET_MANAGER = ""  # (str) set to "true" if you want to always read from google secret manager without using in memory caching. NOT RECOMMENDED in PROD
  1. Add to proxy config.yaml 
model_list:
  - model_name: fake-openai-endpoint
    litellm_params:
      model: openai/fake
      api_base: https://exampleopenaiendpoint-production.up.railway.app/
      api_key: os.environ/OPENAI_API_KEY # this will be read from Google Secret Manager

general_settings:
  key_management_system: "google_secret_manager"

You can now test this by starting your proxy: 

litellm --config /path/to/config.yaml

Quick Test Proxy

Google Key Management Service​

Use encrypted keys from Google KMS on the proxy

Step 1. Add keys to env 

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json"
export GOOGLE_KMS_RESOURCE_NAME="projects/*/locations/*/keyRings/*/cryptoKeys/*"
export PROXY_DATABASE_URL_ENCRYPTED=b'\n$\x00D\xac\xb4/\x8e\xc...'

Step 2: Update Config

general_settings:
  key_management_system: "google_kms"
  database_url: "os.environ/PROXY_DATABASE_URL_ENCRYPTED"
  master_key: sk-1234

Step 3: Start + test proxy

$ litellm --config /path/to/config.yaml

And in another terminal

$ litellm --test

Quick Test Proxy

All Secret Manager Settings​

All settings related to secret management

general_settings:
  key_management_system: "aws_secret_manager" # REQUIRED
  key_management_settings:  

    # Storing Virtual Keys Settings
    store_virtual_keys: true # OPTIONAL. Defaults to False, when True will store virtual keys in secret manager
    prefix_for_stored_virtual_keys: "litellm/" # OPTIONAL.I f set, this prefix will be used for stored virtual keys in the secret manager
    
    # Access Mode Settings
    access_mode: "write_only" # OPTIONAL. Literal["read_only", "write_only", "read_and_write"]. Defaults to "read_only"
    
    # Hosted Keys Settings
    hosted_keys: ["litellm_master_key"] # OPTIONAL. Specify which env keys you stored on AWS