Skip to main content

Clientside LLM Credentials

Pass User LLM API Keys, Fallbacks​

Allow your end-users to pass their model list, api base, OpenAI API key (any LiteLLM supported provider) to make requests

Note This is not related to virtual keys. This is for when you want to pass in your users actual LLM API keys.

info

**You can pass a litellm.RouterConfig as user_config, See all supported params here https://github.com/BerriAI/litellm/blob/main/litellm/types/router.py **

Step 1: Define user model list & config​

import os

user_config = {
    'model_list': [
        {
            'model_name': 'user-azure-instance',
            'litellm_params': {
                'model': 'azure/chatgpt-v-2',
                'api_key': os.getenv('AZURE_API_KEY'),
                'api_version': os.getenv('AZURE_API_VERSION'),
                'api_base': os.getenv('AZURE_API_BASE'),
                'timeout': 10,
            },
            'tpm': 240000,
            'rpm': 1800,
        },
        {
            'model_name': 'user-openai-instance',
            'litellm_params': {
                'model': 'gpt-6-luna',
                'api_key': os.getenv('OPENAI_API_KEY'),
                'timeout': 10,
            },
            'tpm': 240000,
            'rpm': 1800,
        },
    ],
    'num_retries': 2,
    'allowed_fails': 3,
    'fallbacks': [
        {
            'user-azure-instance': ['user-openai-instance']
        }
    ]
}


Step 2: Send user_config in extra_body​

import openai
client = openai.OpenAI(
    api_key="sk-<your-litellm-api-key>",
    base_url="http://0.0.0.0:4000"
)

# send request to `user-azure-instance`
response = client.chat.completions.create(model="user-azure-instance", messages = [
    {
        "role": "user",
        "content": "this is a test request, write a short poem"
    }
], 
    extra_body={
      "user_config": user_config
    }
) # User config

print(response)

Pass User LLM API Keys / API Base​

Allows your users to pass in their OpenAI API key/API base (any LiteLLM supported provider) to make requests

Here's how to do it:

1. Enable configurable clientside auth credentials for a provider​

model_list:
  - model_name: "fireworks_ai/*"
    litellm_params:
      model: "fireworks_ai/*"
      configurable_clientside_auth_params: ["api_base"]
      # OR 
      configurable_clientside_auth_params: [{"api_base": '^https://litellm.*direct\.fireworks\.ai/v1$'}] # regex

Specify the endpoint params you want the user to be able to configure:

Without this opt-in, or general_settings.allow_client_side_credentials: true proxy-wide, a request body that sets api_base or base_url is rejected with Rejected Request: api_base is not allowed in request body

api_key does not need to be listed. A request-body api_key is accepted on any deployment, wildcard or not, and replaces the deployment's configured key for that request only. To send the user's key as a provider header instead, such as x-api-key for Anthropic, see Forward LLM Provider Authentication Headers

(check provider docs for provider-specific auth params - e.g. vertex_project)

2. Test it!​

import openai
client = openai.OpenAI(
    api_key="sk-<your-litellm-api-key>",
    base_url="http://0.0.0.0:4000"
)

# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-6-luna", messages = [
    {
        "role": "user",
        "content": "this is a test request, write a short poem"
    }
], 
    extra_body={"api_key": "my-bad-key", "api_base": "https://litellm-dev.direct.fireworks.ai/v1"}) # clientside credentials

print(response)

More examples:

Pass in the litellm_params (E.g. api_key, api_base, etc.) via the extra_body parameter in the OpenAI client.

import openai
client = openai.OpenAI(
    api_key="sk-<your-litellm-api-key>",
    base_url="http://0.0.0.0:4000"
)

# request sent to model set on litellm proxy, `litellm --model`
response = client.chat.completions.create(model="gpt-6-luna", messages = [
    {
        "role": "user",
        "content": "this is a test request, write a short poem"
    }
], 
    extra_body={
      "api_key": "my-azure-key",
      "api_base": "my-azure-base",
      "api_version": "my-azure-version"
    }) # User Key

print(response)

Let clients choose a deployment's model_id​

Since v1.109.0

If a deployment sets model_id (e.g. a SageMaker inference component, a Bedrock model ID or ARN) then LiteLLM will use the value of that deployment and ignore any model_id present in the request body, unless the deployment has opted in as described below. A SageMaker deployment that sets the X-Amzn-SageMaker-Inference-Component header in extra_headers is pinned in the same way. This applies to both proxy requests and to SDK Router calls.

To let clients choose the value on a deployment, list model_id in its configurable_clientside_auth_params:

model_list:
  - model_name: my-sagemaker
    litellm_params:
      model: sagemaker_chat/<your-endpoint-name>
      model_id: <default-inference-component>
      configurable_clientside_auth_params: ["model_id"]

A request to my-sagemaker with "model_id": "<other-inference-component>" in the body is routed to that component, a request without model_id is routed to <default-inference-component>. general_settings.allow_client_side_credentials: true doesn't opt in so it leaves a pinned model_id in place. A deployment that sets neither model_id nor the SageMaker header still uses the request's model_id. Prior to 1.109.0 a model_id in the request body would take precedence over the deployment's. See Pin the inference component on a deployment for SageMaker examples.

Pass provider-specific params (e.g. Region, Project ID, etc.)​

Specify the region, project id, etc. to use for making requests to Vertex AI on the clientside.

vertex_location, vertex_project, vertex_ai_location, and vertex_ai_project override the region and project set on the deployment, so the proxy rejects them in a request body by default with Rejected Request: vertex_location is not allowed in request body. To let users pick them, list them in configurable_clientside_auth_params on the deployment, or set general_settings.allow_client_side_credentials: true proxy-wide.

model_list:
  - model_name: gemini-2.5-flash
    litellm_params:
      model: vertex_ai/gemini-2.5-flash
      vertex_project: my-project
      vertex_location: europe-west4
      configurable_clientside_auth_params: ["vertex_location"] # 👈 lets the request pick the region

When several deployments share a model name, every one of them must list the param, because the router, not the request, picks which deployment serves it. The same goes for the model group a model_group_alias routes the name to and for every fallback the request or the router's fallbacks can move it onto, since each of them serves the same request. Only Vertex AI, Gemini, and GDC deployments read these four params, so a model group whose deployments all use other providers needs no opt-in for them.

This every-deployment rule covers every configurable_clientside_auth_params entry, api_base included, so a model group where only some deployments list api_base now rejects it. On routes that name the model in the URL, such as /v1beta/models/<model>:generateContent and Azure's /openai/deployments/<model>/chat/completions, the proxy checks the deployments of that model. A request that names more than 100 fallback models is rejected.

info

The proxy rejects these params without an opt-in, and checks every deployment for each opt-in, Since v1.111.0. Earlier versions accept the Vertex params from any request body and read one deployment's opt-in for the whole model group, so add the opt-in to every deployment before you upgrade if your users send them.

Any value passed in the Proxy's request body, will be checked by LiteLLM against the mapped openai / litellm auth params.

Unmapped params, will be assumed to be provider-specific params, and will be passed through to the provider in the LLM API's request body.

import openai
client = openai.OpenAI(
    api_key="sk-<your-litellm-api-key>",
    base_url="http://0.0.0.0:4000"
)

response = client.chat.completions.create(
    model="gemini-2.5-flash",
    messages = [
        {
            "role": "user",
            "content": "this is a test request, write a short poem"
        }
    ],
    extra_body={ # pass any additional litellm_params here
        "vertex_location": "us-central1"
    }
)

print(response)
LiteLLM Enterprise
SSO/SAML, audit logs, spend tracking, multi-team management, and guardrails, built for production.
Learn more →