Azure AI Search - Vector Store (Unified API)
Use this to search Azure AI Search Vector Stores, with LiteLLM's unified /chat/completions API.
Quick Start
You need three things:
- An Azure AI Search service
- An embedding model (to convert your queries to vectors)
- A search index with vector fields
Usage
- SDK
- PROXY
Basic Search
from litellm import vector_stores
import os
# Set your credentials
os.environ["AZURE_SEARCH_API_KEY"] = "your-search-api-key"
os.environ["AZURE_AI_SEARCH_EMBEDDING_API_BASE"] = "your-embedding-endpoint"
os.environ["AZURE_AI_SEARCH_EMBEDDING_API_KEY"] = "your-embedding-api-key"
# Search the vector store
response = vector_stores.search(
vector_store_id="my-vector-index", # Your Azure AI Search index name
query="What is the capital of France?",
custom_llm_provider="azure_ai",
azure_search_service_name="your-search-service",
litellm_embedding_model="azure/text-embedding-3-large",
litellm_embedding_config={
"api_base": os.getenv("AZURE_AI_SEARCH_EMBEDDING_API_BASE"),
"api_key": os.getenv("AZURE_AI_SEARCH_EMBEDDING_API_KEY"),
},
api_key=os.getenv("AZURE_SEARCH_API_KEY"),
)
print(response)
Async Search
from litellm import vector_stores
response = await vector_stores.asearch(
vector_store_id="my-vector-index",
query="What is the capital of France?",
custom_llm_provider="azure_ai",
azure_search_service_name="your-search-service",
litellm_embedding_model="azure/text-embedding-3-large",
litellm_embedding_config={
"api_base": os.getenv("AZURE_AI_SEARCH_EMBEDDING_API_BASE"),
"api_key": os.getenv("AZURE_AI_SEARCH_EMBEDDING_API_KEY"),
},
api_key=os.getenv("AZURE_SEARCH_API_KEY"),
)
print(response)
Advanced Options
from litellm import vector_stores
response = vector_stores.search(
vector_store_id="my-vector-index",
query="What is the capital of France?",
custom_llm_provider="azure_ai",
azure_search_service_name="your-search-service",
litellm_embedding_model="azure/text-embedding-3-large",
litellm_embedding_config={
"api_base": os.getenv("AZURE_AI_SEARCH_EMBEDDING_API_BASE"),
"api_key": os.getenv("AZURE_AI_SEARCH_EMBEDDING_API_KEY"),
},
api_key=os.getenv("AZURE_SEARCH_API_KEY"),
top_k=10, # Number of results to return
azure_search_vector_field="contentVector", # Custom vector field name
)
print(response)
Setup Config
Embed queries through a model group
Since v1.101.0Set litellm_embedding_model to a model_name from your model_list. Each search then embeds the query through the LiteLLM router, so the load balancing, retries, cooldowns and fallbacks you configure for that model group apply to search queries the same way they apply to /v1/embeddings calls on it. You do not set litellm_embedding_config; the credentials stay on the deployments.
model_list:
- model_name: query-embeddings
litellm_params:
model: azure/text-embedding-3-large
api_base: https://my-eastus2-resource.openai.azure.com/
api_key: os.environ/AZURE_EASTUS2_API_KEY
api_version: "2024-10-21"
- model_name: query-embeddings
litellm_params:
model: azure/text-embedding-3-large
api_base: https://my-westus3-resource.openai.azure.com/
api_key: os.environ/AZURE_WESTUS3_API_KEY
api_version: "2024-10-21"
router_settings:
num_retries: 2
vector_store_registry:
- vector_store_name: "azure-ai-search-litellm-website-knowledgebase"
litellm_params:
vector_store_id: "my-vector-index" # Your Azure AI Search index name
custom_llm_provider: "azure_ai"
azure_search_service_name: "your-search-service" # or api_base: https://your-search-service.search.windows.net
api_key: os.environ/AZURE_SEARCH_API_KEY
litellm_embedding_model: "query-embeddings" # model group from model_list
If one deployment fails, the router retries the query embedding on another deployment in the group. To fail over to a different model group, add it under router_settings.fallbacks, for example fallbacks: [{"query-embeddings": ["query-embeddings-backup"]}]. Every deployment in the group, and in any fallback group, must use the same embedding model and dimensions as the vectors in your index.
Single embedding deployment
You can also point the vector store at one embedding deployment directly and pass its credentials in litellm_embedding_config. This form gets no load balancing or failover.
vector_store_registry:
- vector_store_name: "azure-ai-search-litellm-website-knowledgebase"
litellm_params:
vector_store_id: "my-vector-index"
custom_llm_provider: "azure_ai"
azure_search_service_name: "your-search-service"
api_key: os.environ/AZURE_SEARCH_API_KEY
litellm_embedding_model: "azure/text-embedding-3-large"
litellm_embedding_config:
api_base: https://my-eastus2-resource.openai.azure.com/
api_key: os.environ/AZURE_API_KEY
api_version: "2024-10-21"
Start Proxy
litellm --config /path/to/config.yaml
Search via API
curl -X POST 'http://0.0.0.0:4000/v1/vector_stores/my-vector-index/search' \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer $LITELLM_API_KEY" \
-d '{
"query": "What is the capital of France?"
}'
Required Parameters
| Parameter | Type | Description |
|---|---|---|
vector_store_id | string | Your Azure AI Search index name |
custom_llm_provider | string | Set to "azure_ai" |
azure_search_service_name | string | Name of your Azure AI Search service |
litellm_embedding_model | string | Model that embeds the query. On the proxy, use a model group name from model_list (e.g., "query-embeddings") or a provider model (e.g., "azure/text-embedding-3-large") |
litellm_embedding_config | dict | Credentials for the embedding model (api_base, api_key, api_version). Not needed when litellm_embedding_model is a proxy model group |
api_key | string | Your Azure AI Search API key |
Supported Features
| Feature | Status | Notes |
|---|---|---|
| Logging | ✅ Supported | Full logging support available |
| Guardrails | ❌ Not Yet Supported | Guardrails are not currently supported for vector stores |
| Cost Tracking | ✅ Supported | Cost is $0 according to Azure |
| Unified API | ✅ Supported | Call via OpenAI compatible /v1/vector_stores/search endpoint |
| Passthrough | ❌ Not yet supported |
Response Format
The response follows the standard LiteLLM vector store format:
{
"object": "vector_store.search_results.page",
"search_query": "What is the capital of France?",
"data": [
{
"score": 0.95,
"content": [
{
"text": "Paris is the capital of France...",
"type": "text"
}
],
"file_id": "doc_123",
"filename": "Document doc_123",
"attributes": {
"document_id": "doc_123"
}
}
]
}
How It Works
When you search:
- LiteLLM converts your query to a vector using the embedding model you specified
- It sends the vector to Azure AI Search
- Azure AI Search finds the most similar documents in your index
- Results come back with similarity scores
The embedding model can be any model supported by LiteLLM - Azure OpenAI, OpenAI, Bedrock, etc.
Setting Up Your Azure AI Search Index
Your index needs a vector field. Here's what that looks like:
{
"name": "my-vector-index",
"fields": [
{
"name": "id",
"type": "Edm.String",
"key": true
},
{
"name": "content",
"type": "Edm.String"
},
{
"name": "contentVector",
"type": "Collection(Edm.Single)",
"searchable": true,
"dimensions": 1536,
"vectorSearchProfile": "myVectorProfile"
}
]
}
The vector dimensions must match your embedding model. For example:
text-embedding-3-large: 1536 dimensionstext-embedding-3-small: 1536 dimensionstext-embedding-ada-002: 1536 dimensions
Common Issues
"Failed to generate embedding for query"
Your embedding model config is wrong. Check:
- If
litellm_embedding_modelis a model group, the name matches amodel_namein yourmodel_list. A name the proxy does not recognize is sent to the provider as a model name and fails - Otherwise,
litellm_embedding_confighas the right api_base and api_key - Your credentials work
"Index not found"
The vector_store_id doesn't match any index in your search service. Check:
- The index name is correct
- You're using the right search service name
"Field 'contentVector' not found"
Your index uses a different vector field name. Pass it via azure_search_vector_field.