2 posts tagged with "embeddings"

Date: Feb 16, 2026 Duration: ~3 hours Severity: High (for vLLM embedding users) Status: Resolved

Summary​

A commit (dbcae4a) intended to fix OpenAI SDK behavior broke vLLM embeddings by explicitly passing encoding_format=None in API requests. vLLM rejects this with error: "unknown variant \`, expected float or base64"`.

• vLLM embedding calls: Complete failure - all requests rejected
• Other providers: No impact - OpenAI and other providers functioned normally
• Other vLLM functionality: No impact - only embeddings were affected

LiteLLM now supports multimodal embeddings with gemini-embedding-2-preview—generating a single embedding from a mix of text, images, audio, video, and PDF content. Available via both the Gemini API (API key) and Vertex AI (GCP credentials).

Supported Input Types​

Input Formats​

LiteLLM accepts three input formats for multimodal content:

1. Data URIs – Base64-encoded inline: data:image/png;base64,<encoded_data>
2. GCS URLs – Cloud Storage paths (Vertex AI): gs://bucket/path/to/file.png
3. Gemini File References – Pre-uploaded files (Gemini API): files/abc123

Quick Start​

from litellm import embedding
import os

os.environ["GEMINI_API_KEY"] = "your-api-key"

# Text + Image (base64)
response = embedding(
    model="gemini/gemini-embedding-2-preview",
    input=[
        "The food was delicious and the waiter...",
        "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAgAAAAIAQMAAAD+wSzIAAAABlBMVEX///+/v7+jQ3Y5AAAADklEQVQI12P4AIX8EAgALgAD/aNpbtEAAAAASUVORK5CYII"
    ],
)
print(response)

import litellm
from litellm import embedding

litellm.vertex_project = "your-project-id"
litellm.vertex_location = "us-central1"

# Text + Image (GCS URL)
response = embedding(
    model="vertex_ai/gemini-embedding-2-preview",
    input=[
        "Describe this image",
        "gs://my-bucket/images/photo.png"
    ],
)
print(response)

model_list:
  - model_name: gemini-embedding-2-preview
    litellm_params:
      model: gemini/gemini-embedding-2-preview
      api_key: os.environ/GEMINI_API_KEY
  - model_name: vertex-gemini-embedding-2-preview
    litellm_params:
      model: vertex_ai/gemini-embedding-2-preview
      vertex_project: os.environ/VERTEXAI_PROJECT
      vertex_location: os.environ/VERTEXAI_LOCATION

general_settings:
  master_key: sk-1234

litellm --config config.yaml

curl -X POST http://localhost:4000/embeddings \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-embedding-2-preview",
    "input": [
      "The food was delicious and the waiter...",
      "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAgAAAAIAQMAAAD+wSzIAAAABlBMVEX///+/v7+jQ3Y5AAAADklEQVQI12P4AIX8EAgALgAD/aNpbtEAAAAASUVORK5CYII"
    ]
  }'

Input Format Examples​

Supported MIME Types for Data URIs​

• Images: image/png, image/jpeg
• Audio: audio/mpeg, audio/wav
• Video: video/mp4, video/quicktime
• Documents: application/pdf

GCS URL MIME Inference​

For Vertex AI, MIME types are inferred from file extensions:

• .png → image/png
• .jpg / .jpeg → image/jpeg
• .mp3 → audio/mpeg
• .wav → audio/wav
• .mp4 → video/mp4
• .mov → video/quicktime
• .pdf → application/pdf

Optional Parameters​

response = embedding(
    model="gemini/gemini-embedding-2-preview",
    input=["text to embed"],
    dimensions=768,  # Optional: control output vector size
)