Connect corporate users and applications through VPN, Direct Connect, or an approved private access service. Use private DNS and an internal ALB. Place ECS tasks or EKS workers and data stores in private subnets across two availability zones. NAT gateways serve outbound traffic only.
Allow HTTPS only from corporate source ranges or the private access service. Keep routes to those clients in both directions. See internal load balancer access.
Corporate clients use an internal ALB. Private workloads share database and cache endpoints. NAT serves outbound traffic only.
Architecture details and references
One image
The gateway, backend, UI, and migration job use the same image with a different startup argument for each component.
Required module changes
The module creates an internet-facing ALB. In your fork, set internal=true, select private ALB subnets, update the subnet precondition, and restrict port 443 to corporate source ranges. Remove public HTTP ingress. Add private access routes and private DNS. These are source changes, not module inputs.
Base module resources
The module creates one public and one private subnet for each supplied AZ, and one NAT gateway in the first public subnet. Supply at least two AZs. With an existing VPC, provide the subnets and routes yourself. Tasks, Aurora, and Redis share the private subnet set.
Placement and availability
The gateway defaults to two tasks. The backend and UI each default to one. Workload placements are illustrative. Data icons show shared endpoints, not AZ placement. Both AZs use the Aurora writer endpoint and Redis primary endpoint. Aurora has a reader, and Redis has one replica by default. The diagram omits replication paths. Add per-AZ NAT gateways and service replicas to meet your availability requirements.
Security and egress
Supply acm_certificate_arn for HTTPS. Allow corporate traffic only through the private access path. TLS terminates at the internal ALB. The task security group allows all TCP ports from the ALB security group. The diagram labels the target ports. Aurora allows 5432 and Redis allows 6379 from tasks only. Tasks have no public IP. External providers, public registries, and public AWS service endpoints use NAT. Create DNS records separately. Add WAF or ECR repositories if needed.
Optional PrivateLink
Create Bedrock interface endpoints separately. Use com.amazonaws.<region>.bedrock-runtime, private DNS, and endpoint security group ingress 443 from tasks. Grant Bedrock IAM permissions separately. The bedrock endpoint is for the control plane. To expose LiteLLM through PrivateLink, create a separate endpoint service.
The source module creates an internet-facing ALB and opens ports 80 and 443 to 0.0.0.0/0. It has no input to select private ingress. Apply these changes to the module before deployment:
Set aws_lb.this.internal = true. Set its subnets to private subnet IDs in at least two AZs. Update its precondition to check those private subnets. Keep the ACM certificate and HTTPS listener.
Replace the ALB's public ingress rules with port 443 from approved corporate ranges. Remove port 80 ingress if you do not need an internal redirect. Keep workload ingress limited to the ALB security group. Add routes to your private access network.
Platform network and DNS
Create VPN, Direct Connect, or private access routes. Add a private DNS record for the ALB. Keep NAT in public egress subnets.
The module creates one NAT gateway. Add one per AZ if your availability requirements need it. Use separate security groups for the ALB, tasks, and data stores.
Corporate clients use an internal ALB. Private workloads share database and cache endpoints. NAT serves outbound traffic only.
Architecture details and references
One Helm chart
The litellm chart uses one image for all components. This layout uses the default monolith.enabled=false.
Required infrastructure
Helm deploys the LiteLLM workloads. Create private network access, the VPC, internal ALB, private EKS workers, Aurora, Redis, and per-AZ NAT gateways separately. Configure AWS Load Balancer Controller and Ingress for the selected service layout. Set alb.ingress.kubernetes.io/scheme: internal and select private ALB subnets. Set alb.ingress.kubernetes.io/target-type: ip to target pods directly. Instance mode targets NodePorts. Configure secrets and container log collection separately.
Availability and access
Spread replicas across AZs and set disruption budgets. Both AZs use the writer and primary endpoints by default. The diagram omits replication paths. Choose the database and cache replica layout you need. Allow ALB traffic to workload targets and allow database/cache traffic only from application security groups. Use a private EKS API endpoint.
Optional PrivateLink
The interface endpoints shown provide private Bedrock inference access through com.amazonaws.<region>.bedrock-runtime. Enable private DNS and allow HTTPS from workloads on the endpoint security group. Use pod identity and grant Bedrock permissions. Public providers and registries still need outbound access.
Create the private cluster, data stores, network access, and AWS Load Balancer Controller before you install LiteLLM. Set alb.ingress.kubernetes.io/scheme: internal and select private ALB subnets. Use alb.ingress.kubernetes.io/target-type: ip for direct pod targets. Configure the ALB security group for corporate HTTPS sources. See the chart's ingress settings and EKS load balancing.
