Routing

How Routing Works#

When a request arrives at the Gateway, the routing engine determines which provider and model to use. The process follows this order:

1. Model resolution — The requested model name is matched to a configured provider
2. Routing rules — CEL-based rules are evaluated in scope precedence order
3. Health check — The health tracker adjusts traffic weights based on provider health
4. Key selection — A key is selected based on configured weights and model access
5. Fallback chain — If the primary provider fails, fallbacks are attempted in order

Provider Priority#

The Gateway uses a tiered provider strategy for sovereign deployments:

Requests are routed to the provider that owns the requested model. If a model exists on multiple providers (e.g., gpt-4o on both Azure and sovereign cloud), routing rules and key weights determine the selection.

Model Routing#

Models are addressed by their name as configured in the provider's keys[].models array. When multiple providers host the same model name, prefix with the provider key to disambiguate:

# Route to Groq explicitly
curl http://localhost:8080/v1/chat/completions \
  -d '{"model": "groq/llama-3.3-70b-versatile", "messages": [...]}'

# Route to sovereign cloud explicitly
curl http://localhost:8080/v1/chat/completions \
  -d '{"model": "sovereign/gpt-4o", "messages": [...]}'

# Let the Gateway pick the best provider
curl http://localhost:8080/v1/chat/completions \
  -d '{"model": "llama-3.3-70b-versatile", "messages": [...]}'

CEL-Based Routing Rules#

The routing engine supports CEL (Common Expression Language) expressions for dynamic routing decisions. Rules are evaluated in scope precedence order with first-match-wins semantics.

Scope Precedence#

Rules are evaluated from the most specific scope to the least specific:

1. Virtual Key — Rules scoped to a specific API key
2. Team — Rules scoped to a team
3. Customer — Rules scoped to a customer/organization
4. Global — Rules that apply to all requests

The first rule that matches short-circuits evaluation. If no rules match, default routing applies.

Available Variables#

CEL expressions have access to the following variables:

Example Rules#

Route embedding requests to Azure:

request_type == "embedding"

With target provider azure and model text-embedding-3-large-uaenorth.

Route Arabic models to Groq:

model == "allam-2-7b" || model.contains("arabic")

Failover when budget exceeds 80%:

budget_used > 80.0 && provider == "azure"

With fallback chain ["groq/llama-3.3-70b-versatile", "sovereign/gpt-4o"].

Route by header (tenant isolation):

headers["x-tenant-id"] == "ministry-of-finance"

Route by team with request type filter:

team_name == "security-team" && request_type == "chat_completion"

Supported CEL Operators#

Adaptive Load Balancing#

The health tracker uses EWMA (Exponentially Weighted Moving Average) to monitor error rates and latency per provider/model pair. It implements a 4-state machine that automatically adjusts traffic distribution.

Health States#

State Transitions#

Healthy ──→ Degraded ──→ Failed ──→ Recovering ──→ Healthy
              ↑                         │
              └─────────────────────────┘

EWMA Parameters#

Fallback Strategies#

Per-Key Retries#

Each provider's network_config defines retry behavior for transient failures (timeouts, 5xx responses):

{
  "network_config": {
    "max_retries": 3,
    "retry_backoff_initial_ms": 100,
    "retry_backoff_max_ms": 5000
  }
}

Retries use exponential backoff capped at retry_backoff_max_ms.

Cross-Provider Fallback#

Routing rules can define fallback chains that span multiple providers:

{
  "provider": "azure",
  "model": "gpt-4o-uaenorth",
  "fallbacks": ["groq/llama-3.3-70b-versatile", "sovereign/gpt-4o"]
}

If the primary provider fails after exhausting retries, the Gateway attempts each fallback in order. Fallback providers are subject to the same health tracking — a failed provider is skipped in the fallback chain.

Key Weight Distribution#

When a provider has multiple API keys, traffic is distributed by weight:

{
  "keys": [
    { "name": "key-a", "value": "env.KEY_A", "models": ["gpt-4o"], "weight": 0.7 },
    { "name": "key-b", "value": "env.KEY_B", "models": ["gpt-4o"], "weight": 0.3 }
  ]
}

Key A receives approximately 70% of traffic and Key B receives 30%. This is useful for distributing load across multiple API accounts with different rate limits.

Governance Integration#

The routing engine integrates with the Gateway governance layer for hierarchical budget and rate limit enforcement:

• Customer-level budgets — Monthly spend caps per organization
• Team-level budgets — Departmental spending limits
• Virtual key budgets — Per-key token and cost limits
• Rate limits — Requests per minute and tokens per minute, per scope

When a budget or rate limit is exceeded, the Gateway returns a 429 Too Many Requests response. Routing rules can use budget_used and tokens_used variables to proactively shift traffic before limits are hit.

Request Flow Diagram#

  Client Request
       │
       ▼
  ┌─────────────┐
  │  RBAC/OIDC  │ ← Authenticate & authorize
  └──────┬──────┘
         │
  ┌──────▼──────┐
  │  Guardrails │ ← Content safety scan (pre)
  └──────┬──────┘
         │
  ┌──────▼──────┐
  │   Routing   │ ← CEL rules + health weights
  └──────┬──────┘
         │
  ┌──────▼──────┐
  │   Provider  │ ← Upstream API call
  └──────┬──────┘
         │
  ┌──────▼──────┐
  │ Cost Track  │ ← Token + cost recording
  └──────┬──────┘
         │
  ┌──────▼──────┐
  │    Audit    │ ← Event logging
  └──────┬──────┘
         │
       Client Response