Gateway Routing

Overview#

Anar Gateway is the single entry point for all AI model calls across the platform. Written in Go, it routes OpenAI-compatible requests to the appropriate provider based on model name, provider health, and configured priority. Every product connects through the Gateway rather than calling providers directly.

Architecture#

The Gateway exposes an OpenAI-compatible /v1/chat/completions endpoint. Products send standard chat completion requests, and the Gateway handles provider selection, authentication, retries, and load balancing.

Provider Configuration#

The Gateway supports three provider tiers:

Environment Variables#

Configure provider credentials on the Gateway service:

GROQ_API_KEY=gsk_...
AZURE_OPENAI_API_KEY=your-azure-key
AZURE_OPENAI_ENDPOINT=https://your-endpoint.cognitiveservices.azure.com
CORE42_API_KEY=your-core42-key

Model Routing#

The Gateway automatically routes requests to the correct provider based on the model name. The shared library maintains a MODEL_PROVIDER_MAP that determines which provider handles each model:

MODEL_PROVIDER_MAP = {
    "llama-3.3-70b-versatile": "groq",
    "allam-2-7b": "groq",
    "meta-llama/llama-guard-4-12b": "groq",
    "whisper-large-v3": "groq",
    "qwen/qwen3-32b": "groq",
    "gpt-4o-uaenorth": "azure",
    "gpt-4o": "core42",
    "gpt-4.1": "core42",
}

When a product sends a request with model: "llama-3.3-70b-versatile", the Gateway routes it to Groq. A request with model: "gpt-4o-uaenorth" goes to Azure UAE North.

Connecting Products to the Gateway#

Every product backend accepts GATEWAY_URL and GATEWAY_API_KEY environment variables. When set, all LLM calls route through the Gateway instead of hitting providers directly:

GATEWAY_URL=http://gateway:8080/v1
GATEWAY_API_KEY=your-gateway-key

In code, use get_llm_for_model from the shared library to automatically route through the Gateway when configured:

from anar_shared import LLMConfig, get_llm_for_model

config = LLMConfig(
    gateway_url="http://gateway:8080/v1",
    gateway_api_key="your-key",
    default_model="llama-3.3-70b-versatile",
)

llm = get_llm_for_model("llama-3.3-70b-versatile", config)
response = await llm.chat([
    {"role": "user", "content": "Summarize this document"}
])

When gateway_url is set, the get_llm_for_model function creates a GatewayLLM instance that routes all requests through the Gateway. When it is empty, requests go directly to the provider.

Fallback Chains#

The Gateway supports provider fallback. If the primary provider fails (timeout, rate limit, or 5xx error), the request is retried against the next provider in the chain.

Priority Order#

1. Groq -- default for development and testing (fast inference, no data residency)
2. Azure UAE North -- sovereign production workloads (UAE data residency)
3. Sovereign cloud -- last-resort fallback

Health Tracking#

The Gateway includes a health tracker plugin that monitors provider response times and error rates. Providers with elevated error rates are temporarily deprioritized, and requests are routed to healthier alternatives.

Gateway Plugins#

The Gateway ships with the following plugins:

Making Direct Gateway Calls#

The Gateway accepts standard OpenAI-compatible requests:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-gateway-key" \
  -d '{
    "model": "llama-3.3-70b-versatile",
    "messages": [
      {"role": "user", "content": "What is the capital of UAE?"}
    ],
    "temperature": 0.7,
    "max_tokens": 1024
  }'

Streaming is also supported:

curl -X POST http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-gateway-key" \
  -d '{
    "model": "llama-3.3-70b-versatile",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'

Docker Compose Configuration#

In Docker Compose, the Gateway is the first service to start and all product backends depend on it:

gateway:
  build: ./gateway
  ports:
    - "8080:8080"
  environment:
    - GROQ_API_KEY=${GROQ_API_KEY:-}
    - AZURE_OPENAI_API_KEY=${AZURE_OPENAI_API_KEY:-}
    - CORE42_API_KEY=${CORE42_API_KEY:-}
    - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
  healthcheck:
    test: ["CMD", "wget", "--spider", "http://localhost:8080/health"]
    interval: 30s
    timeout: 10s
    retries: 3

Products declare a dependency on the Gateway with a health condition:

chat-backend:
  depends_on:
    gateway:
      condition: service_healthy
  environment:
    - GATEWAY_URL=http://gateway:8080/v1