Gateway

Overview#

Anar Gateway is the foundation layer of the Anar platform. Every AI call across all 14 products flows through the Gateway for unified routing, authentication, cost tracking, and governance. Written in Go with 18 plugins including semantic caching, it adds less than 100 microseconds of overhead at 5,000 requests per second.

The Gateway exposes an OpenAI-compatible API, meaning any client or SDK that works with OpenAI works with Anar Gateway out of the box. This includes the official OpenAI Python and JavaScript SDKs, LangChain, Haystack, and any other tool that speaks the OpenAI protocol.

Key Capabilities#

Multi-Provider Routing#

Route requests across Azure UAE North (sovereign), Groq (fast inference), and sovereign cloud providers. The Gateway handles model mapping, API key injection, and protocol translation transparently.

Adaptive Load Balancing#

EWMA-based health tracking monitors error rates and latency per provider/model pair. A 4-state machine (Healthy, Degraded, Failed, Recovering) automatically shifts traffic away from unhealthy providers with configurable thresholds and cooldown periods.

Enterprise Security#

• RBAC with API key and JWT authentication, four default roles, and custom role definitions
• OIDC integration with any standards-compliant identity provider (RSA + EC key support, JWKS auto-discovery)
• Guardrails integration with Anar Guard for real-time content safety scanning

Semantic Cache#

Frequently repeated queries are served from a built-in semantic cache, reducing latency and provider costs for common request patterns without any client-side changes.

Cost Tracking#

Per-request cost calculation with built-in pricing for 30+ models across OpenAI, Anthropic, Google, Groq, and Azure. Token usage, costs, and tenant attribution are persisted to SQLite or PostgreSQL with batch writing for minimal performance impact.

Audit Logging#

SOC2/GDPR-grade compliance event logging covering model requests, config changes, key operations, auth failures, and MCP tool executions. Events are batched (100 per flush or every 2 seconds) with automatic retention cleanup.

OpenTelemetry#

Native trace export via HTTP and gRPC protocols, with Prometheus metrics endpoint for scraping. Integrates directly with the Anar observability stack (Grafana, Tempo, Loki, Prometheus).

Architecture#

All plugins hook into the request lifecycle via four plugin interfaces (LLMPlugin, HTTPTransportPlugin, MCPPlugin, ObservabilityPlugin), executing pre-request and post-response logic without modifying the core routing engine.

Available Models#

Azure OpenAI (UAE North)#

Groq (Fast Inference)#

Sovereign Cloud (Fallback)#

Quick Start#

docker compose up -d gateway

Verify the gateway is running:

curl http://localhost:8080/health

Send a chat completion:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.3-70b-versatile",
    "messages": [{"role": "user", "content": "Hello from Anar Gateway"}]
  }'

Integration with Anar Products#

All products route AI calls through the Gateway using the standard OpenAI-compatible API:

Test Suite#

The Gateway has 550 tests covering all 18 custom plugins:

cd gateway/src && go test ./...