Observability

Overview#

All 13 Python products and the Go Gateway are instrumented with OpenTelemetry (OTel). Telemetry data flows through the OTel Collector and into the Grafana LGTM stack: Loki (logs), Grafana (visualization), Tempo (traces), and Mimir/Prometheus (metrics).

Stack Architecture#

Instrumenting a Product#

Every Python product calls setup_otel() from the shared library in its main.py:

from fastapi import FastAPI
from anar_shared import harden, setup_otel

app = FastAPI()
harden(app, product="anar-chat")
setup_otel(app, service_name="anar-chat", version="0.1.0")

What setup_otel Does#

1. Checks for the OTEL_EXPORTER_OTLP_ENDPOINT environment variable. If not set, it returns immediately (no-op), making it safe to run without a collector in development.
2. Creates an OTel Resource with service.name, service.version, and deployment.environment.
3. Configures a TracerProvider with BatchSpanProcessor exporting to {endpoint}/v1/traces.
4. Configures a MeterProvider with PeriodicExportingMetricReader exporting to {endpoint}/v1/metrics every 15 seconds.
5. Instruments the FastAPI application with FastAPIInstrumentor, excluding health and doc endpoints.
6. Optionally instruments SQLAlchemy and httpx if those packages are installed.

Excluded URLs#

The following paths are excluded from tracing to reduce noise:

health, readiness, metrics, openapi.json, docs, redoc

Environment Variables#

Running the Observability Stack#

Start the full LGTM stack with Docker Compose using the infra profile:

docker compose --profile infra up -d

This starts five additional services:

• otel-collector (otel/opentelemetry-collector-contrib:0.120.0)
• grafana (grafana/grafana:11.5.2)
• tempo (grafana/tempo:2.7.1)
• loki (grafana/loki:3.4.2)
• prometheus (prom/prometheus:v3.2.1)

Accessing Grafana#

Open http://localhost:3100 in your browser. Anonymous access is enabled with Viewer role by default. Pre-configured datasources connect to Tempo, Loki, and Prometheus automatically.

OTel Collector Configuration#

The collector receives OTLP data over gRPC and HTTP, batches it, and exports to the appropriate backend:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  resource:
    attributes:
      - key: deployment.environment
        value: local
        action: upsert

exporters:
  otlphttp/tempo:
    endpoint: http://tempo:4318
  otlphttp/loki:
    endpoint: http://loki:3100/otlp
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlphttp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch, resource]
      exporters: [otlphttp/loki]

Tracing Across Products#

When a request flows through multiple products (for example, Chat calls Guard for content scanning, then Gateway for LLM inference), the trace context propagates automatically via HTTP headers. The OTel instrumentation on httpx and FastAPI ensures spans are linked into a single distributed trace.

Viewing Traces#

In Grafana, navigate to Explore > Tempo and search by service name:

{ resource.service.name = "anar-chat" }

To find slow requests:

{ resource.service.name = "anar-chat" } | duration > 2s

Querying Logs#

In Grafana, navigate to Explore > Loki and use LogQL:

{service_name="anar-guard"} |= "policy violation"

Viewing Metrics#

Navigate to Explore > Prometheus for request rate and latency metrics:

rate(http_server_request_duration_seconds_count{service_name="anar-chat"}[5m])

Gateway OTel Integration#

The Go Gateway has its own OTel plugin at gateway/src/plugins/otel/ that instruments request routing with trace context propagation. Set OTEL_EXPORTER_OTLP_ENDPOINT on the Gateway service to enable it:

gateway:
  environment:
    - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
    - OTEL_SERVICE_NAME=anar-gateway

Guard Telemetry#

In addition to OTel, products can send structured telemetry events to Guard for governance monitoring using the GuardTelemetry client:

from anar_shared import GuardTelemetry

telemetry = GuardTelemetry(
    guard_url="http://localhost:8002",
    product="anar-chat",
)
telemetry.start()

telemetry.record(
    model="llama-3.3-70b-versatile",
    latency_ms=245.3,
    prompt_tokens=150,
    completion_tokens=80,
    user_id="user@example.com",
)

Events are buffered locally and flushed in batches every 5 seconds to Guard's /api/v1/telemetry/ingest endpoint.