Skip to main content
A
Docs

Eval

Arabic LLM benchmarking and evaluation platform with multi-provider support and model leaderboards.

Overview

Anar Eval is an Arabic LLM benchmarking and evaluation platform for GCC governments. It evaluates language models on Arabic comprehension, translation accuracy, GCC government knowledge, PII detection, and government service QA — providing objective, reproducible scores across models and providers.

Choosing the right LLM for a government deployment requires more than vendor claims. Eval runs standardized benchmarks against real Arabic content and GCC-specific knowledge, producing scores that enable data-driven model selection.

Key Capabilities

Arabic-Focused Benchmarks

Seven benchmark suites with 105 questions test models on capabilities that matter for GCC government use cases: reading comprehension in MSA, EN-AR translation accuracy, knowledge of GCC government institutions, PII detection in Arabic text, and government service QA.

Multi-Provider Evaluation

Run the same benchmarks against models from different providers to compare performance on identical inputs. Currently supports Groq (fast inference) and Azure UAE North (sovereign cloud), enabling side-by-side comparison of latency, accuracy, and cost.

Model Comparison

Side-by-side model comparison across multiple dimensions — accuracy, latency, cost, and safety scores. Compare any two models on the same benchmarks with visual radar charts and detailed per-question breakdowns.

Safety Evaluation

Dedicated safety evaluation suite that tests models against content safety categories, measuring false positive and false negative rates for safety-critical government deployments.

Benchmark History

Track model performance over time with historical evaluation results. See how accuracy, latency, and safety scores evolve across model versions and provider updates.

Model Leaderboard

A ranked leaderboard aggregates evaluation results across all benchmark suites. Models are scored on overall performance, per-category breakdowns, and average latency — giving decision-makers a single view of model capabilities.

Three Scoring Methods

Eval uses three scoring methods to handle different question types:

MethodHow It WorksBest For
Exact matchReference answer found verbatim in model responseFactual questions with definitive answers
Fuzzy matchToken-level similarity using rapidfuzzParaphrased answers, partial matches
LLM-as-judgeAn LLM evaluates semantic correctness (0-100 scale)Open-ended questions, nuanced answers

Architecture

Eval uses a FastAPI backend on port 8007 with a Next.js dashboard on port 3007. Benchmark definitions are stored as JSON files in the benchmarks/ directory, and evaluation results are persisted in SQLite or PostgreSQL.

The evaluation runner executes in the background: submit a run, and Eval processes each question against the model, scores the responses, and stores the results. Multiple evaluations can run concurrently up to a configurable limit.

Background Execution

Evaluation runs execute asynchronously. Submit a run via POST /api/v1/eval/run and poll the results endpoint to check completion status. This prevents long-running evaluations from blocking the API.

Quick Start

cd eval/backend
uv sync
uv run uvicorn anar_eval.main:app --reload --port 8007

Run an evaluation:

curl -X POST http://localhost:8007/api/v1/eval/run \
  -H "Content-Type: application/json" \
  -d '{
    "model_id": "llama-3.3-70b-versatile",
    "provider": "groq",
    "benchmark_ids": ["arabic-comprehension", "gcc-government-knowledge"],
    "temperature": 0.0
  }'

Dashboard

The Eval dashboard provides seven views:

  • Leaderboard — Ranked model scores with overall and per-category breakdowns
  • Benchmarks — Browse benchmark datasets, view questions and expected answers
  • Run Evaluation — Start new evaluation runs with model and benchmark selection
  • Results — Detailed per-question scores and response analysis
  • Model Comparison — Side-by-side comparison with radar charts
  • Safety — Safety evaluation results with category-level scoring
  • History — Historical benchmark performance trends

Observability

Eval is instrumented with OpenTelemetry via anar_shared.setup_otel(). When OTEL_EXPORTER_OTLP_ENDPOINT is set, traces, logs, and metrics are exported to the OTel Collector.

MCP Server

Eval exposes a Model Context Protocol server at /mcp via FastApiMCP, enabling AI assistants to trigger evaluation runs, query benchmarks, and retrieve leaderboard data programmatically.

Gateway Integration

When GATEWAY_URL is set, Eval routes model evaluation calls through Anar Gateway for unified cost tracking.

Test Suite

223 tests cover benchmarks, evaluation runner, scoring methods, leaderboard, model comparison, safety evaluation, benchmark history, and API endpoints.

cd eval/backend && uv run pytest

Next Steps

API ReferenceBenchmarks