Skip to main content
A
Docs

Leaderboard

Ranked model comparison with overall scores, per-category breakdowns, and latency metrics.

Model Leaderboard

The leaderboard aggregates all completed evaluations into a ranked view of model performance. Each model is scored on overall accuracy, per-category performance, and average latency — providing a single source of truth for model selection decisions.

curl http://localhost:8007/api/v1/leaderboard

Response:

[
  {
    "rank": 1,
    "model_id": "llama-3.3-70b-versatile",
    "provider": "groq",
    "overall_score": 85.2,
    "benchmark_scores": {
      "arabic-comprehension": 90.0,
      "gcc-government-knowledge": 80.0,
      "arabic-translation": 82.5,
      "arabic-pii-detection": 88.0,
      "government-service-qa": 85.5
    },
    "category_scores": {
      "Language": 90.0,
      "Domain Knowledge": 82.75,
      "Translation": 82.5,
      "Safety": 88.0
    },
    "total_evaluations": 5,
    "avg_latency_ms": 1200,
    "last_evaluated": "2024-01-15T10:30:00Z"
  }
]

Leaderboard Fields

FieldDescription
rankPosition based on overall score
model_idModel identifier as specified by the provider
providerInference provider (groq, azure)
overall_scoreWeighted average across all completed benchmarks (0-100)
benchmark_scoresPer-benchmark scores
category_scoresAggregated scores by category (Language, Domain, Translation, Safety)
total_evaluationsNumber of completed evaluation runs
avg_latency_msAverage response latency across all evaluations
last_evaluatedTimestamp of the most recent evaluation

Score Categories

Benchmarks are grouped into categories for higher-level comparison:

CategoryBenchmarksWhat It Measures
LanguageArabic ComprehensionMSA reading comprehension and linguistic understanding
TranslationArabic TranslationBidirectional EN-AR translation quality
Domain KnowledgeGCC Government Knowledge, Government Service QAUnderstanding of GCC government operations
SafetyArabic PII DetectionAbility to identify and classify PII in Arabic text

A model's category score is the average of its benchmark scores within that category.

Comparing Models

The leaderboard supports comparing models across multiple dimensions:

By Overall Score

The default ranking sorts by overall score, giving a quick answer to "which model performs best across all Arabic capabilities."

By Category

Category scores reveal model strengths and weaknesses. A model might rank first in Translation but third in Safety — critical information for choosing a model for a specific use case.

By Latency

For real-time applications like chatbots and voice agents, latency matters as much as accuracy. The avg_latency_ms field helps identify models that balance quality with speed.

Provider Comparison

Run the same model through different providers (e.g., Groq vs. Azure) to compare inference latency and any accuracy differences due to quantization or serving infrastructure.

Dashboard Visualization

The Eval dashboard renders the leaderboard with:

  • Ranked table — Sortable by any column, with color-coded scores (green for high, yellow for medium, red for low)
  • Radar charts — Per-model radar charts showing category score distribution, making it easy to visually compare model profiles
  • Latency bars — Horizontal bars showing average response time for each model
  • Trend indicators — Score changes since the model's previous evaluation

Radar Chart

Each model's category scores are plotted on a radar chart with axes for Language, Translation, Domain Knowledge, and Safety. This visualization immediately surfaces models that are well-rounded versus those with significant category gaps.

Keeping the Leaderboard Current

The leaderboard updates automatically as new evaluation runs complete. To ensure models are compared fairly:

  • Run all benchmarks against each model (not just a subset)
  • Use the same temperature setting (0.0 recommended for reproducibility)
  • Re-evaluate models periodically as providers update model versions
  • Compare models on the same provider when latency is a deciding factor
curl -X POST http://localhost:8007/api/v1/eval/run \
  -H "Content-Type: application/json" \
  -d '{
    "model_id": "allam-2-7b",
    "provider": "groq",
    "benchmark_ids": [
      "arabic-comprehension",
      "arabic-translation",
      "gcc-government-knowledge",
      "arabic-pii-detection",
      "government-service-qa"
    ],
    "temperature": 0.0
  }'
BenchmarksCustom Evaluations