Benchmarks

Benchmark Suites#

Eval ships with seven benchmark suites containing 105 questions designed for GCC government AI evaluation. Each suite tests a specific capability with questions relevant to Arabic language processing and government operations.

curl http://localhost:8007/api/v1/benchmarks

Arabic Comprehension#

Tests a model's ability to read and understand Modern Standard Arabic text. Questions present Arabic passages followed by comprehension questions that require understanding vocabulary, grammar, and context.

Example question:

{
  "question": "اقرأ النص التالي وأجب: أعلنت حكومة أبوظبي عن إطلاق برنامج جديد لتطوير الذكاء الاصطناعي...\nما هو البرنامج الذي أعلنت عنه حكومة أبوظبي؟",
  "reference_answer": "برنامج جديد لتطوير الذكاء الاصطناعي",
  "scoring": "fuzzy_match"
}

This benchmark uses fuzzy matching since correct answers may be paraphrased differently across models while retaining the same meaning.

Arabic Translation#

Evaluates bidirectional translation between Arabic and English. Questions include both EN-to-AR and AR-to-EN translation tasks covering formal government terminology, legal language, and everyday citizen communications.

Example question:

{
  "question": "Translate to Arabic: The Ministry of Interior has announced new visa regulations for 2024.",
  "reference_answer": "أعلنت وزارة الداخلية عن لوائح تأشيرات جديدة لعام 2024",
  "scoring": "llm_judge"
}

Translation questions use LLM-as-judge scoring because valid translations vary widely in word choice and sentence structure while conveying the same meaning.

GCC Government Knowledge#

Tests domain knowledge about GCC government structures, policies, and institutions. Models must demonstrate understanding of UAE, Saudi, Qatari, and Omani government organizations, their functions, and recent policy initiatives.

Example question:

{
  "question": "What is the role of SDAIA in Saudi Arabia's AI strategy?",
  "reference_answer": "SDAIA (Saudi Data and AI Authority) leads the national data and AI agenda, developing policies, regulations, and initiatives to position Saudi Arabia as a global leader in AI.",
  "scoring": "llm_judge"
}

Arabic PII Detection#

Evaluates a model's ability to identify personally identifiable information in Arabic text. Questions present Arabic text containing embedded PII — Emirates IDs, Saudi national IDs, phone numbers, IBANs — and ask the model to identify and classify them.

Example question:

{
  "question": "Identify all PII in the following Arabic text: اسمي أحمد محمد ورقم هويتي الإماراتية 784-1990-1234567-1 ورقم هاتفي 0501234567",
  "reference_answer": "Name: أحمد محمد, Emirates ID: 784-1990-1234567-1, Phone: 0501234567",
  "scoring": "fuzzy_match"
}

Government Service QA#

Tests a model's ability to answer citizen questions about government services. Questions cover common topics like visa applications, business licensing, and public service inquiries.

Example question:

{
  "question": "What documents are required to renew a UAE trade license?",
  "reference_answer": "To renew a UAE trade license, you typically need: current trade license copy, tenancy contract, MOA/AOA, passport copies of partners, Emirates ID copies, and NOC from sponsor if applicable.",
  "scoring": "llm_judge"
}

Benchmark JSON Format#

Each benchmark is stored as a JSON file in the benchmarks/ directory:

{
  "id": "arabic-comprehension",
  "name": "Arabic Comprehension",
  "category": "Language",
  "description": "Reading comprehension in Modern Standard Arabic",
  "questions": [
    {
      "id": "q-001",
      "question": "...",
      "reference_answer": "...",
      "scoring": "fuzzy_match",
      "metadata": {
        "difficulty": "medium",
        "topic": "government_policy"
      }
    }
  ]
}

Scoring Methods in Detail#

Exact Match#

The reference answer must appear verbatim in the model's response. Case-insensitive. Used for questions with definitive, short answers.

Fuzzy Match#

Uses rapidfuzz for token-level similarity comparison. The score is the percentage similarity between the reference answer tokens and the model response tokens. A threshold of 70% is considered a pass.

LLM-as-Judge#

A separate LLM (default: llama-3.3-70b-versatile via Groq) evaluates the model's response against the reference answer and assigns a score from 0-100. The judge prompt includes the question, reference answer, and model response, asking for a semantic correctness assessment.

Score 0-100: How semantically correct is the model's response compared to the reference answer?
Consider: factual accuracy, completeness, relevance, and language quality.