Custom Evaluations

Creating Custom Benchmarks#

The built-in benchmark suites cover common Arabic AI capabilities, but every government organization has unique requirements. Custom benchmarks let you evaluate models against your specific use cases — whether that is legal document analysis, healthcare terminology, or departmental knowledge.

Benchmark File Format#

Custom benchmarks follow the same JSON schema as built-in benchmarks. Create a JSON file and place it in the benchmarks/ directory:

{
  "id": "custom-legal-arabic",
  "name": "Arabic Legal Document Analysis",
  "category": "Domain Knowledge",
  "description": "Evaluates model comprehension of GCC legal terminology and contract analysis",
  "questions": [
    {
      "id": "legal-001",
      "question": "ما هي الآثار القانونية لبند التحكيم التالي في عقد إماراتي: 'يحال أي نزاع إلى مركز أبوظبي للتحكيم'؟",
      "reference_answer": "This clause subjects disputes to arbitration at the Abu Dhabi Arbitration Centre (formerly Abu Dhabi Commercial Conciliation and Arbitration Centre) under UAE Federal Arbitration Law No. 6 of 2018, waiving the right to litigation in ordinary courts.",
      "scoring": "llm_judge",
      "metadata": {
        "difficulty": "hard",
        "topic": "arbitration_law"
      }
    },
    {
      "id": "legal-002",
      "question": "Define 'force majeure' as applied in GCC contract law and give an example.",
      "reference_answer": "Force majeure in GCC contract law refers to an unforeseeable, unavoidable event beyond the parties' control that prevents contractual performance. Examples include natural disasters, government actions, or pandemics. UAE Civil Code Articles 273 and 287 address force majeure.",
      "scoring": "llm_judge",
      "metadata": {
        "difficulty": "medium",
        "topic": "contract_law"
      }
    }
  ]
}

Required Fields#

Question Fields#

Choosing a Scoring Method#

Running Custom Benchmarks#

Once your benchmark JSON is in the benchmarks/ directory, it appears in the benchmark listing and can be included in evaluation runs:

# Verify the benchmark loaded
curl http://localhost:8007/api/v1/benchmarks/custom-legal-arabic

# Run it against a model
curl -X POST http://localhost:8007/api/v1/eval/run \
  -H "Content-Type: application/json" \
  -d '{
    "model_id": "llama-3.3-70b-versatile",
    "provider": "groq",
    "benchmark_ids": ["custom-legal-arabic"],
    "temperature": 0.0
  }'

Custom benchmark results appear in the standard results listing and contribute to leaderboard scores under their specified category.

Designing Effective Benchmarks#

Question Quality#

• Write questions that have clear, defensible correct answers
• Include both Arabic and English questions to test bilingual capability
• Vary difficulty levels to distinguish between model tiers
• Cover edge cases relevant to your domain

Reference Answers#

• Write comprehensive reference answers that cover all key points
• For llm_judge scoring, the reference answer guides the judge — more detail produces better scoring
• Include Arabic text in reference answers for Arabic questions

Sample Size#

• Aim for at least 10 questions per benchmark for statistically meaningful results
• Group related questions under one benchmark rather than creating many small benchmarks
• Include questions of varying difficulty (easy, medium, hard) to spread scores

Example: Healthcare Benchmark#

{
  "id": "healthcare-arabic",
  "name": "Arabic Healthcare Terminology",
  "category": "Domain Knowledge",
  "description": "Medical terminology and healthcare policy knowledge in Arabic",
  "questions": [
    {
      "id": "health-001",
      "question": "ما هي متطلبات ترخيص المنشأة الصحية في أبوظبي حسب هيئة الصحة؟",
      "reference_answer": "Requirements include: facility layout approval, equipment certification, qualified medical staff with DOH-verified credentials, medical waste management plan, infection control protocols, and civil defense clearance.",
      "scoring": "llm_judge",
      "metadata": {"difficulty": "medium", "topic": "healthcare_licensing"}
    }
  ]
}

Environment Variables#