API Reference

Base URL#

http://localhost:8007/api/v1

Endpoints#

GET /api/v1/benchmarks#

List all available benchmarks with their categories, question counts, and descriptions.

curl http://localhost:8007/api/v1/benchmarks

GET /api/v1/benchmarks/{id}#

Get full benchmark details including all questions, reference answers, and scoring methods.

curl http://localhost:8007/api/v1/benchmarks/arabic-comprehension

POST /api/v1/eval/run#

Start an evaluation run. The evaluation executes in the background — the response returns immediately with evaluation IDs.

Request:

{
  "model_id": "llama-3.3-70b-versatile",
  "provider": "groq",
  "benchmark_ids": ["arabic-comprehension", "gcc-government-knowledge"],
  "temperature": 0.0
}

Response:

{
  "eval_ids": ["eval-a1b2", "eval-c3d4"],
  "message": "Started 2 evaluation(s)"
}

GET /api/v1/results#

List all evaluation results across all models and benchmarks.

curl http://localhost:8007/api/v1/results

Returns an array of evaluation result summaries including model, provider, benchmark, overall score, and status (running, completed, failed).

GET /api/v1/results/{id}#

Get detailed results for a specific evaluation, including per-question scores, model responses, and latency measurements.

curl http://localhost:8007/api/v1/results/eval-a1b2

The detailed result includes:

• Overall benchmark score
• Per-question breakdown with model response, reference answer, score, and scoring method
• Aggregate statistics: average latency, total tokens, pass rate

GET /api/v1/leaderboard#

Returns the ranked model leaderboard aggregating all completed evaluations.

curl http://localhost:8007/api/v1/leaderboard

Response:

[
  {
    "rank": 1,
    "model_id": "llama-3.3-70b-versatile",
    "provider": "groq",
    "overall_score": 85.2,
    "benchmark_scores": {
      "arabic-comprehension": 90.0,
      "gcc-government-knowledge": 80.0
    },
    "category_scores": {
      "Language": 90.0,
      "Domain Knowledge": 80.0
    },
    "total_evaluations": 2,
    "avg_latency_ms": 1200,
    "last_evaluated": "2024-01-15T10:30:00Z"
  }
]

Models are ranked by overall_score in descending order. Only models with at least one completed evaluation appear on the leaderboard.

Supported Providers#

Environment Variables#

MCP Server#

Eval exposes a Model Context Protocol server at /mcp, enabling integration with AI assistants and development tools that support the MCP specification.