Compares model capabilities across benchmark tasks, alongside API costs, speed and context limits.
Evaluates
Specific model versions and reasoning settings
Method
The Intelligence Index combines results across task categories; costs and runtime efficiency are reported separately.
Comparison scope
The index uses English tasks, mostly text with some image inputs. Match index versions and reasoning settings. Image and video generation quality have separate evaluations.
The original HLE tests expert knowledge and reasoning with 2,500 questions across disciplines. HLE-Rolling revises the question set over time. HLE-Diamond contains 1,000 carefully reviewed questions.
Evaluates
Model answers to academic questions with text and images
Method
Responses are graded against reference answers. Original HLE reports accuracy, confidence intervals and calibration error for models’ stated confidence.
Comparison scope
Compare results on the same question set, matching text-only or multimodal scope, tool access, graders and reasoning settings.
Tests agents on complex tasks in isolated terminal environments, identifying both the model and agent harness.
Evaluates
Model and terminal-agent combinations
Method
Uses task verifiers to measure completion, alongside costs, resource usage and confidence intervals.
Comparison scope
The standalone leaderboard uses 4.1. The coding component of AA Intelligence Index v5.0.1 uses 4.0. Within a version, match harnesses, resource limits and trial counts.
Evaluates long workflows across desktop apps and browsers, distinguishing full completion from partial credit.
Evaluates
Computer-use agents and their interaction settings
Method
Checks task state after execution and reports completion, steps, elapsed time and costs.
Comparison scope
Model coverage remains limited. Compare within the same version and full or offline task set. Match interaction settings and step budgets, and distinguish completion from partial credit.