AI benchmark scores: compare the test and conditions before the model

Start with the historical episode, not a current leaderboard claim

The 4 February 2025 source provides chapter names but no numerical results or detailed verdict. It moves through AIME 2024, Codeforces, GPQA Diamond, MATH-500, MMLU-Pro and SWE-bench Verified before DeepSeek R1 versus OpenAI o1 at 5:37. The definitions below are a separately sourced reading aid, not a reconstruction of Luke’s unseen scores.

1. Record the benchmark name, model version, metric and evaluation date before comparing two reported results.
2. Keep missing values and unspecified conditions visible rather than infer a winner from a model’s reputation.

Distinguish mathematics, science and broader knowledge

AIME is a mathematics competition; MATH-500 uses 500 held-out MATH problems. Primary references: https://maa.org/wp-content/uploads/2025/08/Final-MAA-Impact-Report-2024.pdf and https://github.com/openai/prm800k.

GPQA Diamond is a curated science-question subset, while MMLU-Pro extends multi-subject evaluation with harder reasoning questions and ten answer options. References: https://arxiv.org/html/2311.12022 and https://arxiv.org/abs/2406.01574.

1. Use the episode at 0:24, 1:54, 2:40 and 3:05 to keep those different tasks distinct.
2. Ask whether a result represents the sort of question your own application needs to answer before treating it as evidence of general usefulness.

Separate competitive coding from repository repair

Codeforces centres on programming contests and ratings; SWE-bench Verified is a human-filtered set of 500 software issues. Primary references: https://codeforces.com/help and https://www.swebench.com/.

1. Follow Codeforces at 1:03 and SWE-bench Verified at 4:29, checking whether the reported metric is a rating, percentage or resolved-task measure.
2. Do not read a contest rating as a percentage of production bugs fixed, or assume repository-task performance measures every kind of coding work.

Read the evaluation conditions and test your actual job

DeepSeek’s official R1 evaluation labels metrics separately and describes sampling conditions: https://github.com/deepseek-ai/DeepSeek-R1/blob/main/README.md. That is a provider report, not an independently reproduced result here. Differences in prompts, tools and permitted attempts need to remain part of a comparison rather than disappear behind one large number.

1. Compare the same benchmark version and metric under stated conditions, keeping provider-reported and independently checked results distinguishable.
2. Then try a bounded real task with a known result and record quality, errors and effort separately from the benchmark score.

The takeaway

Benchmarks are useful evidence about defined tests. They do not supply a timeless winner for every task or a substitute for checking your own use case.