What GDPval is

What GDPval is

❓ What is GDPval, and what real gap in AI evaluation does it fill?

GDPval (GDP-valued evaluation; also written GDPval / GDPVal; sometimes mis-typed “GPPval”) is an OpenAI benchmark that scores AI models on real professional knowledge-work deliverables, not on exam quizzes. The first public version was introduced on 25 September 2025 in an OpenAI research post and detailed in Patwardhan et al., GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks (arXiv:2510.04374, 5 October 2025).

The problem it attacks is simple: classic benchmarks tell you a model can pass tests or fix software tickets. They do not tell you whether a model can produce the kinds of artifacts that already move money in the real economy—legal briefs, nursing care plans, slide decks, engineering drawings, spreadsheets, customer-support packages—when those artifacts are judged by people who do that job for a living.

Definition

Definition: GDPval is a pairwise, expert-graded benchmark of model-written work products against expert-written work products on tasks drawn from high-wage, predominantly digital occupations in the largest U.S. GDP sectors.

Explanation: OpenAI starts from GDP sectors (not from AI hype topics), picks knowledge-heavy occupations that pay a large share of wages in those sectors, asks veterans of those jobs to turn real work into prompts + reference files + gold deliverables, then has other experts blindly rank model vs human deliverables. The headline score is not “percent correct.” It is how often the model’s deliverable is judged better than or as good as the expert’s.

Different from: This is different from MMLU-style exams (synthetic multiple-choice knowledge) and different from single-domain coding benches (SWE-Bench / SWE-Lancer) because the unit is a multi-file professional deliverable across many occupations, graded with professional taste, not only unit tests.

Hard-to-vary test: You cannot swap “exam questions” for “expert deliverables” and keep the same claim about economic relevance, because exam correctness does not encode client-facing structure, aesthetics, file formats, or occupational standards. You cannot swap “automated pass/fail” for “expert pairwise preference” without losing the subjective quality bar real managers use.

Refutability: The evaluation would be undermined if independent occupational experts systematically reverse the pairwise rankings, if tasks are not representative of the claimed O*NET work activities, or if one-shot scoped prompts turn out to share almost no variance with real job performance.

Reach example: The same design—GDP-weighted occupations × expert gold deliverables × pairwise win rate—can be reused inside a bank or hospital on internal work products to build a private “mini-GDPval.”

Criticism note: The weakest point is not the pairwise idea; it is coverage and ecology. v1 is one-shot, well-scoped, and digital-task heavy. Real jobs include ambiguity, politics, multi-week iteration, and physical work GDPval deliberately excludes.

Why “GDP” is in the name

OpenAI explicitly started from Gross Domestic Product as an economic anchor: pick industries that contribute most to U.S. GDP, then occupations that dominate wages inside those industries and are mostly knowledge work. The point is not that the score is GDP. The point is that the task sample is aimed at work that already carries large economic weight, so capability gains are harder to dismiss as toy-problem progress.

Paper estimate: the 9 sectors / 44 occupations in scope collectively earn on the order of $3 trillion annually in wages/compensation terms used by the authors—large enough that even modest task-level productivity shifts matter.

Place in the evaluation ladder

OpenAI positions GDPval as a step after academic and domain benches:

1. Academic multi-subject exams (e.g., MMLU)
2. Applied engineering / science benches (SWE-Bench, MLE-Bench, PaperBench)
3. Market-priced work (SWE-Lancer)
4. Cross-occupation economic deliverables (GDPval)

Core mechanism (one paragraph)

The hard-to-vary causal chain is: economic sector weight → high-wage digital occupations → expert-authored real work products → multi-modal model attempts → blinded expert pairwise comparison → win rate over time, then (separately) win rate + human review time + redo probability → expected time/cost with oversight. Quality and economics are linked by the win rate w, not by raw token speed alone.

What is open

• Gold set: 220 tasks (5 per occupation × 44), prompts + reference files on Hugging Face (openai/gdpval)
• Full set: 1,320 tasks (30 per occupation) — used internally / broader analysis
• Public automated grader service: experimental, at evals.openai.com
• Primary metric for science claims: human expert pairwise grades, not the auto-grader alone

A concrete seed: an industry expert with ~14 years’ experience spends on the order of seven hours producing a gold deliverable (design package, brief, plan). A frontier model returns a candidate in minutes. Another expert spends roughly an hour-plus deciding which package they would rather ship. That triangle—create, generate, judge—is the entire GDPval theater.