Your text-to-SQL model isn't as wrong as your benchmark says. The gold SQL is.

You run BIRD-dev, read an execution accuracy of 0.512, and the instinct is immediate: start writing planner directives to close the gap. We had the same instinct. Before acting on it we did one thing that changed the whole plan — we bucketed the losses. Not skimmed a few failures; tagged all 238 mismatches with a structural differ and counted what actually went wrong in each.

19% of our losses were one DISTINCT — added correctly

The biggest bucket was startling: 46 of 238 mismatches (19%) differ from the gold SQL only by a DISTINCT the model added and gold didn't. COUNT(DISTINCT customer_id) where gold wrote COUNT(*). SELECT DISTINCT x where gold wrote a plain SELECT x. Read the pairs one by one and a large share of them are the model being more correct than the annotation.

-- Question: how many patients had an abnormal lab result?
-- Gold (BIRD annotation):
SELECT COUNT(T1.ID)
FROM Patient AS T1
JOIN Laboratory AS T2 ON T1.ID = T2.ID
WHERE T2.result = 'abnormal';
-- one patient with 5 abnormal labs counts 5 times

-- Model:
SELECT COUNT(DISTINCT T1.ID)
FROM Patient AS T1
JOIN Laboratory AS T2 ON T1.ID = T2.ID
WHERE T2.result = 'abnormal';
-- counts patients — the thing the question asked for

A patient-to-labs join is one-to-many. Counting patients after that join needs COUNT(DISTINCT T1.ID); gold's bare COUNT(T1.ID) over-counts the fan-out. Execution accuracy compares result sets, gold's result set is wrong, so the scorer marks the model wrong for being right.

This is not one benchmark having a bad day

Independent measurement backs the pattern. The Kang lab at UIUC audited BIRD and found 52.8% of instances carry annotation errors (VLDB 2026, arXiv:2601.08778) — wrong gold SQL, ambiguous questions, schema mismatches — and released a corrected evaluation set. When half the answer key has errors, the number the leaderboard prints is not a measurement of your engine. It's a measurement of your engine and the answer key's noise, entangled.

The trap: optimizing into the noise

Here is why this matters beyond bruised pride. If you chase the biggest loss bucket with a prompt directive — "avoid DISTINCT unless explicitly requested" — the benchmark number goes up. You will feel good about it. And you will have taught your model to drop a DISTINCT it should keep, silently degrading real-world answers over one-to-many joins, which are everywhere. That is overfitting to wrong gold: trading production correctness for benchmark points.

The fix: classify your loss mass before touching the prompt

The differ that buckets losses is about a hundred lines and needs no LLM — parse both queries, diff the structure, tag the difference class (extra_DISTINCT, extra column, wrong aggregate, wrong filter…), histogram the tags. Then apply one rule: only write a directive for a bucket where gold is right and the model is wrong.

The same audit that disqualified the DISTINCT bucket qualified another: 7 losses where the model concatenated two requested columns into one (first_name || ' ' || last_name) against a gold that correctly returned two columns — 7 clean losses, zero gold-noise, and no winning query used the pattern. That bucket earned a directive, and re-scoring the de-concatenated predictions against the real databases confirmed it: wrong-to-right flips, zero regressions.

The rule

A benchmark number is a floor bounded by its gold quality, not a measure of your engine. Before you optimize any metric, audit what it's actually counting — because past a point, the returns on prompt engineering aren't limited by your model. They're limited by the answer key.

(This audit is standing practice for nlqdb, the data layer you ask in English: every eval run's losses are bucketed structurally before any planner change ships, so the directives we write target real engine mistakes — not annotation noise.)