OenoBench

OenoBench puts 3,266 multiple-choice wine questions to 16 configurations of large language models — viticulture, winemaking, the wine business, wine regions, grape varieties, producers — at difficulty tiers calibrated to the WSET / Court of Master Sommeliers ladder. The first release shows a frontier that has nearly finished memorising wine and has not learned to reason about it: the moment a question stops being recall, every configuration tested gives up between 26.6 and 39.6 percentage points of its accuracy.

o3 leads the release at 83.6%, and the order behind it settles less than a ranking implies: six configurations land inside a 2.6-point band at the top. Choosing among them is a question of price, and it is settled in the leaderboard’s cost analysis rather than in this column.

The collapse when recall stops helping is the run’s one result without an exception: the confidence interval excludes zero for all 16 of the 16 configurations. And at the top, the memorising half of the subject is close to exhausted: 5 configurations answer the closed-book questions — the ones a model can meet from memory alone — at 97% or better, and the same configurations manage 65.2% to 70.3% where a question has to be reasoned out from supplied context. For these models the hard part of wine is no longer remembering it.

The wine business is the worst domain for every configuration in the run — 16 of 16, at a mean of 56.5% — against 79.4% on producers, the best domain for 15 of them. The models know who makes wine far better than how wine is sold, which is probably a fair portrait of what there was to read: producers publish about themselves constantly, and the trade publishes its margins almost never. The blind spot, in other words, is the trade.

Of the 4 matched pairs in the run — one base model scored with and without its thinking mode — the interval clears zero for 1 of them: DeepSeek-R1 vs DeepSeek-V3, a lift of 6.8 points from a base running at 70.3%. The other 3 pairs — all frontier — show nothing, and the Claude Opus 4.7 pair lands at −0.1pp exactly. A thinking mode corrects mistakes the base model still makes; at the top of this subject there are too few left to correct. Paying for extended reasoning is a repair, not an upgrade.

The run also measures its own examiners. The questions were drafted by five model families (plus a set of deterministic templates), so each configuration that carries the measurement can be scored on its own family’s questions against everyone else’s — and the gap follows the lab. Every Anthropic configuration scores higher on Anthropic-written questions, Claude Haiku 4.5 by 10.0 points, and Qwen 2.5 7B leans the same way; every Google configuration scores lower on its own, Gemini 2.5 Flash by 10.4 points; OpenAI’s stay flat. In 7 of the 13 measured configurations the interval excludes zero — a model unable to stay neutral about who wrote the exam, in either direction — and that is why no single family writes the questions here.

The release

How to cite

Both forms cite the dated run file, which never moves once published.

@misc{khudov-oenobench-v2026-05-04,
  author  = {Nikita Khudov},
  title   = {OenoBench: a wine-knowledge benchmark for large language models},
  version = {v2026-05-04},
  year    = {2026},
  url     = {https://vinumexmachina.com/oenobench/oenobench-v2026-05-04.json},
  note    = {CC BY-SA 4.0}
}

How the question set is built

What the leaderboard reports

Headline accuracy

One number, and the least interesting one.

1

Per domain

Accuracy within each domain.

6

Per difficulty tier

Accuracy at each rung of the WSET / CMS ladder.

4

Closed-book vs contextual

Is the question answerable from parametric memory, or does it need reasoning over supplied context?

2

Reasoning-mode lift

The same base model scored with and without extended reasoning.

Δ

Self-preference bias

Each evaluator's accuracy on questions generated by its own family against questions from other families. Reported separately, because it is a property of the benchmark's own construction rather than of the model.

Δ