OenoBench Leaderboard

Step plot. The corpus is ordered by the closed-book split: the first 1,601 questions are answerable from parametric knowledge alone, the remaining 1,665 require contextual reasoning. The pen holds 0.974 in the first regime and 0.703 in the second — a drop of 0.271 for o3 — inside a ±1 binomial standard-error band of 0.0159 and 0.0457 at a reference window of 100. The dashed rules are the unweighted mean over all configurations (16): 0.896 and 0.570, a drop of 0.326. Accuracy runs from 0.30 to 1.00. The trace is a step, not a rolling curve: the run file publishes regime aggregates rather than per-question results.

Dumbbell chart. The configurations (16) are ranked first to last by overall accuracy, one row each. A rule joins the row's accuracy on questions that require contextual reasoning (n 1,665) to its accuracy on closed-book questions (n 1,601), on a scale from 0.30 to 1.00, with the drop between the two printed at the right of the row. The smallest drop is GPT-5 at 0.266; the largest is DeepSeek-V3 at 0.396. The overall leader is o3.