TreasureX Researchq-fin.GNFLARE Scores May Measure the Grader, Not the Model
Dung Cao, Palaash Gang, Arjun Veluri · October 24, 2026
A new audit of FLARE — the financial benchmark suite used by PIXIU and FinBen to rank language models — suggests many published scores reflect how answers are extracted, not how good the model is. The authors show that FLARE's own code and the evaluation harness it borrows from disagree on which answer to credit: one picks the first listed option that appears anywhere in the output, the other picks whichever appears earliest. On identical model outputs, the two rules can differ by up to 0.42 in accuracy, and no leaderboard discloses which rule was used.
The evidence is visible in already-published results. Of 34 released Headlines scores, nine exactly match what the benchmark's code returns when no output actually equals "Yes" — meaning the score is an artifact of parsing. Only four clear the majority-class floor. On Financial PhraseBank, four of seven models fall below that floor under one rule and clear it under the other.
Why it matters: investors and analysts increasingly cite benchmark rankings to judge AI tools for finance. If a score partly measures an undisclosed grading rule, comparisons between models are unreliable. The authors argue a FLARE score needs three unreported numbers — ambiguity rate, coverage, and the majority floor — and offer a tool, floorbench, that computes them from outputs already generated. The practical takeaway is to treat single benchmark numbers with caution and ask how they were scored.
Source: arXiv, https://arxiv.org/abs/2610.33604
Educational content only, generated with AI assistance from public research abstracts. Not investment advice. Thank you to arXiv for use of its open access interoperability.