TreasureX Researchq-fin.GNSame Text, Different Numbers: Why LLM-Based Research Measures Depend on the Model You Pick
Hamid Boustanifar, Sasan Mansouri · September 25, 2026
The key idea
A growing number of researchers and analysts feed corporate text — earnings call transcripts, filings, press releases — into large language models and treat the output as a data point: a sentiment score, a clarity score, an uncertainty score. This paper asks a blunt question: if you swap one LLM for another, do you get the same number?
The authors had seven LLMs from different providers score S&P 500 earnings call transcripts on thirteen constructs, including sentiment, management clarity, uncertainty, answer specificity, and climate and political risk. The answer is largely no. Cross-model rank correlations averaged just 0.52, and differences shared across providers explained only about a third of the variation in scores. The rest looks model-specific rather than reflecting genuine ambiguity in the disclosure.
Why it matters
When the same text yields different variables depending on the model, downstream results change too. The authors find that coefficient magnitudes, signs, and statistical significance move around across models. Averaging several models stabilizes the ranking of transcripts for most constructs, but the absolute score levels still shift depending on which models are in the mix.
Practical implication
Treat LLM-derived variables as model-contingent measurements, not objective facts. For investors and analysts building text-based signals, that means validating across multiple providers, reporting which models were used, and being skeptical of precise-looking scores that may not survive a model swap.
Source: arXiv, "Same Text, Different Numbers: The Divergence of LLM-Based Measures" by Hamid Boustanifar and Sasan Mansouri — https://arxiv.org/abs/2609.31013
Educational content only, generated with AI assistance from public research abstracts. Not investment advice. Thank you to arXiv for use of its open access interoperability.