TreasureX Researchcs.LGHow LLM Judges Really Score Text: A Peek Under the Hood
Himil Vasava, Ming Jiang · September 1, 2026
Large language models (LLMs) are increasingly used to judge the quality of AI-generated text, such as summaries. But how do they actually decide on a score? A new study from arXiv (2609.01604) takes a mechanistic look inside two popular evaluator models, Themis and Prometheus, to understand their inner workings.
The researchers found that these models follow a structured, two-stage process. In the first stage (layers below 15), attention mechanisms compare the summary to the original text, focusing on local errors and routing that information to a specific position. In the second stage (higher layers), a neural network component integrates this signal and produces the final rating. Interestingly, the decision 'crystallizes' at a specific late layer, suggesting a clear point where the score is essentially determined.
A key finding is that fine-tuning doesn't build this pipeline from scratch. Instead, it modifies an existing architecture found in the base model, primarily by suppressing certain contributions and shifting the crystallization point. This suggests that LLM judges are not black boxes but have interpretable mechanisms.
For investors, this research matters because it could lead to more reliable and transparent AI evaluation tools. Understanding how judges work can help improve training signals for AI models, potentially making them more accurate and trustworthy. This could impact companies that rely on AI for content generation, quality control, or automated feedback, as better evaluation methods could reduce errors and improve product quality.
Source: arXiv (https://arxiv.org/abs/2609.01604)
Educational content only, generated with AI assistance from public research abstracts. Not investment advice. Thank you to arXiv for use of its open access interoperability.