Sources and further reading

Cited works

• American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. American Educational Research Association, Washington, DC.

  The joint testing standards, cited for the requirement to identify the domain of legitimate replications before reliability means anything. Cited in Reliability and generalizability.

• Anthropic (2026). Demystifying evals for AI agents. Anthropic Engineering. Link(opens in a new tab)

  Practitioner guidance behind the error-analysis workflow and the warning that a strong model scoring near 0% usually means a task or harness bug, both adopted by this handbook. Cited in Task distributions and dataset engineering; Population, environment, harness, and validity domain; and Task validity and outcome validity.

• Victor Barres et al. (2025). τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv. Link(opens in a new tab)

  τ²-bench, the handbook's example of a dual-control environment, in which the agent and a simulated user both act through tools and the score combines several assertion categories. Cited via the Berkeley slides in Evaluating agents; Reading agent benchmarks; and Task distributions and dataset engineering.

• Denny Borsboom, Gideon J. Mellenbergh, and Jaap van Heerden (2004). The concept of validity. Psychological Review, 111(4):1061–1071.

  The causal account of validity, the main alternative to the argument-based view this handbook uses. Cited in The validity argument.

• Robert L. Brennan (2001). Generalizability Theory. Springer.

  The standard reference for generalizability theory, the source of the variance-component view and the absolute coefficient Φ. Cited in Reliability and generalizability.

• Donald T. Campbell and Donald W. Fiske (1959). Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin, 56(2):81–105.

  The original multitrait–multimethod design for separating trait variance from method variance. Cited in Sources of validity evidence.

• Mark Chen et al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Link(opens in a new tab)

  The origin of the pass@k name, with its at-least-one meaning, and of the unbiased pass@k estimator. Cited in Estimands, pass rates, and intervals.

• Mingguang Chen, Licheng Wang, and Bo Qu (2026). Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops. arXiv:2607.07663. Link(opens in a new tab)

  The RSI survey, used for the evaluator design space of self-improving systems, its evidence on grounding and self-confirmation, and its open problems; cited across the handbook, most heavily in Self-improvement and evaluator co-evolution, and always paraphrased. Section numbers in this handbook follow arXiv v1 (July 2026); a later version, v2, has since appeared.

• Sangwoo Cho, Kushal Chawla, Pengshan Cai, Zefang Liu, Chenyang Zhu, Shi-Xiong Zhang, and Sambit Sahu (2026). Ask, don’t judge: Binary questions for interpretable LLM evaluation and self-improvement. Link(opens in a new tab)

  BinEval, which replaces a holistic judgment with atomic yes-or-no questions whose answers can be inspected, cited via the RSI survey (§3.1) in support of binary per-criterion verdicts. Cited in LLM judges in practice and Rubrics and criterion cards.

• Lee J. Cronbach and Paul E. Meehl (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4):281–302.

  The classic statement of construct validity and the nomological network. Cited in Constructs, facets, and nuisances; and Sources of validity evidence.

• Bentley DeVilling (2025). The mirror loop: Recursive non-convergence in generative reasoning systems. Link(opens in a new tab)

  The Mirror Loop study, in which informational change declined over rounds of ungrounded self-critique and a single verification step restored forward movement; cited via the RSI survey (§5.2) for the bottom of the verification hierarchy. Cited in The verification hierarchy, demoted; and Self-improvement and evaluator co-evolution.

• Steven Young Eulig (2026). Position: Correct answer, wrong mechanism – when AI scientists defend general claims their own data contradicts. Link(opens in a new tab)

  A position paper on AI scientists that reach correct answers while defending wrong mechanisms, the basis for measuring outcome, mechanism fidelity, and epistemic honesty separately. Cited via the RSI survey (§6.3) in Evaluating agents; Signals and resolution; and Case D (Autonomous research agent).

• Leo Gao, John Schulman, and Jacob Hilton (2023). Scaling laws for reward model overoptimization. International Conference on Machine Learning (ICML). Link(opens in a new tab)

  The scaling laws for reward-model overoptimization: optimizing a learned proxy reward eventually lowers quality as judged by a stronger reference, the canonical exploitability curve. Cited in Adaptive validity; Evaluators inside training loops; Goodhart pressure and self-confirming loops; and Exposure, modification, and access.

• D. Hadfield-Menell, A. Dragan, P. Abbeel, and S. Russell (2016). Cooperative Inverse Reinforcement Learning. arXiv:1606.03137. Link(opens in a new tab)

  Cooperative inverse reinforcement learning, which makes asking questions part of inferring an objective; the handbook builds on it for evaluators that ask where their posterior is uncertain. Cited in Inverse evaluation design.

• D. Hadfield-Menell, S. Milli, P. Abbeel, S. Russell, and A. Dragan (2017). Inverse Reward Design. arXiv:1711.02827. Link(opens in a new tab)

  Inverse reward design, which reads a hand-written reward as evidence about what its designer intended; the handbook takes from it the point that a rubric documents a construct rather than defining it. Cited in Inverse evaluation design.

• Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou (2024). Large language models cannot self-correct reasoning yet. ICLR 2024. Link(opens in a new tab)

  Evidence that LLMs largely cannot self-correct their reasoning without external feedback, cited via the RSI survey (§3.1) for the weakness of intrinsic signals. Cited in The verification hierarchy, demoted; and Mechanism families.

• Kung-Hsiang Huang et al. (2025). CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environments. NAACL 2025. Link(opens in a new tab)

  CRMArena, read in coordinates as exact match and F1 against a fixed key, with turns, tokens, and cost reported beside the score. Cited via the Berkeley slides in Reading agent benchmarks.

• Hamel Husain (2025). Evals FAQ. Online practitioner reference. Link(opens in a new tab)

  Practitioner source for error analysis, from open coding of real traces to saturation, adopted by this handbook. Cited in Task distributions and dataset engineering.

• Mykel J. Kochenderfer, Sydney M. Katz, Anthony L. Corso, and Robert J. Moss (2026). Algorithms for Validation. MIT Press, forthcoming 2026. Link(opens in a new tab)

  The validation-algorithms text this handbook follows for the validation triad, metrics versus specifications, robustness margins, falsification, rare-event estimation, and runtime assurance; cited in Evaluation is measurement; Rare-event assurance; Falsification and failure regions; and Guarantee before you estimate. A draft is available online.

• Klaus Krippendorff (2004). Content Analysis: An Introduction to Its Methodology. Sage, second edition.

  The standard treatment of agreement coefficients, cited with Reidsma and Carletta (2008) for reliability being necessary but never sufficient for validity. Cited in Reliability and generalizability.

• Zsolt Lang and Jenő Reiczigel (2014). Confidence limits for prevalence of disease adjusted for estimated sensitivity and specificity. Preventive Veterinary Medicine, 113(1):13–22.

  Confidence limits for a prevalence corrected for estimated sensitivity and specificity, the purpose-built interval the handbook points to for a Rogan–Gladen-corrected pass rate. Cited in The judge as a noisy sensor.

• Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe (2023). Let’s verify step by step. arXiv preprint arXiv:2305.20050; ICLR 2024. Link(opens in a new tab)

  The evidence that step-level process supervision can outperform outcome-only supervision when training on MATH. Cited in Signals and resolution; Evaluators inside training loops; and Case A (Mathematical reasoning).

• Jianzhe Lin (2026). Self-improvement can self-regress: The rise-and-collapse failure mode of LLM self-training. Link(opens in a new tab)

  A study of LLM self-training in which single-attempt success on code rose and then collapsed within a run despite a verifiable reward. Cited via the RSI survey (§4.1) in Evaluators inside training loops.

• Samuel Messick (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9):741–749.

  Validation as scientific inquiry into score meaning, the root of the unitary, argument-based view of validity. Cited in The validity argument.

• OpenAI (2024). Introducing SWE-bench Verified. Link(opens in a new tab)

  SWE-bench Verified, the handbook's example of a benchmark audit that found invalid tasks and tests understating capability; cited in Task validity and outcome validity; Task distributions and dataset engineering; Reading agent benchmarks; and Case B (Coding agent). Cited via the Berkeley course slides, which give the link but not the year; the year is the post's publication date.

• OpenAI (2025). GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks. Link(opens in a new tab)

  GDPval, read in coordinates as a blinded panel of occupation experts comparing a model's deliverable with an expert's own. Cited via the Berkeley slides in Reading agent benchmarks; and Task distributions and dataset engineering.

• Rafael Rafailov et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290. Link(opens in a new tab)

  Direct preference optimization, which trains a policy on preference pairs without a separate reward model and RL loop, and so changes the optimizer but not the measurement problem. Cited in Evaluators inside training loops.

• Sebastian Raschka (2025). Understanding the 4 Main Approaches to LLM Evaluation (From Scratch): Multiple-Choice Benchmarks, Verifiers, Leaderboards, and LLM Judges.

  The practical four-way map of LLM evaluation (multiple choice, verifiers, leaderboards, LLM judges), which an earlier version of this site taught as its four evaluation methods. Re-expressed on the six axes in Vocabulary without category errors and the Crosswalk.

• Dennis Reidsma and Jean Carletta (2008). Reliability measurement without limits. Computational Linguistics, 34(3):319–326.

  A caution against treating agreement as the goal: simplifying an annotation scheme until coders agree can cost validity. Cited in Reliability and generalizability; and LLM judges in practice.

• Stefan Riezler and Michael Hagmann (2024). Validity, Reliability, and Significance: Empirical Methods for NLP and Data Science. Springer, second edition.

  Validity, reliability, and significance as separate empirical questions, with the bias, illegitimate, and circular feature taxonomy and mixed-effects reliability analysis. Cited in Task distributions and dataset engineering; Reliability and generalizability; and Comparing systems.

• Walter J. Rogan and Beth Gladen (1978). Estimating prevalence from the results of a screening test. American Journal of Epidemiology, 107(1):71–76.

  The prevalence correction for an imperfect screening test, which the handbook applies to judge pass rates as the Rogan–Gladen correction. Cited in The judge as a noisy sensor.

• Zhan Shi, Bing He, Yisi Sang, Hanqing Lu, and Benoit Dumoulin (2026). A-evolve-training: Autonomous post-training of a 30b model. Link(opens in a new tab)

  A-Evolve-Training, an autonomous post-training loop that noticed its development metric decoupling from the external target. Cited via the RSI survey (§6.2) in Channels, independence, and anchor burn; and Self-improvement and evaluator co-evolution.

• Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal (2024). AI models collapse when trained on recursively generated data. Nature, 631:755–759. Link(opens in a new tab)

  Model collapse under recursive training on generated data, in which the tails of the distribution vanish first. Cited via the RSI survey (§5.3) in Goodhart pressure and self-confirming loops; and Anti-patterns and failure vocabulary.

• Dawn Song (instructor) (2025). LLM Agent Evaluations & Project Overview. Lecture slides, Agentic AI MOOC, UC Berkeley, October 2025. Link(opens in a new tab)

  Berkeley lecture slides, cited by slide number for agent-benchmark case studies, outcome-validity checks, and evaluation by lifecycle stage, mainly in Reading agent benchmarks; Task validity and outcome validity; and Task distributions and dataset engineering. The title slide is dated 2025.10.6.

• Chuyi Tan, Peiwen Yuan, Xinglin Wang, Yicheng Li, Shaoxiong Feng, Yueqi Zhang, Jiayi Shi, Ji Zhang, Boyuan Pan, Yao Hu, and Kan Li (2025). Breaking the self-confirming loop: Diagnosing and mitigating systemic reward bias in self-rewarding RL. Link(opens in a new tab)

  Systemic reward bias in self-rewarding RL, where rewards tied to the model's own confidence favor the mistakes it is surest of. Cited via the RSI survey (§5.3) in Goodhart pressure and self-confirming loops; Evaluators inside training loops; and Anti-patterns and failure vocabulary.

• Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi, and Naohiko Matsuda (2026). LLM judges have dark current: A psychometric datasheet for LLM-as-a-judge evaluation. arXiv preprint arXiv:2606.15610. Link(opens in a new tab)

  The judge datasheet, which replaces a single agreement number with probes such as dark current, a judge's baseline preference when there is no real quality signal. Cited in LLM judges in practice and Decision claims.

• Zhun Wang, Tianneng Shi, et al. (2025). CyberGym: Evaluating AI Agents' Cybersecurity Capabilities with Real-World Vulnerabilities at Scale. arXiv. Link(opens in a new tab)

  CyberGym, read in coordinates as a differential execution oracle in which sanitizers judge pre- and post-patch builds, with a post-cutoff analysis against contamination. Cited via the Berkeley slides in Reading agent benchmarks; and Task distributions and dataset engineering.

• Zonglin Yang, Xingtong Liu, and Xinyan Xu (2026). SciIntegrity-bench: A benchmark for evaluating academic integrity in AI scientist systems. Link(opens in a new tab)

  SciIntegrity-Bench, integrity dilemmas in which admitting failure is the only correct response. Cited via the RSI survey (§6.3) in Evaluating agents; and Case D (Autonomous research agent).

• Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan (2024). τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045. Link(opens in a new tab)

  τ-bench, the source of the all-k pass^k and an example of grading a database end state against an annotated goal. Cited in Estimands, pass rates, and intervals; Evaluating agents; Reading agent benchmarks; and Task distributions and dataset engineering.

• Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang (2024). ReST-MCTS*: LLM self-training via process reward guided tree search. Link(opens in a new tab)

  ReST-MCTS*, cited via the RSI survey (§4.1) for the finding that keeping training solutions by final-answer correctness admits right answers reached by wrong reasoning. Cited in Evaluators inside training loops.

• Zhengxin Zhang, Ning Wang, Sainyam Galhotra, and Claire Cardie (2026). How far are we from true auto-research?. Link(opens in a new tab)

  The benchmark the RSI survey calls ResearchArena, in which manuscript-only automated review put the best agent's papers on a par with the average human ICLR submission, a picture that artifact-aware review and human meta-review showed to be overstated. Cited via the survey (§6.3) in Reading agent benchmarks; and Case D (Autonomous research agent).

• Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. Link(opens in a new tab)

  The MT-Bench and Chatbot Arena study that made LLM-as-a-judge a measurable method, reporting judge–human agreement and documenting position, verbosity, and self-enhancement biases. Cited in LLM judges in practice.

• Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, and Daniel Kang (2025). Establishing best practices for building rigorous agentic benchmarks. arXiv preprint arXiv:2507.02825. Link(opens in a new tab)

  The agentic benchmark checklist: task validity and outcome validity as named links, and outcome-validity checks by output type. Cited in Task validity and outcome validity; The validity argument; Evaluation is measurement; LLM judges in practice; and Cases A and B.

• Chen Zhu, Xiaolu Wang, and Weilong Zhang (2026). (human) attention is (still) all you need: Human oversight makes AI-assisted social science reliable. Link(opens in a new tab)

  A pre-specified study of human-in-the-loop research in economics, which found that reliability depended on how the work was divided between people and machine. Cited via the RSI survey (§6.3) in Frame review and governance.