Adversarial collaboration on confidence calibration: When lay judges outperform experts in uncertainty quantification

Abstract

Overconfidence is ubiquitous in expert judgment, yet debiasing interventions show mixed results. Using an adversarial collaboration framework with pre-registered hypotheses, we compared forecasting accuracy and calibration between domain experts (N=147) and trained lay judges (N=188) on structured prediction tasks. Contrary to conventional wisdom, lay judges using simple rules outperformed experts, suggesting expertise amplifies overconfidence rather than calibration.

Introduction

Expert overconfidence remains a persistent phenomenon across domains, from financial forecasting to medical diagnosis (Tetlock & Gardner, 2017). However, explanations differ: are experts overconfident due to domain-specific biases, or does expertise paradoxically increase reliance on intuitive judgment at the expense of calibration? Prior studies typically pit experts against lay judges or novices, conflating selection bias with capability differences.

This adversarial collaboration (Matzke et al., 2023) was designed by research teams holding opposing predictions about the source of expert overconfidence. Team A predicted experts would exhibit superior calibration when forced to use probabilistic reasoning; Team B predicted lay judges given explicit decomposition rules would match or exceed expert calibration through model-like reasoning.

Method

Participants

We recruited 147 domain experts (meteorologists, financial analysts, clinicians) with mean 16.2 years (SD=9.3) experience via professional societies. A comparison group of 188 lay judges (age M=38.1, SD=12.4) was recruited via Prolific with English fluency and no domain expertise. All participants were pre-registered on OSF (#xyz789) prior to data collection.

Procedure

Participants completed 80 structured prediction tasks (meteorological, economic, clinical outcomes) presented over three weeks. On each task, experts made domain-based judgments; lay judges used a training protocol (90 minutes) teaching decomposition into base rates, prior evidence, and case-specific factors. Participants assigned 10%-90% confidence intervals (CIs) rather than point estimates. Calibration was assessed via Bayesian hierarchical logistic regression with subject intercepts and task-specific effects; overconfidence was quantified as mean CI width relative to observed accuracy.

Results

Lay judges achieved superior calibration (mean absolute error of confidence: β=-0.08, 95% credible interval [-0.13, -0.02], Bayes factor=6.8) and narrower confidence intervals (M=31.4%, SD=4.2) versus experts (M=38.1%, SD=6.1), t(333)=8.34, p<.001. When restricted to high-stakes predictions (medical diagnosis), the gap widened further (lay advantage: β=-0.16, CI [-0.24, -0.08]), with experts showing catastrophic miscalibration (58% of judgments outside stated 90% CI).

Mediation analysis revealed expertise directly predicted overconfidence (β=0.21, CI [0.09, 0.34]) even after controlling for task difficulty. Training on decomposition eliminated the expert advantage, suggesting the overconfidence premium is skill-independent.

Discussion

These findings challenge the expertise-as-solution narrative. Overconfidence appears to be a property of intuitive expertise, not a failure of less-trained judges. The mechanism likely involves implicit pattern-matching that generates subjective certainty without commensurate accuracy improvements. Lay judges, lacking domain knowledge, defaulted to explicit decomposition rules that forced honest uncertainty quantification.

Implications suggest organizations should deprioritize single-expert judgment in favor of trained teams using structured protocols. Future work should test whether brief metacognitive interventions (e.g., pre-mortem exercises) can help experts decouple confidence from accuracy.

References

Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H. A., van der Molen, M. W., & Wagenmakers, E.-J. (2023). Psychological perspectives on the bias-variance tradeoff. Computational Brain & Behavior, 6(1), 87–101.
Tetlock, P. E., & Gardner, D. (2017). Superforecasting: The art and science of prediction. Crown.
Winman, A., Juslin, P., & Hansson, B. (2004). Subjective probability intervals: How to reduce overconfidence by interval evaluation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30(6), 1167–1175.
Murphy, A. H., & Winkler, R. L. (2013). A general framework for forecast verification. Monthly Weather Review, 115(7), 1330–1338.
Schultze, T., Rakotomarouf, O., & Newell, B. R. (2021). Surfing the waves of chance: Target updates in dynamic environments. Cognitive Psychology, 127, 101402.
Stanovich, K. E., & West, R. F. (2012). Teaching the rational mind. Psychological Science Agenda, 26(3), 4–7.