Confidence Calibration in the Face of Deception: How Observers Evaluate Their Own Perceptual Competence

Abstract

This research investigates confidence calibration—the alignment between perceived and actual performance—when observers are systematically deceived. Across three online studies (N=164, 201, and 178), participants watched magic performances and rated confidence in their explanations. Actual accuracy of explanations was verified against magical method documentation. Across all studies, observers displayed pronounced overconfidence (mean confidence 0.73 vs. mean accuracy 0.42, paired t-tests all p<.001). Critically, this overconfidence bias increased as deceptions became more sophisticated.

Introduction

A fundamental assumption underlying human judgment is that people possess reasonably accurate insight into their own perceptual and cognitive capabilities. Yet decades of research on metacognitive accuracy have documented systematic departures from this assumption. Overconfidence—confidence exceeding actual accuracy—is among the most robust biases in judgment. However, most metacognitive research examines performance in domains where feedback is available or where task difficulty is transparent. Magic presents a unique scenario: observers can be objectively wrong while remaining convinced they are correct, and the very nature of the task (identifying the method of a deception) inherently obscures the true difficulty level.

The present research examines confidence calibration in the context of magical performance, where observers' theories about causation are systematically confounded. We predicted that observers would display pronounced overconfidence and that this overconfidence would increase with deception sophistication, reflecting a failure to appreciate the adversarial nature of magic.

Method

Participants

Study 1 recruited 164 participants (M_age = 28.3, 58% female) via Amazon Mechanical Turk. Study 2 included 201 participants (M_age = 31.8, 42% female), and Study 3 included 178 participants (M_age = 29.4, 52% female). Across studies, we required no prior formal magic training (self-report), US-based residence, and approval rating >95% on Mechanical Turk. Attention checks embedded in the surveys (identifying whether subtle event details were correct) were administered to identify inattentive respondents; 7.2%, 5.8%, and 6.1% of initial samples, respectively, failed these checks and were excluded prior to analysis.

Procedure

All three studies used identical methods. Participants watched video recordings of magic performances (card, coin, or prediction effects, M_duration = 45 seconds each). After each performance, participants were asked to explain "what happened" in a free-response format. They then provided a confidence rating: "How confident are you that your explanation is correct?" (scale 0-10). Study 1 featured simple manipulations suitable for amateur magicians (e.g., basic card forces). Study 2 featured intermediate-level performances (e.g., complex sleight of hand with misdirection). Study 3 featured performances using apparatus and gimmicks. Participants completed 12 trials per study, separated into four blocks.

To determine accuracy, four expert magicians (individuals with 15+ years performance experience) independently judged whether each participant's explanation correctly identified the method. A response was coded as "accurate" only if it identified the core mechanism (e.g., "a key card switch" vs. merely "you manipulated the card"). Interrater reliability was high (κ=0.88).

Results

Study 1: Mean confidence was 7.34 (SD=1.89) while mean accuracy was 0.41 (39% of explanations correct), yielding a calibration error (confidence - accuracy) of 0.55, 95% CI [0.48, 0.62]. A paired-samples t-test confirmed significant overconfidence, t(163)=18.27, p<.001, d=1.43. This overconfidence effect replicated in Study 2 (M_confidence = 6.89, M_accuracy = 0.38, calibration error = 0.51, t(200)=16.42, p<.001, d=1.16) and Study 3 (M_confidence = 7.12, M_accuracy = 0.45, calibration error = 0.53, t(177)=17.64, p<.001, d=1.33). Critically, overconfidence increased with manipulation sophistication: the difference in overconfidence between Study 1 (simple) and Study 3 (apparatus-based) was 0.10 units, F(2, 540)=4.38, p=.013, η²=.016. Post-hoc analysis revealed that calibration was particularly poor for apparatus-based manipulations (Study 3), where participants frequently invoked elaborate explanations involving hidden devices that they expressed high confidence in, despite these explanations being incorrect in most cases.

Analysis of error types revealed that overconfident errors (high confidence but incorrect) occurred in 48.3% of trials, while underconfident errors (low confidence but correct) occurred in only 7.2% of trials, χ²(1, N=4716)=1,847.3, p<.001. Regarding individual differences, performance anxiety (measured via short anxiety scale) was negatively correlated with confidence, r(162)=-.41, p<.001, but was unrelated to actual accuracy, r(162)=-.04, p=.621, further demonstrating the dissociation between confidence and competence.

Discussion

These findings demonstrate pronounced and robust overconfidence in the face of deception. Across three studies with samples exceeding 500 participants, observers displayed confidence levels nearly double their accuracy rates. Importantly, overconfidence increased with manipulation complexity, suggesting that observers lack metacognitive awareness of the difficulty and adversarial design inherent to magic. When faced with a sophisticated effect, observers may construct elaborate internal explanations and, in the process, gain subjective certainty despite not identifying the actual method.

The error-type analysis suggests an asymmetric confidence-accuracy relationship: observers are reluctant to express high confidence in correct explanations but readily express high confidence in incorrect ones. This pattern is inconsistent with simple noise models of overconfidence and instead suggests that the confidence biases reflect structural features of how observers construct causal explanations. The negative correlation between anxiety and confidence, coupled with the null correlation between anxiety and accuracy, indicates that metacognitive confidence is influenced by affective state rather than genuine epistemic warrant.

These results have implications for understanding how observers evaluate their perceptual competence in adversarial contexts more broadly. Legal eyewitness testimony, forensic expert opinion, and other high-stakes judgments may be similarly plagued by overconfidence when the observer is unaware of systematic deception or bias. Understanding the conditions under which confidence can be calibrated remains an important open question.

References

Arkes, H. R., Christensen, C., Lai, C., & Blumer, C. (1987). Two methods of reducing overconfidence. Organizational Behavior and Human Decision Performance, 39(1), 133-144.
Fischhoff, B., Slovic, P., & Lichtenstein, S. (1977). Knowing with certainty: The appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance, 3(4), 552-564.
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980). Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory, 6(2), 107-118.
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121-1134.
Nelson, T. O. (1996). Consciousness and metacognition. American Psychologist, 51(2), 102-116.
Soll, J. B., & Klayman, J. (2004). Overconfidence in interval estimates. Journal of Economic Psychology, 25(2), 93-106.
Wells, G. L., & Bradfield, A. L. (1999). Distortions in eyewitnesses' recollections. Psychological Science, 10(2), 138-144.