No evidence for retroactive facilitation of recall: a six-lab pre-registered adversarial replication

Abstract

We report a six-laboratory adversarial replication of retroactive facilitation effects — the claim that studying a random subset of previously recalled words improves memory for the unstudied words. Using pre-registered protocols agreed upon between proponent and sceptic laboratories, and analysed with Bayesian methods specified in advance, we find no credible evidence for the effect across a combined sample of 1,847 participants. We discuss the implications for the replication literature and for the standards that should govern extraordinary claims in psychological science.

Introduction

Retroactive facilitation of recall (RFR) is the claim that studying a random subset of words previously recalled in a free-recall task improves subsequent recall of the unstudied words — words not present in the study phase, not semantically related to the studied items, and selected randomly (Bem, Tressoldi, Rabeyron, & Duggan, 2016; Savva, Moutoussis, & Bhatt, 2017). If genuine, RFR would represent evidence of backward causation: a later study episode retroactively facilitating a memory that had already been formed.

Proponents report small-to-medium effects in controlled laboratory conditions (d ≈ 0.2–0.3) and point to several independent replications. Sceptics note that prominent reports used analytic procedures that inflate false-positive rates, that the theoretical mechanism is unspecified, and that many reported replications used adapted protocols that may not test the same phenomenon (Ritchie, Wiseman, & French, 2012; Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011). The adversarial collaboration model — in which proponent and sceptic teams jointly design a replication protocol, agree on analysis procedures and decision criteria in advance, and independently collect data — has been proposed as the appropriate standard for evaluating extraordinary claims (Matzke et al., 2015). We applied this model to RFR.

Method

Adversarial Collaboration Process

Protocol development proceeded over 18 months and involved three rounds of written exchange and two in-person workshops. Three laboratories were designated as sceptic sites (Magic Institute, University of Amsterdam, Ohio State University) and three as proponent sites (University of Edinburgh, Cornell University, Radboud University Nijmegen). The final protocol was pre-registered on AsPredicted (#87341) and OSF before any data collection began. Each site agreed in writing to accept the Bayesian decision criterion as definitive for their laboratory's contribution.

Participants and Procedure

Each laboratory recruited a minimum of 300 participants from local student pools. Final total N = 1,847 after pre-registered exclusion criteria were applied. Participants completed a free-recall task on a 48-word list, then after a 10-minute distractor task were shown a study phase containing 24 of the words they had recalled, selected randomly. A final recall test measured memory for all 48 original words. The critical dependent variable was final-recall performance for unstudied previously-recalled words in the study condition versus a matched no-study control.

The primary analysis used a Bayesian hierarchical model with site as a random factor, specified by the consortium statistician before data collection. The decision criterion for a positive result was BF₁₀ > 10; for a null result, BF₀₁ > 3.

Results

The primary Bayesian analysis yielded strong evidence for the null (BF₀₁ = 18.7). The point estimate of the RFR effect was d = 0.02 (95% credible interval: −0.06 to 0.10), substantially below the pre-registered threshold for theoretical meaningfulness (d = 0.15). Results were consistent across all six laboratories: site-level Bayes factors ranged from BF₀₁ = 2.3 (Cornell) to BF₀₁ = 11.4 (Magic Institute), with no site producing evidence for an effect. Exploratory moderator analyses across time of day, list composition, retention interval, and participant age identified no condition under which a credible effect emerged.

Discussion

The adversarial replication finds no credible evidence for retroactive facilitation of recall under conditions agreed upon by proponent and sceptic teams as a fair test. The null result is consistent across six laboratories, nearly 1,900 participants, and an analysis plan pre-specified before data collection. We take no pleasure in this conclusion: the adversarial collaboration process was a model of collegial engagement, and the proponent teams accepted the null result with intellectual honesty.

We recommend that claims of backward-causal memory effects be held to the standard established here before further investigative resources are committed. More broadly, the adversarial collaboration framework used in this project — jointly agreed protocols, pre-registered analysis plans, and consensus decision criteria — offers a template the field might apply more widely to contested empirical claims. All data and analysis code are publicly archived.

References

Bem, D. J., Tressoldi, P., Rabeyron, T., & Duggan, M. (2016). Feeling the future: A meta-analysis of 90 experiments on anomalous anticipation. F1000Research, 4, 1188.
Matzke, D., Nieuwenhuis, S., van Rijn, H., Slagter, H. A., van der Molen, M. W., & Wagenmakers, E.-J. (2015). The effect of horizontal eye movements on free recall. PeerJ, 3, e893.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
Ritchie, S. J., Wiseman, R., & French, C. C. (2012). Failing the future: Three unsuccessful attempts to replicate Bem's retroactive facilitation effect. PLoS ONE, 7(3), e33423.
Savva, M., Moutoussis, K., & Bhatt, M. (2017). Anomalous facilitation effects in free recall. Journal of Anomalous Psychology, 126(2), 143–159.
Wagenmakers, E.-J., Wetzels, R., Borsboom, D., & van der Maas, H. L. J. (2011). Why psychologists must change the way they analyze their data. Journal of Personality and Social Psychology, 100(3), 426–432.