Hierarchical Bayesian Learning in Non-Stationary Environments: A Computational Framework and Neural Implementation

Abstract

Learning in dynamic environments requires tracking both specific contingencies and abstract change-point statistics. We present a hierarchical Bayesian framework formalizing how agents simultaneously learn task structure (L1), environmental volatility (L2), and volatility change-points (L3). Behavioral experiments with 46 participants and fMRI data from 38 participants demonstrate that humans employ nested Bayesian inference consistent with the model. Anterior cingulate and dorsolateral prefrontal cortex track prediction errors at higher hierarchical levels. Open-source PyMC implementation (github.com/magic-institute/hbl-package/) enables accessible computational modelling for neuroscience and psychology research.

Introduction

Learning theory has long grappled with the challenge of non-stationarity: in real environments, previously learned contingencies often change, and the rate of change itself varies. Classical reinforcement learning models assume stationary task structure, making them brittle when applied to dynamic settings. Hierarchical Bayesian approaches offer a principled solution: maintain beliefs at multiple levels of abstraction, from specific task contingencies to beliefs about environmental volatility itself. However, implementing hierarchical Bayesian models in realistic neuroscience contexts has been computationally intractable, and neural signatures of hierarchical inference remain incompletely understood.

Recent advances in probabilistic programming and layer-resolved fMRI now enable direct comparison of hierarchical Bayesian predictions to neural data. We developed a formal computational framework wherein learning operates at three nested levels: (L1) task-specific prediction error correction; (L2) volatility tracking through detection of unexpected prediction errors; (L3) change-point inference about the stability of volatility itself. We tested human learning of probabilistic category structures in non-stationary environments and identified dissociable neural circuits implementing each hierarchical level.

Method

Participants

Experiment 1 recruited 46 participants (Mage=24.3, SD=5.2; 55% female) via Prolific to perform an adaptive category-learning task. Experiment 2 enrolled 38 participants (Mage=23.9, SD=4.8; 52% female) for identical protocol during 3T fMRI scanning. All participants provided informed consent (Protocol #RI-2022-073) and received performance-contingent payments (mean payout CAD$18.50, range $12–$25).

Procedure

Both experiments used a probabilistic category-learning paradigm with dynamically changing category structures. On each trial, participants observed a visual stimulus (dot clouds varying in color, size, and density) and predicted which of two categories it belonged to. True category membership was determined by a hidden rule (e.g., "color matters more than size") that changed unpredictably at certain points. Task structure explicitly manipulated three factors: (1) baseline volatility (stable vs. volatile environments); (2) volatility change-points (abrupt vs. gradual transition periods); (3) prediction error magnitude at different hierarchical levels.

The hierarchical Bayesian learning (HBL) model formalized learning as nested inference:

L1 (task): P(rule_t | observations) — prediction of which rule is active L2 (volatility): P(volatility_t | prediction_errors) — inference that the rule-changing rate has itself changed L3 (change-point): P(change_point_t | volatility_trajectory) — detection that volatility is now changing

We implemented the full model using PyMC v4.0 with hierarchical normal priors on all parameters. For each participant, we fit the model separately to learning blocks 1–4 (training) and tested predictions on block 5 (held-out validation). Behavioral predictions were accuracy, response time, and confidence ratings. fMRI data were analysed using layer-resolved imaging (0.8 mm voxels) to identify regional specificity. We entered trial-wise model predictions (L1, L2, L3 prediction errors) as regressors in a Bayesian hierarchical GLM using HierBayesFMRI v2.1.

Results

Behavioural results strongly supported the hierarchical model. Participants' accuracy trajectories closely matched model predictions (r=0.64, 95% CrI [0.54, 0.72]). Critically, when the model's L2 (volatility) and L3 (change-point) predictions diverged from a single-level learning model, participants' behaviour followed the hierarchical model (difference score: β=0.31, t₄₅=2.8, p=0.007). Response times showed sensitivity to L2 prediction errors (τ for L2 error: β=87 ms, 95% CrI [42, 138]), distinct from responses to L1 errors, indicating hierarchical separation.

fMRI results revealed functional dissociation: dorsal anterior cingulate cortex (dACC) tracked L2 prediction errors (t=4.1, cluster FWE p<0.001, cluster size 156 voxels, coordinates [2, 18, 26]), while dorsolateral prefrontal cortex (dlPFC) tracked L3 change-point detection (t=3.6, p_FWE<0.001, cluster size 118 voxels, coordinates [44, 28, 22]). L1 task errors recruited a distributed network including posterior parietal cortex and ventral striatum, consistent with previous studies. Connectivity analysis revealed that dACC→dlPFC coupling during L2 volatility signals correlated with individual learning efficiency in volatile blocks (r=0.42, p=0.012).

Discussion

These results provide converging evidence that humans implement hierarchical Bayesian learning through nested inference mechanisms, with separable neural implementations across cortical regions. The strong predictive power of the hierarchical model across both behaviour and fMRI indicates that the three-level framework captures core aspects of human learning in non-stationary environments. The dissociation between dACC (volatility tracking) and dlPFC (change-point detection) aligns with broader theories of hierarchical reinforcement learning and extends them to the learning-to-learn domain.

These findings have implications for understanding learning deficits in neuropsychiatric conditions characterized by impaired tracking of environmental dynamics (e.g., anxiety disorders, attention-deficit/hyperactivity disorder). The open-source PyMC implementation provided here democratizes hierarchical Bayesian modelling, enabling researchers without advanced computational expertise to fit these models to their own datasets. Future work should examine whether the hierarchical learning architecture is preserved across neurodevelopment and psychiatric populations, and whether targeted training in volatility discrimination can improve adaptive learning in real-world domains.

References

Behrens, T. E., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. (2007). Learning the value of information in an uncertain world. Nature Neuroscience, 10(12), 1214–1221.
Browning, M., Behrens, T. E., Jocham, G., O'Reilly, J. X., & Bishop, S. J. (2015). Anxious individuals have difficulty learning the causal statistics of volatile environments. Nature Neuroscience, 18(6), 807–809.
Friston, K. J., Stephan, K. E., Montague, P. R., & Dolan, R. J. (2015). Computational psychiatry: the brain as a phantastic organ of hypothesis testing. The Lancet Psychiatry, 2(12), 1072–1080.
Kruschke, J. K. (2008). Bayesian approaches to associative learning: From passive to active learning. Learning & Behavior, 36(3), 210–236.
Nassar, M. R., & Frank, M. J. (2016). Taming the beast: Extracting generalizable knowledge from computational models of cognition. Current Opinion in Behavioral Sciences, 11, 49–54.
Piray, P., Dezfouli, A., Heskes, T., Daw, N. D., & O'Doherty, J. P. (2019). Hierarchical Bayesian inference for concurrent model fitting and comparison for group studies. PLOS Computational Biology, 15(7), e1007043.
Tenenhaus, A., & Tenenhaus, M. (2011). Regularized generalized canonical correlation analysis. Psychometrika, 76(2), 257.