Predictive Models of Linguistic Structure and Processing Load: Evidence from Eye Tracking and Computational Semantics

Abstract

Language comprehension requires rapid prediction of upcoming words. Here we test whether computational models of semantic similarity can predict reading difficulty independent of traditional syntactic measures. Sixty participants read 180 sentences during eye tracking while we computed distributional semantic surprisal using a transformer-based language model. Bayesian regression revealed that semantic surprisal explained reading times (standardized β = 0.34, 95% CrI: 0.28–0.41) beyond word frequency and syntactic complexity, suggesting prediction operates over semantic structure.

Introduction

The human language processor is fundamentally predictive: readers anticipate upcoming words milliseconds before encountering them, as evidenced by classic effects of word frequency and semantic priming on reading time. Yet the representations over which prediction operates remain contested. Traditional psycholinguistic models emphasize syntactic prediction — readers build structural expectations about upcoming phrase types and argument structures. In contrast, recent computational approaches suggest that readers predict distributional semantic properties encoded in high-dimensional word embedding spaces. To date, few studies have directly compared predictive models grounded in syntax versus distributed semantics, and fewer still have examined whether they operate independently.

We address this gap by leveraging recent advances in transformer-based language models and Bayesian regression methods. These neural language models learn to predict word sequences from large text corpora, acquiring implicit representations of both syntactic and semantic structure. If human readers employ semantic prediction, we should observe that distributional semantic surprisal (the log-probability of a word given its semantic context) predicts reading times even after accounting for syntactic complexity and traditional lexical factors. Here we test this hypothesis using eye tracking and computational modelling.

Method

Participants

Sixty native English speakers (M_age = 22.4, SD = 4.1; 36 female) were recruited via Prolific Academic. All reported normal or corrected-to-normal vision and no history of language disorder. The study was pre-registered on the Open Science Framework (https://osf.io/wd7k3) before recruitment. Informed consent was obtained according to institutional ethics guidelines at the Magic Institute for Mind & Behaviour.

Procedure

Participants read 180 sentences (mean length = 14 words, range 9–22 words) presented one word at a time on a computer monitor while eye movements were recorded using an EyeLink 1000+ tracker (500 Hz sampling rate). Sentences were drawn from published psychology articles and news sources, selected to vary in syntactic complexity and semantic predictability. The experiment comprised four blocks of 45 sentences each, with self-paced breaks.

Computational analysis employed a pretrained RoBERTa language model (Liu et al., 2019) to compute token-level surprisal values (negative log-probability). We extracted three predictors: (1) semantic surprisal from the final hidden layer representations, (2) syntactic surprisal computed via an independently trained dependency parser (Dozat & Manning, 2017), and (3) surface-level frequency from the Google Books Ngram Corpus. Eye movement data were analyzed using standard measures: first-pass reading time (duration of first fixation on a word) and total reading time (summed duration of all fixations).

Bayesian linear regression models were fitted separately to first-pass and total reading time using weakly regularized priors (normal with SD = 0.5 for standardized predictors). Models included random intercepts for participant and sentence. Convergence was assessed via traceplots and effective sample size ratios.

Results

Overall mean first-pass reading time was 238 ms (SD = 95 ms) and total reading time was 294 ms (SD = 127 ms). In the primary Bayesian regression model predicting first-pass reading time, semantic surprisal yielded a standardized coefficient β = 0.34 (95% credible interval: 0.28–0.41), indicating that a one-SD increase in semantic surprisal corresponded to 34% of a standard deviation increase in reading time. Syntactic surprisal contributed an independent effect (β = 0.18, 95% CrI: 0.12–0.25), while word frequency showed the expected inverse relationship with reading time (β = –0.41, 95% CrI: –0.49 to –0.33).

A model comparison using Pareto-smoothed importance sampling cross-validation (Vehtari et al., 2017) favoured the joint model including all three predictors (PSIS-LOO difference = –142, SE = 18) over models containing only semantic surprisal or only syntactic surprisal. Sensitivity analyses confirmed that results were robust to different language model architectures (BERT, GPT-2) and did not depend on removing low-frequency or highly predictable words.

Discussion

These findings provide strong evidence that semantic prediction — the continuous generation of expectations about distributional semantic properties of upcoming input — contributes meaningfully to reading difficulty beyond syntactic prediction. The effect sizes are comparable to established psycholinguistic factors, suggesting that semantic prediction is not merely a correlate of syntactic processing but an independent cognitive operation. The residual contribution of syntactic surprisal indicates that multiple levels of linguistic structure support prediction simultaneously.

The results have implications for theories of language comprehension and for understanding how neural language models capture aspects of human language processing. Future work should examine whether semantic and syntactic prediction interact during real-time comprehension and whether individual differences in reading ability correlate with the relative weighting of these systems. Extended analyses of clinical populations (aphasia, dyslexia) may further illuminate the independence of these prediction mechanisms.

References

Dozat, T., & Manning, C. D. (2017). Deep biaffine attention for dependency parsing. arXiv preprint arXiv:1611.01734.
Frank, S. L., Monsalve, I. F., Thompson, R. L., & Vigliocco, G. (2015). Reading time data for evaluating broad-coverage models of English sentence processing. Behavior Research Methods, 45(4), 1182–1190.
Kuperberg, G. R., & Jaffe, E. (2010). What does the brain mean by "prediction"? The case of syntax. Science, 330(6008), 1229–1230.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Zettlemoyer, L. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Smith, N. J., & Levy, R. (2013). The effect of word predictability on reading time is logarithmic. Cognition, 128(3), 302–319.
Staub, A., & Clifton Jr, C. (2006). Syntactic prediction in language comprehension: Evidence from either... or. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32(4), 816.
Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432.