1. Introduction
Spoken language understanding is a complex cognitive activity. Listeners need to decode their interlocutors’ intended meaning from the acoustic signal they produce, but the percept of this signal is high-dimensional and corrupted by listener-internal, speaker-internal, and environmental noise. Thus, any given cue in the signal inevitably leaves some degree of uncertainty about the underlying linguistic unit (e.g. phonemes, syllables, words). In speech perception, this is typically referred to as the lack of invariance problem: there is no one-to-one mapping between acoustic cues and phonetic units (Lisker & Abramson, 1967). The fundamental problem of real-time spoken language processing, then, is how listeners arrive at decisions about the identity of linguistic categories (like sounds, words, and syntactic structures) from inconclusive evidence.
One way for listeners to mitigate this problem is to make inferences based on multiple cues in the input, reducing (but not fully eliminating) uncertainty. In speech, each segment contains a multitude of cues that are relevant to inferring its category. In American English, for example, syllable-initial voicing (distinguishing, e.g., /t/ vs. /d/) is cued by voice-onset time, fundamental frequency, speech rate, burst duration, and other acoustic properties (Cooper et al., 1952; Kingston & Diehl, 1994; Liberman, 1957; Port, 1979). Listeners are able to combine these phonetic cues to infer an underlying phonemic category (Liberman, 1957; Lisker & Abramson, 1970). However, cues to segment identity do not always appear concurrently; they are also temporally distributed across the signal. For example, an important cue to syllable-final voicing is the duration of the previous vowel (Klatt, 1976). This temporal distribution also includes higher-level cues beyond acoustics. Later lexical context, for example, can provide cues to the identity of earlier segments – e.g., -ask following a segment acoustically manipulated to range between /t-d/ suggests that the earlier segment was more likely to be /t/ (task is a word, while dask is not). Indeed, listeners integrate these lexical cues with earlier acoustic information in spoken word recognition experiments (Ganong, 1980).1 This effect is particularly striking because it implies that listeners can maintain subcategorical information about the initial segment /t-d/ in memory over the course of the word: in order to successfully integrate early acoustic and later lexical cues, listeners must have access to the early cue in memory, so it can be integrated with the later cue. But given human memory limitations, it is impossible for listeners to maintain every bit of the complex acoustic signal in memory indefinitely. Thus, it is critical that we reconcile how listeners are able to integrate long-distance cues with the fundamental limitation of memory. The goal of the present work is to investigate to what degree listeners have access to subcategorical information about prior speech input over time, and what limitations there are (if any) on this process.
These questions are important to ask, because they address the foundational underpinnings of our theoretical understanding of language processing. In particular, a long-held belief in the psycholinguistic literature, often referred to as the immediacy assumption, is that listeners categorize incoming input as fast as possible and immediately discard low-level information in favor of categorical representations (Christiansen & Chater, 2016; Just & Carpenter, 1980). This assumption is driven by the (correct) observation that there are memory limitations on language processing. However, there is a large empirical and theoretical literature potentially suggesting that listeners can maintain significant amounts of subcategorical information for long periods of time. Indeed, most influential formal models of speech perception assume that some degree of low-level information is maintained by listeners over time. In connectionist models like TRACE and its descendants, this manifests as lingering activation for competitors as a result of network dynamics (Magnuson et al., 2020; McClelland & Elman, 1986); in Bayesian models like Shortlist B, all relevant acoustic and sentential information is combined to produce a word categorization, implying that listeners keep track of these cues over time (Norris & McQueen, 2008). However, the predictions of these models have primarily been tested in isolated word recognition (Ganong, 1980; Gwilliams et al., 2018; McMurray et al., 2002, 2009; Toscano et al., 2010), and only sometimes are quantitatively compared to formal models.
An emerging area of empirical research investigates these effects at longer timescales. An influential study by Connine and colleagues presented participants with sentences like the following:
(1)
When the ?ent in the fender was noticed, we sold the car.
They varied the acoustic features of the ? segment between /t-d/, and a later word in the sentence semantically biased toward a tent or dent interpretation of the target word (fender, as in (1) above, vs. forest; Connine et al., 1991). They found that participants’ categorizations of the target word were influenced both by the acoustics of the initial target and the later semantic context. Other studies using similar stimuli and methods ranging from categorization to visual world eye-tracking have yielded similar results at strikingly long distances (up to 35 syllables away from a target word; Bicknell et al., 2025; Brown-Schmidt & Toscano, 2017; Bushong & Jaeger, 2019, 2025; Falandays et al., 2020; Szostak & Pitt, 2013; Zellou & Dahan, 2019).
These findings have been interpreted to constitute evidence for subcategorical information maintenance. The key assumption is that if both early and late cues are used in categorization, listeners must have maintained subcategorical information about the early cue.2 This assumption is based on a (usually implicit) comparison between two basic models of listener strategies, which in this article we will call ideal integration and categorize-&-discard. Figure 1 shows a basic demonstration of the ideal integration strategy, using example stimuli from the studies presented in this article (inspired by Connine et al., 1991). Here, we use the term ideal in the sense of ideal observer models, which estimate the statistically optimal solution to perceptual cue integration problems (see, e.g., Ernst & Banks, 2002).3 The listener maintains some information about an initial stimulus varying between /t-d/ – at minimum, their degree of uncertainty about whether the sound was /t/ or /d/, but potentially more detailed, such as the value of a relevant acoustic cue, like voice-onset time (VOT). When they encounter the later biasing context fender, they are able to integrate information from both sources to come to a categorization decision. Notice that if the listener makes an initial categorical commitment to /t/ or /d/, this could not happen. Figure 2 demonstrates this categorize-&-discard approach: by the time the listener reaches fender, they are already committed to a /t/ response and have no access to a gradient representation with which to integrate the contextual information. This is the basic line of reasoning that has led researchers in this field to argue for subcategorical information maintenance.4
Figure 1: Schematic of the ideal integration model, with maintenance of subcategorical information over time.
![]()
Figure 2: Schematic of the categorize-&-discard model, without maintenance of subcategorical information over time.
![]()
Surprisingly, this interpretation of the literature has gone largely unchallenged, even though there are many other possible word recognition strategies that listeners could engage in beyond the ideal integration and categorize-&-discard options presented above. Indeed, it is quite possible to derive the central qualitative finding of these studies – that listeners’ behavioral responses depend on both the acoustic properties of the target sound (e.g., the /t-d/ stimulus) and subsequent sentential context (fender vs. forest) – without requiring maintenance of subcategorical information. Imagine, for example, that listeners initially categorize the /t-d/ sound as /t/ or /d/, discarding all gradient subcategorical information about it. After they encounter the subsequent context, they can choose to switch their response if the context conflicts with their original categorization. As we describe in more detail below, this strategy would yield categorization responses that exhibit dependence on both the acoustic and subsequent contextual cues: exactly the qualitative pattern observed in previous research. In short, there are plausible scenarios under which the available empirical evidence is qualitatively compatible with models of spoken word recognition that do not allow subcategorical information maintenance. Thus, it is clear that relying on qualitative outcomes like the presence or absence of acoustic and contextual effects in experiments is not sufficient for distinguishing between different theories of subcategorical information maintenance. Instead, we need to mathematically formalize these theories and fit them quantitatively to behavioral data.
The goal of the present study is to develop formal models of subcategorical information maintenance and test them in behavioral experiments. We formalize and evaluate a range of plausible listener strategies, from all-or-nothing approaches (like the ideal integration and categorize-&-discard models), to more nuanced strategies. Critically, by mathematically formalizing our models and fitting them quantitatively to behavioral data, we make our theoretical assumptions explicit, in contrast to previous work, which evaluates qualitative data patterns that could (in principle) be compatible with different theories.
First, we describe the formalization of each model, then present four behavioral experiments against which we fit our computational models. Figure 3 shows the general structure of the modeled task, closely following Connine et al. (1991). Listeners hear sentences that contain a target word whose onset varies acoustically between /t-d/ (by manipulating voice-onset time, VOT); additionally, a subsequent word in the sentence biases toward a particular interpretation of the target word. The listeners’ task is to categorize the target word.
Figure 3: General design of stimuli for all experiments in this article.
![]()
3. Experimental methods
We fit our models to four previously conducted behavioral experiments in our lab that used the same general paradigm (see Figure 3). Experiment 1 was previously reported in Bicknell et al. (2025) as Experiment 2; Experiment 3 was reported in Bushong and Jaeger (2019) as the “high-conflict” group. Experiments 2 and 4 have not been previously reported.
Our experimental materials, full datasets, and analysis scripts can be found in our GitHub repository at https://doi.org/10.5281/zenodo.15237589.
3.1 Participants
Participants were recruited from Amazon Mechanical Turk. Each experiment took approximately 30 minutes to complete and subjects were compensated $3.00 for their participation in the experiment.7 48 participants were recruited for Experiments 1—2, and 60 were recruited for Experiments 3—4. All experiments were approved by the University of Rochester Research Subjects Review Board (RSRB).
3.2 Materials
Our experiments are inspired by the paradigm first introduced by Connine and colleagues (Connine et al., 1991). Table 1 shows an example sentence item. Following Connine and colleagues, we manipulated context (tent-biasing vs. dent-biasing), distance (near, 3 syllables vs. far, 6–9 syllables), and voice-onset time (VOT, the acoustic cue distinguishing /t/ from /d/; 6 continuum steps in each experiment). For the purposes of the present work, we do not evaluate any differences between context distance conditions, though this is an important avenue for future research to explore.
Table 1: Example sentence item in each biasing context and distance condition.
| Subsequent Context |
Distance |
Sentence |
| Tent-biasing |
Near (3 syllables) |
When the [t/d]ent in the forest was well camouflaged, we began our hike. |
| Dent-biasing |
Near (3 syllables) |
When the [t/d]ent in the fender waswell camouflaged, we sold the car. |
| Tent-biasing |
Far (6–9 syllables) |
When the [t/d]ent was noticed in theforest, we stopped to rest. |
| Dent-biasing |
Far (6–9 syllables) |
When the [t/d]ent was noticed in thefender, we sold the car. |
Each participant heard seven sentence frames in each of the context, distance, and VOT condition combinations, resulting in a total of 168 sentences in each experiment.8
3.3 Procedure
Participants were instructed to listen to the sentence and report whether they heard the word tent or dent. Between experiments, we manipulated whether participants could make a response only after they had heard the entire sentence (“forced-response”), or were permitted to respond anytime during the sentence stimulus (“free-response”).
We chose this comparatively simple paradigm because it allows a clear linking function between the input (acoustic cues and subsequent context) and listeners’ categorization decisions (for more discussion, see 2.1.1). By contrast, the link between subjective probabilities and more complex measures, such as fixation latency in visual-world eye-tracking experiments (Brown-Schmidt & Toscano, 2017; McMurray et al., 2009), or MEG responses (Gwilliams et al., 2018), is less well understood. This rich temporal information has the potential to give us additional insight into the mechanisms listeners use when integrating incoming information, but it is not necessary to answer the basic question we seek to address in the current work.
3.4 Acoustic manipulation
We created a continuum between /t/-/d/ by following the procedure of previous studies (Bicknell et al., 2025; Connine et al., 1991). From our recordings of the full sentence stimuli, we took one recording of dent with a relatively short VOT (10 ms) and one recording of tent with a relatively long VOT (85 ms). Then, we replaced the /d/ portion of the dent recording by successively replacing more and more portions of the /t/ in tent (i.e., 15 ms VOT was created by taking the closure and burst of the /t/ recording plus 15 ms of VOT and pasting this onto the ent portion of the dent recording). The continuum created by this process then replaced the original target words produced in the full sentence recordings.
Experiments 1–2 use the same stimulus set used in Bicknell et al. (2025), and we used the same VOT steps as reported in that study. For Experiments 3–4, we developed a new stimulus set with an expanded set of sentence frames. We used the same VOT manipulation process on these stimuli; after stimulus creation, we conducted a norming study in order to choose the VOT points we would present to participants. The full details of the norming study are presented in SI §2 at the GitHub repository for this study.
3.5 Data exclusions
Following previous work using this paradigm (Bicknell et al., 2025; Bushong & Jaeger, 2019), participants were excluded from data analysis if they showed no effect of VOT, as defined by significance of a VOT coefficient in a simple logistic regression fitted to each subject. This resulted in the exclusion of 8, 11, 9, and 12 participants, respectively. For the free-response experiments, we removed trials where participants responded before hearing the biasing subsequent context (defined as 200 ms after context word offset, to account for motor planning). See Table 2 for the number of observations remaining for each experiment after exclusions.
Table 2: Overview of each experiment after data exclusions. For the free-response experiments, we removed trials where participants responded before subsequent context, resulting in many fewer trials than their forced-response counterparts.
| Experiment |
Response Type |
# Participants |
# Observations |
VOT Steps |
| Experiment 1 |
Forced-Response |
40 |
6,720 |
10, 40, 50, 60, 70, 85 |
| Experiment 2 |
Free-Response |
37 |
3,470 |
10, 40, 50, 60, 70, 85 |
| Experiment 3 |
Forced-Response |
51 |
8,568 |
10, 30, 35, 40, 50, 85 |
| Experiment 4 |
Free-Response |
48 |
4,723 |
10, 30, 35, 40, 50, 85 |
3.6 Model fitting and comparison
We employed Bayesian non-linear mixed-effects regression to test each of our formal cognitive models. The advantages of this approach are (i) we can directly fit the equations derived above for each of the models, rather than relying on testing qualitative predictions (like patterns of significant results), and (ii) the Bayesian approach allows us to derive measures of evidentiary support based on posterior predictive accuracy. To implement these models, we used the nonlinear formula feature of the brms package in R (Bürkner et al., 2017; R Core Team, 2016).
We follow common practice and use weakly regularizing priors to facilitate model convergence. For fixed effect parameters, we use Student priors centered around zero, with a scale of 2.5 units (following Gelman, 2008) and 3 degrees of freedom. For random effect standard deviations, we use a Cauchy prior, with location 0 and scale 2, and for random effect correlations, we use an uninformative LKJ-Correlation prior, with its only parameter set to 1 (Lewandowski et al., 2009), describing a uniform prior over correlation matrices. Each model was fit using four chains, with 1,000 post-warmup samples per chain (after thinning to every 4th sample to reduce auto-correlations), for a total of 4,000 posterior samples for each analysis. Each chain used 2,000 warmup samples to calibrate Stan’s No U-Turn Sampler. All analyses reported here converged (e.g., all
1 ≤ R^s ≪ 1.01
\documentclass[10pt]{article}
\usepackage{wasysym}
\usepackage[substack]{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage[mathscr]{eucal}
\usepackage{mathrsfs}
\usepackage{pmc}
\usepackage[Euler]{upgreek}
\pagestyle{empty}
\oddsidemargin -1.0in
\begin{document}
\[
1\; \le \;\hat Rs\; \ll \;1.01
\]
\end{document}
![]()
).
To compare models against each other, we used the Watanabe-Aikake Information Criterion (WAIC, also known as Widely Applicable Information Criterion; Watanabe & Opper, 2010). The WAIC is a measure for the comparison of non-nested models. It is an approximation of Bayesian leave-one-out (LOO) cross-validation, which provides a measure of a model’s predictive accuracy – specifically, its estimated log predictive density (elpd; Gelman et al., 2014; Watanabe & Opper, 2010). LOO is very computationally intensive, requiring re-fitting of the same model many times. Considering the complexity of our models, refitting each model to thousands of observations for every experiment is computationally infeasible.
WAIC saves on this expensive computation by starting with a biased estimate of a model’s elpd (based on its within-sample predictive accuracy), and correcting for its effective number of parameters. This is particularly important in our case, because several of our models have the same number of fitted parameters, but have a higher effective number of parameters (compare the ideal integration and ambiguity-dependent models). We chose the WAIC, as opposed to other information criteria, because it averages over the posterior density of the model, rather than relying on point estimates. This makes the WAIC useful in evaluating mixed-effects models like ours, which contain many parameters that may result in singular estimates (Gelman et al., 2014). Continuing forward, we will refer to the WAIC-estimated elpd as elpdwaic.
There is no general rule of thumb for what differences in elpdwaic between models constitute evidence for a difference. One proposal by Vehtari9 is 5 times the standard error (SE) of the difference – 2.5 SEs, to cover the 95% interval on the difference, and multiplication by 2, since this is the upper limit on the error of the 99% interval estimated by Bengio and Grandvalet (2004). For our purposes here, we will classify 2.5 SE <elpdwaic diff < 5 SE as weak evidence, and elpdwaicdiff>5 SE as strong evidence.
3.7 Assessing individual differences
Recent studies of cue integration in spoken word recognition have increasingly noted that there is sizable individual variability in cue use and weighting (Crinnion et al., 2024), including in some of our recent work using this paradigm (Bushong & Jaeger, 2025). Thus, it is possible that the best-performing models fitted to an entire experiment might not accurately characterize any particular individual subject. The inclusion of random effects over subjects mitigates this issue slightly, but is inadequate for assessing whether different listeners use wholly different strategies.
To characterize possible individual differences in listener strategies, we fit each of our five models to each individual participant across the four experiments. After this process, we excluded from further analysis any subject for whom at least one model did not converge (which we define as at least one
R^ ≥ 1.01
\documentclass[10pt]{article}
\usepackage{wasysym}
\usepackage[substack]{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage[mathscr]{eucal}
\usepackage{mathrsfs}
\usepackage{pmc}
\usepackage[Euler]{upgreek}
\pagestyle{empty}
\oddsidemargin -1.0in
\begin{document}
\[
\hat R\; \ge \;1.01
\]
\end{document}
![]()
or ≤0.99, a slightly looser criterion than our standard for the aggregate models). We then conducted the elpdwaic model comparisons within each individual participant. We calculated which model was the best fit for each subject and the degree of evidence for that model over the next-best-fitting model (defined, as above, as a difference in elpdwaic of >2.5 SE for weak evidence >5 SE for strong evidence).
4. Results
At the whole-experiment level, the model comparisons yielded strikingly similar results across all experiments. Models with subcategorical information maintenance always outperformed models without subcategorical information maintenance; in fact, there was strong evidence against the categorize-&-discard, categorize-discard-&-switch, and context-only models, compared to the ideal integration and ambiguity-dependent models, in twenty-three out of twenty-four comparisons across the experiments. The ideal integration model was the best-fitting across the board, strongly outperforming the ambiguity-dependent model in Experiments 1–2, and weakly outperforming it in Experiments 3–4. For full pairwise model comparisons for each experiment, see Table 3.
Table 3: Pairwise comparison of model fits (elpdwaic) for Experiments 1–4. Each cell shows the fit difference and the standard error of the difference in parentheses. Negative values indicate that the model listed in the row is a better fit than the model in the column (i.e., the top left cell shows the ideal integration model is a better fit than the ambiguity-dependent model for Experiment 1). Italicized cells indicate weak evidence for a difference (elpdwaic difference > 2.5 SEs), with bolded cells indicating strong evidence (difference > 5 SEs).
| Experiment 1 |
ambiguity |
cat.-discard |
cat.-discard-switch |
context-only |
| ideal |
–25.6 (4.6) |
–55.6 (9.9) |
–1009.7 (42) |
–2415.4 (52.4) |
| ambiguity |
|
–30 (7.9) |
–984.2 (43.9) |
–2389.8 (53) |
| cat-discard |
|
|
–954.1 (45.5) |
–2359.8 (53.4) |
| cat-discard-switch |
|
|
|
–1405.6 (33) |
| Experiment 2 |
ambiguity |
cat.-discard |
cat.-discard-switch |
context-only |
| ideal |
–30.7 (5.5) |
–187.3 (17.8) |
–294.8 (25.3) |
–861 (33.9) |
| ambiguity |
|
–156.6 (16.7) |
–264.1 (27.2) |
–830.3 (34.3) |
| cat-discard |
|
|
–107.5 (30.9) |
–673.7 (38.9) |
| cat-discard-switch |
|
|
|
–566.2 (22.9) |
| Experiment 3 |
ambiguity |
cat.-discard |
cat.-discard-switch |
context-only |
| ideal |
–26.3 (6.6) |
–180.3 (17.6) |
–1481.4 (42.7) |
–2662.2 (57.1) |
| ambiguity |
|
–154 (16.4) |
–1455.2 (44.9) |
–2635 (57.6) |
| cat-discard |
|
|
–1301.1 (48) |
–2481.9 (59.5) |
| cat-discard-switch |
|
|
|
–1180.8 (44.3) |
| Experiment 4 |
ambiguity |
cat.-discard |
cat.-discard-switch |
context-only |
| ideal |
–17.1 (5.8) |
–130.9 (15.6) |
–579.5 (33.2) |
–1187.5 (42.8) |
| ambiguity |
|
–113.8 (14.9) |
–562.4 (35.1) |
–1170.4 (42.7) |
| cat-discard |
|
|
–448.6 (38.5) |
–1056.7 (45.4) |
| cat-discard-switch |
|
|
|
–608.1 (32.4) |
To illustrate the fit of the different models to listeners’ responses, we visualize the predictions of all models for Experiment 2 in Figure 6.10 It is clear from these fits why the a priori plausible categorize-discard-&-switch model (and its more extreme counterpart, the context-only model) performed so badly: the model predicts quite a shallow effect of VOT, which does not fit well to the relatively steep average slope we observe in behavior. By contrast, the ideal integration and ambiguity-dependent models fit the VOT effect quite well, while also explaining the presence of the context effect.
Figure 6: Predictions of the five models fit to Experiment 2 in proportion space (left panel), log-odds space (center panel), and context effect predictions (right panel). Point ranges in the left panel show means and bootstrapped 95% confidence intervals over empirical by-subject means. Dashed lines and shaded regions are mean and 95% highest-density continuous interval (HDCI) of model predictions, drawn from 1,000 random posterior samples.
![]()
4.1 Individual results
Of 176 participants, at least one model failed to converge for 25, leaving us with 151 participants who had analyzable results. The results of the model comparisons are summarized in Figure 7. Unlike the models fit to whole experiments, the results for individual participants were less clear. For every participant, the best-fitting model was not statistically distinguishable from the next-best-fitting model (i.e., elpdwaic difference <2.5 SE). Numerically, for most participants, the best-fitting model was the ideal integration model (62, 41% of participants), followed by categorize-&-discard (61, 40.4%), ambiguity-dependent (14, 9.3%), categorize-discard-&-switch (13, 8.6%), and context-only (1, .6%).
Figure 7: Summary of models fit to individual participants. Each panel represents a model, and each position on the y-axis indicates the model it is compared against. Each point represents an individual subject. The position on the x-axis is the degree of evidence for the model represented by the panel. Shaded regions indicate degree of evidence for or against the model (gray: inconclusive evidence, light green/light red: weak evidence for/against, green/red: strong evidence for/against). Note that there was no subject for whom the best-fitting model performed significantly better than the next-best-fitting model. So while, for example, there are many instances of the ideal integration model being a significantly better fit than either the ambiguity-dependent or categorize-&-discard model (see top-left panel), it was never the case that the model was a significantly better fit than both of those models within an individual participant.
![]()
5. General discussion
There is a substantial body of work that seeks to answer the question of whether listeners are able to maintain subcategorical information about previous input (Bicknell et al., 2025; Brown-Schmidt & Toscano, 2017; Connine et al., 1991; Falandays et al., 2020; Ganong, 1980; McMurray et al., 2009; Szostak & Pitt, 2013; Zellou & Dahan, 2019, inter alia). The inferences made by these studies have rested on the assumption that observing effects of both initial acoustic input and later contextual information on behavioral responses constitutes evidence that listeners have maintained gradient subcategorical information about prior input. While some studies have proposed conceptual cognitive models that can be compared to behavior (Bicknell et al., 2025; Connine et al., 1991), there has been no concerted effort to formalize and quantitatively test these alternatives.
Here, we formalized five cognitive models that allow us to distinguish different kinds of information maintenance using results from perceptual categorization studies. Two of these models, ideal integration and ambiguity-dependent, were based on prior conceptual proposals in the literature (Bicknell et al., 2025; Connine et al., 1991). We introduced three additional models that assume listeners do not maintain any uncertainty about prior input after initial word recognition: the categorize-&-discard models and context-only model. The categorize-discard-&-switch is a novel contribution to this literature – to our knowledge, such a cognitive process has not been proposed before to explain subsequent context effects. At first blush, this new model seemed to provide an alternative explanation for behavioral patterns that reflect both early and later cues: if listeners simply switch their categorizations when later information conflicts with initial categorizations, one would expect this pattern.
The quantitative comparison of the competing models yielded strikingly consistent results across experiments: the ideal integration models always outperformed the four non-ideal models. The ambiguity-dependent model was also a strong contender, but it always patterned after the ideal integration models, and in two of our four experiments, the evidence against it in favor of the ideal integration model was strong. The three models that assume listeners discard subcategorical information were systematically worse, with our novel proposal, the categorize-discard-&-switch, patterning consistently second-worst. On the whole, these results very strongly suggest that listeners are capable of maintaining subcategorical information about input over long perceptual timescales (3–9 syllables).
Since there may be variability in strategies between participants, we also assessed model fits within individuals. These results were less conclusive, because within each participant, the best-fitting models were statistically indistinguishable from the next-best fit, making it difficult to draw firm conclusions. The qualitative pattern of results, however, showed some divergence from the models fit to whole datasets: while the ideal integration model was the best-fitting model for a plurality of participants, the categorize-discard-&-switch model was a close second, and the ambiguity-dependent model was the best fit for only a small fraction of the total participants (as in the full-experiment results, the categorize-discard-&-switch and context-only models were the worst-performing). We discuss these results further in 5.2.
The failure of the categorize-discard-&-switch model is illustrative of the importance of formalizing and quantitatively testing theories. On its face, it appears to be a plausible competitor to models that assume maintenance of subcategorical information. It also predicts effects of both acoustic and contextual cues across time, and calls into question the assumption in previous work that finding these effects must imply maintenance of subcategorical information (Bicknell et al., 2025; Brown-Schmidt & Toscano, 2017; Connine et al., 1991). When we quantitatively evaluated this model, however, it provided a very poor fit to the data. This work, thus, highlights the importance of directly fitting quantitative predictions of cognitive models to behavioral data. With an eye to the future, we see two major avenues for advancement in this area.
5.1 What kind of subcategorical information do listeners maintain?
While the present work reveals that listeners can maintain gradient representations of previous input, it is unclear what kind of information is contained in these representations. Throughout this paper, we use the general term subcategorical information to refer to any kind of representation of past input that is below the level of a categorical decision. But how detailed these representations are has significant implications for the language processing system. For example, listeners could maintain information about specific cue values over time, which would likely be a highly resource-intensive process. By contrast, listeners may maintain something as general as a probability distribution over possible categories, which would be less resource-intensive, but still sufficient to perform ideal cue integration (under some simplifying assumptions). It is also possible that there is some mixture of representations maintained over different timescales; listeners may maintain fine phonetic detail over limited timescales, moving to uncertainty over categories as more time passes. In this work, we aimed to show that maintenance of some kind of subcategorical information is possible over long timescales, but our paradigm cannot adjudicate between these different types of representations.
This issue is not trivially solvable. In a series of neuroimaging studies, Gwilliams et al. (2018, 2022) use MEG to reveal across time the neural activity of brain regions known to be associated with phonetic processing. In particular, Gwilliams et al. (2022) are able to decode phonetic features from these regions (as subjects listen to natural speech) for modest perceptual distances (∼300 ms). However, these data do not necessarily disambiguate whether listeners have access to more detailed information: indeed, uncertainty about phonetic feature identity may paradoxically lead to worse decoding accuracy (particularly in noisy natural speech), precisely because listeners have access to information more detailed than the binary phonetic feature category level, leading to a higher degree of uncertainty at the category level. Furthermore, listeners could, in principle, make perceptual commitments while continuing to maintain subcategorical detail over time – what pattern of neural responses this would predict is unclear.
There is a second line of work that tackles the problem of representational detail behaviorally, using the perceptual recalibration paradigm. Caplan et al. (2021) find that lexical labeling following exposure to acoustically manipulated words failed to induce perceptual recalibration effects (in contrast to lexical labeling preceding acoustic information). Perceptual recalibration requires that listeners be able to track acoustic cues and re-map them to phonemic categories, so the absence of recalibration suggests listeners do not have access to representations as detailed as acoustic cue values at the time of lexical labeling. However, other work using a different accent adaptation paradigm has found effects with delayed lexical labeling (Burchill et al., 2018).
Given the results from the above lines of work, we find it likely that listeners in our studies maintain a more general uncertainty, such as a probability distribution over phonemic categories, rather than a more detailed representation of acoustic feature values. However, we cannot rule it out, and testing these questions is very tricky. Careful model-building and highly controlled experiments are likely to be key to future work in this area.
5.2 How general is subcategorical information maintenance?
How generalizable our results are to naturalistic language comprehension depends on two factors: (i) how well our models, which are fit to entire experiments, capture what any particular individual listener does; and (ii) whether our task is reflective of typical language use.
Psycholinguistic experiments generally assume a modal language user – that is, we operate under the assumption that most humans share the same fundamental language production and comprehension processes, with some limited exceptions (for recent discussion of this issue, see McMurray et al., 2023). This is in contrast to an approach which views psycholinguistic processes as fundamentally variable and under which each individual’s behavioral patterns are considered. Thus, it is important in our work to at least begin to address to what degree our average results (here, the models fit to entire experiments) are sufficient descriptors of individual participants. To tackle this issue, we fit each of our models to individual participants. Unfortunately, given the small amount of data at the individual level, the results were inconclusive. The most notable result to us was that the categorize-&-discard model performed much better on an individual level than at the aggregate level; in particular, it was the best-fitting model for nearly the same number of participants as the ideal integration model. The ambiguity-dependent model, by contrast, performed much worse for individuals than in the aggregate models. To some degree, this is likely driven by the strong VOT effects present within subjects. However, we want to avoid speculating about these results too much – there were no participants who had a statistically clear best-fitting model. Ultimately, to address the question of whether there is individual variation in the mechanisms of subcategorical information maintenance, we need a different approach. Future work should collect significantly more data per participant and fit one model that implements a mixture over the base models – in this way, one could get an estimate of the degree to which each model captures variation in strategies between individuals. Such an approach could also help us understand whether there may even be changes in strategy between trials.
A second concern about the generalizability of the present work concerns task. The experiments presented here use highly predictable, repetitive stimuli: participants are always asked about the first phoneme of the third word of the sentence, which is predictably followed by additional sentence context 3–9 syllables later. Thus, it is worth considering to what extent our results here reflect real, day-to-day language comprehension, versus a learned task strategy that develops based on exposure to our particular stimuli. To some degree, we can address this question empirically. If participants in our studies show effects of VOT and context from the very beginning of the experiment, this would constitute evidence (albeit limited) that use of these time-disjoint cues is not a task-dependent strategy that requires repeated exposure to our stimuli. To that end, we conducted additional trial analyses (presented in SI §3 at the GitHub repository for this study) on each of our experiments. We find that there is strong evidence for effects of both context and VOT from the very first trial of the experiment. Of course, this does not constitute evidence that there are no task-specific adaptations that may result in behavioral patterns that are not present in natural language comprehension. To mitigate this problem, future work using a paradigm like ours should take steps to draw listeners’ attention away from critical manipulations, including introducing filler items, probing alternative words in the sentences, and developing a larger set of sentence items to reduce repetition.
Even if we take as a given that the effects we find here are not task-dependent, it is worth asking whether subcategorical information maintenance is a static, unchanging mechanism of language comprehension, or is a strategy that is malleable and under listeners’ control. To this point, we have used these terms interchangeably, but they imply quite different things about the language processing system. Consider what it would mean for subcategorical information maintenance to be a general mechanism: it would imply that listeners always maintain subcategorical representations about every segment of speech input on an indefinite timescale – the memory demands this process would imply seem immense (and, to some degree, contradictory to the general principle of incrementality in language processing; Christiansen & Chater, 2016). Some work has begun to test whether subcategorical information maintenance can change across time; for example, Bushong and Jaeger (2025) propose that the expected utility of context modulates whether listeners maintain subcategorical information; they find that when sentence context is less informative, listeners subsequently down-weight its use in a spoken word recognition task. Some lexical garden-path studies have also started to investigate whether individual listeners’ perceptual abilities modulate acoustic-lexical cue integration (Kapnoula et al., 2021). And, as we mentioned above, extensions to our current work to model mixtures of strategies may also begin to elucidate these processes. As of yet, however, there are no concrete theories of how perceptual, attentional, and memory processes together play a role in maintaining and updating linguistic representations of uncertainty in real time spoken language understanding.11 We see this as a fruitful area for future work to address.