While ambiguous attachment sentences such as the ones used in Swets et al. (2008) are often complex and fairly unnatural, ambiguous pronominal references are found in everyday language and in less complex sentence constructions. A study by Stewart et al. (2007) examined the processing of ambiguous pronouns using a self-paced reading task. Their experimental design manipulated the ambiguity of the pronominal reference and the processing depth. Participants were presented with sentences that contained two referents and a personal pronoun that was either ambiguous, as in (1), or could refer to only one of the referents due to the presence of disambiguating gender information, as in (2) and (3).

Depth of processing was manipulated through offline comprehension questions. To probe shallow processing, participants were asked questions requiring a yes/no answer on only a third of the target sentences (e.g., Did Paul lend Kate the CD?) and a third of the filler items. Half of these questions targeted the information content of the main clause and half targeted the information content of the embedded clause. In the deep processing condition, questions on all experimental items and fillers targeted the information content of the embedded clause, and, hence, required resolution of the pronoun (e.g., Who left for the holidays?).

The results showed that in the shallow processing condition the total sentence reading times in the different conditions did not differ from each other (although numerically the ambiguous condition was read fastest). In the deep processing condition, ambiguous sentences were read more slowly than first and second character reference sentences (i.e., an ambiguity penalty or disadvantage). Stewart et al. (2007) proposed that readers invest processing resources to interpret the ambiguous pronoun when engaged in deep processing only, and not under shallow processing.

A second experiment, in which participants read sentences in a cumulative word-by-word self-paced reading paradigm paired only with shallow processing comprehension questions, again showed no effect of ambiguity on response times to the pronoun region and to the word after the pronoun in the first sentence. However, the critical sentences in this experiment were followed by a second sentence in which the ambiguous pronoun from the first sentence was disambiguated (e.g., for (1): He went to the Bahamas and sent Rick a postcard from the hotel, for which disambiguation occurs at the repeated name of the character to whom the pronoun does not refer). In the second sentence, a significant effect of ambiguity was found for the disambiguating region. The region that contained the repeated name was read more slowly in the ambiguous conditions than in the unambiguous conditions.

Based on the results of both experiments, Stewart et al. concluded that, under deep processing, readers attempt to assign a referent to the ambiguous pronoun even in a context where no disambiguation is available to indicate which assignment is correct. In contrast, under shallow processing, ambiguous pronouns would ultimately be interpreted but only when the reader is able to do this with certainty. The authors further argued that readers delayed the resolution of the ambiguous pronoun until disambiguating information was available, such that initial underspecification becomes more specified when disambiguating information is encountered, even under shallow processing.

In contrast to the results by Stewart et al. (2007), Grant et al. (2020) reported an ambiguity advantage for both structural and referential ambiguities in two experiments measuring eye movements during reading. We focus here on the results of the referential ambiguity study, for which the conditions are shown in (4) and (6).1

In the ambiguous condition, as in (4), both nouns (e.g., brother and waiter) were possible referents, while in the unambiguous conditions, as in (5) and (6), only one of the nouns was a possible referent. Rather than using personal names as in Stewart et al. (2007), gender information was based on a mix of lexical-semantic cues (e.g., brother), morphological cues (e.g., waitress), and stereotypical bias (e.g., beautician). The results showed an overall ambiguity advantage: similar to the results for structural ambiguities, Grant et al. reported significantly shorter reading times for ambiguous pronouns compared to unambiguous pronouns in go-past time (defined as the sum of all fixations from first entering a region until leaving it to the right).

To rule out that the ambiguity advantage was due to the lack of salience for the referents, a second experiment was run, in which, similar to the design in Stewart et al. (2007), the main clause subject and object formed the potential referents. The referents were either definite descriptions that had definitional gender (e.g., king), or were proper names that were gender unambiguous (e.g., Olivia). The conditions were otherwise similar to those in the first experiment, as illustrated in (7) to (9).

The results again revealed an ambiguity advantage effect in three out of five reading measures (first pass, go-past, and total time) in the post-critical region, with lower reading times for the ambiguous condition compared to either unambiguous condition. Grant et al. (2020) interpreted these results along the lines of an Unrestricted Race Model (cf. Van Gompel, Pickering, & Traxler, 2000) in which different referents ‘race’ to be selected as an antecedent for a pronoun, with the one that becomes available most quickly being adopted as the referent. If the antecedent that wins turns out to not match the pronoun’s gender features, the reference of the pronoun needs to be re-evaluated. The slower reading times in the unambiguous conditions reflect a penalty due to reanalysis: on some portion of trials, the processor selects the ultimately incorrect reference, which must then be reanalyzed. Such reanalysis never occurs for ambiguous trials as the parser will never be revealed to be wrong. Hence, reading times are predicted to be faster for ambiguous sentences compared to unambiguous ones.

In sum, Stewart et al. (2007) showed an ambiguity penalty when all sentences were followed by a targeted comprehension question in the deep processing condition (Experiment 1), and no significant difference between ambiguous and unambiguous sentences when 4/24 experimental trials were followed by a targeted comprehension question in the shallow processing condition (Experiment 1 and 2). Grant et al. (2020) showed ambiguity advantages when 12/30 sentences (Experiment 1) and 12/24 sentences (Experiment 2) were followed by a targeted comprehension question. However, these studies cannot be compared directly as different methodologies were used: Stewart et al. (2007, Experiment 1) analyzed whole-sentence self-paced reading times and Grant et al. (2020) analyzed eye-tracking measures while reading.

There is some evidence suggesting that a less demanding task may lead comprehenders to underspecify certain syntactic relations. Ferreira and Yang (2019), for example, discuss how readers may flexibly adjust their reading strategies depending on their goals, and that the ultimate comprehension level may depend on reader or listener engagement or motivation. The results by Stewart et al. (2007) are suggestive in this respect, showing an ambiguity penalty only under deep processing conditions, and not under shallow processing. Swets et al. (2008), focusing on structural ambiguities, further showed that the processing of structurally ambiguous sentences is sensitive to depth of processing. They examined structural ambiguities in the form of sentences with a relative clause that could attach to one of two sites, as in (10), and sentences in which the relative clause occurred with a disambiguating reflexive pronoun, as in (11) and (12).

Depth of processing was manipulated through the number and difficulty of the comprehension questions that followed the sentences. The results showed an ambiguity advantage only when comprehension questions were sparse and superficial, but an advantage for disambiguated low attachment sentences when comprehension questions were frequent and targeted the ambiguity.

Swets et al. argued that the comprehension system creates an underspecified representation when there is no basis on which to fully commit to an attachment decision in an ambiguous sentence. In other words, the syntactic representation in the globally ambiguous case may remain underspecified under shallow processing, but not under deep processing (see also Ferreira, Bailey, & Ferraro, 2002). Building on the results by Swets et al. (2008), Logačev and Vasishth (2016a) provided further evidence that, given a sufficiently high degree of task difficulty, sentences with attachment ambiguities are read more slowly than unambiguous sentences.

The goal of the present study was to examine to what extent the processing of referential ambiguities is modulated by processing depth, as manipulated through comprehension questions. If deep processing conditions result in an ambiguity penalty, and shallow processing conditions in an ambiguity advantage, this could potentially explain the discrepancy in earlier results on ambiguous referential processing (Grant et al., 2020; Stewart et al., 2007). To examine the effects of processing depth on the processing of referential ambiguities, we ran a conceptual replication of the first experiment in Stewart et al. (2007).

The materials consisted of Dutch sentences similar to the English sentences used in Stewart et al. (2007) and in Grant et al. (2020, Experiment 2). The sentences were ambiguous or disambiguated through gender features. In the ambiguous sentences, as in (13), the pronominal reference could refer to either character. In the disambiguated sentences, the pronoun could only refer to the gender-consistent referent, which was the first-mentioned character in the first character reference (14), or the second-mentioned in the second character reference (15). Sample items, including translations, are given below (a full stimulus list is provided as supplemental material; see Data Accessibility Statement).

Processing depth was manipulated through the type of comprehension questions that participants were asked to answer. Sentences in the shallow processing condition were followed by questions that probed the information content of the main clause (e.g., Wie leende het boek aan Gijs? ‘Who lent the book to Gijs?’) and which did not require the participant to resolve the pronominal reference in the embedded clause. Sentences in the deep processing condition were followed by a question that probed the information content of the embedded clause and that could not be answered without resolving the pronominal reference (e.g., Wie ging op vakantie? ‘Who went on holiday?’). Note that we manipulated the required depth of processing of the embedded clause, not of the entire sentence. We return to this point in the Discussion.

If depth of processing modulates ambiguity resolution, we predict that participants show an ambiguity advantage under shallow processing conditions, but an ambiguity penalty under deep processing conditions. Hence, under deep processing, we predict that reading times for ambiguous pronouns would be longer than those for unambiguous pronouns as readers will attempt to resolve the reference. We predict shorter reading times for ambiguous pronouns under shallow processing as ambiguous pronominal references may remain underspecified under certain conditions, as was argued for English by Stewart et al. (2007).

We also examined question responses and question answering times to shed further light on the way in which pronominal resolution occurs, testing whether readers used a heuristic that favored one referent over the other. Referents in subject position are often argued to hold a privileged status, which predicts that the first-mentioned character may be more often co-indexed with the pronoun, even in the absence of disambiguating information (e.g. Arnold, 1998; Arnold, Eisenband, Brown-Schmidt, & Trueswell, 2000; McDonald & MacWhinney, 1995; Stewart et al., 2007; Van Rij, Van Rijn, & Hendriks, 2013).

Our study improved on the original design in Stewart et al. (2007, Experiment 1) in several ways, providing a different methodology, a larger sample size, and up-to-date statistical analyses. We used a word-by-word self-paced reading task (Just, Carpenter, & Woolley, 1982), which allowed us to localize the effects and obtain a more detailed picture of the time course of processing, compared to an analysis based on whole-sentence reading times in Stewart et al. (2007). In addition, we used a non-cumulative (moving window) paradigm, instead of a cumulative reading task. The cumulative paradigm has been shown to be problematic for studying on-line parsing, as participants develop a reading strategy in which they reveal several segments of a stimulus at a time and then read them all at once, or even wait to process a sentence until it is entirely revealed (Ferreira & Henderson, 1990; Just et al., 1982).

Finally, given the fact that the replication crisis has surfaced also in the pronoun interpretation literature (see e.g., Chow, Lewis, and Phillips 2014’s failed attempts to replicate the findings in Badecker and Straub 2002), we included more experimental items. While the original study included 24 target sentences, we included a total of 60 target sentences. We performed an a-priori power analysis to determine the sample needed to observe an effect with at least 80% power (see below). While it is possible that the lack of a significant ambiguity advantage in the shallow processing condition in Stewart et al. (2007, Experiment 1) was due to a lack of power, our experiment should have appropriate power to detect the effect. We further used linear mixed-effects models to analyze the data, rather than traditional ANOVAs.

Participants were 156 non-dyslexic native speakers of Dutch who reported having normal or corrected-to-normal vision (mean age = 24.9; sd = 4.36). Participants were recruited from the participant pool at the Max Planck Institute for Psycholinguistics, and anyone who took part in the norming study was excluded from participation. Participants received a payment of €6 for their participation. Informed consent was obtained from each participant. The study was approved by the ethics board of the Faculty of Social Sciences of Radboud University.

The number of participants was determined through an a-priori power analysis conducted in G*Power (Faul, Erdfelder, Lang, & Buchner, 2007), which suggested that a sample size of 39 participants (13 per list) would give us at least 80% power to observe a main effect of sentence type with an effect size f of 0.15 or larger (which was calculated based on the results in Stewart et al. 2007, Experiment 2).2 To account for the interaction with processing depth, we included twice as many participants per cell, hence, including four times as many participants, leading to a total of 156 participants.

As in Stewart et al. (2007), we used a 2 × 3 mixed design, consisting of a between-participants factor ‘depth of processing’ of the embedded clause with two levels (shallow processing, deep processing), and a within-participants factor ‘sentence type’ with three levels (ambiguous, first character reference, second character reference). The materials consisted of 60 target sentences, with an ambiguous version, a first character reference version, and a second character reference version each. Because self-paced reading experiments typically show spill-over effects, in that difficulty induced by a given word slows down the reading times of subsequent words, the sentences were created such that the word immediately following the pronoun (PRONOUN+1) was always a closed class item (a preposition or determiner) and the word after that (PRONOUN+2) was an open class item. We made sure that the frequency of the open class items was not extremely low (i.e., >1 per million words) or extremely high (≤100 per million). The mean word frequency of the PRONOUN+1 items was 17862.88 per million words (sd = 8345.13); the mean frequency of PRONOUN+2 items was 33.65 per million (sd = 29.28). Word frequencies were retrieved from the SubtLex-NL database (Keuleers, Brysbaert, & New, 2010).

We ensured that the critical regions of interest in the sentences in the different conditions were identical. To do so, we switched the order of the female and male proper names in the first and second character reference sentences (e.g., Abel lent Zoë and Zoë lent Abel) which ensured the use of the same pronoun across conditions. All of the critical sentences occurred with the third person masculine personal pronoun hij ‘he’, as the Dutch feminine pronoun zij or ze ‘she’ can also refer to the third person plural pronoun ‘they’, in which case a sentence as in (13) would be disambiguated only at the sentence-final verb. Dutch stereotypically male or female proper names were selected from the Dutch First Name database (Nederlandse Voornamenbank3). The different versions (ambiguous, first character reference, second character reference) of each experimental item were rotated across three lists, so that each participant saw every item only once. We further included 30 filler sentences that also consisted of a main and an embedded clause, but differed from the experimental items in that they only contained one character. Twenty of the filler sentences occurred with the feminine personal pronoun ze ‘she’ and ten occurred without any personal pronoun or with the masculine personal pronoun hij ‘he’. Filler items were identical for each of the lists.

To ensure that participants could not use plausibility information to guide the interpretation of the pronoun in the critical sentences, we conducted a norming study to establish that the first and second character reference interpretations were equally plausible. The norming materials consisted of 70 unambiguous sentences. Following the procedure reported in Stewart et al. (2007), a first character reference and a second character reference version were included for each sentence. The norming study consisted of two lists, with first and second character references of the same sentence occurring on different lists. Each list contained equal numbers of first and second character reference sentences. In addition, we included 20 sentences that were implausible on the basis of pragmatic or world knowledge (e.g., Paul lachte Sophie uit toen hij van zijn fiets viel ‘Paul laughed at Sophie when he fell off his bike’ and Rick stuurde een onderzeeër om de bosbrand te blussen ‘Rick sent a submarine to extinguish the forest fire’). The norming study was run online, using an online tool for web experiments (Frinex) developed by the technical group of the MPI.

A total of 40 native speakers of Dutch (n = 20 per list; mean age = 23.28; sd = 2.92) were asked to rate each sentence for plausibility of the event it described (Hoe waarschijnlijk is het dat dit gebeurt? ‘How likely is this happening?’) on a 7-point Likert scale, with 1 corresponding to ‘highly implausible’ (zeer onwaarschijnlijk in Dutch) and 7 to ‘highly plausible’ (zeer waarschijnlijk). Based on the results of the norming study, a set of 60 sentences were selected for which the difference between the mean ratings for the two versions of the same sentence was minimal. The mean difference between the first and second character reference sentences for these 60 sentences was 0.40 (sd = 0.36), with a range of 0 to 1.2. The mean plausibility of the sentences referring to the first-mentioned character was 5.45 (sd = 0.84), and the mean plausibility of the sentences referring to the second-mentioned character was 5.39 (sd = 0.83).

All sentences in the main experiment were followed by a two-choice comprehension question. As described above, these questions served to manipulated the processing depth. In the shallow processing condition, all questions probed the information content of the main clause. Hence, the embedded clause was irrelevant to the comprehension question and participants were not required to resolve the pronominal reference. Sentence (13) above, for instance, was followed by the question Wie leende het boek aan Gijs? ‘Who lent the book to Gijs?’, with answer options Abel (correct) or Zoë (incorrect). To answer this question, participants were merely required to remember who the characters were in the sentence, since only one of the two answer options occurred in the sentence. A third of the sentences were followed by a question that did not mention any of the characters (e.g., Wie werd er uitgelachen? ‘Who was laughed at?’ after the sentence Jordy lachte Marc uit toen hij de grap begreep ‘Jordy laughed at Marc when he understood the joke’), with answer options Jordy (incorrect) and Marc (correct). In these cases, participants needed to remember who did what as expressed in the main clause, but they, crucially, did not need to resolve the pronominal reference in the embedded clause. Half of the shallow processing questions targeted the first-mentioned referent, and half targeted the second-mentioned referent (i.e., Aan wie leende Abel het boek? ‘To whom did Abel lend the book?’).

In the deep processing condition, all sentences were followed by a question that probed the information content of the embedded clause, such that the questions could not be answered without resolving the pronominal reference. Sentence (13) above was paired with the question Wie ging op vakantie? ‘Who went on holiday?’ with answer choices Abel and Gijs. In the ambiguous condition, both answers reflected potential interpretations of the pronoun, while in the disambiguated sentences only one of the two choices was correct (e.g., for (18): Abel (correct) or Zoë (incorrect)).

The experiment was run online, using the same online tool for web experiments as was used for the norming study (Frinex). In the experiment, participants read sentences in a word-by-word self-paced reading paradigm (Just et al., 1982). Participants used their own keyboards to respond. Upon pressing the space bar, a trial began and the first word of the sentence appeared. Participants continued to press the space bar to read each successive word, and as each word appeared, preceding words were masked again. Participants were instructed to read at a natural pace and to make sure they understood what they were reading so that they could answer the comprehension questions accurately. Reading times were measured for each word from the time it appeared on the screen until the space bar was pressed.

After participants read the complete sentence, a comprehension question appeared on the screen with two answer choices, to which participants responded with a key press. Participants were randomly assigned to either the shallow processing or the deep processing condition. In the shallow processing condition, the answers consisted of the correct character name and a character name that had not appeared in the preceding sentence. In a portion of the questions no character was mentioned in the question (e.g., Wie vertelde over het probleem? ‘Who mentioned the problem?’), in which case the two character names from the sentence were given as possible answers. In the deep processing condition, the two character names that had appeared in the preceding sentence were always depicted on the screen as possible answers. The positions of the two answers on the screen were randomized. After answering, participants proceeded to the next trial by pressing the space bar.

We generated twelve pseudo-random orders of item presentation to which participants were randomly assigned. The experimental session was preceded by a set of 8 practice trials. The practice trials were meant to familiarize the participant with the experiment and to mimic the different conditions of the between-participants manipulation of processing depth: in the shallow processing condition, practice trials were followed by questions targeting main clause information, while questions targeted the information content of the embedded clause were used in the deep processing condition.

We recorded reading times (RTs) for each word, and accuracy and question-answering times for the comprehension questions after each sentence. Eight participants were removed for technical reasons (only part of their responses were properly recorded, likely due to connection issues or because they continuously pressed the space bar for at least a portion of the experiment). We then calculated the mean accuracy on the comprehension questions after filler sentences for each participant, and removed one participant with <75% accuracy from further analyses. All remaining 147 participants had a high accuracy on the filler items (on average 99% accurate on the filler items under both deep and shallow processing conditions).

The reading and question-answering time data were analyzed with linear mixed-effects models, using the lme4 package (Bates, Kliegl, Vasishth, & Baayen, 2015) in the R environment (R Core Team, 2016) (version 3.6.0). Post-hoc contrasts were performed using the emmeans package (Lenth, Singmann, Love, Buerkner, & Herve, 2018) and p-values were computed using the Satterthwaite approximation for degrees of freedom, as implemented in the lmerTest package (Kuznetsova, Brockhoff, & Christensen, 2016). We used the optimiser “bobyqa” to help with convergence issues.

Unreasonably short and long latencies (<100 ms and >2500 ms) were excluded prior to analysis, which resulted in a loss of less than 1% of the data. No additional outlier removal was performed. Reading times were log-transformed (natural log) to normalize residuals and reduce the effect of extreme data points. Log-transformed reading times were then regressed against two factors that are known to affect reading times in self-paced reading tasks: word length and the list position of the sentence in the stimulus list (i.e., longer words predict longer reading times and later list positions predict faster reading times; Hofmeister, 2011; Hofmeister & Vasishth, 2014). The model estimating these effects included by-participant slopes of word length and list position. Both word length and list position were log-transformed with a natural logarithm and then centered. The data from all sentences (experimental and filler) were used to produce maximally general estimates (Hofmeister & Vasishth, 2014).

We analyzed the residual log reading times in five separate regions of interest: the pronoun itself (pronoun), the two regions immediately following the pronoun (pronoun+1 and pronoun+2), and the two regions preceding it (pronoun-1 and pronoun-2). The words used in each of these regions were identical across conditions. We included the two words following the pronoun as self-paced reading experiments often show spill-over effects, i.e., effects on words following the region of interest in the sentence. To establish whether there were any differences between the conditions prior to the pronoun, we further included the two words preceding the pronoun: the pronoun-1 region (which was always the first word of the embedded clause) and the pronoun-2 region. Linear mixed-effects models for the different regions of interest were fitted with fixed effects of sentence type and depth of processing and their interactions. sentence type was coded with successive differences contrast coding (using the contr.sdif() function in the mass package; Venables & Ripley, 2002), comparing the differences between the means of the second and first levels (ambiguous vs. first character reference), and the third and second levels (second character reference vs. ambiguous). depth of processing was coded with scaled sum contrasts (or effect coding; –0.5 and 0.5). We fitted the models with the maximal random effects structure justified by the design (Barr, Levy, Scheepers, & Tily, 2013), which included by-participant and by-item intercepts, by-participant slopes for sentence type, and by-item slopes for sentence type, depth, and their interactions. Random slopes were removed when the model did not converge; random slopes correlated above 0.95 were also removed to avoid overfitting. This resulted in models that included by-participant and by-item intercepts and a by-item slope for depth in the pronoun, pronoun+1, and pronoun+2 regions, and models with by-participant and by-item intercepts and by-item slopes for sentence type, depth, and their interactions for the pronoun-1 and pronoun-2 regions.

For the question-answering times, unreasonably high (>10000 ms) and low (<700 ms) response times were removed. This resulted in 0.85% of the response time data being excluded. Response times were log-transformed (natural log) and log response times more than 2.5 standard deviations from the condition mean (per participant) were excluded, resulting in the removal of a further 1.22% of the data. Log-transformed question-answering times were analyzed with linear mixed-effects models that included a fixed effect of sentence type and depth of processing and their interactions. As with the analysis of reading times, sentence type was coded with successive differences contrast coding, comparing the ambiguous condition to the first character reference and to the second character reference conditions. depth of processing was coded with scaled sum contrasts and random intercepts for subjects and items were included.

A summary of the non-transformed reading times across the experimental conditions is given in Table 1. Analyses were conducted on residual log reading times, as shown in Figure 1. Table 2 gives the output of the mixed-effect models.

Throughout the pronoun region and the two regions following it, reading times were strongly affected by processing depth. The shallow processing condition elicited faster average reading times (280 ms on average) than the deep processing condition (342 ms on average) when comparing reading times in the pronoun region and the two spill-over regions. A main effect of depth was found for each of these three regions (pronoun region: β = –0.057, p < 0.001; pronoun+1 region: β = –0.043, p < 0.001; pronoun+2 region: β= –0.037, p = 0.012). When considering the whole-sentence reading times, the deep processing condition also led to longer average RTs (4746 ms) compared to the shallow processing condition (3950 ms). However, the effect of depth was only marginally significant in the region immediately preceding the pronoun (pronoun-1 region: β = –0.026, p = 0.069) and was not significant in the region before that (pronoun-2 region: p = 0.280). This indicates that participants who were asked ‘superficial’ questions in our shallow processing condition read the embedded clause faster than participants who were asked questions that specifically targeted the information content of that embedded clause. While the manipulation of processing depth affected the processing of the embedded clause (which started at the word in the pronoun-1 region), this further shows that it did not necessarily affect the generic processing depth over the entire sentence.

Zooming in on the reading times per region of interest, the results in the pronoun region did not reveal a main effect of sentence type when comparing the ambiguous condition to the first character (p = 0.478) and second character reference (p = 0.893) conditions. Crucial for the aim of the current experiment are the interactions between sentence type and depth. For the pronoun region, these interactions were significant when comparing ambiguous and first character reference sentences (β = –0.030, p = 0.033) and when comparing second character reference and ambiguous sentences (β = 0.031, p = 0.026).4 This suggests that there is an effect of processing depth (or task demand) on the difference in reading times between ambiguous and unambiguous sentences in this region, with shorter reading times for ambiguous sentences under shallow processing but longer reading times under deep processing, which would be in line with earlier findings of an ambiguity advantage. Numerically, the ambiguous condition was read faster under shallow processing compared to the unambiguous conditions. However, post-hoc comparisons (with a correction for multiple comparisons using the Tukey method), revealed no significant differences in reading times between the different sentence types under shallow processing (first-second: p = 0.787; first-ambiguous: p = 0.575; ambiguous-second: p = 0.219) or under deep processing (first-second: p = 0.855; first-ambiguous: p = 0.111; ambiguous-second: p = 0.304).

The results in the pronoun+1 region, which consisted of highly frequent words such as prepositions, showed a difference between the reading times in the ambiguous and first character reference conditions across levels of depth (β = 0.014, p = 0.033).5 This suggests that the reading times for the first character reference sentences were faster compared to the ambiguous sentences when looking at both processing depths. However, and as shown in Table 2, none of the interactions between sentence type and depth were significant.

The pronoun+2 region showed a main effect of sentence type when comparing reading times for ambiguous to the first character reference (β = 0.019, p = 0.008) and the second character reference sentences (β = –0.020, p = 0.006). This shows that, across levels of depth, ambiguous sentences were read more slowly than either of the unambiguous ones, although this effect is probably driven by the deep processing condition as discussed below. The interaction between sentence type and depth was significant when comparing the ambiguous and the first character reference sentences (β = –0.050, p < 0.001) and the ambiguous and second character reference sentences (β = 0.035, p = 0.013).6 This again suggests that the difference in reading times between ambiguous and unambiguous sentences depends on the depth of processing that participants engaged in. Post-hoc comparisons indeed showed that the ambiguous sentences were read significantly more slowly than the unambiguous sentences under deep processing (first-ambiguous: β = –0.044, p < 0.001; ambiguous-second: β = 0.037, p < 0.001). No differences were found between the unambiguous conditions (p = 0.794). The longer reading times for ambiguous sentences reflect an ambiguity disadvantage or penalty under deep processing. Under shallow processing, no significant differences between any of the sentence types were found (first-ambiguous: p = 0.817; first-second: p = 0.704; ambiguous-second: p = 0.980).

Finally, we analyzed the two regions preceding the pronoun which, similar to the regions following the pronoun, were identical across conditions. In the pronoun-1 region, the model revealed a marginally significant interaction when comparing the ambiguous sentences to the first character reference sentences (β = –0.038, p = 0.053), in the same direction as the interaction in the pronoun region. Although this interaction was only marginally significant (in both the full and reduced models) and should be interpreted with caution, this could indicate that participants may have learned over the course of the experiment that an ambiguous pronoun may follow after having seen two stereotypically male proper names in the main clause (as opposed to one male and one female name). No other effects were significant. The model for the pronoun-2 region revealed no significant effects (see Table 2).

The analysis of the question responses showed high accuracy rates across conditions, as summarized in Table 3. When examining answers to questions after ambiguous sentences in the deep processing condition (for which participants had to resolve the pronominal reference), a strong preference to resolve the ambiguous pronominal reference to the first character reference was observed: out of 1460 answers, 1040 (71%) referred to the first character and 420 (29%) to the second character (Figure 2A). This is in line with the consistent finding of a tendency for the referent of the subject or first-mentioned noun phrase to be more accessible than other entities (e.g. Arnold, 1998; Arnold et al., 2000; McDonald & MacWhinney, 1995; Van Rij et al., 2013). It is likely that the fact that referents in subject position are more prominent or salient resulted in this subject bias when resolving pronoun reference.

The analysis of question answering times (see Figure 2B and Table 4 for the complete model output) showed a difference between the ambiguous and the first character reference conditions (β = 0.184; p < 0.001) and between the ambiguous and the second character reference conditions (β = 0.172; p < 0.001) across levels of depth. depth did not result in a significant main effect (p = 0.292). However, the interactions between sentence type and depth were significant and showed a larger difference between ambiguous and first character reference sentences (β = 0.318; p < 0.001) and between ambiguous and second character reference sentences (β = 0.308; p < 0.001) under deep processing compared to shallow processing. Post-hoc comparisons (with a correction for multiple comparisons using the Tukey method) indeed showed no significant differences between sentence types under shallow processing (first-second: p = 0.838; first-ambiguous: p = 0.124; second-ambiguous: p = 0.347). Under deep processing conditions, however, participants took significantly longer to answer comprehension questions after ambiguous sentences compared to both first character reference (β = 0.344, p < 0.001) and second character reference sentences (β = 0.326, p < 0.001). No difference between the unambiguous sentences was found (p = 0.341).

Our reading time results replicate the results reported in Stewart et al. (2007, Experiment 1), using a different methodology (i.e., measuring non-cumulative word-by-word reading times rather than whole-sentence reading times), a larger sample size to provide adequate power, up-to-date statistical analyses, and a language that is different from but related to English.

Stewart et al. (2007) found that whole-sentence self-paced reading times in the ambiguous and unambiguous conditions did not differ significantly from each other under shallow processing, while ambiguous sentences were read more slowly (i.e., an ambiguity penalty) than unambiguous sentences under deep processing. These results for whole-sentence reading times showed a similar pattern to our results for word-by-word reading times: in our results, the conditions also did not differ from each under under shallow processing, while an ambiguity penalty was found under deep processing. In contrast, Grant et al. (2020) observed a significant ambiguity advantage in the post-critical region when comparing sentences with an ambiguous pronominal reference to unambiguous sentences in an eye-tracking while reading experiment.

A relevant question is why we did not find a significant ambiguity advantage under shallow processing. One potential reason is that participants in our study, and in the experiments reported in Stewart et al. (2007), did not read the embedded clause well enough to lead to differences between the sentence types. Since, in our study, none of the comprehension questions in the shallow processing version targeted the content of the embedded clause, this may have lead some participants to only skim this part of the sentence. However, considering the fact that this was a self-paced reading experiment in which participants were presented each word of the sentence in a cumulative way (different from an eye-tracking experiment in which it is easier to skip over parts of the sentence), this seems unlikely. Moreover, Stewart et al. (2007) did include some (4/24) comprehension questions that targeted the embedded clause and did not find a significant ambiguity advantage either.

A different explanation relating to the shallow comprehension questions, suggested to us by a reviewer, is that our ‘shallow’ questions may not have been shallow enough to observe an ambiguity advantage. While none of the shallow questions targeted the pronominal reference, the questions did refer to the referents (asking who did something). This may have invited strategic attention to the referents, which may have wiped out a possible ambiguity advantage. In their study, Grant et al. (2020) also included questions that probed the comprehension of the pronoun, but did so after only a proportion of the trials. It could, therefore, be the case that the occurrence of a question relating to a referent on all trials motivated a less superficial type of referential processing even in the shallow condition, resulting in the absence of a significant ambiguity advantage.

It could also be the case that the processing system handles the resolution of structural (i.e., attachment) ambiguities and pronominal reference ambiguities differently. Unlike structural/attachment ambiguities, which have typically been shown to facilitate processing (e.g. Clifton & Staub, 2008; Traxler et al., 1998; Van Gompel et al., 2005, 2001), referential ambiguities would then more often hinder processing (Badecker & Straub, 2002; Stewart et al., 2007). While this position is not unreasonable considering that referential and structural ambiguities are distinct phenomena, the findings by Grant et al. (2020) suggests that, when compared directly and under equivalent conditions, structural and referential ambiguity show similar patterns in eye movements during reading. This suggests that readers react to referential and structural ambiguity in a similar fashion.

Another possible explanation is that the ambiguity advantage for pronominal references may constitute such a small effect that even more statistical power and/or a more sensitive experimental paradigm is needed to observe it. For instance, it could be that the effect is more pronounced in an eye-tracking study, explaining why Grant et al. (2020) did find a significant ambiguity advantage.

Moving back to comparisons to earlier findings, we used a word-by-word self-paced reading task, while Stewart et al. (2007, Experiment 1) examined whole-sentence reading times. This allowed us to localize the effects and obtain a more detailed picture of the time course of processing. The results showed an ambiguity disadvantage in the pronoun+2 region, which was a post-critical region. While spill-over effects are expected in self-paced reading experiments, this could potentially indicate that referential ambiguity does not immediately result in an ambiguity penalty, but that this effect is delayed. This is similar to what Logačev and Vasishth (2016a) reported for an ambiguity disadvantage in structurally ambiguous sentences, where a relatively late ambiguity disadvantage was observed at the last word of the relative clause. It is also in line with Stewart et al. (2007), who argued that readers may delay the resolution of ambiguous pronouns. A different line of literature on the processing of gender features also suggests that ambiguous pronouns do not disrupt comprehension immediately (e.g. Greene, McKoon, & Ratcliff, 1992; Rigalleau, Caplan, & Baudiffier, 2004), but may do so after a delay.

Finally, the question-answering results showed a larger difference between the ambiguous and both of the unambiguous conditions when questions required resolving the pronominal reference compared to when questions were only superficial. Similar results were reported in Stewart et al. (2007) and Swets et al. (2008), who showed that question-answering times were significantly longer after ambiguous than after unambiguous sentences when ‘deep’ processing questions were asked, but no such difference was found when questions were only superficial.

Our results provide additional evidence, along the lines of Swets et al. (2008) and Stewart et al. (2007), that the difference in reading times between referentially ambiguous and unambiguous sentences depends on the readers’ goals, in our case whether the manipulation of ‘depth of processing’ required the reader to answer questions that targeted the antecedent of the pronoun in the embedded clause. This has important implications for models of sentence processing. In particular, it suggests that ambiguity resolution is a flexible process, such that the way in which ambiguous sentences are processed (relative to unambiguous ones) depends on the processing depth or task demands.

In previous studies of structural and referential ambiguities, ambiguity penalties have been taken as evidence for constraint-based parsing models or competition-based models, in which the parallel consideration of multiple structures gives rise to competition effects in processing (e.g., Badecker & Straub, 2002; MacDonald, 1994). In contrast, ambiguity advantages have been taken as support for non-competitive models such as the Unrestricted Race Model (e.g. Van Gompel et al., 2000), according to which potential syntactic analyses engage in a race before a single analysis is adopted. In a race-based model, disambiguated sentences are predicted to take longer to process than ambiguous ones because processing delays occur when subsequent information is inconsistent with a previously adopted analysis and the initial analysis needs to be revised (Van Gompel et al., 2000). Alternatively, an ambiguity advantage is predicted because the adopted analysis is the one that is completed the fastest, leading to shorter times on average when there are several candidates, as in globally ambiguous cases compared to unambiguous cases (Logačev & Vasishth, 2016a). The current results warrant the formulation of a model of ambiguity resolution that takes into account the effect of processing depth on ambiguity resolution, and that should be able to capture both ambiguity advantages under shallow processing and ambiguity penalties under deep processing.

One proposal that has been put forward to account for moderating effects of processing depth or task demands is based on underspecification (cf. Ferreira et al., 2002; Karimi & Ferreira, 2016; Stewart et al., 2007; Swets et al., 2008). This proposal is in line with the ‘good-enough’ processing approach, which argues that readers or listeners may construct sentence representations that are only ‘good-enough’, i.e., not necessarily complete and fully specified, but that suffice to accomplish the communicative goal at hand (e.g., Christianson, 2016; Christianson, Hollingworth, Halliwell, & Ferreira, 2001; Ferreira & Lowder, 2016; Ferreira & Patson, 2007). For structural ambiguities, Swets et al. (2008) argued that readers may strategically underspecify the representation of structurally ambiguous sentences to save time, unless disambiguation is required by task demands as under deep processing. Committing to one reading and building the corresponding syntactic structure is argued to take time. Therefore, not having to make a commitment because of underspecification under shallow processing is expected to result in a reading time advantage for ambiguous sentences. The longer question answering times for ambiguous sentences under deep processing, as also found in our study, have been taken as additional evidence in favor of the delay of a commitment to a particular reading (Swets et al., 2008).

For referential ambiguities, Stewart et al. (2007) argued that ambiguous pronouns may initially be underspecified under shallow processing, resulting from the absence of co-indexation between the pronoun and its antecedent. According to this approach, readers invest processing resources to interpret the ambiguous pronoun only when they are required to engage in deep processing; under shallow processing ambiguous pronouns may remain underspecified. A sentence such as Abel lent Gijs the book before he went on holiday will be understood by the comprehender to mean that someone went on holiday, but it will be left undecided whether that was Abel or Gijs. Because of this underspecification, under shallow processing reading times for ambiguous sentences are expected to be faster or equivalent compared to unambiguous sentences. By contrast, under deep processing, reading times for ambiguous sentences are expected to be slower.

The finding of processing depth affecting ambiguous sentence processing can also be explained without the notion of underspecification. Logačev and Vasishth (2016a) proposed a Stochastic Multiple-Channel Model of structural ambiguity resolution as an extension of Van Gompel et al.’s (2000) Unrestricted Race Model. To account for the flexible nature of ambiguity resolution, Logačev and Vasishth (2016a) stipulated a stopping rule that is determined by task demands. If the task does not require more than one permissible relative clause attachment to be constructed, the parser stops after one attachment has been computed (i.e., a first-terminating stopping rule). However, if the task requires access to all available attachment options, the parser may choose to wait for all permissible structures to be built (i.e., an exhaustive stopping rule). Under a first-terminating stopping rule, an ambiguity advantage is predicted, because the parser needs to wait for only one particular attachment in unambiguous conditions and the fastest will be adopted. However, when the exhaustive stopping rule is used, an ambiguity disadvantage is predicted because the parser needs to wait for both attachments to be computed in the ambiguous condition, which will lead to more instances of long completion times.

This approach can easily be applied to referential ambiguities as well. Under deep processing, as implemented in the experiment reported here, the task suggests to the participant that an exhaustive stopping rule should be used, because the task requires resolving the pronominal references or because the reader is aware that the pronouns sometimes have more than one referent. In that case, the parser waits for both referents that form potential antecedents for the ambiguous pronoun to be computed (and then possibly discards the least plausible of them). This predicts an ambiguity penalty under deep processing, because computing both referents will always take longer than computing one, even in a parallel processing system (Logačev & Vasishth, 2016a). In contrast, under shallow processing conditions, one referent will suffice and an ambiguity advantage is predicted as the adopted referent is the one that takes the least time, leading to shorter time on average when there are more candidates as in the ambiguous cases.

In sum, both underspecification and race-based models are, in modified form, able to account for the finding that processing depth may influence the difference in reading times between referentially ambiguous and unambiguous sentences: the former through strategic underspecification of the co-indexation between the pronoun and its antecedent; the latter through the use of different stopping rules determined by task demands. A difference between the approaches is that at least one referent is selected as an antecedent to the pronoun in the Stochastic Multiple-Channel Model, while the co-indexation is not necessarily specified under an underspecification model. Hence, important questions are what the resulting representations look like under shallow processing and, under an underspecification approach, whether the resulting representations are partially specified or non-specified (see Logačev & Vasishth 2016b, for discussion). As pointed out by Van Gompel et al. (2001), the relevant parts of ambiguous sentences need to be processed to some extent under an underspecification approach as well, in order for the comprehension system to be able to determine what can be left underspecified (ambiguous cases) and what cannot (unambiguous cases). In response, Swets et al. (2008) argued that the relevant part is processed just enough to identify disambiguation cues and inappropriate interpretations, but not deeply enough to fully disambiguate when there are two appropriate interpretations. On the basis of the present data we cannot tell which representations were generated while the sentences were read and exactly how specified these were under shallow processing. It is possible that both referents remained available as possible antecedents, as predicted under an underspecification account. It is also possible that a specific referent was selected, either at random, or, more likely, because stored knowledge was taken into account (such as the bias for referents in subject position to be more accessible). Further research is needed to distinguish between these options and evaluate the validity of both models.

Thinking about the broader implications of the current findings, an important question is how representative our materials and task are for other reading/listening situations. With respect to the task, people do not usually face comprehension questions. Instead, depth of processing or the processing strategy more generally may be driven by both general constraints (e.g., whether the conversation or text is judged to be important) and local factors (e.g., information structure). The consequences of shallow/deep processing may also vary across situations. In our experiment, participants read isolated sentences in which the ambiguity always concerned a choice between two referents. Considering an underspecifiation model, these materials may have biased participants towards partial specification (e.g., either Abel or Gijs went on holiday), while a context in which the potential referents are not so clearly specified may bias participants towards complete underspecification (e.g., someone went on holiday).

In conclusion, the present results show, once again, that the way in which ambiguous sentences are processed depends on the depth of processing that participants engage in. This finding may not only be true for ambiguity resolution, but likely generalizes to other areas of language processing as well. For instance, recent work by Laurinavichyute and von der Malsburg (2021) showed that the illusion of ungrammaticality (i.e., a slow-down in reading due to agreement attraction in sentences like The key to the cabinets is rusty) disappeared when participants engaged in deep processing. In addition, a study by Schotter, Bicknell, Howard, Levy, and Rayner (2014) showed that the type of proofreading task that participants engaged in affected properties of word processing (such as word frequency and predictability effects). An important methodological implication of these findings is that the type of comprehension questions or tasks included in an experiment may affect the manner of processing. Care must, therefore, be taken in the choice of these comprehension questions and tasks (see also Ferreira & Yang, 2019). At a theoretical level, the present findings illustrate the need for models of ambiguity resolution that take into account the effect of processing depth on ambiguity resolution, and that can flexibly account for both ambiguity advantages and ambiguity penalties.