Understanding the inner workings of human language and its cognitive underpinnings has been increasingly shaped by experimental data. With a field that builds its theories on a rapidly growing body of experimental evidence, it is of critical importance to evaluate and substantiate existing findings in the literature because evidence provided by a single study is limited (e.g., Amrhein et al., 2019). Scientists are trained to ensure the reliability and generalizability of scientific findings by conducting direct replication studies, i.e., studies that aim to arrive at the same scientific conclusions as an initial study by collecting new data and completing new analyses but using the same methodology (for a comprehensive overview of different terminological uses, see Barba, 2018).

Replications are an integral part of cumulative experimental science (e.g., Campbell, 1969; Rosenthal, 1990; Zwaan et al., 2018). Yet many scientific disciplines do not replicate a lot. Researchers from diverse fields such as psychology (Makel et al., 2012), educational science (Makel & Plucker, 2014), ecology (Kelly, 2019), criminology (McNeeley & Warner, 2015), and economics (Mueller-Langer et al., 2019) report on very low numbers of published replications, ranging from 0.02% in ecology to 2% in criminology.

One reason for the observed lack of replication studies is the asymmetric incentive system in academia that rewards novel confirmatory findings over direct replications and null results: Replication studies are not very popular because the necessary time and resource investments are not appropriately rewarded (e.g., Koole & Lakens, 2012; Nosek et al., 2012). Both successful replications (Madden et al., 1995) and repeated failures to replicate (e.g., Doyen et al., 2012) are rarely published. Even if they are, replications usually appear in less prestigious outlets than the original findings. These dynamics lead to an abundance of positive findings in the absence of possible conflicting negative evidence (see also Fanelli, 2010) and the widely held view that replications lack prestige, originality, or excitement (e.g., Lindsay & Ehrenberg, 1993).

This perceived lack of prestige additionally comes with the sentiment that direct replications are unnecessary and/or uninformative. This sentiment expresses itself in two parts: Direct replications are claimed to be theoretically uninformative and conceptual replications are claimed to be sufficient to assess the robustness of a field’s empirical foundation (e.g., Crandall & Sherman, 2016; Stroebe & Strack, 2014). However, both of these assumptions are problematic (e.g., Zwaan et al., 2018): A repeated demonstration that an effect cannot be replicated is an important contribution to the field and aids it in calibrating researchers’ (un)certainty in the existence of a phenomenon. Moreover, failed direct replications might uncover important moderators and boundary conditions that explain the discrepancy between an original study and a replication. Conducting a direct replication operates on the assumption that all critical elements to reproduce the original effect are understood. If the replication fails, that strong assumption has to be questioned, thus relevant auxiliary hypotheses must be reconsidered, which in turn might weaken the theory. On the other hand, successful direct replications add important data to the discourse, allowing for more precise estimation of theoretically relevant parameters, and thus help to strengthen the derivation chain between theory and predictions (Meehl, 1990).

Given these arguments, we consider direct replications theoretically informative and a worthwhile endeavor. Conceptual replications, on the other hand, i.e., replication attempts that have changed multiple critical design properties of the original study, are often upheld as being more valuable than direct replications because they are assumed to simultaneously address concerns about reliability of an original claim and they are able to extend the original findings.

Conceptual replications are often considered sufficient for a field to move forward under the stipulation that repeated successful conceptual replications will occur only then when the prior research identified a true effect. However, there is increasing evidence that this strong assumption is empirically not supported. Without replicating individual studies, biases caused by questionable research practices (John et al., 2012), small sample size (Button et al., 2013) and publication bias (Fanelli, 2012) can lead to a set of studies that appear to form a coherent empirical foundation of an underlying theory, even if the underlying empirical claims cannot be replicated: There are now a number of widely studied theories and effects that have been supported by dozens, if not hundreds, of conceptual replications, but appear to crumble in light of meta-analyses or systematic direct replication attempts (e.g., Shanks et al. 2015; Wagenmakers et al., 2016). Moreover, conceptual replications can introduce interpretational ambiguity. A failed conceptual replication can never be considered evidence against the original claim. It is always possible to attribute a failed conceptual replication to the methodological changes that were made (e.g., Pashler & Harris, 2012).

In sum, direct replications are an under-appreciated tool to evaluate and cement the empirical and theoretical foundation of a field and must be considered an important complementary tool to conceptual replications.

The observed lack of replication studies across disciplines threatens the very fabric of cumulative progress in experimental science, because experimental results are often taken for granted without ever being replicated, which leads to a related problem: If we don’t try, we won’t fail. The recent past has shown that if we try, we fail more often than we would like to: Coordinated efforts to replicate published findings have uncovered alarmingly low rates of successful replications in fields such as psychology (Open Science Collaboration, 2015), economics (Camerer et al., 2016), and social sciences (Camerer et al., 2018), a state of affairs that has been referred to as the “replication crisis” (Fidler & Wilcox, 2018).

The replication crisis is not rooted in a singular cause, but pertains to a network of different practices and incentive structures, all of which conjointly lead to an increase in results that are not replicable. Researchers have identified practices that might have contributed to the widespread lack of replicability, including but not limited to too small sample sizes (e.g., Button et al., 2013; Vasishth et al., 2018), lack of data and materials sharing (e.g., Nosek et al., 2015), use of anti-conservative statistical methods (e.g., Yarkoni, 2019), large analytical flexibility (e.g., Simmons et al., 2011), and lack of generalizability across diverse contexts and populations (Henrich et al., 2010).

These limitations are present, and maybe even exacerbated in experimental linguistic research: Access to certain linguistic populations is often limited or too cost-intensive, making it difficult to collect sufficiently large samples. Experimental linguistic research is resource-intensive because of equipment cost and complexity, elaborateness of data collection procedures, and computational requirements of data analysis and curation. This often results in studies with small sample sizes and, consequently, with low statistical power (e.g., Casillas, 2021; Kirby & Sonderegger, 2018). Statistical analyses in linguistics are often ignoring important assumptions (e.g., Winter & Grice, 2021) and are characterized by a large number of researcher degrees of freedom (Roettger, 2019). Moreover, claims about human language are often based on a small set of languages, limiting their generalizability (e.g., Levisen, 2019; Majid & Levinson, 2010).

In light of the large overlap in research practices between linguistics and neighboring disciplines for which low replication rates and failures of attempts to replicate have been attested, there are rising concerns about both replication rates and replicability in the field of experimental linguistics (e.g., Marsden et al., 2018; Roettger & Baer-Henney, 2019; Sönning & Werner, 2021). A number of failed replication attempts reported in various subfields of linguistics indicate that these concerns have to be taken seriously (e.g., Chen, 2007; Jäger et al., 2020; Morey et al., 2021; Nieuwland et al., 2020; Nieuwland et al., 2018; Papesh, 2015; Stack et al., 2018; Vasishth et al., 2018; Westbury, 2018).

Despite these known problems, there might be only very few published direct replications in linguistics. In their detailed assessment of replications in second language (L2) research, Marsden et al. (2018) explored 67 self-labeled L2 replication studies for a wide variety of characteristics. Their results indicate that for every 400 articles, only one replication study is published which translates into 0.25% of published studies containing a replication. Following Makel et al. (2012), we will refer to the proportion of published articles containing at least one replication as the replication rate. Moreover, the sample of Marsden et al. (2018) did not include a single direct replication study, i.e., a replication that strictly followed the design of the initial study. This is a state of affairs that is worrisome and warrants further investigation. To our knowledge, there is no systematic assessment of replication rates across experimental linguistics beyond Marsden et al., (2018). The present paper aims at filling this gap. To gauge the past and current replication landscape in experimental linguistics, track progress over time, and calibrate future policy and training initiatives, it will be useful to assess the prevalence of replications across experimental linguistics and explore their contributing factors.

The present study assesses the frequency of articles containing replications as well as the typology of replication studies that have been published in a representative sample of experimental linguistic journals from 1945 to 2020. Given the arguments presented above, we are primarily interested in the prevalence of direct replications in the field. Our study aimed at answering two main questions: “How many published papers in experimental linguistics contain at least one direct replication?” and “Are there factors that affect the replication rates and are they found either at the journal level (e.g., journal policies, open access, journal impact factor, etc.) or at the study level (e.g., composition of authors, investigated language, etc.)?” The study consisted of two analyses: First, we assessed the frequency of articles mentioning the term replication (search string: replicat*) across 98 linguistic journals. Second, we manually categorized the type of replication studies (direct, partial, conceptual) in a subset of twenty journals. We then related their replication rates to factors like the years of publication, and the citation counts of both initial and replication study.

The key dependent variable of the first part of this study was the rate of replication mention for journals relevant to the field of experimental linguistics.

The second part of the study had two aims: First, the term replication is commonly used in ambiguous ways, so articles containing the search term were further analyzed to determine whether they indeed reported a replication study or whether they used the term in a different way. Second, we further investigated what types of replication studies are published and whether replications are becoming more frequent over time. Our target estimand is the proportion of experimental articles containing at least one replication.

The current study aimed at providing a comprehensive survey of published replications in experimental linguistic research. By analyzing the publication history of over 50,000 articles across 98 journals that publish experimental linguistic research, our study found that 4.5% of experimental linguistic publications used the term replicat* in title, abstract or keywords. A more thorough analysis of 274 sampled experimental articles containing the term replicat* revealed that only around half of the hits represented actual replication studies, reducing the effective replication rate to 2.5%. This rate is slightly higher than reports of comparable investigations in psychology (1.6%, Makel et al., 2012), educational science (0.1%, Makel & Plucker, 2014), and economics (0.1%, Mueller-Langer et al., 2019). The higher rate might be due to a methodological choice, however. Due to large plurality of methods in linguistics, we calculated the replication rate based on only those articles that contained the term experiment* (as opposed to all articles in the sample), reducing the denominator substantially.

A closer look at the nature of replication studies revealed that the majority of replication studies were studies that diverged from the initial study in at least one design choice. Only 7.3% were direct replications, i.e., studies that directly repeated an initial study without self-reported changes to the design, and only five of these were replications conducted by an independent team of researchers. Taking together replicat* mention rate and actual replication rate, 0.08% of experimental studies are independent direct replications in the field of linguistics. In other words, only 1 in 1,250 experimental linguistic articles contains an independent direct replication. This clearly indicates that replication attempts, and especially independent direct replication attempts, are still very rare in the experimental linguistics literature.

Before interpreting the results and offering possible ways forward, we need to discuss two important caveats to our study. First, if research articles were not framed as experimental, then they were not included in the analysis. Similarly, if experimental articles were not framed as replications, then they were not categorized as such. These are clear limitations to our search strategy and might lead to an underestimation of the true replication rate. Assuming the false negative rate is not zero, the reported replication rates might change after correction. To circumvent this methodological problem, a large sample of articles would have to undergo manual coding, which is not feasible for a large-scale assessment. Future research using alternative assessment methods (possibly machine learning techniques) or more in-depth investigation of either subfields (e.g., Marsden et al., 2018) or specific journals might result in different replication rates. However, the existence of replication studies that are not referred to as such might also reflect a more general problem: If studies are not framed as replications by using the term replication, readers’ ability to connect research to its intellectual precedents is severely limited.

Second, our assessment of replication types relied on two assumptions. On the one hand, we assume that the authors disclosed changes to the initial study in a transparent way. On the other hand, we assume that if changes were disclosed, we were able to extract and interpret these changes accurately. Neither of these assumptions must hold, thus any rates that are generated here are necessarily only a rough proxy of the true replication rate. Nevertheless, given that our findings seem to align well with evidence from other fields as well as an in-depth analysis of a subfield of linguistics (Marsden et al., 2018), we are confident that our conclusion holds.

Although the present study is the first systematic assessment of replication rates in linguistics, our conclusions are hardly surprising. Academic incentive systems do not reward replication studies. Neither journals nor funders encourage them. For example, Martin and Clarke’s (2017) survey results suggest that in 2015 only 3% of psychology journals explicitly state that they will consider publishing replications. Similarly, out of the 98 journals in our sample, only 2 encouraged direct replications. And even if one manages to publish a replication, replication studies are characterized by much lower yearly citation counts compared to corresponding initial studies, leading to a lack of perceived prestige (e.g., Koole & Lakens, 2012; Marsden et al., 2018; Nosek et al., 2012). Direct replications simply do not seem worth their costs.

In order to overcome the asymmetry between the cost of direct replication studies and the presently low academic payoff for it, we must re-evaluate the value of direct replications. Funding agencies, journals, but also editors and reviewers, need to start valuing direct replication attempts as much as they value novel findings. For example, we could either dedicate existing journal space to direct replications (e.g., as its own article type) or create new journals that are specifically dedicated to replication studies. Journals could help to normalize replication studies by calls for special issues dedicated to replications of influential findings like, e.g., the recent call by the Journal of Memory and Language.10 Another alternative is the Pottery Barn rule, implemented by, for example, Royal Society Open Science: Once the journal has published a study, it commits to publishing all direct replications of this study.11

At the same time, we should attempt to find more resource-efficient ways to both identify replication targets and conduct replication studies. We believe, most people would agree that not every study needs direct replication. Take for example the McGurk effect, i.e., perceiving a sound that lies in-between an auditory presented component of one sound and a visually presented component of another one (McGurk & MacDonald, 1976). This phenomenon is probably replicated in dozens of linguistic classrooms every semester across the globe. On the other hand, it might be a good idea to evaluate more critically whether a given study is worth replicating. Resources can be saved if studies with poor experimental design, unsuitable measurement approach or inept model specifications are ruled out from direct replication attempts (Yarkoni, 2019). Finding convenient yet effective tools to identify worthwhile replication targets is an active meta-scientific field (e.g., Coles et al., 2018; Hardwicke et al., 2018; Isager et al., 2021a) and feasible algorithms are currently developed and tested (Isager et al., 2021b). When it comes to more accessible ways to conduct replication studies, several authors have suggested involving our students more rigorously (e.g., de Leeuw et al., 2019; Frank & Saxe, 2012; Grahe et al., 2012; Roettger & Baer-Henney, 2019), possibly creating a rich learning experience for our students while at the same time reducing the resource costs of replication studies. Alternatively, resources can be pooled across multi-lab replication efforts, effectively reducing the costs for individual researchers and labs (e.g., Frank et al., 2017; Nieuwland et al., 2018; Open Science Collaboration, 2015). The StudySwap platform, for example, allows researchers to identify independent labs for conducting a replication attempt of one’s study, thus helping researchers to assess the independent replicability of their findings prior to publication (Chartier et al., 2018).

We are confident that the field of linguistics can function as a role model for neighboring fields. Although major meta-scientific discourses are held in other fields, linguistics has demonstrated quick uptake of methodological reforms time and time again. A point in case is the swift uptake of Registered Reports,12 a new article form in which a study proposal is reviewed before the research is undertaken. While the uptake across disciplines is slow, linguistics has already at least 12 high-impact journal outlets that offer Registered Reports. Moreover, an increasing number of reproducibility initiatives founded in the field during the last few years give hope that the field is continuing to evaluate their past, current, and future practices and successfully face the challenges ahead. This paper was an attempt to contribute to this development. We hope our assessment allows future efforts to track progress over time and calibrate policies across experimental linguistics.