All quantitative analyses were conducted using R Core Team (2025).

Focusing on experimental linguistic research, we used Kobrock and Roettger (2023) as a point of departure. They list 100 linguistic journals that had at least a hundred articles published at the time of assessment (2021) and a high ratio of articles containing the search string “experiment*” in title, abstract and/or keywords. Out of these 100 journals, we selected all journals with at least 10% of articles containing the search string “experiment*”. Out of the remaining 37 journals, we selected those journals that encouraged the use of APA style (American Psychological Association, 2020), either in the main body of the text or specifically regarding statistics, in the author guidelines, resulting in nine remaining journals. Moreover, to access the articles in .pdf format, the articles had to be either accessible to us through our library license, or open access, resulting in a final list of eight journals: Applied Psycholinguistics (APS), Bilingualism: Language and Cognition (BLC), Linguistic Approaches to Bilingualism (LAB), Language and Speech (LaS), Language Learning and Technology (LLT), Journal of Language and Social Psychology (LSP), Journal of Child Language (JCL), and Studies in Second Language Acquisition (SLA). An anonymous reviewer raised the justified concern that the resulting sample might not represent the vast majority of work in experimental linguistics. To address this concern, we additionally included those five journals that have published the highest absolute number of experimental articles (according to Kobrock & Roettger, 2023), regardless of the above mentioned constraints, resulting in the inclusion of the following psycholinguistic outlets: Journal of Memory and Language (JML), Language, Cognition and Neuroscience (LCN, formerly Language and Cognitive Processes), Journal of Psycholinguistic Research (JPR), Journal Of Speech Language And Hearing Research (SLH), and Brain and Language (BAL).

We included only original research articles within the publication years of 2000–2023, excluding any book reviews, response articles, commentaries, editorials, corrigenda, errata, advertisements, etc.

We used the R package Statcheck (Version 1.5.0; Nuijten & Epskamp, 2024) to automatically detect statistical reporting inconsistencies. Statcheck works as follows: After converting articles in .pdf or .html format to plain text, Statcheck searches for specific strings that correspond to a NHST result using regular expressions. That way, Statcheck can detect results of t-tests, F-tests, Z-tests, χ2-tests, correlation tests, and Q-tests, as long as the test result fulfills three conditions: (a) the test result is reported completely, including the test statistic, the degrees of freedom (if applicable), and the p-value; (b) the test result is in the body of the text, i.e. Statcheck usually misses information in tables; and (c) the test result is reported in APA style. Given these constraints, Statcheck is estimated to detect roughly 60% of all reported NHST results (Nuijten et al., 2016). Statcheck uses the reported test statistic and degrees of freedom to recalculate the p-value, compares the reported and recalculated p-value and, if there is a mismatch, flags the test as containing an “inconsistency.” The algorithm takes into account that tests might have been performed as one-tailed by identifying the search strings “one-tailed,” “one-sided,” or “directional” in the body of the text. Moreover, Statcheck considers p = .000 and p < .000 as inconsistent, because p-values of exactly zero are mathematically impossible and the APA manual (American Psychological Association, 2020) advises reporting very small p-values as p < .001. Validity checks of Statcheck suggest that inter-rater reliability between manual coding and Statcheck is high, i.e. 0.76 for inconsistencies and 0.89 for decision inconsistencies (Nuijten et al., 2016). The overall accuracy of Statcheck is estimated to be between 96.2% to 99.9% (Nuijten et al., 2017; but see Schmidt, 2017). We thus consider Statcheck a valid proxy for the prevalence of statistical reporting inconsistencies.

Articles from Linguistic Approaches to Bilingualism spanned 2011–2023; articles from Language, Cognition and Neuroscience spanned 2015–2023. Statcheck could not parse 291 articles, likely related to issues with rendering the Chi-Squared symbol in the conversion from .pdf to .txt, resulting in some gaps in coverage. This procedure resulted in 13,065 research articles, which were submitted to analysis.

The aims of this study were explicitly exploratory and hypothesis-generating, thus, analyses remain merely descriptive, reporting on the proportion of articles/tests that are statistically (in)consistent. Our investigation set out to explore the following research questions: How prevalent are statistical inconsistencies (and decision inconsistencies) in our sample? (4.1) Do inconsistency rates vary across journals and/or publication years? (4.2) Is there evidence for bias, i.e. are processes that result in inconsistencies more likely to produce lower p-values? (4.3) And do decision inconsistencies matter, i.e. are they merely typos or do they have the potential to lead to erroneous theoretical interpretations? (4.4)

The results are summarized in Table 1. Out of 13,065 articles, 6,268 articles contained statistical tests that Statcheck could assess (48%), amounting to 82,991 assessable p-values. Interestingly, the five journals that did not explicitly encourage APA style had virtually identical rates of assessable articles, compared to the original sample. Overall, 10,421 p-values were flagged as inconsistent (12.6%), out of which 1,226 were considered decision inconsistencies (1.5%) (see Table 1).

On average, 49% of assessable articles contained one or more inconsistencies (journals range from 42 to 56%) and 12% contained one or more decision inconsistencies (journals range from 10 to 15%) (see Figure 1). The proportion of inconsistencies ranged from 7 to 22% across journals (1 to 7% for decision inconsistencies). These rates appear to be stable across year of publication (see Figure 2).

If inconsistencies were bias-free, we would expect different types of inconsistencies to be equally frequent. However, this is not the case. Inconsistencies that report the p-value as being larger or smaller than a reference value (e.g. p > 0.05 or p < 0.05, respectively) are not equally prevalent in the sample: There were 4.4% of inconsistent tests with p being reported as larger than a reference, but 6.7% of inconsistent tests with p being reported as smaller than a reference. So even if we assumed these inconsistencies were merely typos of the comparison sign (e.g. < instead of >), inconsistencies are biased towards producing smaller p-values.

These biases are also reflected in decision inconsistencies. Of all decision inconsistencies (n = 1,226), 68% represent cases in which a reported significant result (p < 0.05) is recalculated as non-significant (p > 0.05), i.e. non-significant results are more than twice as likely to be erroneously reported as significant than the other way around. The latter pattern, however, seems to have decreased over time. Reproducing Nuijten et al. (2016), Figure 3 plots the development of the bias observed for decision inconsistencies over time. The prevalence of decision inconsistencies in significant p-values seems to have slightly decreased over the years, while the prevalence of decision inconsistencies in non-significant p-values seems to have slightly increased over the years.

The question arises as to whether the inconsistencies detected have the potential to matter theoretically. This is a difficult question to address empirically, because experimental behavioral research is often characterized by weak links between theory and predictions (Scheel et al., 2021), making it difficult even to extract hypotheses and determine whether hypotheses have been corroborated or not from published articles (Scheel, 2022). What we can assess is whether a given test result has the potential to be theoretically relevant, i.e. whether theoretical claims can be made based on it. We approximated an answer to this question as follows: First, we extracted a random subsample (15%, n = 164) of articles which contain at least one decision inconsistency, distributed as equally as possible across publication years. We restricted the subsample to only those decision inconsistencies that involve either a larger than (>) or smaller than (<) comparison of the p-value, to allow unambiguous interpretation of the results. We then manually extracted the relevant sentence which the test statistic is embedded in and assessed whether there was a mismatch between the reported p-value significance and the authors’ interpretation of the p-value significance, such that cases that are likely typos (mismatch) and true decision consistencies (match) could be separated. This ratio let us approximate the true decision inconsistency rate. We considered true decision inconsistencies theoretically relevant: For example, if an article claimed that two groups are different (or not) based on an erroneously reported significant test, we considered this claim to potentially have theoretical consequences. Out of the 164 cases assessed, most were straightforwardly interpretable (155, 95%). The remaining 9 cases were coded as unclear, i.e. even after assessing the manuscript in detail, it remained unclear to us how the p-value was interpreted. Unclear cases occurred mostly due to tables or footnotes with lists of test results without clear reference. Of the interpretable test reports, 132 matched the erroneously reported p-value (85%), representing true decision inconsistencies that potentially have theoretical consequences. Inversely, 23 cases mismatched the erroneously reported p-value (15%), possibly representing genuine typos, e.g. the article interpreted the results in line with the computed p-value and not the erroneously reported p-value.

The present study found a large amount of statistical reporting inconsistencies across a sample of 13,065 experimental linguistic articles, containing 82,991 assessable p-values. 12.6% of all p-values were flagged as inconsistent, and 1.5% were flagged as decision inconsistencies, i.e. the reported p-value is on the opposite side of the alpha threshold from the recalculated p-value. A manual validation of a subset of decision inconsistencies revealed that 85% of these decision inconsistencies are interpreted in line with the inconsistent p-value, i.e. a paper claims that e.g. two groups are different (or not) based on an inconsistently reported significant test. On average, 49% of assessable articles contained at least one inconsistency and 12% contained at least one decision inconsistency. The present examination did not indicate strong differences in inconsistency rates across journals or publication years.

The present study can be considered a conceptual replication of previous studies investigating statistical reporting inconsistency across different disciplines (Bakker & Wicherts, 2011, 2014; Caperos & Pardo, 2013; Veldkamp et al., 2014; Wicherts et al., 2011) and most recent assessments using the automatic tool Statcheck (Buckley et al., 2023; Colombo et al., 2018; Groß, 2021; Nuijten et al., 2016). The discovered inconsistency rates fall in line with these studies, which report on inconsistency rates between 4% and 14%, with between 10% and 63% of articles containing at least one inconsistency. Moreover, the observed rates of inconsistencies and decision inconsistencies are virtually identical to rates reported by Nuijten et al. (2016) for the psychological science literature.

Even if the prevalence of these inconsistencies could be largely attributed to inconsequential typos or rounding errors (an assumption we cannot test without access to the data), the sheer amount of the inconsistencies that have made it through peer review should concern us. They are human errors. If such a substantial amount of errors is found in plain sight, the question naturally arises as to how many errors during the data analysis itself remain undetected. We should ask ourselves, if the tip of the iceberg is already so large, what is the volume of the submerged iceberg?

Our results suggest biases as well. The rate of decision inconsistencies was twice as high for p-values reported as significant than for those reported as non-significant. These biases have been reported for other disciplines as well (Bakker & Wicherts, 2011; Nuijten et al., 2016) and could indicate a systematic bias in favor of lower p-values, in general, and a bias towards significant results, in particular. Our data do not speak to the causes of these biases, but possible reasons include the following:

First, researchers might intentionally round down p-values because they think lower p-values are more convincing to reviewers and readers. This practice has been admitted to by 1 in 5 surveyed psychological researchers (John et al., 2012). Given that a non-trivial number of quantitative linguists have admitted to committing questionable research practices (and even fraud) (Isbell et al., 2022), we cannot exclude the possibility that some of the inconsistencies in our sample were, indeed, intentional. It is our strong belief, however, that the majority of inconsistencies are unintentional and caused by other mechanisms.

Second, researchers might scrutinize non-significant results more than significant results, or are less likely to double-check significant results than non-significant results, because results that confirm their hypothesis feed into their confirmation bias (Nickerson, 1998). For example, Fugelsang et al. (2004) let researchers evaluate data that are either consistent or inconsistent with their prior expectations. They showed that when researchers encounter results that are not in line with their expectations, they are likely to blame the methodology, while results that confirmed their expectations were rarely critically scrutinized.

Third, the observed bias might merely be a reflection of publication bias (Franco et al., 2014; Sterling, 1959) with (erroneously) reported significant p-values being more likely to be published than (erroneously) reported non-significant ones. Publication bias is a well-established pattern in experimental linguistic research, with many recent meta-analyses discussing possible evidence for it (Isbilen & Christiansen, 2022; Lehtonen et al., 2018; Lu et al., 2024). For example, De Bruin et al. (2015) showed that studies with results supporting the bilingual-advantage theory were more likely to be published, while studies with results challenging the theory were significantly less likely to be published.

Regardless of what might possibly cause biases in the processes that generate inconsistencies, our data also suggest a positive development. The over-proportional occurrence of decision inconsistencies for p-values reported as significant has decreased over time.

While we believe our work offers an important contribution to improving statistical reporting practices in experimental linguistics, the present assessment and the conclusions we can draw from them are limited. First, our sample is limited to only a subset of experimental linguistic journals. However, given the selection of journals and their standing in the field, and given that the inconsistency rates of our study are not only comparable to similar studies from other disciplines, but also relatively stable across journals and time, our findings should be considered relevant for experimental linguistics at large.

Second, given the constraints on automatically detecting test statistics, Statcheck misses reported values that either diverge from APA reporting standards or are reported in tables. However, inconsistency rates in our own sample have been shown to be similar for results in APA format vs. results that diverge from APA format (Bakker & Wicherts, 2011; Nuijten et al., 2016).

Third, Statcheck overestimates inconsistency rates, because it might not accurately detect corrections for multiple comparisons (Schmidt, 2017). Nuijten et al. (2017), however, show that not only were there only a small proportion of flagged inconsistencies related to multiple comparisons, but also that these multiple comparisons themselves were often erroneously reported. They conclude that “[a]ny reporting inconsistencies associated with these tests and corrections could not explain the high prevalence of reporting inconsistencies” (Nuijten et al., 2017, p. 27).

More elaborate automatic tools for the extraction of statistical information might allow for a more detailed and more accurate assessment of statistical reporting in the future (e.g. Kalmbach et al., 2023). Despite its limitations, Statcheck provides a rough proxy for true inconsistency rates in the published literature, and we hope the reader agrees that the prevalence of inconsistencies is a state of affairs that should be reflected upon.

There are concrete actionable steps that the field of experimental linguistics can take to reduce statistical reporting inconsistencies. In order to avoid simple copy-and-paste errors related to working in two separate programs for writing the manuscript and conducting the statistical analysis, authors should consider literate programming, i.e. an integration of analysis code and prose into a single, dynamic document (Casillas et al., 2023; Knuth, 1984). Several implementations of literate programming are freely available to researchers, including common R markdown files (Rmd) and Quarto markdown files (qmd). Literate programming can ensure that values derived from the statistical analysis are automatically integrated into the manuscript document, avoiding errors that might happen during a manual transfer from one program to the other.

Authors should generally engage in transparent and reproducible practices that can reduce human errors or at least make them detectable by sharing their derived data (i.e. the anonymized data table that was analyzed) as well as a detailed description of their statistical protocol, ideally in form of reproducible scripts. Sharing reproducible analyses with reviewers allows the reviewers to reproduce the authors’ analyses, possibly detect errors or even inappropriate statistical choices before publication, thus improving the quality and robustness of the final product. Moreover, publicly sharing their analyses has numerous benefits to the authors themselves beyond error detection: Open data and materials can facilitate collaboration (Boland et al., 2017), increase efficiency and sustainability (Lowndes et al., 2017), and are cited more often (Colavizza et al., 2020).

Reviewers can, additionally, check the statistical reporting consistency in the manuscript by using tools such as Statcheck (Nuijten & Epskamp, 2024, http://statcheck.io) or p-checker (Schönbrodt, 2015, http://shinyapps.org/apps/p-checker/). Reviewers could consider requesting data and scripts during peer review. Such requests might be particularly justified when inconsistencies are apparent. Explicitly requesting that authors share data might instill additional care and quality checks when authors prepare their materials, but might also allow the reviewers to carefully reproduce the results and critically evaluate all choices made in the statistical analysis (Sakaluk et al., 2014). Recent evidence suggests that experimental linguistics is still characterized by a pluralism of statistical approaches, even when researchers are trying to answer the same research question (Coretta et al., 2023). Some of these approaches might be more appropriate than others (Sonderegger & Sóskuthy, 2024; Vasishth, 2023), so more thorough evaluations of how researchers arrive at their statistical conclusions might elevate their analytical robustness. Moreover, a turn towards inferential frameworks that do not focus on binary decision procedures, might alleviate some of the biases we observed in the direction of inconsistencies (Cumming, 2014; Vasishth et al., 2018).

Journal editors could explicitly recommend consistency checks with algorithms such as Statcheck during peer review, a practice that has been taken up on by several journals from neighboring disciplines (Psychological Science,1 Advances in Methods and Practices in Psychological Science,2 Stress & Health (Barber, 2017)). Editors could also demand, recommend or at least encourage data sharing for publication in their journal. Data sharing policies have been shown to substantially increase the reproducibility of analyses (e.g. Hardwicke et al., 2018; Laurinavichyute et al., 2022), and a number of linguistic journals have already implemented such policies, including journals within our sample. Having said that, open data policies which, for example, the Journal of Memory and Language introduced in 2018, did not seem to have affected the proportion of statistical inconsistencies after their introduction. The inconsistency rates for JML (and other journals) were rather stable across time, so open data alone might not resolve the issue without further changes to the research eco-system.

Researchers make errors. Researchers have biases. This is who we are as humans, and there is not much we can do about our nature. Being aware of this fact and how it might affect research might help us to make possibly negative consequences detectable and preventable.