Out of the garden and into the multiverse: Open science practices, multidimensional degrees of freedom, and the speech sciences
The early 2010’s saw the reproducibility crisis take hold of the psychological sciences. As a consequence, a push for increased transparency and reproducible methodology (e.g. open access publishing, preregistration, sharing study materials, data, and analyses, performing study replications, declaring conflicts of interest, ect.) to help mitigate the effects of questionable research practices (e.g., p-hacking, HARKing, researcher degrees of freedom) has followed. The resulting methodological framework and associated techniques, now referred to as open science, have reshaped research methods in psychology and have slowly but surely made their way into adjacent fields. In this talk I will consider challenges related to open science practices and researcher degrees of freedom in the field of linguistics.
Language-related data are complicated and typically subject to myriad operationalizations. Analytic flexibility in data analysis—commonly referred to as researcher degrees of freedom (Gelman & Loken, 2014; Simmons, Nelson, & Simonsohn, 2011)—can result in substantially different conclusions based on the same data set, a situation some refer to as the inference crisis (see Rotello, Heit, & Dubé, 2015; Starns et al., 2019). The so-called “garden of forking paths” associated with researcher degrees of freedom directly affects our ability to replicate findings and accumulate knowledge.
Moreover, the scientific study of language spans numerous disciplines and encompasses social, cultural, cognitive, technological, and biomedical studies of human nature and behavior. As linguistics becomes increasingly empirical and quantitative, unique challenges arise regarding the methodological practices aimed at improving reproducibility/replicability. One of the proposed solutions to the replicability crisis has been the implementation of open science practices. As a consequence, today’s speech researcher must be up to date on the current protocols of open science in order to produce research projects that meet these new methodological standards. However, an important consideration often overlooked in the wake of the open science movement deals with how current linguists train the next generation of researchers. Few, if any, researchers have had explicit instruction on the practices of open science as part of their professional training.
In this talk I will present research on open science and transparency practices in the field of linguistics. I will show that, by and large, linguistics does not engage in open science and transparency practices, though there is reason to believe this is changing for the better. Furthermore, I will present data from the Many Speech Analyses Project showing that the problem of researcher degrees of freedom is likely much more serious than our current understanding suggests. To conclude, I will discuss what we learn from methodological reform, many analyst-type studies, and big team science, as well as what current trends mean for linguistics, generally—and for some sub-fields of linguistics, specifically—. I will present solutions aimed at reducing researcher degrees of freedom and fostering open science practices in linguistics.