Substantial financial resources are invested in scientific research, amounting to 2% of GDP in the EU and translating to 5,600 researchers per million inhabitants in the Netherlands (UNESCO Science Report, 2021). Yet, some consider a sizable part of this investment misspent. The output of research projects is typically published in academic journals. Still, necessary information to judge the quality of the output often remains inaccessible (e.g., code, data, or details on null results), raising several concerns. First, non-transparency regarding essential parts of the research process complicates building on existing research, leading to inefficiencies in scientific progress. Second, the unavailability of said resources makes research more difficult to verify, with potentially adverse outcomes for society (e.g., Open Science Collaboration, 2015; Camerer et al., 2021).

One natural solution to these issues is for researchers to become more open – to conduct open science. Open Science aims to “increase transparency, accountability, equity and collaboration, and knowledge production by increasing access to research results, articles, methods, and tools” (Ross-Hellauer, 2022, p. 363). Its proponents include UNESCO, which in November 2021 published the first international framework on open science – adopted by 193 countries attending UNESCO’s General Conference (UNESCO Recommendation on Open Science, 2021).

Tilburg University has already made significant steps toward increasing transparency and accountability – by encouraging researchers to publish articles with open access 1 and share data and protocols in institutional repositories upon article acceptance. 2 A small but growing community at Tilburg University also practices open science before publication (e.g., Sokolova et al., 2020; Wichmann et al., 2022; Zeelenberg et al., 2021), creates tools to verify the accuracy of statistical reporting (e.g., metaresearch.nl), develops tutorials for researchers and students (e.g., tilburgsciencehub.com), or provides a platform for knowledge exchange (e.g., Open Science Community Tilburg). However, systemic change in how research is conducted across the whole research cycle is still in its infancy.3 For example, while replicability of studies is encouraged broadly, many replication packages do not replicate.4

If open science is such a natural solution to the integrity and efficiency issues faced by the scientific community, why is it not universally adopted? And how can we facilitate its broader adoption at Tilburg University? To answer these questions, we first discuss common concerns raised against the adoption of open science in section 2. In section 3, we highlight some of the key benefits of practicing open science and conclude, based on scientific research, that it is a desirable goal for Tilburg University and society at large. Finally, in section 4, we make recommendations for Tilburg University on how adopting open science across the research cycle can be accelerated.

Why are Researchers Hesitant to Engage in Open Science?

Given Tilburg University’s strategic goal of pursuing high-quality research, it seems surprising that some researchers remain hesitant to embrace open science. For example, the data editor at the American Economic Association recalls: “In a simple check we conducted in 2016, we emailed all 117 authors that had published in a lower ranked economics journal between 2011 and 2013 (Vilhuber, 2020). The journal has no data deposit policy and only requires that authors promise to collaborate. We sent a single request for data and code. Only 48 (41%) responded, in line with other studies of the kind (Stodden et al., 2018)” (Vilhuber, 2020). In this section, we discuss some of the critical concerns raised against the adoption of open science.

Early-career researchers face adverse incentives and fear loss of competitive edge

Early career researchers are in the process of building a research program, and career progression in many disciplines hinges on publishing in top-tier journals. Open science practices do not necessarily conform with the incentives faced by early-career researchers. For example, given the time required to develop publication-ready code, open science can often be seen as an unnecessary burden, a “nice-to-have but not necessary” add-on to the already challenging task of publishing. In many institutions, writing reproducible code and sharing data are not explicitly valued. Having enough top-tier publications is often the necessary and sufficient criterion for tenure, making open science a luxury researchers can decide to invest in – or not.

In a similar vein, sharing data openly is viewed by some as risky. Constructing a high-quality data set or coding new routines takes time and effort, representing a high barrier to competition from other researchers. Suppose other researchers seek to contribute to the same research stream. In that case, they either need to incur similar up-front investments (e.g., in data collection) or collaborate with those who already have collected the data. Having such a barrier to competition is viewed as necessary by some researchers. The fear of many early-career researchers is that public sharing of data and code is equivalent to a firm freely giving away its patents to its competitors – doing all the challenging work and giving others the chance to scoop your next idea before you can get to it. A recent Science article voiced similar concerns openly regarding global research collaborations (Serwadda et al., 2018).

We believe that these two issues – open science practices being viewed as nice-tohave but not crucial and potentially even career harming – are the main reasons why many early-career researchers remain hesitant to adopt open science principles. Established researchers, in turn, face additional concerns that we discuss next.

Established researchers face high learning costs and may put their reputation at risk The risk of losing a competitive advantage extends to established researchers. Sharing proprietary data and/or code only with potential collaborators is a widespread practice. It is sometimes also a path to new publications. If a colleague has a promising new idea requiring said proprietary data, there is a high chance of collaboration.5 Like early career researchers, established researchers may fear that such collaborations are less likely to happen if data or code become publicly available.

More senior researchers also face considerable learning costs when seeking to adopt open science practices (e.g., documenting data sets and implementing file versioning takes time and effort). Simultaneously, many researchers are exceedingly pressed for time in challenging roles at universities and journals. Understandably, if the learning costs do not outweigh the perceived benefits, it is hard to justify the personal investment.

Finally, established researchers risk reputational damage. In many disciplines, mistakes of any form seem significantly reputation damaging. For example, a widely publicized Excel input mistake invalidated the findings reported in Reinhart and Rogoff (2010) and continues to be a source of ridicule for the authors. In such an environment – analyses requiring increasingly complex code and mistakes in the code being a source of potentially serious reputational damage – it is understandable that researchers might be reluctant to potentially have strangers “snoop around” in their data and code (Gelman, 2017; Allen and Mehler, 2019).

Which Benefits does Open Science Entail?

While we believe the concerns listed in the previous section are legitimate, we now highlight the benefits associated with practicing science more openly.

Enhanced quality and credibility

We believe open science leads to a more transparent and robust scientific process, ensuring that society has access to high-quality research results. One core issue at the heart of empirical research is the inherent subjectivity in the research design for complex analyses. For example, Silberzahn et al. (2018) provided 29 analyst teams with the same experimental data set and research question. Effect sizes varied widely, and most of the explained variation could be attributed to the complexity of the analysis task, leading to subjectivity in research design choices. A similar study was conducted among 70 analyst teams in neuroscience, yielding the same general conclusions (Botvinik-Nezer et al., 2020).

Subjectivity in research design choices poses a real risk to scientific discovery, potentially leading to a high number of published and undetected false positive results (LeBel et al., 2017; Steegen et al., 2016). The pressure to show statistically significant results coupled with design subjectivity is often considered a prime reason for the low rate of replicability of studies (e.g., Simmons et al., 2011; Pashler et al., 2012).6 Low replicability of a wide range of papers, in turn, casts doubt on the validity of claims of that literature, endangers society’s trust in scientific research, and potentially slows down the rate of scientific progress (National Academies of Sciences, Engineering, and Medicine, 2019). It is thus vital to be aware of the choices made in a study, which means being able to fully reproduce each step of the analysis.7

By sharing code and/or data, open science strongly promotes reproducibility and replicability. Current open science practices ensure a copy of work is available in high-quality repositories. The goal is to achieve the transparency required for evaluating the reproducibility and subjectivity of a study’s main takeaways. In addition, by providing public repositories, researchers not only make code available but may also be able to incorporate community requests to disclose additional details, such as diagnostic parameters that have not been reported in the paper earlier.

Enhanced discoverability and impact

Publishing material across the whole research cycle may lead to a broader scientific impact among research communities other than one’s field. For example, a software algorithm developed for an empirical study on power imbalances between music platforms and their suppliers can be used in other research fields to classify music genres more accurately (Pachali and Datta, 2022). Research on the differential impact of open (vs. not openly developed) science shows that open science contributions get cited and downloaded more often (e.g., Lawrence, 2001; Wang et al., 2015). The enhanced visibility is helpful for individual scholars and research institutions at large. To the extent that material becomes more discoverable, open science also naturally supports collaboration, multidisciplinary, interdisciplinary, and sustainability and enables more complex research with diverse teams.

Open science output also becomes more easily discoverable, owing to its public accessibility on platforms used by scholars and the larger public (e.g., journalists and policymakers). The enhanced visibility of a researcher, in turn, may lead to a significant expansion in a researcher’s network and potentially new (academic) collaborations. For example, a well-known case study in chemical research (Woelfle et al., 2011) shows how setting up an open project helped accelerate the scientific discovery process because relevant experts could identify themselves, rather than the lead researchers needing to rely on their network to identify the right people to ask for input.

Inclusivity and diversity

Open science promises to foster inclusivity and diversity. It is subject to debate whether it really does in all aspects, especially whether open science promotes equity between financially well- and less-well-situated research institutions (e.g., Ross-Hellauer et al., 2022). However, the open project case study of Woelfle et al. (2011) attracted a much broader and more diverse team of experts working on it than would have been possible in a more closed project. Recent network analyses also suggest that the open science literature has a more collaborative structure and uses more communal and pro-social language than does the comparable but largely independently developing reproducibility literature (Murphy et al., 2020).

Valuing open science output may also lead to the tighter inclusion of scholars with diverse talents. For example, research on complex choice models could rarely be carried out without expert knowledge of optimizing the underlying computer code. At the same time, open science makes available the process of research not only in the form of papers but also in the form of computer code, supporting multiple ways for researchers to learn about a particular problem.

Recommendations to Build a Strategic Advantage with Open Science

Open science is a contemporary and innovative way of conducting research and has the potential to accelerate research progress and increase efficiency through enhanced verification and collaboration. The goals of open science – to “increase transparency, accountability, equity and collaboration, and knowledge production by increasing access to research results, articles, methods, and tools” (Ross-Hellauer, 2022) – closely align with Tilburg University’s shared values; to be curious, caring, connected, and courageous researchers. Open science, by promoting transparent research, helps us search for and better evaluate new knowledge and insights, fostering curiosity. Open science promotes a diverse culture, true to the strategic goal of being caring scholars that respect each other and draw strength from differences. Freely sharing code, data, or other materials helps us to connect— to learn from other disciplines and embrace variety. Finally, open science fosters courage; it requires courage to put your code out in the open, to share your data freely, to expose yourself to very public criticism. In sum, we believe embracing open science is beneficial for our university and strengthens our shared values.

Among the research community in Tilburg, however, a feeling persists that open science practices are nice-to-have, but either not crucial or even risky for career progression. Thus, while open science is generally believed to be beneficial for society, whether it benefits individual researchers enough is not always clear, creating inertia in the adoption of open science. Below, we list recommendations that may alleviate personal concerns and pave the way for better science, i.e., open science, at Tilburg University and beyond.

First, reiterating calls by others (e.g., Gelman 2017; Allen and Mehler, 2019), we recommend fostering a culture where making mistakes is acceptable, thereby addressing one of the key concerns against the adoption of open science. To set the right incentives, sloppy work must have negative consequences. However, stigmatization is undesirable and unwarranted. In this respect, academia can take a cue from professional software development. Commercial software is rarely bug-free, despite many testing routines and extensive training for writing robust code. Why expect research to be bug-free when those systems are not in place, and many researchers are not professionally trained coders? We believe a change of culture would be a major step toward reducing this obstacle. One way to foster a mindset change could be to promote opt-in initiatives such as a “bug bounty hunt,” rewarding both the discovery (for the hunters) and the severity (for the participating researchers) of software bugs in replication packages prior to a paper’s first journal submission. Such an initiative may not only foster a climate where making mistakes is acceptable (and even rewarded) but may also directly lead to replication packages that do replicate.

Second, since many view open science as just nice-to-have and to increase the inclusivity of diverse types of research talent, we recommend conducting (experimental) research on how open science contributions at Tilburg University’s schools can be measured and valued. An evaluation does not need to be quantitative. A more subjective approach to evaluating open science contributions may involve peer feedback of (non-peer-reviewed) software or datasets and their documentation. Understanding how to best value these and similar contributions seems key to increasing the incentives for wider adoption of open science practices and could even make an impact on the broader scientific community. In addition, further integrating open science contributions in existing systems (e.g., linking a researcher’s GitHub profile to Pure) would already increase the internal visibility (and likely appreciation) of these and similar contributions.

Third, we recommend introducing open science practices to all our educational programs, both in the initial stages (in which students still discover a way of working), and in the later stages (in which students work on their Bachelor’s and Master’s theses). Such a step would establish open science as an alternative mode of conducting research in just a few years and thereby drastically reduce the upfront investments that early-career researchers will have to incur. In our teaching, we have learned that students enjoy collaborating on public coding projects or publishing datasets as a team. Embracing Tilburg Science Hub or similar platforms would be a relatively low-cost way for faculty and students alike to gradually learn to conduct open science. To promote working publicly on research projects, Tilburg University could invest in a campus license for state-of-the-art development tools for students and staff (e.g., GitHub, Bitbucket, or similar coding platforms), making software code developed at Tilburg University accessible to a broader community.

Fourth, open science practices are still developing at a rapid pace. To help spread evolving best practices, we advocate for researcher-led open science support. Such support services could assist researchers in developing and improving scientific software and code, test replication packages, or help turn prototype code into stable software packages. The initiative could be embedded in Tilburg’s new Data Competence Center but could also exist at the level of schools (e.g., TiSEM-funded Tilburg Science Hub), or departments. Templates of such initiatives exist, as some leading research institutions have already set up labs where procedures for documenting data and reviewing code are standardized, leading to large efficiency gains for lab members (e.g., http://whitaker-lab.netlify.app at The Alan Turing Institute, https://github.com/gslab-econ by Stanford and Harvard scientists).

Fifth, we encourage researchers to discuss the introduction of open science practices in journals more openly, going beyond the “gold route” to publishing open access. If more (top) journals embraced open science, such as by hosting and verifying replication packages, output quality and impact factors may rise, and incentives for researchers may increase to start doing open science. This shift is already happening in leading journals, such as the American Economic Journal (AER). In the review process, early-career researchers interested in open science can request authors to provide details on a paper more openly, such as through code or data. In a similar vein, researchers can start a conversation about the adoption of open science principles within departments. We observe young(er) researchers like working openly. Some research groups at Tilburg University have explicitly adopted open science principles. Research groups interested in becoming more open can follow the advice in Lowndes et al. (2017, 2019). We note that the transition to open science may equip departments with a unique competitive edge in their field, and therefore advise departments to start the conversation with team members.

Conclusion

In this essay, we have reflected on how open science can contribute to better science, and we have made recommendations for further transitioning to a more open research culture at Tilburg University. While we believe the arguments for a broad adoption are convincing, one should always remain “open” to different approaches of doing research, including a more proprietary take on software, code, and data.

One limitation of this essay is that both authors are quantitative scholars engaging in data-intensive research. For more qualitative disciplines, open science is unlikely to be conducted through writing software code. We hence encourage scholars to contribute to a conversation about the adoption of open science in their schools. Similarly, Tilburg University is home to more open science initiatives than we could mention in this essay. We, therefore, encourage scholars curious about the practice of open science at Tilburg University to consult the resources provided by the Open Science Community.8

Despite these limitations, we hope our essay contributes to an open discussion about open science at Tilburg University. While open science may not be the only way to ensure the execution of research with utmost integrity, we are confident about its vital role in ensuring the credibility of science and leading to new ways of working together, which – by the way – can be a lot of fun.

References

Allen, Christopher, and David MA Mehler. 2019. “Open Science Challenges, Benefits and Tips in Early Career and Beyond.” PLoS Biology 17, no. 5: 1–14.

Botvinik-Nezer, Rotem, Felix Holzmeister, Colin F. Camerer, Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette A. Mumford, and R. Alison Adcock. 2020. “Variability in the Analysis of a Single Neuroimaging Dataset by Many Teams.” Nature 582, no. 7810: 84–88.

Camerer, Colin F., Anna Dreber, Felix Holzmeister, Teck-Hua Ho, Jürgen Huber, Magnus Johannesson, Michael Kirchler, Gideon Nave, Brian A. Nosek, and Thomas Pfeiffer. 2018. “Evaluating the Replicability of Social Science Experiments in Nature and Science between 2010 and 2015.” Nature Human Behaviour 2, no. 9: 637–44.

Collaboration, Open Science. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349, no. 6251: aac4716.

Gelman, Andrew. 2017. “Ethics and Statistics: Honesty and Transparency Are Not Enough.” Chance 30, no. 1: 37–39.

Gelman, Andrew, and John Carlin. 2014. “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors.” Perspectives on Psychological Science 9, no. 6: 641–51.

Lawrence, Steve. 2001. “Free Online Availability Substantially Increases a Paper’s Impact.” Nature 411, no. 6837: 521–521.

LeBel, Etienne P., Lorne Campbell, and Timothy J. Loving. 2017. “Benefits of Open and High-Powered Research Outweigh Costs.” Journal of Personality and Social Psychology 113, no. 2: 230.

Lewis, Jake, Susan Schneegans, and Tiffany Straza. 2021. UNESCO Science Report: The Race against Time for Smarter Development. Vol. 2021. UNESCO Publishing.

Lowndes, Julia S. Stewart, Benjamin D. Best, Courtney Scarborough, Jamie C. Afflerbach, Melanie R. Frazier, Casey C. O’Hara, Ning Jiang, and Benjamin S. Halpern. 2017. “Our Path to Better Science in Less Time Using Open Data Science Tools.” Nature Ecology & Evolution 1, no. 6: 1–7.

Lowndes, Julia S. Stewart, Halley E. Froehlich, Allison Horst, Nishad Jayasundara, Malin L. Pinsky, Adrian C. Stier, Nina O. Therkildsen, and Chelsea L. Wood. 2019. “Supercharge Your Research: A Ten-Week Plan for Open Data Science.” Nature. https://www.nature.com/articles/d41586-019-03335-4

Murphy, Mary C., Amanda F. Mejia, Jorge Mejia, Xiaoran Yan, Sapna Cheryan, Nilanjana Dasgupta, Mesmin Destin, Stephanie A. Fryberg, Julie A. Garcia, and Elizabeth L. Haines. 2020. “Open Science, Communal Culture, and Women’s Participation in the Movement to Improve Science.” Proceedings of the National Academy of Sciences 117, no. 39: 24154–64.

National Academies of Sciences, Engineering. 2019. “Reproducibility and Replicability in Science.” National Academies Press.

Pachali, Max J., and Hannes Datta. 2022. “What Drives Demand for Playlists on Spotify?” Available at SSRN.

Pashler, Harold, and Christine R. Harris. 2012. “Is the Replicability Crisis Overblown? Three Arguments Examined.” Perspectives on Psychological Science 7, no. 6: 531–36.

Reinhart, Carmen M., and Kenneth S. Rogoff. 2010. “Growth in a Time of Debt.” American Economic Review 100, no. 2: 573–78.

Ross-Hellauer, Tony. 2022. “Open Science, Done Wrong, Will Compound Inequities.” Nature 603, no. 7901: 363–363.

Ross-Hellauer, Tony, Stefan Reichmann, Nicki Lisa Cole, Angela Fessl, Thomas Klebel, and Nancy Pontika. 2022. “Dynamics of Cumulative Advantage and Threats to Equity in Open Science: A Scoping Review.” Royal Society Open Science 9, no. 1: 211032.

Serwadda, David, Paul Ndebele, M. Kate Grabowski, Francis Bajunirwe, and Rhoda K. Wanyenze. 2018. “Open Data Sharing and the Global South—Who Benefits?” Science 359, no. 6376: 642–43.

Silberzahn, Raphael, Eric L. Uhlmann, Daniel P. Martin, Pasquale Anselmi, Frederik Aust, Eli Awtrey, Štěpán Bahník, Feng Bai, Colin Bannard, and Evelina Bonnier. 2018. “Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results.” Advances in Methods and Practices in Psychological Science 1, no. 3: 337–56.

Simmons, Joseph P., Leif D. Nelson, and Uri Simonsohn. 2011. “False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.” Psychological Science 22-11.

Steegen, Sara, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. 2016. “Increasing Transparency through a Multiverse Analysis.” Perspectives on Psychological Science 11, no. 5: 702–12.

UNESCO. 2021. UNESCO Recommendation on Open Science Report. UNESCO Publishing.

Vilhuber, Lars. 2020. “Reproducibility and replicability in economics.” Harvard Data Science Review. https://doi.org/10.1162/99608f92.4f6b9e67

Wang, Xianwen, Chen Liu, Wenli Mao, and Zhichao Fang. 2015. “The Open Access Advantage Considering Citation, Article Usage and Social Media Attention.” Scientometrics 103, no. 2: 555–64. https://doi.org/10.1007/s11192-015-1547-0.

Wichmann, Julian R.K., Abhinav Uppal, Amalesh Sharma, and Marnik G. Dekimpe. 2022. “A Global Perspective on the Marketing Mix across Time and Space.” International Journal of Research in Marketing 39, no. 2: 502–21. https://doi.org/10.1016/j.ijresmar.2021.09.001.

Woelfle, Michael, Piero Olliaro, and Matthew H. Todd. 2011. “Open Science Is a Research Accelerator.” Nature Chemistry 3, no. 10: 745–48. https://doi.org/10.1038/nchem.1149.

Sokolova, Tatiana, Satheesh Seenivasan, and Manoj Thomas. 2020. “The Left-Digit Bias: When and Why Are Consumers Penny Wise and Pound Foolish?” Journal of Marketing Research 57, no. 4: 771–88. https://doi.org/10.1177/0022243720932532.

Zeelenberg, Marcel, Terri G. Seuntjens, Niels van de Ven, and Seger M. Breugelmans. 2022. “Dispositional Greed Scales.” European Journal of Psychological Assessment 38, no. 2: 91–100. https://doi.org/10.1027/1015-5759/a000647.