Each individual nudge is made possible by using a survey or similar tool to measure and quantify the availability of digital artifacts, reproducibility of published results, and replicability of findings. We welcome discussion to improve the survey tool and to improve the reproducibility of our science. The authors translated definitions of availability, reproducibility, and replicability into a question Qualtrics Research Core Qualtrics online survey Fig.
The survey progressed from soliciting metadata about the article Questions 1—4 , to testing availability of artifacts Q5—9 , and ultimately testing reproducibility of results Q10— Green or yellow shaded answers Fig. Selecting a red-shaded answer stopped progression and directed the reviewer to a final question that asked how many minutes the reviewer spent to reach their stopping point Q This time to complete was self-reported by reviewers rather than using the built-in Qualtrics timer so reviewers could consider the entire time spent reading and assessing the published article and artifacts, rather than the time completing the survey.
The authors developed the tool over four months in Fall and pre-tested it in early on a sub-sample of five articles that spanned the availability and reproducibility progression. From our experience pre-testing and to improve use of the tool, we reworded some questions, altered the survey logic, discussed and addressed inter-reviewer variability.
Instead, we calculated time spent using papers from the remaining sample. Journals were selected based on impact and to cover a range of hydrology and water resources topics. Stratified random sampling was approximately proportional to the number of articles published in each journal in , with extra weight placed on articles with a set of reproducibility-related keywords Table 1.
Of the articles published in the six journals in that had at least one keyword, we sampled articles, principally to retain at least 15 non-keyword articles for each journal with an approximately non-keyword to keyword ratio overall.
Each author was randomly assigned 60 articles stratified by journal to assess the availability of article artifacts Q1—9. After identifying all publications that had the available artifacts, we re-assigned reviewers to assess whether the published results could be reproduced Q1— The Qualtrics online format allowed us to both simultaneously and asynchronously assess journal articles and store survey responses in an accompanying Qualtrics database.
After all availability and reproducibility assessments were complete, we exported results from the Qualtrics database to a text file which was processed in R to generate figures, tables, and results presented in this article. Time spent to complete the survey Q15 was analyzed for three key stopping points: no artifacts available Q5 , availability of artifacts Q9 , and reproducibility of results Q Sampled articles were sorted into six mutually exclusive categories that were stopping points in the survey: Data-less or review, Author or Third Party Only, No availability, Some availability, Available but not reproducible, and Some or all reproducible.
The resampling approach generated 5, random populations. Each population had 1, articles. In each population, we inserted the articles we manually assessed, assuming that we exactly knew the reproducibility of these articles. Estimates for the remaining 1, unsampled articles were simulated based on survey results for the sampled articles in their stratified category, i. For each random sampled population, the proportion of unsampled articles in each reproducibility category was randomly simulated using the multinomial uncertainty approach of Sison and Glaz 51 , This produced 5, sample populations equal in size and distribution journal and keyword to the true population of articles published in , while incorporating uncertainty due to unsampled papers.
The survey tool, Qualtrics results, and all code used for analysis presented in this article are available online through the permanent repository Please cite this repository for any use of the related data or code. Additionally, results can be reproduced using RStudio deployed in the cloud using MyBinder through the GitHub website. All relevant data presented in this article are available online through the permanent repository A pdf image of the survey tool is also available in the permanent repository How to cite this article : Stagge, J.
Assessing data availability and research reproducibility in hydrology and water resources. Sandve, G. Aarts, A. Estimating the reproducibility of psychological science. Science , 1—8 Baker, M. Nature , — Gil, Y. Toward the geoscience paper of the future. Earth and Space Science 3 , — Brembs, B. Frontiers in Human Neuroscience 12 , 37 Stodden, V. An empirical analysis of journal policy effectiveness for computational reproducibility.
Proceedings of the National Academy of Sciences , — Kovacevic, J. How to Encourage and Publish Reproducible Research. Peng, R. Reproducible research and Biostatistics. Biostatistics 10 , — Setting the default to reproducible in computational science research. SIAM News 46 , 4—6 Easterbrook, S. Open code for open science? Nature Geoscience 7 , — Leek, J. Opinion: Reproducible research can still be wrong: Adopting a prevention approach. Pulverer, B. Reproducibility blues. Goodman, S. What does research reproducibility mean? Science Translational Medicine 8 ps12 Melsen, L.
Water Resources Research 53 , — Plesser, H. Reproducibility vs. Frontiers in Neuroinformatics 11 , 76 Akmon, D. The application of archival concepts to a data-intensive environment: working with scientists to understand data management and preservation needs.
Archival Science 11 , — Hutton, C.
Most computational hydrology is not reproducible, so is it really science? Water Resources Research 52 , — Casadevall, A. Reproducible Science. Infection and Immunity 78 , — Drummond, C.
Reproducible research: a minority opinion. Fary, M. Shen, Y.
International Journal of Digital Curation 10 , — Shiffrin, R. Scientific progress despite irreproducibility: A seeming paradox. Diekema, A. The Journal of Academic Librarianship 40 , — Wallis, J. Kaufman, D. PAGES 2k special-issue editorial team. Past Discuss , 1—10 Gabridge, T. Bracke, M. Pinfield, S. Weller, T. Journal of eScience Librarianship 4 , 2 Horsburgh, J. Essawy, B. Integrating scientific cyberinfrastructures to improve reproducibility in computational hydrology: Example for HydroShare and GeoTrust. Gillman, M.
Calibration of a modified temperature-light intensity logger for quantifying water electrical conductivity. Measuring water use, conservation, and differences by gender using an inexpensive, high frequency metering system. Environmental Modelling and Software 96 , 83—94 Neuwirth, C. System dynamics simulations for data-intensive applications. Environmental Modelling and Software 96 , — Xu, W. Detecting spatial patterns of rivermouth processes using a geostatistical framework for near-real-time analysis.
Environmental Modelling and Software 97 , 72—85 Buscombe, D.
Shallow water benthic imaging and substrate characterization using recreational-grade sidescan-sonar. Environmental Modelling and Software 89 , 1—18 Yu, C. Consistent initial conditions for the Saint-Venant equations in river network modeling. Hydrology and Earth System Sciences 21 , — Di Matteo, M. Multiobjective optimization of distributed stormwater harvesting systems.
Journal of Water Resources Planning and Management Engdahl, N. Lagrangian simulation of mixing and reactions in complex geochemical systems. Landscape-scale water balance monitoring with an iGrav superconducting gravimeter in a field enclosure. Sattar, A. Three dimensional modeling of free surface flow and sediment transport with bed deformation using automatic mesh motion. Environmental Modelling and Software 97 , — Nosek, B.
Promoting an open research culture.