Open and reproducible research
Motivations and tools
Let’s envision the research process
In computational research the traditional publication is simply advertisement for the research stored in implemented algorithms.
— Jon Claerbout, 1998, “The idea of reproducible research”
Why bother “reproducing” research: a case study
Why bother “reproducing” research
Within a few years, this work was used for framing national economic decisions, e.g. in the 2013 budget from the US Speaker of the House:
A well-known study completed by economists Ken Rogoff and Carmen Reinhart confirms this common-sense conclusion. The study found conclusive empirical evidence that gross debt (meaning all debt that a government owes, including debt held in government trust funds) exceeding 90 percent of the economy has a significant negative effect on economic growth.
— Paul Ryan’s 2013 “Path to prosperity” budget proposal
Why bother “reproducing” research
But it turns out that the 2010 paper might not provide a solid foundation…
We were unable to replicate the RR results from the publicly available country spreadsheet data although our initial results from the publicly available data closely resemble the results we ultimately present as correct.
Reinhart and Rogoff kindly provided us with the working spreadsheet from the RR analysis. With the working spreadsheet, we were able to approximate closely the published RR results.
From this spreadsheet, three sets of concerns:
- Selective exclusion of available data and data gaps
- Spreadsheet coding error
- Unconventional weighting of summary statistics
Spreadsheet coding error:
A coding error in the RR working spreadsheet entirely excludes five countries, Australia, Austria, Belgium, Canada, and Denmark, from the analysis. The omitted countries are selected alphabetically and, hence, likely randomly with respect to economic relationships.
This spreadsheet error, compounded with other errors, is responsible for a −0.3 percentage- point error in RR’s published average real GDP growth in the highest public debt/GDP category
Selective exclusion of available data and data gaps:
More significant are RR’s data exclusions with three other countries: Australia (1946–1950), New Zealand (1946–1949), and Canada (1946–1950).
The exclusions for New Zealand are of particular significance. This is because all four of the excluded years were in the highest, 90 percent and above, public debt/GDP category. Real GDP growth rates in those years were 7.7, 11.9, −9.9, and 10.8 percent. […] The exclusion of the missing years is alone responsible for a reduction of −0.3 percentage points of estimated real GDP growth in the highest public debt/GDP category.
Unconventional weighting of summary statistics:
After assigning each country-year to one of four public debt/GDP groups, RR calculates the average real GDP growth for each country within the group, that is, a single average value for the country for all the years it appeared in the category.
For example, real GDP growth in the UK averaged 2.4 percent per year during the 19 years that the UK appeared in the highest public debt/GDP category while real GDP growth for the US averaged −2.0 percent per year during the 4 years that the US appeared in the highest category.
Both were weighed equally in the final analaysis, despite one country contributing 19 years of data, vs. 4 years for the other.
Why bother “reproducing” research, pt. 2
A series of papers in 2006 by a Duke Univ. researcher suggested a genetic basis for chemotherapy response.
This led to to several new clinical trials, involving hundreds of patients.
Researchers at MD Anderson Cancer Center were unable to replicate analysis and uncovered serious fraud
But try as they might, Coombes and Baggerly could not reproduce the results. Even worse, they found serious errors in Potti’s work. Columns of data had been shifted and sample labels switched. When Coombes and Baggerly corrected the problems, the correlations vanished. In other words, patients who used the gene screening to choose a chemotherapy were essentially flipping a coin.
- The clinical trials were suspended, patients suffered needless emotional trauma, and large amonts of resources were wasted
See this news report for details
Why bother “reproducing” research, pt. 3
In 2019, a researcher reading a 2014 paper about social behavior in spiders and noticed irregularities in the underlying data
This led the corresponding author to do a deep dive into the dataset, which reveealed a series of irregularities in this and other related datasets generated in Jonathan Pruitt’s lab
Ultimately, this led to >15 paper retractions, and drained time and mental resources from the ecology research community, especially junior researchers. . . .
See this blog post by Dr. Kate Laskowski for details
Why bother creating reproducible research
Good science is done in community and in relationship with other work
Transparency and reproducibility is one of the easiest ways to make our research more meaningful and robust.
If the scientific process is to be “self-correcting” reproduction and verification of analyses is essential
In this course, we will touch on how to build reproduciblity throughout the research cycle.
Reproducible, replicable, robust, and generalizable
Making sense of the linguistic menagerie
Image from The Turing Way
As defined here, reproducibility is the most trivial form of verification.
So, please consider this course to be just one step on journey to doing good science!
What is “open” science, and how does relate to reproducible science?
Many ways to understand “open” science
Open science can be understood as a set of practices
From the Center for Open Science
Open science can be understood in terms of principles
Open science is a set of principles and practices that aim to make scientific research from all fields accessible to everyone for the benefits of scientists and society as a whole.
Open science is about making sure not only that scientific knowledge is accessible but also that the production of that knowledge itself is inclusive, equitable and sustainable.
Open science can be understood as a movement
Open Science is the movement to make scientific research, data and their dissemination available to any member of an inquiring society, from professionals to citizens. It impinges on principles of scientific growth and public access including practices such as publishing open research and campaigning for open access, with the ultimate aim of making it easier to publish and communicate scientific knowledge.
From Orion Open Science
Open science can be understood as an ethos
The ethos of Open Science lies in the belief that broader access to meticulously curated scientific data will contribute to a richer accumulation of knowledge. All space-related biological research data are precious national data resources.
From NASA’s Open Data Science Repository
Open Science can be understood as an idea
I have an informal definition I sometimes use: “Open science is the idea that scientific knowledge of all kinds should be openly shared as early as is practical in the discovery process.”
By scientific knowledge “of all kinds” I include journal articles, data, code, online software tools, questions, ideas, and speculations; anything which can be considered knowledge.
The “as is practical” clause is included because very often there are other factors (legal, ethical, social, etc) that must be considered. This can get extremely complex, and requires, in my opinion, extended discussion.
A nice feature of the above informal definition is that it makes it clear that when the journal system was developed in the 17th and 18th centuries it was an excellent example of open science. The journals are perhaps the most open system for the dissemination of knowledge that can be constructed — if you’re working with 17th century technology. But, of course, today we can do a lot better.
From a 2011 post on the Open Knowledge Foundation listserv