Preface

This book grew out of a recurring and dispiriting experience: an analysis that ran perfectly on the machine where it was written, and nowhere else. The script that produced last year’s figures now errors on a colleague’s laptop. The supplement that accompanied a published paper will not knit, because a package it silently depended on has moved on. The reviewer who asks, reasonably, for the code receives a folder that does everything except reproduce the result. None of these failures is a failure of statistics. Each is a failure of computational reproducibility, and each was avoidable.

The health sciences have absorbed the lesson that a result must be reproducible in principle, and journals now ask for the code. What the field has not yet absorbed is that making a computational result reproducible in practice is a technical discipline in its own right. It has its own tools, its own failure modes, and its own standards of evidence. In this book we teach that discipline to an audience that already has the quantitative training to appreciate why it matters.

The organizing idea

The organizing idea of the book is that reproducibility is layered, and that honesty about which layer one has reached is itself a scientific virtue. We develop a ladder of four levels. A locatable analysis (L0) is under version control and can be found, but nothing about its computational environment is fixed. A pinned analysis (L1) records the exact versions of the packages it used. A pinned-environment analysis (L2) fixes, in addition, the operating system, the language runtime, and the system libraries, most often inside a container. A verified analysis (L3) has gone one step further: its recorded computational outputs have been regenerated from scratch and shown to match. In the chapters that follow we teach how to reach each level and how to tell which one an analysis has actually reached. Just as important, we consider when a lower level is the honest and sufficient choice.

Running underneath the ladder is a distinction we return to constantly: the difference between capturing reproducibility and validating it. Pinning packages and pinning the environment are acts of capture; they fix the inputs and thereby set the level. Continuous integration, dependency checking, and verification are acts of validation; they confirm the level without capturing anything new. Conflating the two is the source of a great deal of false confidence.

The tool, and why it is here

The book uses zzcollab, an R framework for building reproducible research compendia, as its concrete apparatus. The choice is deliberate and, we hope, honestly motivated. Reproducibility is learned by building something that reproduces, and that requires a real toolchain. zzcollab composes the established R tools into one workflow organized by the levels ladder. This allows us to teach the concepts and demonstrate them in the same breath. A reader who prefers a different toolchain will find that the concepts transfer without loss. That is, the ladder, the compendium, and the capture-validation distinction are properties of the problem, not of any one program.

What this book covers

Following the introduction, the remaining twelve chapters are arranged in four parts, which we outline here.

  1. Foundations. The reproducibility crisis and its specifically computational character; a framework of reproducibility levels and the determinants that set them; version control with git as the foundation of a locatable analysis; and the research compendium as the unit of reproducible work.
  2. Capturing the environment. Pinning packages with a lockfile; pinning the operating system and runtime with a container; and the alternatives, from Nix to Podman to the high-performance-computing runtime Apptainer.
  3. Validation. Literate programming and the rendered report; automating the analysis with make and targets; testing and code quality; and continuous integration and computational verification, the acts that turn a claimed level into a confirmed one.
  4. Collaboration and stewardship. Sharing a compendium with a team and with the world: the team environment model, package registries, archival deposition, and the FAIR principles.

What this book does not cover

The book is not a statistics text. We assume the reader can already conduct an analysis, and we concern ourselves only with making that analysis reproducible. It is not a comprehensive manual for any single tool; where a tool’s own documentation is the right reference, we point to it. Nor is it a treatise on replicability in the scientific sense, the question of an independent study reaching the same conclusion on new data. That is a vital matter, but a different one. We are careful throughout to keep the computational question of re-execution distinct from it.

Nor is the book a manual of computer-system validation for regulated work. Where we touch 21 CFR Part 11, qualification, or validation in the regulatory sense, we do so to keep the vocabulary straight and to mark where a regulated setting asks for more. A reader preparing analyses for submission should treat those passages as orientation only. The governing documents are the regulations themselves, the sponsor’s standard operating procedures, and guidance such as the R Validation Hub’s (Nicholls et al., 2020).

Reading it in one quarter

We have written the book for a one-quarter, ten-week course, but not every chapter fits that schedule. The Conventions page marks a core path through the thirteen chapters. It also names the material a ten-week offering has no room for, namely Backends and Runtimes in full and the latter half of Automating the Analysis. An instructor teaching to that pace will find a week-by-week table built from the same core path in the instructor guide, which is distributed separately to course instructors and is not part of this book. A reader with more time, or working alone, can simply read the book in order. Nothing marked optional is weaker material, only material a shorter term cannot fit.

Chapter template

Each content chapter follows the same sequence. It opens with learning objectives and an orientation that motivates the chapter from a concrete research problem. It then identifies the analyst’s contribution that no tool automates, and proceeds through content sections with worked R and shell examples. It closes with a section on collaborating with a language model, exercises, and further reading. The pattern repeats deliberately, so that by the third chapter the reader knows where to find each component.