3  Levels of Reproducibility

Reproducibility has the potential to serve as a minimum standard for judging scientific claims when full independent replication of a study is not possible.

Roger D. Peng, Reproducible Research in Computational Science (2011)

By the end of this chapter you will be able to look at any repository, your own or someone else’s, and say exactly which of four rungs it has reached. You will also be able to say what still stands between it and the next one, rather than reach for the unfalsifiable claim ‘fully reproducible.’

NoteBefore you start

Reading time. About 40 minutes.

Prerequisites. Chapter 1 (the introduction), for the first, informal pass at the determinant stack this chapter makes precise, and Chapter 2 (Chapter 2), for the two silent defaults of R itself; no other chapter is assumed. Statistically, nothing beyond reading a regression output, a hazard ratio or an odds ratio with its confidence interval, is required. No R fluency beyond reading a short script is assumed; the chapter’s own code chunks are simple enough to follow without having written R before.

By the end of this chapter you will be able to answer:

  • Can you enumerate the layers of the determinant stack on which a computational result stands, and say which layers common practice pins and which it leaves to chance?
  • Can you place an analysis on the four-rung ladder of this book (L0 locatable, L1 pinned packages, L2 pinned environment, L3 verified) and name the artifact that marks each transition?
  • Can you distinguish capturing reproducibility from validating it, and explain why continuous integration adds no new pinned layer?
  • Can you distinguish a result that is reproduced from one that is replicated, and navigate the shifting use of those two words across the literature?
  • Can you state plainly the level an analysis has reached, and no higher, including the candid admission that no present framework pins the whole stack?

Outline.

  • Orientation: a result that silently changed on a second machine.
  • The analyst’s contribution: the three judgments no tool automates.
  • The stack a result stands on: the nine layers, and which rungs secure which.
  • The ladder: capture sets the level, validation confirms it.
  • Reproduced, replicated, and a warning about the words.
  • Two numbering schemes, and which this book uses.
  • The ladder grades every archetype.
  • No framework pins the whole stack.
  • Worked example: one analysis walked up all four rungs.
  • The cohort, continued: placing this book’s own compendium on the ladder, today.

A colleague re-runs a 2018 heart-failure analysis, written under R 3.5, on a laptop with a current release of R. The script calls set.seed(42) before a bootstrap built on sample(), yet the bootstrap interval has moved. In the same run, levels(d$arm) on a column read by read.csv() returns NULL. Drawing on Chapter 1 and Chapter 2, is this a replication failure? For each discrepancy, name the R release and the determinant layer responsible, and say what would close it.

Neither change in the re-run 2018 analysis is a replication failure. The same code ran on the same data, so by the definitions of Chapter 1 and Chapter 2 this is a failure of computational reproducibility, with a computational cause. The moved interval traces to R 3.6.0, which changed the default sample.kind: the seed was fixed, but the sampler that turns it into draws was not (Table 2.1). The NULL levels trace to R 4.0.0, which flipped the stringsAsFactors default to FALSE, so arm now arrives as a character vector. Both are defaults of the R runtime, and pinning the R release, which a container does at L2, fixes both. Each can also be closed in the analysis code: an explicit RNGkind() call, the only fix for an override, and an explicit stringsAsFactors argument, as load_cohort() writes it.

3.1 Orientation

Consider a registry study of thirty-day readmission after heart failure. The analyst fits a mixed-effects model, draws a forest plot, and reports an adjusted odds ratio of 1.24. The script runs, the manuscript renders, the paper is accepted. Two years later a reader requests the code, downloads it, and runs it on a fresh laptop. The model fits, but differently. A package that the analysis relied upon has changed its default optimizer, and the estimate now returned is 1.19. Which number is the finding? The discrepancy is computational in origin. The model specified was the same on both machines; what differed was everything underneath it, and the analyst never recorded it. Its size, however, is a statistical finding. An estimate that moves from 1.24 to 1.19 when only the optimizer changes signals a flat or ill-conditioned likelihood, or a fit that has not converged. It should be checked, for example by refitting under several optimizers with lme4::allFit(), and reported as optimizer-sensitive.

This chapter is about naming that ‘everything underneath’, and about grading how much of it a given analysis has secured. We introduced in Chapter 1 the observation that a computational result depends on a stack of determinants beneath the code and the data, and that a published script pins almost none of them. Here we make that observation precise. After setting out the judgments that remain with the analyst, we shall lay out the stack layer by layer. We then define the ladder that this book uses to say how far down the stack a particular effort has reached: L0 locatable, L1 pinned packages, L2 pinned environment, and L3 verified. With the ladder in hand, we consider how the words ‘reproduced’ and ‘replicated’ have shifted across the literature, and how the framework’s own numbering differs from ours. Lastly, we examine how the ladder grades each archetype, admit that no framework pins the whole stack, walk one analysis up all four rungs, and then place this book’s own compendium on the ladder.

The ladder is the conceptual backbone of everything that follows. Every later chapter is, in effect, an account of how to climb one rung of it. Pinning packages with a lockfile (Chapter 6) is the L0-to-L1 step. Pinning the environment with a container (Chapter 7) is the L1-to-L2 step. And verifying the outputs (Chapter 12) is the L2-to-L3 step. Before we can teach the climbing we need the ladder itself, together with the discipline of naming, without flinching, the rung one has actually reached.

In the registry-study story above, the code and the data were unchanged between the two runs. What changed, why does calling the discrepancy purely ‘statistical’ mischaracterize its origin, and what statistical question does it nonetheless raise?

A package the analysis depended on changed its default optimizer between the two runs. That change sits on the package-versions layer of the determinant stack, not on the analysis code or the data. The model specified was identical on both machines; what differed was an unpinned determinant beneath it. Calling the discrepancy purely ‘statistical’ hides a computational failure behind a word that belongs to a different kind of disagreement, one about the data or the model. Its size is still a statistical question. An estimate that moves when only the optimizer changes comes from a poorly conditioned or unconverged fit, so it should be refit under other optimizers and reported as optimizer-sensitive. Pinning the package makes that fragile estimate reproducible; it does not make it trustworthy.

3.2 The analyst’s contribution

We begin with three judgments at the center of this chapter that belong to the analyst and are automated by no tool. The machinery of later chapters exists to support them.

  1. Choosing, and declaring, the honest level. A framework can compute which rung a repository has reached and print it; it cannot decide what the author will claim. The judgment that a paper’s reproducibility statement will match the level actually secured, and never assert above it, is the analyst’s alone. It is the single habit this book most wishes to instill.

  2. Matching the investment to the stakes. Climbing the ladder costs effort, and not every analysis warrants the climb. A two-day exploratory calculation may honestly rest at L0; a registry study destined for a guideline warrants L2 or L3. Deciding where on the ladder a given project should sit, given its lifespan, its audience, and the cost of being wrong, is a design decision.

  3. Naming what remains exposed. No rung of the ladder pins the whole stack. The analyst who reaches L2 must still ask whether the layers left unpinned (the numerical libraries, the random-number discipline, the integrity of an external data source) matter for this particular analysis. Those that do must be closed by hand. Telling a live exposure from an inert one is a matter of subject-matter judgment.

A vendor’s reproducibility dashboard reports that a project has reached ‘L2’. Which of the three judgments above, if any, has the dashboard made for the analyst, and which remain the analyst’s alone?

The dashboard can detect the capture artifacts (a lockfile and a container recipe) and so can compute the rung mechanically; that much is automatable. It cannot decide whether L2 was the right level to target for this analysis given its stakes. Nor can it know which unpinned layers, among the numerical libraries, the RNG kind, or an external data source, are live exposures for this particular question. Judging the match of effort to stakes, and naming what remains exposed, stay with the analyst no matter how good the dashboard is.

NoteThe vocabulary of this chapter

New terms are defined again on first use; they are collected here for reference.

  • Determinant stack. The layered set of things a computational result depends upon, from the hardware at the base through the environment and the packages to the analysis code and the data at the top. A layer is ‘pinned’ when it has been fixed to a recorded value.
  • Capture. Fixing an input so that it cannot drift: writing a lockfile, building a container image. Capture sets levels L0 to L2; L3 is reached by validation.
  • Validation. Confirming that the captured inputs actually suffice, by rebuilding and re-running. Validation does not pin anything new; it checks.
  • The level ladder (L0 to L3). The four-rung scale, L0 locatable, L1 pinned packages, L2 pinned environment, L3 verified, used throughout this book to grade how far down the stack an analysis has been secured.
  • Reproduced. Obtaining the same computational result by re-executing the same code on the same data. Contrast replicated, obtaining a consistent finding from a new study with new data.
  • Lockfile. A file, such as renv.lock, recording the exact version of every package an analysis used.
  • Container image. A packaged environment, built from a recipe, that carries the operating system, the system libraries, and the language runtime, and provides the same environment wherever a container runtime for its platform runs.
  • RNG kind. The random-number algorithm in force, set by RNGkind(). Two runs with the same seed but a different kind draw different numbers.

3.3 The stack a result stands on

We turn now to the determinants, because the ladder is only intelligible once we see what it grades. A computational result is the output of running some code against some data. However, the code and the data are only the two topmost of a tall stack of dependencies. The result is a function of the whole stack, not of its top two layers alone. Arranging the determinants from the hardware upward to the analysis logic yields the nine layers this book uses; a finer split, such as one that separates the document toolchain, yields ten or more.

At the base sits the hardware: the processor architecture, its instruction-set extensions, and the behavior of its floating-point unit. Above it lie the numerical libraries, the BLAS and LAPACK implementations that perform the linear algebra beneath every regression and every matrix decomposition. These matter more than their obscurity suggests. A different BLAS, or the same BLAS run with a different thread count, reorders the floating-point summations and can change a result in its low-order digits. Above these sit the operating system and its system libraries: the kernel, the C library, the locale data, the fonts, and the document toolchain of pandoc and LaTeX. Then comes the R runtime itself, whose default behaviors shift between releases. The random-number change of R 3.6.0 and the stringsAsFactors change of R 4.0.0 both illustrate the point.

Above the runtime sit the layers most analysts think of first. The package versions, the exact release of every R package the analysis loads, are the layer that churns fastest and breaks analyses most often. The session configuration is a cluster of settings that shape the session before any user code runs. It comprises the random-number kind set by RNGkind(), the contrasts used for factor variables, and the number of printed digits. It also comprises the locale that governs sorting, and the time zone that governs date arithmetic. Then come the two layers the analyst actually writes. These are the analysis code, including every set.seed() call and the order of execution, and the input data, its content and its integrity. Finally, threaded through the whole stack, sits intrinsic non-determinism: the use of the wall-clock, race conditions in parallel code, and iteration over unordered structures are each capable of producing a different answer on two runs of identical inputs.

Common practice pins very few of these layers. A bare published script pins exactly one layer with any reliability, the analysis code, since the code is the file itself. If the reader is fortunate, it pins a second, the data, when it has been deposited alongside. Every other layer is left to whatever the next machine happens to provide. Figure 3.1 maps the stack against the rung of the ladder that first secures each layer. Three layers (the hardware, the numerical libraries, and intrinsic non-determinism) are secured by no rung of the capture ladder at all. The numerical libraries are the case most often misread. A container image determines which BLAS is installed, but the ladder does not require that library to be fixed, and the thread count and the processor still vary between machines. No rung therefore guarantees this layer.

plot_determinant_stack(
  captures = determinant_layers()$layer,
  title = 'Which rung first secures each layer')
A horizontal bar chart of the nine-layer determinant stack, hardware at the bottom and intrinsic non-determinism at the top, each bar colored by, and labeled with, the rung that secures it across the book. This chapter captures Hardware, Numerical libraries (BLAS, LAPACK), Operating system, system libraries, R runtime, Package versions, Session configuration, Analysis code, Input data, Intrinsic non-determinism; those layers are drawn at full color, labeled as captured or validated here, and, where validated, with a heavy outline. Every layer is highlighted, so none is shown faded. From the base, the rungs are: Hardware, unpinned by any rung; Numerical libraries (BLAS, LAPACK), unpinned by any rung; Operating system, system libraries, L2 pinned environment; R runtime, L2 pinned environment; Package versions, L1 pinned packages; Session configuration, L2 partially pinned; Analysis code, L0 locatable; Input data, L0 locatable; Intrinsic non-determinism, unpinned by any rung.
Figure 3.1: The determinant stack, from the hardware at the base to the data at the top, colored by the rung of the ladder that first secures each layer. Package versions are secured at L1, the operating system and R runtime at L2, and the analysis code and data are merely locatable at L0. The session-configuration layer is only partially pinned at L2: the locale and the time zone are fixed, but the RNG kind, the contrasts, and the number of printed digits are not. Three further layers, the hardware, the numerical libraries, and intrinsic non-determinism, are pinned by no rung of the capture ladder. Every chapter that draws this stack draws it with the same function, as the Conventions page explains, so that no two chapters can disagree about what a layer is called or which rung secures it.

Two settings of the session-configuration layer deserve a word, because they are the settings most often forgotten by an analyst who has already adopted a lockfile and a container. The random-number kind is the more dangerous. An analysis that seeds its generator and draws a sample is reproducible only if the algorithm behind the seed is also fixed. The change of the default sample.kind in R 3.6.0 broke exactly this, silently, for every analysis that had set a seed under the old default: the same seed now drew different samples, so no seeded result regenerated digit for digit. Whether a result also changed as a statistical matter is a separate question. Chapter 11 answers it on the book’s cohort (Table 11.2): the shift is a fraction of one Monte Carlo standard error, so the bound reproduces statistically although not seed-exactly. Given that simulation studies, multiple imputation, and bootstrap confidence intervals are staples of health research, the RNG kind is a first-order concern. The locale is the second. sort() and order() on a character vector depend on LC_COLLATE, so an analysis developed under one locale and re-run under another can reorder a table without any error. The decimal mark is a different matter. R keeps LC_NUMERIC at ‘C’ and warns if code changes it, so base parsing reads ‘.’ as the decimal mark whatever the system locale. A file written with decimal commas is read as text unless the reading function is told otherwise, for example with dec = ',' in read.csv().

The RNG kind also shows why a setting can belong to two layers at once. The R release sets the default kind, so the default is part of the R runtime and is fixed at L2. Code or a project .Rprofile can override the default, and an override is session configuration, fixed only by an explicit RNGkind() call in the analysis code. The same distinction between a default and an override recurs across the stack, and the table below states it for the settings this book returns to most often.

Setting What fixes the default What can override it What closes the override
RNG kind (sample.kind) The R release (R runtime, L2) Code, .Rprofile, a package An explicit RNGkind() call in the analysis
stringsAsFactors The R release (R runtime, L2) The argument in each call Writing the argument in the code
Locale and time zone The image or the host (L2 when the image sets them) Sys.setlocale(), Sys.setenv(TZ = ) Setting them in the image or in the analysis
BLAS and LAPACK The image or the host (no rung) The library path, the thread count No rung; fixing the thread count narrows it, and a stated tolerance absorbs what remains

An epidemiologist reports that a colleague, re-running her cohort analysis, obtained a least-squares regression coefficient identical to five decimal places but a different value in the sixth. The fit was computed as solve(crossprod(X), crossprod(X, y)) on a large design matrix. The code, the data, the package versions, and the operating system were all identical. Which layer of the determinant stack is the most likely culprit, and is the discrepancy a bug?

Neither the code nor the data nor the pinned layers can be at fault, since all were identical. A discrepancy confined to the low-order digits is the signature of the numerical-library layer. crossprod() hands the matrix product to the BLAS, and solve() hands the linear system to LAPACK. The two machines used a different BLAS implementation, or the same one with a different thread count, and so the floating-point summations were reordered. It is not a bug in the analysis. It is the reason that byte-for-byte identity across different hardware is a stronger claim than number-for-number agreement on a fixed architecture. And it is a layer that no rung of the capture ladder pins.

3.4 The ladder: capture sets the level, validation confirms it

With the stack in view we can define the ladder. The central design idea, which we take from the framework this book uses and develop as a general principle, is that the features acting on reproducibility divide into two kinds that are constantly confused. Separating them is what makes an honest grade possible.

The capture axes fix inputs, and therefore set the level. There are two of them: the package backend, which pins the package layer, and the computational environment, which pins the operating system, the runtime, and the system libraries. Adding a capture feature raises the level; removing one lowers it. The validation layer confirms the captured level without capturing anything new. Its members are continuous integration, the dependency-consistency check, and the output-verification command. Removing a validation feature leaves a repository exactly as reconstructable as before; it only stops the automated checking. This is why continuous integration, which we develop in Chapter 12, is best understood as the harness that runs the verification rather than as a layer of reproducibility in its own right. Pinning is capture; re-running to confirm is validation.

The four rungs follow from this distinction. Levels L0 through L2 are reached by the capture axes, and L3 is reached by the validation layer. Table 3.1 states each rung, what it pins, the artifact that marks it, and what the rung means for a reader of health research.

library(tibble)
library(knitr)

ladder <- tibble::tibble(
  Level = c('L0 locatable',
            'L1 pinned packages',
            'L2 pinned environment',
            'L3 verified'),
  `What is pinned` = c(
    'Nothing computational; the source is under version control',
    'The exact version of every R package',
    'The operating system, the R runtime, and the system libraries as well',
    'Nothing new; the recorded outputs are shown to regenerate within a stated tolerance from a clean rebuild'),
  `Primary artifact` = c(
    'A version-controlled repository',
    'A lockfile such as renv.lock',
    'A container image or a Nix environment',
    'A passing baseline test, run after a clean rebuild, that compares regenerated outputs with values recorded in a committed baseline file within a stated tolerance'),
  `Health-research reading` = c(
    'A reviewer can find the code but cannot yet rebuild it',
    'A colleague installs the same packages the analysis used',
    'The analysis yields the same numbers on a different machine',
    'The reported table has been shown to regenerate, not merely asserted to'))

knitr::kable(ladder)
Table 3.1: The reproducibility ladder used throughout this book. Each rung names what it pins, the primary artifact that marks it, and what reaching it means for a reader of health research.
Level What is pinned Primary artifact Health-research reading
L0 locatable Nothing computational; the source is under version control A version-controlled repository A reviewer can find the code but cannot yet rebuild it
L1 pinned packages The exact version of every R package A lockfile such as renv.lock A colleague installs the same packages the analysis used
L2 pinned environment The operating system, the R runtime, and the system libraries as well A container image or a Nix environment The analysis yields the same numbers on a different machine
L3 verified Nothing new; the recorded outputs are shown to regenerate within a stated tolerance from a clean rebuild A passing baseline test, run after a clean rebuild, that compares regenerated outputs with values recorded in a committed baseline file within a stated tolerance The reported table has been shown to regenerate, not merely asserted to

L3 is the one rung that cannot be read off a file listing. An analysis is at L3, verified, when its recorded outputs have been shown to regenerate within a stated tolerance from a clean rebuild. The showing is done by a baseline test, that is, a test that recomputes the reported quantities and compares them with values recorded in a committed baseline file. The comparison is made within a tolerance written into the test. A clean rebuild whose tests pass, but which compares nothing against a recorded baseline, demonstrates that the environment builds and the code runs. That is L2 confirmed, not L3. But why a tolerance, and not exact equality? Estimates are floating-point numbers, and exact equality would fail on harmless differences in the last digits while saying nothing about whether the substantive result moved. In the framework this book uses, zzc verify --full applies this definition. It records L3 only when the image rebuilds, the test suite passes, and at least one baseline test (a file named inst/tinytest/test-baseline*.R) exists and appears in the test log. Otherwise it reports the project as rebuilt and tested, L2, with no baseline comparison. Continuous integration that re-runs the check on every change keeps an L3 claim current. However, it is the baseline comparison, not the automation, that earns the rung.

The rungs are cumulative. An L2 analysis is also locatable and package-pinned; the label names the highest rung reached. The transition that matters most for a first-year student to internalize is the one between L1 and L2. A lockfile alone, the L1 artifact, pins the packages but not the R version, the operating system, or the system libraries a package links against. A lockfile restored under a different R version may fail to install or, more insidiously, may install and behave differently. The container image, the L2 artifact, fixes what the lockfile cannot. The two are complements, not substitutes, and a genuinely pinned-environment analysis carries both.

The ladder consolidates several graded views of reproducibility that the literature has offered in different vocabularies. Peng (2011) arranged computational work along a spectrum, from the merely published, through code available, code and data available, and linked and executable, to full replication. Our L0, L1, and L2 track the lower and middle reaches of that spectrum. The artifact-review badges used by the ACM and several computing venues express a related ordering, from artifacts that are available, through artifacts evaluated as functional and then reusable, to results independently reproduced (Association for Computing Machinery, 2020). The badges and the rungs grade different things and should not be equated. The ‘functional’ and ‘reusable’ badges grade how well an artifact is documented and exercised. They do not grade the package-versus-environment pinning depth that separates L1 from L2. Likewise, reaching L3 is not the same act as earning a venue’s reproduction badge, which is awarded by reviewers under that venue’s own policy. The reader who needs a badge should read the venue’s current definition directly. The L-scheme adds no new definition of reproducibility. It maps these graded views onto artifacts a tool can detect, so that the level of a repository can be determined mechanically and reported honestly, rather than asserted by its author.

A laboratory adds a continuous-integration workflow that rebuilds its compendium and runs the tests on every push. A colleague objects that this cannot improve reproducibility, because it pins no new layer of the determinant stack. Is the colleague correct, and has the laboratory wasted its effort?

The colleague is correct on the narrow point and wrong on the conclusion. Continuous integration is a validation feature, not a capture feature: it pins no new layer, and removing it would leave the compendium exactly as reconstructable as before. What it adds is confirmation. It re-runs the captured analysis in a clean environment that carries none of the author’s undeclared state. It therefore catches the silent decay that a captured-but-never-retested analysis suffers, and the ‘works on my machine’ failure at its source. The laboratory has moved from L2 asserted to L2 confirmed: the environment is shown to build and the code to run. A baseline test, comparing regenerated outputs with recorded values, is what would make it L3.

3.5 Reproduced, replicated, and a warning about the words

The ladder grades re-execution, and re-execution is only one of the things the word ‘reproducibility’ is made to carry. We need to fix the terms before reading the literature, because the same two words are attached to opposite meanings across the sources a reader will meet.

The National Academies, in their 2019 consensus report, adopt the distinction this book follows (National Academies of Sciences, Engineering, and Medicine, 2019). Reproducibility is obtaining consistent computational results using the same input data, the same code, and the same conditions of analysis. That is, it is a statement about re-execution, and it is what the ladder grades. Replicability is obtaining consistent results across studies aimed at the same question, each with its own data. It is a statement about whether a scientific finding holds on new data. The two are independent. A result can be perfectly reproducible and entirely wrong, if the code faithfully regenerates a mistaken number every time. Conversely, a finding can replicate across laboratories while no single analysis in the set is computationally reproducible. Leek and Peng press exactly this point, that a reproducible analysis can still be wrong, and that reproducibility is therefore a floor and not a ceiling (Leek & Peng, 2015). Patil, Peng, and Leek go further and offer a statistical framing of the distinction. They cast replication as a question about the agreement of estimates across studies rather than about the re-running of code (Patil et al., 2016).

How do other vocabularies relate to this one? The current ACM artifact-badge vocabulary, in its 2020 revision, aligns its direction of use with the National Academies convention rather than inverting it (Association for Computing Machinery, 2020). However, the two draw the boundary differently. Under the National Academies convention, ‘Reproduced’ means obtaining a prior result using the same code and data; the same team re-running their own analysis qualifies. Under the ACM ‘Results Reproduced’ badge, the re-running must be done by a team other than the original authors, using the author’s provided artifacts. Both conventions therefore point ‘Reproduced’ toward re-execution with the original artifacts and ‘Replicated’ toward independence. The ACM badge, however, specifically requires a different team, while the National Academies definition does not.

The genuine word-swap that trips students belongs to two other places. One is the pre-2020 ACM badging, which attached the two words in the opposite direction along the re-execution axis. The other is a set of biomedical and other field-specific usages, which a reader may still encounter, where ‘reproduce’ names an independent study rather than a re-run. A reader who carries the sense of the words from one of those older documents into a current one will be misled. The misreading happens silently and without any error to warn them, in exactly the manner of a shifted package default. This book therefore fixes the meaning and states it once. Throughout, ‘reproduced’ carries the National Academies sense. L3 means that the captured analysis, rebuilt from its pinned definition and re-run, regenerates its own recorded outputs within a stated tolerance, as checked by a baseline test. It is deliberately not replication in the sense of an independent study on new data, which no software framework can supply. When the reader meets ‘reproduced’ or ‘replicated’ in a journal’s author instructions or a venue’s badging policy, the first task is to determine which convention that document uses.

WarningAnother warning about the words: ‘validation’

This chapter uses ‘validation’ and the L3 label ‘verified’ for a narrow act: confirming that a previously captured value still holds, as a baseline test does. Regulated industry work, pharmaceutical and other FDA-adjacent settings among them, attaches ‘validation’ to something larger: a formal, documented process confirming that a system is fit for its intended use, as under 21 CFR 11.10(a) (21 CFR Part 11, n.d.). European regulators expect something comparable of a sponsor’s analysis tools (European Medicines Agency, GCP Inspectors Working Group, 2023). A graduate moving into that kind of work will meet the regulated sense on arrival; this book’s narrower one does not prepare them for the paperwork that comes with it.

A manuscript’s cover letter states that the authors ‘reproduced the primary finding in an independent cohort.’ Under the National Academies convention this chapter follows, is that reproduction or replication, and why does the sentence as written risk being misread?

It is replication: an independent cohort means new data, and the National Academies convention attaches ‘replicated’ to a new study and ‘reproduced’ to re-running the original artifacts on the original data. The sentence risks being misread because of the current ACM badge sense. A reader applying it would expect the word ‘reproduced’ to mean an independent team re-ran the authors’ own artifacts, not that a second cohort was collected. The fix is to write ‘replicated’, not ‘reproduced’, or to spell out which convention is meant.

ImportantThe honest level

This chapter defines the ladder rather than climbing it. Its one executed placement, the cohort compendium in ‘The cohort, continued’ below, is computed from the repository’s artifacts each time the chapter renders, not asserted here. Every rung above the one that section prints is left to the chapters this ladder points toward.

3.6 Two numbering schemes, and which this book uses

Before applying the ladder to particular archetypes, we need to dispose of a source of confusion. A reader who consults the documentation of the framework this book uses will encounter a second numbering scheme, and the two are easily confused.

The framework’s teaching materials include a vignette that presents a five-level ramp. Level 1 is a basic R project with manual package management, and Level 2 is the addition of renv. Level 3 is the addition of Docker, Level 4 the addition of unit testing, and Level 5 the addition of continuous integration. That scale counts which tools a project has adopted, and it is a pedagogical on-ramp, meant to be walked from the bottom as a project matures. It is a useful thing, but it is not the scale of this book.

The scale of this book is the architectural ladder of the preceding sections, L0 through L3. It grades something different: how far down the determinant stack a result has been locked, and whether the lock has been verified. The two axes are related but not identical. Adopting renv (the tool scheme’s Level 2) is what lifts a project to L1. Adopting a container (the tool scheme’s Level 3) is what lifts it to L2. The tool scheme’s testing and continuous-integration levels belong to the validation layer that reaches L3. However, one can imagine a project that has adopted every tool and still verified nothing. The architectural ladder, not the tool count, is what tells the reader whether the result has been shown to reproduce.

To avoid ambiguity we adopt a single convention. This book uses the L0 to L3 architectural ladder throughout, in every chapter and every worked example. Where the framework’s own documentation speaks of ‘Level 1’ through ‘Level 5’, it is using the tool-adoption ramp, and the reader should read it as such and not conflate it with our L0 to L3. When we write a bare ‘L2’, we always mean the architectural rung.

A student writes in a lab notebook, ‘the project is at Level 2, so it is container-pinned.’ Using this chapter’s two scales, is the inference safe?

No. On the tool-adoption ramp, Level 2 means renv has been added, which is the architectural chapter’s L1, not its L2; the container arrives at the ramp’s Level 3. The student’s sentence silently mixed the two scales’ numbers. The safe habit is to never write a bare number without saying which scale it belongs to. This book’s own convention, stated above, is to let an unqualified ‘L2’ always mean the architectural rung.

3.7 The ladder grades every archetype

Does the ladder apply equally to every kind of compendium? Table 5.2 in Chapter 5 shows how each archetype reads it. A data-analysis project and a manuscript read it the same way. An R package reaches L2 by R CMD check and L3 by a baseline on its function outputs, and a blog applies the ladder post by post. The case most easily missed is the simulation study. Its exposure on the session-configuration layer is acute, because its outputs are functions of the random-number stream. An unseeded or wrongly seeded simulation can therefore sit at L2, with a perfectly pinned environment, and still fail to reproduce. For this archetype the discipline of fixing the RNG kind and the seed is the difference between a nominal L2 and a real one. The ladder is a property of the determinant stack, which every archetype shares, and not of any particular directory layout.

An R package has renv.lock, a Dockerfile, and a full testthat-style test suite that passes in a clean rebuild, but none of its tests compare a function’s output with a value recorded earlier. What rung has it reached, and what is missing for the next one?

It is at L2, pinned environment, confirmed: the lockfile and the container pin the packages and the environment, and the clean rebuild and passing test suite show the package builds and runs. It is not at L3, because none of its tests is a baseline test. Passing means the code ran without error, not that a key function’s output still matches a recorded reference value within a tolerance. Reaching L3 needs at least one test that compares a fixed input’s output against a value committed earlier.

3.8 No framework pins the whole stack

Reaching L2 or L3 does not pin the entire determinant stack, and no present framework pins all of it.

By its own published accounting, which splits the stack more finely than this chapter into about ten determinants, the framework this book uses robustly pins about four. These are the operating system and system libraries (as strongly as its base image is pinned), the R package versions (nominally, through the lockfile), the locale and time zone, and the source code. It leaves others to convention or to the base image. The random-number kind is not pinned by default. The BLAS and LAPACK implementation is inherited from the base image (the generated Dockerfile sets OMP_NUM_THREADS=1, which fixes the thread count but not the library). The integrity of external data is recorded by provenance but not by checksum, and the fonts and the graphics device are pinned only for some profiles. We know of no integrative framework that currently pins these layers, and an honest L2 or L3 claim names them. The reader will therefore find, at each rung of the ladder in the chapters ahead, an explicit account of what is secured and what remains exposed. That account includes the framework this book teaches.

A compendium reaches L3 by this chapter’s definition: it rebuilds from clean, its tests pass, and a baseline test compares a reported hazard ratio with a recorded value within a stated tolerance. Is it now safe to call the analysis fully reproducible, with nothing left to disclose?

No. L3 shows that the pinned definition regenerates the reported quantity within a stated tolerance. It does not pin the numerical libraries, the hardware, or intrinsic non-determinism, none of which any rung of the capture ladder secures. An honest statement at L3 still names these as exposed, for example that the bootstrap interval has not been checked for agreement across a different BLAS or CPU architecture. ‘Fully reproducible’ claims a closed stack that no present framework, including the one this book teaches, actually closes.

3.9 Worked example

To make the ladder concrete, consider a single analysis walked up all four rungs. A biostatistician has a folder containing analysis.R, which reads cohort.csv, fits a Cox model with the survival package, and writes a hazard ratio to results.csv. We shall place it on the ladder and name each transition.

At the start the folder is on the analyst’s laptop and nowhere else. It is at no rung of the ladder at all, because it cannot even be found by a second person. The first act, placing the folder under version control and pushing it to a repository, lifts it to L0, locatable. A reviewer can now find analysis.R and read it. Nothing computational is pinned; a reviewer who runs it will install whatever version of survival happens to be current, which may not be the version the analyst used.

The analyst now runs renv::init() and renv::snapshot(), which we develop in Chapter 6. This writes a renv.lock recording that the analysis used survival version 3.5-7, together with the exact version of every other package in the closure. The folder is now at L1, pinned packages. A colleague who restores from the lockfile obtains the same survival the analyst used. But the colleague’s laptop runs a newer R and a different operating system, and one of the packages links against a system library that is absent, so the restore fails to build.

To close that gap the analyst adds a container, developed in Chapter 7, built from a dated rocker base image pinned by its content digest. The image carries a fixed operating system, a fixed R, and the system libraries the packages need, and the lockfile restores inside it. The folder is now at L2, pinned environment. It will build the same environment on the colleague’s laptop, on a reviewer’s, and on a server. Because the analyst computes bootstrap confidence intervals for the hazard ratio, the analyst also sets the RNG kind with an explicit RNGkind() call in analysis.R, and passes a seed at the call that draws the resamples. Neither goes in the project .Rprofile. A seed set at session start makes every draw depend on whatever consumed random numbers earlier in the session, the rule Chapter 11 develops. This closes a session-configuration exposure that L2 does not close by default.

Finally the analyst adds a verification step, developed in Chapter 12. The step rebuilds the image from clean and re-runs analysis.R. It then runs a baseline test comparing the regenerated hazard ratio and its confidence limits with the values recorded in a committed baseline file, within a stated relative tolerance. It passes, and a continuous-integration service re-runs it on every change. The folder is now at L3, verified. That is, the reported hazard ratio has been shown to regenerate, not merely asserted to. The honest reproducibility statement for the paper can now name the rung, the artifacts, and the layers that remain outside the guarantee. No rung pins the hardware, the numerical libraries, or intrinsic non-determinism. For this analysis they amount chiefly to the cross-architecture numerical determinism of the bootstrap.

The biostatistician’s colleague restores the lockfile but skips the container, then reports the analysis as ‘reproducible.’ Which rung has the colleague actually reached, and what single artifact would the colleague need to add to reach the next one?

Restoring the lockfile reaches L1, pinned packages, and no further. The operating system, the R runtime, and the system libraries the colleague’s machine happens to carry are still whatever that machine provides, not what the original analyst used. Calling this ‘reproducible’ without qualification overstates by at least one rung. The single artifact that closes the gap to L2 is the container image, built from the same recipe the original analyst used, which pins the layers the lockfile cannot reach.

3.10 The cohort, continued

The worked example above follows a composite biostatistician. We now perform the same placement exercise on this book’s own running cohort compendium, as it stands in this repository when the chapter is rendered.

raw <- load_cohort('../data/raw_data/cohort_raw.csv')
cohort <- clean_cohort(raw)
nrow(raw)
#> [1] 338
nrow(cohort)
#> [1] 337

The extract loads and cleans without error, and clean_cohort() drops the one duplicated mrn the Conventions page describes, leaving the row count shown above. Whether the compendium is locatable, and whether anything beneath the code is pinned, is a question about which artifacts the repository holds, so we check rather than assert it.

repo_state <- c(
  source_code = file.exists('../../R/cohort.R'),
  data_generator = file.exists('../scripts/simulate_cohort.R'),
  lockfile = file.exists('../../renv.lock'),
  dockerfile = file.exists('../../Dockerfile')
)
repo_state
#>    source_code data_generator       lockfile     dockerfile 
#>           TRUE           TRUE           TRUE          FALSE
rung <- if (repo_state[['lockfile']] && repo_state[['dockerfile']]) {
  'L2'
} else if (repo_state[['lockfile']]) {
  'L1'
} else {
  'L0'
}
rung_name <- c(
  L0 = 'locatable', L1 = 'pinned packages',
  L2 = 'pinned environment'
)[[rung]]
next_artifact <- c(
  L0 = 'a lockfile, `renv.lock`',
  L1 = 'a container recipe, a `Dockerfile`',
  L2 = 'a baseline test that passes after a clean rebuild'
)[[rung]]

The code is under version control, so a reader of this book can find R/cohort.R and read exactly what ran. The extract itself is not committed; the repository holds the generator, analysis/scripts/simulate_cohort.R, which writes the synthetic extract from a fixed seed. That much is L0, locatable. The lockfile entry above is TRUE and the Dockerfile entry is FALSE, so on its capture artifacts this compendium is at L1, pinned packages. What it still lacks for the next rung is a container recipe, a Dockerfile. Any layer that no present artifact pins is left to whatever R, system libraries, and packages happen to be installed on the machine that runs this chunk. The chapters on pinning packages (Chapter 6) and pinning the environment (Chapter 7) teach the artifacts that mark L1 and L2. The verification chapter (Chapter 12) is the one that can lift the compendium to L3, once a baseline test compares a quantity computed from cohort against a value recorded today; the chunk above does not look for one, so it never reports L3. The reader who returns to this cell after reading those later chapters is looking at the same lines of code. What changes, as artifacts are added, is the rung those lines print.

Reading only the repo_state output above, what rung does this book’s own cohort compendium sit at, and what is the single artifact whose absence keeps it from the next rung?

L1, pinned packages, as the repo_state chunk above prints. The code is under version control, which is L0. L1 requires a lockfile, renv.lock, recording the exact version of tibble and every other non-base package the two functions call; base packages such as stats and utils have no record of their own because the R version fixes them. L2 requires a container recipe as well. The artifact the compendium still lacks for the next rung is a container recipe, a Dockerfile. Chapter 6 teaches the lockfile and Chapter 7 the container recipe.

3.11 Collaborating with an LLM

We turn finally to the use of a large language model, which is a capable assistant for reasoning about levels. However, it is confident in a register that this chapter has just taught the reader to distrust, and its confidence is not calibrated to the honest level.

ImportantRestricted data

One rule comes first (habit 3). Never paste or upload protected health information, identifiable records, or data held under a data use agreement, nor outputs derived from them, into a model or a coding agent unless three conditions all hold. The institution has approved that endpoint for that class of data. The IRB protocol and the consent permit the processing. The data use agreement, or the provider’s terms, permits it. If any of the three is unclear, the answer is no.

The third condition is the one most often missed. Institutional approval of a tool does not override the terms under which a data provider released the data. NIH, for example, treats sharing controlled-access data through prompts with public generative AI tools, which it glosses as third-party tools, as a breach of the data use certification. It treats a model trained on such data as a data derivative, to be shared only with other approved users and not kept after the project closes (National Institutes of Health, 2025).

Describe the structure instead: the column names and types, or a small synthetic example generated from the data dictionary, never edited from real rows. A coding agent sees whatever files are in its working tree, so run it in a checkout or container where the real data is not mounted.

An agent also acts, so confine it to a branch and keep merges, pushes, and deposits for yourself (Section 2.10.1).

A directory listing can itself be identifying when file names carry site or participant codes. List only the code, configuration, and lock files, and leave data directories out. The following triples illustrate productive prompts and the verification each demands.

Prompt. ‘Here is the listing of my project’s code and configuration files, with the data directories omitted. What reproducibility level has it reached on the L0 to L3 ladder, and what would it take to reach the next rung?’

Watch for. Reading a renv.lock as L1 and a Dockerfile as L2 is the right heuristic as far as it goes. However, no one can tell from a listing whether the lockfile is current, whether the image is pinned by a mutable tag or an immutable digest, or whether a verification run has ever passed. Check whether the answer reports L3 on the strength of a continuous-integration file that has never been run green.

Verification. Confirm each rung against the artifact itself, not its filename. For L1, check that the lockfile is in step with the code. For L2, check that the base image carries a digest and that the library is not shadowed by a bind mount. For L3, find an actual passing run of a baseline test after a clean rebuild, and read the test to confirm it compares outputs with recorded values under a stated tolerance. The presence of a file is capture asserted; a green baseline run is validation observed.

Prompt. ‘Below are the four rungs of a reproducibility ladder with their definitions, followed by a description of my project. Place the project on the ladder. For each rung, name the artifact in the description that meets its definition, or say what is missing. The rungs are cumulative. Say what you could not determine from the description.’

Supply the rows of Table 3.1 and the paragraph on cumulative rungs as the definitions, since a model cannot apply a scale it has not been given. Describe the project in prose: what is under version control, whether a lockfile exists, what the Dockerfile’s FROM line says, and what the tests compare.

Watch for. Check whether the answer applies the cumulative rule, or credits L2 to a Dockerfile in a project with no lockfile. Check whether it treats a base image named by tag as pinned in the same way as one named by digest. Check also whether it awards L3 to a test suite that compares nothing with a recorded baseline, which this chapter calls L2 confirmed, not L3.

Verification. Place the project yourself, from the artifacts, before reading the answer, and compare the two placements rung by rung. Then give the model a case whose answer you know, such as a project with a Dockerfile and no lockfile, and confirm that it does not award L2. A model that misplaces the known case should not be trusted on the unknown one.

Prompt. ‘My simulation gives slightly different numbers each time even inside the container. Why, and is it a bug?’

Watch for. Check whether the answer blames a layer that the container has already pinned, such as a package version or the operating system. Check also that it distinguishes an unseeded RNG (a real defect) from low-order-digit variation due to BLAS threading (an inherent limit at L2).

Verification. Trace the discrepancy to a layer of the determinant stack. If the numbers differ in all digits, look to the RNG kind and the seed, which the analyst controls. If they differ only in the low-order digits, suspect the numerical-library layer, which no rung of the capture ladder pins, and decide whether that residual matters for the inference at hand.

A better prompt. ‘My simulation, run twice in the same container, gives the two outputs pasted below. Also pasted are the sessionInfo() output, which names the BLAS and LAPACK libraries, the RNGkind() output, the line of my code that sets the seed, and the thread count. Say which layer of the determinant stack each difference most plausibly comes from, and for each give one command I can run to confirm or exclude it.’

3.12 Exercises

  1. For an analysis of your own, list the nine layers of the determinant stack and, for each, state whether your current setup pins it, and by what artifact. Identify the lowest layer you have left unpinned.

  2. Take a published paper in your field that provides code. Without running it, place it on the L0 to L3 ladder from the materials the authors supply, and write the one-sentence reproducibility statement the paper should have carried.

  3. Explain, to a colleague who believes that a Docker image makes an analysis ‘fully reproducible’, two determinants that the image does not pin and one circumstance in which each would cause two runs to disagree.

  4. The framework’s documentation grades a project ‘Level 4’ on its five-level tool-adoption scale. State what this does and does not tell you about the project’s rung on the L0 to L3 architectural ladder, and name one artifact you would inspect to settle the question.

  5. A colleague reports that a finding ‘replicated’ in a second cohort. A reviewer reads this as an ACM artifact badge and asks for the container. Explain the miscommunication in terms of the two conventions for the word, and state which sense the colleague almost certainly intended.

  6. Construct a small simulation in R that draws a sample under a fixed seed, and show that changing the RNG kind with RNGkind() changes the draw while leaving the seed unchanged. Explain which layer of the determinant stack this demonstrates and why a container alone would not protect against it.

  7. Run cohort <- clean_cohort(load_cohort('analysis/data/raw_data/cohort_raw.csv')) and report nrow(cohort). Then check file.exists() for renv.lock and Dockerfile at the project root, as the repo_state chunk in ‘The cohort, continued’ does. From that output alone, state the single rung of the L0 to L3 ladder this book’s own cohort compendium occupies, and name the one committed artifact, absent when you run the check, that would be needed to lift it to the next rung.

  8. This exercise returns to the computational-versus-statistical distinction of Chapter 2 (Chapter 2). That chapter shows that the R 3.6.0 sampler correction changed the draws a seeded sample() call produces, with identical code and an identical seed. Name the exact layer of this chapter’s determinant stack responsible, and state which rung, if any, of the L0 to L3 ladder would have protected a 2018 analysis against this specific change.

3.13 Solutions

Attempt an exercise before expanding its solution. Two of these ask you to work with an analysis or a paper of your own, so they have no single correct answer; for those the box says what a good answer establishes.

No single correct answer. What a good answer establishes is that you can name the layers in order and attach an artifact, or an admission, to each.

Work up from the base. The hardware is pinned by nothing short of running on the same machine. The numerical libraries are pinned by no rung. The container image determines which BLAS is installed, but the ladder does not require it to be fixed, and the thread count and the processor still vary. The operating system and system libraries are pinned by the container image. The R runtime is likewise pinned by the container image, and merely recorded by the lockfile’s R block. The package versions are pinned by renv.lock. The session configuration is partly pinned by the image through the locale and the time zone, and not pinned at all for the RNG kind, the contrasts, and the printed digits. The analysis code is pinned by version control, while the input data are pinned by a checksum or an archived deposit, and are often only locatable. Lastly, intrinsic non-determinism, such as parallel reduction order, is pinned by nothing.

Your lowest unpinned layer is most likely either the numerical libraries or the session configuration, and both answers are common and honest. If your lowest unpinned layer is the input data, you have discovered something more urgent than anything this chapter teaches, because the analysis is not reconstructable at all.

No single correct answer. What a good answer establishes is that the rung is read off the deposited artifacts, not off the quality of the paper or the reputation of the authors.

The procedure is mechanical. Is there code and data that can be located? That is L0. Is there a lockfile, or an equivalent exact version record? That is L1. Is there a container recipe, or another artifact pinning the runtime and the system libraries? That is L2. Is there evidence that the outputs were regenerated from the pinned definition and compared with recorded values within a stated tolerance, shown by a baseline test and a CI log or verification report of it passing? That is L3. A CI log that only shows the code ran is not enough. Stop at the highest rung for which an artifact is actually present.

The reproducibility statement should then claim that rung and no more. For the common L0 case: ‘The analysis code and the derived dataset are deposited at [location]. The software environment was not recorded, so re-execution may require package versions that differ from those used here.’ The second clause is the part most papers omit, and it is the part that makes the statement honest rather than merely modest.

Two determinants an image does not pin, with a circumstance for each.

The numerical libraries. An image built on a base that links against a BLAS chosen at run time, or one run on hosts that expose different thread counts or processors, will reorder floating-point summations. Two runs of the same image on two machines can then return a least-squares coefficient, computed through crossprod() and solve(), agreeing to five decimal places and differing in the sixth. The image pinned the software and not the arithmetic beneath it.

The session configuration, specifically the RNG kind. An image fixes the R version, and the R version determines the default sampler, so this exposure is narrower than it first appears. It bites when the analysis sets RNGkind() itself, or inherits it from a project .Rprofile that is mounted rather than baked in, or when a package changes the generator as a side effect. Two runs then draw different samples from the same seed, and a bootstrap interval moves.

A third acceptable answer is the input data: an image mounting a host directory reproduces the environment and not the extract, so a changed file underneath produces a different result with the image unchanged. A fourth is intrinsic non-determinism from parallelism, where the reduction order of a parallel sum varies run to run on the same machine.

In general, a container pins the environment, and the environment is not the whole stack.

‘Level 4’ on the framework’s scale means the project has adopted a particular set of tools: an R project, renv, Docker, and unit testing. It is a statement about tool adoption, and it is read from the bottom up, so Level 4 implies the first three as well.

What it tells you about the architectural ladder is therefore quite a lot, but indirectly and with one gap. The presence of renv implies the machinery for L1; the presence of Docker implies the machinery for L2. What it does not tell you is whether any of it has been verified, because the tool scale counts adoption and the architectural ladder grades locking and verification. A project can hold every tool in the list and have never rebuilt from its pinned definition. In that case it sits at L2 and is asserted rather than shown to reproduce. L3 requires a baseline comparison, which neither Level 4 nor Level 5 implies by itself. Unit tests and continuous integration can both be present while nothing is compared with a recorded result.

The artifacts to inspect are a baseline test and the continuous-integration record. The record should be a log showing that the container was rebuilt from the recipe, the analysis re-run, and its outputs compared with recorded values within a stated tolerance. Absent that, the honest reading of ‘Level 4’ is L2. Inspecting renv.lock and the Dockerfile tells you the capture artifacts exist; only the CI record tells you the capture works.

The colleague almost certainly meant the National Academies sense: a second cohort, collecting its own data, found a consistent result. That is replication, a statement about the durability of the finding, and it is the sense in which the word is ordinarily used in health research.

The reviewer read it in an artifact-badging frame, where the pair of words attaches to re-execution rather than to new data. Under the current 2020 ACM vocabulary the two conventions actually agree, so the current standard cannot explain the request. The miscommunication traces to one of two places. One is the pre-2020 ACM badging, which attached the two words in the opposite direction. The other is a field usage in which ‘reproduce’ names an independent study rather than a re-run. A reviewer carrying the sense of the words from an older document into a current one is misled silently, with no error to warn them.

The practical resolution is that the container request is a category error given what the colleague did. A container would let the reviewer re-execute the original analysis, which is not what the colleague claimed and not what a second cohort demonstrates. The colleague should answer by naming the convention, stating that the second cohort collected its own data, and offering the artifact that actually matches the claim, which is the second cohort’s own analysis code.

The demonstration is the worked example of Chapter 2 in miniature.

draw <- function(kind) {
  suppressWarnings({
    RNGkind(sample.kind = kind)
    set.seed(42)
    sample(1:10, 5)
  })
}
original <- RNGkind()
draw('Rounding')
draw('Rejection')
suppressWarnings(RNGkind(
  kind = original[1], normal.kind = original[2],
  sample.kind = original[3]
))

The two calls share a seed and return different draws. Restoring the original generator afterward matters, since RNGkind() sets session state that would otherwise persist and silently affect everything downstream.

The layer demonstrated is the session configuration layer, which sits above the package versions and below the analysis code. It is the layer this chapter singles out as only partially pinned at L2. An image fixes the locale and the time zone, and the R version it carries fixes the default sampler. However, the RNG kind, the contrasts, and the number of printed digits are all settable from inside the session and are not properties of the image.

A container alone would not protect against this because the change is made by the analysis, inside the container, after the environment is fixed. Two runs of the identical image produce different draws if the code sets the sampler differently, or if a mounted .Rprofile does. The protection lies in an explicit declaration, which a heavier container cannot supply. Concretely, set RNGkind() in the analysis, record it alongside the seed, and treat it as part of the analysis code rather than part of the environment.

The solution for this exercise is in the instructor guide, which is distributed separately to course instructors. Self-study readers who want to check their answer can compare the autogradable value against those reported in the chapter’s own ‘The cohort, continued’ section.

The layer is the R runtime layer of this chapter’s determinant stack, not the package-versions layer above it, and not the analysis code further up. The sampler that turns a seed into a draw is, by default, a behavior of R itself. The change arrived with a new R release rather than with a new package version or an edited script. Exercise 6 places the RNG kind in the session configuration instead because there the analysis overrides the default; here the default itself moved.

By this chapter’s own stack figure, the R runtime is secured only at L2, pinned environment, the same rung that secures the operating system and system libraries. A lockfile at L1 fixes the packages but not the R version underneath them. So an L0 or L1 analysis in 2018 would have had no protection at all. Whichever R release the person re-running it happened to install would supply whichever sampler that release shipped. Only an analysis built from a container image pinned to an R release predating 3.6.0 would have run the old sampler regardless of when it was restored. That protection is exactly what the L1-to-L2 step of this chapter’s worked example buys, and what a lockfile alone does not.

3.14 Cheat sheet

Rung or rule What it means or checks
L0 locatable Source under version control; nothing computational pinned
L1 pinned packages A lockfile (renv.lock) records every package’s exact version
L2 pinned environment A container or Nix recipe also fixes the OS, R runtime, and system libraries
L3 verified A baseline test compares regenerated outputs with recorded values, within a stated tolerance, after a clean rebuild
Capture vs. validation Capture (a lockfile, a container) sets L0 to L2; validation (CI, a baseline test) adds nothing pinned, confirms the captured level, and is what reaches L3 (habit 5)
Reproduced (this book) Same code, same data, re-run; contrast replicated: a new study, new data
zzc verify --full Reports L3 only if the image rebuilds, tests pass, and a test-baseline*.R file appears in the test log
Honest statement Name the rung, the artifact, and what remains exposed; never claim ‘fully reproducible’ (motto)

3.15 Further reading

  • National Academies of Sciences, Engineering, and Medicine (2019) is the National Academies consensus report that fixes the reproducible-versus-replicable terminology this book follows; read its opening chapter for the definitions and the reasoning behind them.
  • Peng (2011) introduces the reproducibility spectrum that our ladder consolidates, and argues for reproducibility as a minimum standard when full replication is impractical; the earlier Peng (2009) applies the idea to epidemiology directly.
  • Leek & Peng (2015) makes the case that reproducibility is a floor and not a ceiling, and Patil et al. (2016) offers a statistical framing of the distinction between reproducing a computation and replicating a finding.
  • Association for Computing Machinery (2020) documents the artifact-review badges whose ‘available’, ‘functional’, ‘reusable’, and ‘reproduced’ ordering grades related but different things from our ladder; read it alongside National Academies of Sciences, Engineering, and Medicine (2019) to see how the 2020 revision aligns the two vocabularies.
  • The Turing Way Community (2022) crosses the reproducibility axis with a what-varies axis, same or different data against same or different analysis, and is the most accessible community handbook for a first-year student.
  • For the determinant stack and the capture-versus-validation distinction as a framework applies them, Sandve et al. (2013) and Wilson et al. (2017) give the practices, the former stressing the recording of random-number seeds and exact versions that the ladder’s session-configuration layer names.