11  Testing and Code Quality

Program testing can be used to show the presence of bugs, but never to show their absence!

Edsger W. Dijkstra, Notes on Structured Programming (1972)

A function that silently sums instead of averaging will pass every rung of the reproducibility ladder and still be wrong on every machine that runs it. By the end of this chapter you will be able to say, of any computational result, whether it has merely been reproduced or actually checked. To make that distinction in your own analyses, the chapter develops the four kinds of test, among them the baseline test that Table 11.2 demonstrates on the book’s own running cohort.

NoteBefore you start

Reading time. About 65 minutes.

Prerequisites. Chapter 3, for the L0-L3 ladder and the capture-versus-validation vocabulary this chapter relies on throughout; Chapter 5, for the inst/tinytest/ layout and the R-package archetype. Statistically, no more than comfort reading a coefficient table from a fitted model and a percentile bootstrap interval; no new statistics are introduced. R knowledge assumed: writing and calling a function, vectors, and basic control flow.

Objectives. By the end of this chapter, can you:

  • state the reproducibility paradox, and explain why reproducibility machinery cannot, by itself, detect a wrong answer?
  • place testing in the validation register rather than the capture register, and say what that does and does not change about an analysis’s position on the L0-L3 ladder?
  • write a unit test, an integration test, a data-validation test, and a baseline (reproducibility) test in tinytest, naming the oracle behind each?
  • tell behavioral testing from implementation testing, and say why the second kind breaks under an honest refactor?
  • explain why lintr, styler, and a pre-commit hook belong to the validation register?

Outline. We begin with the analyst’s irreplaceable judgments, state the paradox precisely, and treat a pre-specified analysis plan as an oracle. We next introduce tinytest‘s bare-expectation style, work the four kinds of test on the cohort and the retinopathy trial, and separate Monte Carlo error from a change of sampler. We then distinguish behavior from implementation, treat linting, styling, and the pre-commit gate, and set out what ’validation’ means in regulated work, before turning to how one analyst reviews another’s code. The later sections show how the emphasis shifts across archetypes, give a worked example of the paradox itself, discuss collaborating with an LLM, and apply all four kinds of test to the running cohort. The chapter closes with exercises, a cheat sheet, and further reading.

Three failures reach you in one week. (a) A collaborator runs a project that has no renv.lock, installs newer versions of its packages than the author used, and a summary table changes. (b) A colleague’s renv::restore() of a complete lockfile stops because a spatial package cannot find GDAL. (c) A figure still shows last month’s counts after the raw extract was corrected, although the report rendered without error. For each, name the determinant layer that failed and the earlier chapter’s remedy.

Of the three failures in this warm-up, (a) is the package-version layer. With no lockfile the project has not reached L1, and the remedy is the L1 step of Chapter 6: a lockfile written by renv::snapshot() and read by renv::restore(). (b) is the operating-system and system-library layer. A lockfile pins the R packages and nothing beneath them, and GDAL is not an R package. The remedy is the container of Chapter 7, which reaches L2 only with a digest-pinned base, a dated snapshot, and the lockfile restored inside it. (c) is a stale derived file, not an environment layer at all: the correction never reached the figure. The remedy is a pipeline in which the raw extract is a format = 'file' target, so that targets rebuilds everything downstream of it (Chapter 10).

11.1 Where this sits on the stack

Testing is this book’s clearest case of pure validation. It is also, among the chapters so far, one that captures nothing at all. That is, a test suite adds no package to the lockfile and no layer to the image. Figure 11.1 marks that asymmetry on the shared determinant stack. The two layers this chapter writes code against, the analysis code and the input data, are validated here, not captured. The surrounding chapters captured them already, and this chapter only confirms they still behave.

plot_determinant_stack(
  validates = c('Analysis code', 'Input data'),
  title = 'Testing validates code and data; it captures neither')
A horizontal bar chart of the nine-layer determinant stack, hardware at the bottom and intrinsic non-determinism at the top, each bar colored by, and labeled with, the rung that secures it across the book. This chapter validates Analysis code, Input data; those layers are drawn at full color, labeled as captured or validated here, and, where validated, with a heavy outline. The remaining 7 layers are shown faded, for context, as secured elsewhere in the book. From the base, the rungs are: Hardware, unpinned by any rung; Numerical libraries (BLAS, LAPACK), unpinned by any rung; Operating system, system libraries, L2 pinned environment; R runtime, L2 pinned environment; Package versions, L1 pinned packages; Session configuration, L2 partially pinned; Analysis code, L0 locatable; Input data, L0 locatable; Intrinsic non-determinism, unpinned by any rung.
Figure 11.1: The determinant stack of Chapter 3, with analysis code and input data marked as validated rather than captured. This chapter pins no new layer; a unit test, a data-validation test, and a baseline test each confirm that the code and the data already captured by earlier chapters still produce the answer they are supposed to, which is the capture-versus-validation distinction the chapter develops.

11.2 Orientation

Consider a registry study in which an analyst has written a small function to compute a mean biomarker value for each treatment arm. The function carries a simple bug: it sums the values but forgets to divide by their count. The analysis runs without error. It produces a table, a figure, a p-value. Sent to a colleague, cloned onto a second machine, restored from a lockfile inside a pinned container, it runs again and produces the identical table, the identical figure, the identical p-value. Every rung of the ladder of Chapter 3 is satisfied: the source is locatable, the packages are pinned, the environment is pinned. The result is perfectly reproducible, and it is wrong.

This is the reproducibility paradox, and it is the pivot of this chapter. A buggy function that returns the wrong answer returns the same wrong answer everywhere, on every machine, at every later date. Reproducibility machinery, faithfully applied, does not detect the error; it propagates it with mechanical fidelity. The lesson, pressed by Leek and Peng in their aptly titled note that reproducible research can still be wrong (Leek & Peng, 2015), is that reproducibility is a floor and not a ceiling. It guarantees that a result can be regenerated. It says nothing whatever about whether the result is correct.

The remedy is testing. But why does testing belong in a book about reproducibility, rather than in a separate book about software engineering? The reason is precisely the paradox. Pinning a package version, building a container, restoring a lockfile: each of these captures a determinant of the computation, and none of them examines whether the computation is right. Testing captures nothing. A test adds no package to the lockfile and no library to the image. What a test does is confirm that a function, a pipeline, or a dataset behaves as the analyst intended. It belongs, in the vocabulary this book has used since Chapter 1, to the validation register rather than the capture register, alongside the continuous integration and output verification of Chapter 12. Capture fixes the inputs; validation checks that the fixed inputs still produce the answer they are supposed to. A reproducible analysis without tests is trusted; a reproducible analysis with tests is checked.

There is a further and more constructive link to the ladder. The highest rung, L3 verified, is reached not by capturing more but by confirming that a recorded computational output regenerates. That confirmation is itself a test, of a particular kind we shall meet below as the baseline test, so the topmost level is defined by a test.

A colleague reports that their analysis reaches L2 pinned environment and argues this means the reported hazard ratio must be correct. What is wrong with the argument?

L2 is a claim about capture, that the packages and the environment are pinned so the same code produces the same output everywhere. It is not a claim about correctness: a pinned, containerized analysis still reproduces a wrong hazard ratio exactly, if the code that computes it is wrong. Only a test, comparing against an independent oracle, bears on whether the number is right.

11.3 The analyst’s contribution

Testing tooling runs the checks, tallies the passes and failures, and reports them. It does not originate any of the judgments that give a test its value. Three of them underlie everything that follows.

  1. The analyst decides what ‘correct’ means. A test compares a computed value against an expected one, and the expected one has to come from somewhere outside the code under test. This source of independent truth is called an oracle. Examples are a mean of one, two, and three that the analyst knows to be two; a count of rows that a data dictionary fixes at some known number; a coefficient that a textbook formula supplies. No tool knows the right answer. The analyst supplies it, and a test is only as trustworthy as the oracle behind it.

  2. The analyst chooses which behaviors to pin. A function has an infinity of properties, most of them incidental to its purpose. The analyst decides which few constitute the function’s contract, the behaviors a future change must not break, and writes tests for those. This is the distinction, developed below, between testing behavior and testing implementation.

  3. The analyst encodes the domain constraints. Consider the rules that an age lies between zero and one hundred and twenty, that a treatment arm is one of a fixed set of labels, and that a probability sits in the unit interval. These are not facts the computer can infer. They are clinical and scientific knowledge. A data-validation test is the place where that knowledge is written down in executable form, so that a violation halts the analysis rather than passing silently into a result.

NoteThe vocabulary of this chapter
  • Unit test. A focused check that one small piece of code, typically a single function, behaves as intended on known inputs.
  • Integration test. A check that several components behave correctly when composed into a pipeline, from raw input to final result.
  • Data-validation test. A check on the data itself, not the code: that a value lies in a plausible range, that no impossible ages appear, that the expected columns are present.
  • Reproducibility test. A check that a recorded computational result regenerates. The form that can fail is the baseline test, which compares a fresh result with a value recorded in an earlier session and committed, within a stated tolerance. Two runs in one session do not qualify.
  • Oracle. The source of independent truth a test compares against; the known correct answer.
  • tinytest. A lightweight, zero-dependency R testing framework in which a test file is a plain script of bare, top-level expectation calls (Loo, 2020).
  • Expectation. A single assertion, such as expect_equal(actual, expected), that passes or fails.
  • Linting. Static analysis of source code for stylistic and likely-error patterns, performed by lintr, without running the code.
  • Styling. Automatic reformatting of source code to a consistent layout, performed by styler.
  • Pre-commit hook. A script that git runs automatically before a commit is recorded, used here to run the linter and the styler as a gate.

Name the three judgments that a testing tool cannot make on its own, and that fall to the analyst instead.

The first is what ‘correct’ means for a given input, supplied by an oracle outside the code under test. The second is which behaviors of a function constitute its contract and so deserve a test. The third is which domain constraints, such as a plausible age range, the data itself must satisfy.

11.4 The reproducibility paradox, stated precisely

A computational result depends on the code, the data, and the stack of determinants beneath them that Chapter 3 enumerated. The capture tools of Part II fix those determinants, so that the mapping from input to output is held constant across machines and across time. But holding a mapping constant says nothing about whether the mapping is the intended one. If the code encodes the function ‘sum’ where the analyst meant the function ‘mean’, then capture faithfully preserves ‘sum’, and every reproduction returns the sum. The error is not in the reproduction; the reproduction is flawless. The error is in the mapping, and the only instrument that examines the mapping is a test.

Two consequences follow, and they cut in opposite directions. On the one hand, a result that fails to reproduce is thereby shown to be untrustworthy, which is a genuine service; irreproducibility is a detectable defect. On the other hand, a result that does reproduce has been shown only to be stable, not to be right. Stability is easily mistaken for correctness by an analyst who has watched the same number appear three times. Testing exists to break the false inference from ‘it came out the same’ to ‘it came out correct’.

ImportantThe honest level

This chapter’s own practice reaches a baseline test on one bootstrap bound (Table 11.2) and a short, illustrative suite of unit, integration, and data-validation checks on hand-picked examples. It establishes that those particular results regenerate and that those particular functions behave as intended. It says nothing whatsoever about any function, dataset, or computation this chapter did not choose to test.

If the code encodes ‘sum’ where the analyst meant ‘mean’, what does capturing the environment do to that error, and what does it not do?

Capture preserves the error exactly: every reproduction on every pinned machine returns the same sum, forever. It does nothing to detect the error, because capture examines only whether the mapping from input to output is held constant, never whether the mapping is the intended one. Only a test, comparing against an independent oracle, examines the mapping itself.

11.5 The analysis plan as an oracle

The oracle introduced above sits at the level of a single function: a hand-computed mean, a row count a data dictionary fixes. The same structure scales up to the level of the whole analysis’s primary result, and doing so closes a gap the paradox otherwise leaves open.

A pre-specified analysis plan states in advance the primary endpoint, the model, the covariates, and the comparison that will answer the study’s main question. It is also called a statistical analysis plan, or, when it is registered publicly before the data are seen, a pre-registration. Written and dated before the analyst has looked at the outcome data, it is computed independently of any code’s output in exactly the sense the Vocabulary box above requires of an oracle. That is, it specifies what the correct analysis looks like, not by running anything, but by committing in writing to a choice that the data cannot later be allowed to revise.

A function-level oracle, such as the hand-computed mean described above, catches a coding error: the function returns the wrong number for a known input. It says nothing about whether the analysis itself asks the right question. Consider an analysis built by trying several models, several covariate sets, or several endpoints and reporting the one that reached significance. It can be bug-free in every function and pass every unit test. It can also still reproduce exactly on a second machine, because reproduction only checks that the same code returns the same output. The reproducibility paradox stated above applies again, one level up. A fully reproducible analysis can still be the wrong analysis, chosen with the benefit of having already seen the answer. A pre-specified plan is the oracle that guards against this failure, because it fixes the intended analysis before the data could have shaped it.

The two oracles differ in what independence they supply. A function’s oracle is independent of the code that computes the value; a pre-specified plan is independent of the data that will later be analyzed. Both are necessary, and neither substitutes for the other. A bug-free implementation of an unregistered, data-driven analysis is still untrustworthy in the way the plan would have prevented. Conversely, a faithful implementation of a pre-specified plan can still contain a coding error that only a function-level oracle would catch.

In a research compendium, a pre-specified plan is a dated document, committed to the repository (or deposited externally, as on the Open Science Framework) before the raw data arrive, or before they are unblinded. Checking an analysis against it is not an expect_equal() call but a review step: do the fitted model’s formula, the stated primary comparison, and the decision rule actually implemented match what the plan specified? Where the match can be stated mechanically, for example that fit_model() fits the formula the plan names, a test can assert it directly. Such a test compares the model’s formula (as text, or as a parsed call) against a string recorded from the plan rather than against anything the analysis itself computed.

A colleague’s bootstrap confidence interval regenerates bit for bit on a second machine, and every unit test in the package passes. Is this enough to trust the reported result as the study’s primary finding? What additional check, and what additional oracle, would you ask for?

No. Reproducing exactly and passing every unit test shows that the code is a faithful, bug-free implementation of whatever analysis it runs. Neither check examines whether that analysis is the one the study committed to before seeing the data. The additional check is whether the fitted model, the endpoint, and the comparison match a pre-specified analysis plan dated before the data arrived. That plan is the oracle: a source of what the correct analysis should look like, supplied independently of any result the code could produce.

11.6 tinytest: bare expectations

The framework this book uses adopts tinytest (Loo, 2020) as its testing tool. The choice rewards a moment’s explanation because it shapes how the tests look. tinytest is lightweight and carries no dependencies of its own, which matters in a reproducible compendium. Every additional test dependency enlarges the lockfile and the container image, and a testing framework that dragged in a dozen packages would tax the very machinery it is meant to protect. The more visible consequence is stylistic. A tinytest test file is an ordinary R script containing bare, top-level expectation calls, with no wrapping function around them.

A reader who has met testthat will notice the difference at once. Where testthat nests each assertion inside a test_that('description', { ... }) block, tinytest writes the assertion directly:

# inst/tinytest/test-calculate_mean.R
expect_equal(calculate_mean(c(1, 2, 3)), 2)
expect_equal(calculate_mean(5), 5)
expect_true(is.nan(calculate_mean(numeric(0))))

Each line is a complete test. The file is also a runnable script, so during development an analyst can source it directly and watch the expectations evaluate. The available expectations cover the common cases (expect_equal(), expect_true(), expect_false(), expect_identical(), expect_error(), expect_inherits(), and a few others). A compendium’s tests are collected in inst/tinytest/ and driven by a small tests/tinytest.R runner that R CMD check invokes. tinytest runs each file with the test directory as its working directory. A test that reads a project file from the root therefore reaches it through ../../, as the listings below do. We follow the convention throughout: bare top-level expectations, one concern per file, named test-<thing>.R.

A tinytest test file contains no test_that() call and no function definition around its expectations. What does that tell you about how to run one line of it during development?

Because each expectation is a bare, top-level statement, the file is an ordinary runnable R script. An analyst can source() the whole file, or send a single line to the console, and watch that one expectation evaluate immediately, with no need to locate it inside a nested block first.

11.7 The four kinds of test

A research compendium wants four kinds of test, and they form a progression from the smallest unit of code to the reproduction of a whole result. We take them in turn, illustrating each on synthetic data of the sort a health study might produce.

11.7.1 Unit tests of functions

A unit test checks that one function behaves as intended on inputs whose correct outputs are known independently. The mean function of the orientation, here called calculate_mean, is the canonical case. Its contract is that it returns the arithmetic mean, and the oracle is arithmetic done by hand:

# inst/tinytest/test-calculate_mean.R
expect_equal(calculate_mean(c(2, 4, 6)), 4)
expect_equal(calculate_mean(c(-1, 0, 1)), 0)
expect_true(is.nan(calculate_mean(numeric(0))))
expect_equal(calculate_mean(c(1, NA, 3),
                            na.rm = TRUE), 2)

The first two lines pin the ordinary behavior against hand-computed answers. The third and fourth pin the edge cases, the empty vector and the missing value, which are exactly the inputs on which an untested function tends to misbehave. A unit test suite is, in effect, a written record of what the analyst believes each function promises.

11.7.2 Integration tests of the pipeline

A unit test exercises a function in isolation, but a health analysis is rarely one function. It is a pipeline, in which raw data is loaded, cleaned, modeled, and summarized, and the components can be individually correct yet compose incorrectly. An integration test runs the pipeline end to end and checks the result:

# abridged from inst/tinytest/test-cohort.R
raw <- load_cohort('../../analysis/data/raw_data/cohort_raw.csv')
cohort <- clean_cohort(raw)
fit <- fit_model(cohort)
summary_tbl <- summarise_fit(fit)

expect_true(is.numeric(coef(fit)))
expect_true(all(c('term', 'estimate') %in%
                names(summary_tbl)))
expect_equal(nrow(summary_tbl), 3L)

The first expectation tests the contract that coef(fit) returns numeric coefficients, which is what summarise_fit() needs to proceed. Testing expect_inherits(fit, 'lm') would also pass here, because glm inherits from lm, but it would break the moment the function was upgraded to fit_cox() returning a coxph, which does not. Testing the interface (‘does coef() return numeric values?’) survives that migration; testing the class does not. The book’s own compendium does not perform this migration: fit_cox() was added beside fit_model() rather than replacing it, and its tests sit in a file of their own, inst/tinytest/test-fit-cox.R.

The integration test does not re-check what the unit tests already cover. Instead it checks the seams: that the output of clean_cohort is a valid input to fit_model, and that the model summary has the expected shape. A pipeline that passes its unit tests can still fail here, if a column is renamed in one step and read under its old name in the next.

11.7.3 Data-validation tests

The third kind of test turns its attention from the code to the data. Much of what goes wrong in a health analysis is not a coding error but a data defect. Examples are an age of two hundred, a negative laboratory value, a treatment label misspelled, or a column silently dropped by an upstream export. A data-validation test encodes the analyst’s domain knowledge as executable constraints and fails when the data violates them.

Before writing constraints, we look first at what we are constraining. Table 11.1 tabulates the retinopathy trial introduced in the conventions. It is included here for a reason beyond illustration: every cell in it can be counted by hand from the data, which is precisely the property an oracle needs. A test asserting that the untreated arm contains 197 eyes, written as that literal number, is checkable by someone who does not trust the function that produced it. A test asserting that it contains nrow(subset(d, trt == 0)) eyes is not a test at all; it is the function agreeing with itself.

library(tibble)
library(knitr)

d <- survival::diabetic
d$arm <- factor(d$trt, levels = c(0, 1),
                labels = c('Untreated', 'Laser'))

arms <- tibble(
  Arm = levels(d$arm),
  Eyes = as.integer(table(d$arm)),
  Events = as.integer(tapply(d$status, d$arm, sum)),
  `Event rate` = sprintf(
    '%.3f', as.numeric(tapply(d$status, d$arm, mean))
  )
)

kable(arms, align = 'lrrr')
Table 11.1: Eyes and events by treatment arm in the diabetic retinopathy trial (survival::diabetic), the real dataset this book computes on. Each count is verifiable by hand from the 394 records, which is what makes the table usable as a test oracle rather than merely as a description.
Arm Eyes Events Event rate
Untreated 197 101 0.513
Laser 197 54 0.274

The difference between the arms is large and in the expected direction. That is a reason to be careful rather than pleased, because a validation test should assert what the protocol requires, not what this particular extract happens to show. An assertion that the event rate in the laser arm is below one third would pass today. It would fail on a different trial, or on a corrected export, for reasons having nothing to do with a defect.

# abridged from inst/tinytest/test-cohort.R
raw <- load_cohort('../../analysis/data/raw_data/cohort_raw.csv')
cohort <- clean_cohort(raw)

# the expected columns are present
expect_true(all(c('mrn', 'age_at_enroll', 'sex', 'incident_dm')
                %in% names(raw)))

# ages lie within the enrollment criteria, not merely
# within the observed range
expect_true(all(cohort$age_at_enroll >= 18 &
                cohort$age_at_enroll <= 105, na.rm = TRUE))

# sex takes only its two coded levels once cleaned
expect_true(all(cohort$sex %in% c('F', 'M')))

# outcome missingness stays within what the analysis
# can tolerate
expect_true(mean(is.na(cohort$incident_dm)) < 0.15)

These are the tests that most directly repay a health scientist, because they catch the errors that no amount of correct code can prevent, the errors that live in the data before the analysis begins. In a data-analysis or simulation archetype, where there may be few reusable functions to unit-test, the data-validation tests often carry most of the suite’s weight.

11.7.4 Reproducibility tests

The fourth kind closes the loop back to the ladder. A reproducibility test checks that a recorded result regenerates. Its most tempting form runs a seeded procedure twice in one session and asserts that the two results agree:

# a same-session check: it cannot detect what it
# appears to guard against
expect_equal(bootstrap_ci(cohort, B = 2000, seed = 2024),
             bootstrap_ci(cohort, B = 2000, seed = 2024))

Note first where the seed lives. bootstrap_ci() takes it as an argument and sets it internally, with withr::with_seed(), which also restores the caller’s generator state on exit. As a result, the function is deterministic given its arguments and does not disturb any random stream around it. A procedure that depends on the caller setting a seed first is one set.seed away from irreproducible, and moving the seed into the signature makes forgetting it impossible.

WarningPitfall

A same-session check cannot fail for the reason it seems to serve. Both calls run in the same R, under the same random-number defaults, with the same package versions. When an upgrade changes how sample() draws, as R 3.6.0 did, both calls change together and still agree. What the check does catch is narrower: a procedure that ignores its seed. That is worth one line in a unit test, but it is not a reproducibility test.

A reproducibility test that can fail compares today’s result with one recorded in an earlier session and committed to the repository. We call it a baseline test. It has three parts: a script, run once, that records the baseline; the committed baseline file; and a test file that recomputes the quantities and compares them with a stated tolerance.

# analysis/scripts/generate_baseline.R, abridged. Run once
# from the project root to record the baseline, then commit
# inst/baseline/cohort_baseline.rds.
# The book's own baseline was recorded this way, and the
# file is now committed at that path; regenerate it only
# if the generator, the seed, or the model change.
result <- run_canonical_analysis(
  'analysis/data/raw_data/cohort_raw.csv'
)
dir.create('inst/baseline', recursive = TRUE,
           showWarnings = FALSE)
saveRDS(result, 'inst/baseline/cohort_baseline.rds')
# inst/tinytest/test-baseline-cohort.R, abridged: the
# real file also searches fallback paths for both files
raw_path <- '../../analysis/data/raw_data/cohort_raw.csv'
if (!file.exists(raw_path)) {
  exit_file('raw extract absent; run simulate_cohort.R')
}
baseline <- readRDS(
  system.file('baseline', 'cohort_baseline.rds',
              package = 'reproducibility'))
result <- run_canonical_analysis(raw_path)

expect_equal(result$n_rows, baseline$n_rows,
  info = 'cleaned row count matches baseline')
expect_equal(result$n_events, baseline$n_events,
  info = 'event count matches baseline')
expect_equal(result$hr, baseline$hr, tolerance = 1e-8,
  info = 'Cox HR matches committed baseline')
expect_equal(result$lower, baseline$lower,
  tolerance = 1e-8,
  info = 'lower CI bound matches committed baseline')
expect_equal(result$upper, baseline$upper,
  tolerance = 1e-8,
  info = 'upper CI bound matches committed baseline')

Three details carry the weight. The path is relative to inst/tinytest/, where tinytest runs the file. The guard lets the file exit early under R CMD check, which installs the package without analysis/. The mirror in inst/testdata/ covers the unit and data-validation tests; the baseline test requires the raw extract and skips cleanly without it. Finally, the tolerance is stated and chosen, not left at a default. 1e-8 relative is wide enough to absorb floating-point summation reordered by a different BLAS or thread count (Chapter 7), and far too narrow for a change in the fitted coefficients to pass. A comparison with no tolerance would fail on that reordering, and a check that fails for reasons unrelated to the analysis trains its reader to ignore it.

The test can be watched failing for the reason it exists. The chunk below plays both sessions on the synthetic cohort. It records the 97.5 percent bootstrap bound for mean BMI under the sampler R used before version 3.6.0 (sample.kind = 'Rounding'), as an analysis recorded on an older R would have. It then reruns it with the same seed and sampler, and reruns it again with the same seed under the current default ('Rejection'). The comparison is the one expect_equal() makes, all.equal() at the stated tolerance. withr::with_seed() sets the sampler only for the duration of each call, and the final stopifnot() confirms the session’s generator is as it was.

library(knitr)

cohort <- read.csv('../data/raw_data/cohort_raw.csv')
bmi <- cohort$bmi[!is.na(cohort$bmi)]

boot_upper <- function(x, seed, sample_kind, reps = 2000) {
  withr::with_seed(
    seed,
    {
      means <- replicate(reps, mean(sample(x, replace = TRUE)))
      unname(quantile(means, 0.975))
    },
    .rng_sample_kind = sample_kind
  )
}

kind_before <- RNGkind()[3]

# R warns whenever the pre-3.6.0 sampler is selected;
# the warning is expected here and muffled.
baseline <- suppressWarnings(boot_upper(bmi, 2024, 'Rounding'))
rerun_same <- suppressWarnings(boot_upper(bmi, 2024, 'Rounding'))
rerun_new <- boot_upper(bmi, 2024, 'Rejection')

tol <- 1e-8
passes <- function(current) {
  isTRUE(all.equal(current, baseline, tolerance = tol))
}

checks <- data.frame(
  run = c('recorded baseline (Rounding)',
          'rerun, same seed and sampler (Rounding)',
          'rerun, same seed, current default (Rejection)'),
  upper_bound = c(baseline, rerun_same, rerun_new),
  passes = c('(baseline)',
             ifelse(passes(rerun_same), 'yes', 'no'),
             ifelse(passes(rerun_new), 'yes', 'no'))
)

stopifnot(identical(RNGkind()[3], kind_before))

kable(checks, digits = 4,
      col.names = c('Run', '97.5% bootstrap bound',
                    'Passes at tolerance 1e-8'))
Table 11.2: A baseline test across sessions. The bound recorded under the pre-3.6.0 sampler is reproduced exactly by a rerun with the same seed and sampler, and is not reproduced seed-exactly by a rerun with the same seed under the current default sampler, so the comparison at tolerance 1e-8 passes once and fails once. A same-session check would have passed in both cases.
Run 97.5% bootstrap bound Passes at tolerance 1e-8
recorded baseline (Rounding) 29.1400 (baseline)
rerun, same seed and sampler (Rounding) 29.1400 yes
rerun, same seed, current default (Rejection) 29.1436 no

The rerun under the current sampler moves the bound by 0.0036 units of BMI, from 29.1400 to 29.1436. That is a small number, and it is not a numerical artifact: the resamples themselves are different. The baseline test reports it; two calls in one session, both under whichever sampler the session uses, would have agreed with each other and reported nothing.

Failing the seed-exact comparison is not the same as failing to reproduce the result, because the bound is itself a Monte Carlo estimate. The chunk below estimates its Monte Carlo standard error (MCSE) as the standard deviation of the bound over 40 seeds, at the table’s 2000 replicates, and then applies a two-tier check to the two reruns.

mc_bounds <- vapply(1:40, function(s) boot_upper(bmi, s, 'Rejection'),
                    numeric(1))
mcse <- sd(mc_bounds)
two_tier <- compare_to_baseline(c(rerun_same, rerun_new),
                                c(baseline, baseline), mcse = mcse)
two_tier
#> [1] "exact"                    "within Monte Carlo error"

# Decimals an MCSE supports: the MCSE must be below half a
# unit of the last reported digit.
digits_supported <- function(mcse) max(0, floor(-log10(2 * mcse)))
d_bmi <- digits_supported(mcse)
# An MCSE from 40 seeds is itself uncertain by about 11
# percent, so two significant figures are all it supports.
sig2 <- function(x) {
  formatC(signif(x, 2), digits = 2, format = 'fg', flag = '#')
}
stopifnot(round(baseline, d_bmi) == round(rerun_new, d_bmi))

The MCSE of the bound is 0.015, so the sampler change is about 0.2 of one MCSE. The seed-exact comparison fails, yet the change is a fraction of one MCSE, and the statistical result reproduces.

This is the two-tier baseline check. The first tier compares at the numerical tolerance 1e-8; a pass means the draws themselves regenerated. On a failure, the second tier accepts a difference of at most three MCSE, with the MCSE recorded beside the baseline value. compare_to_baseline() in the book’s package implements both tiers, and it returns ‘exact’ for the same-sampler rerun and ‘within Monte Carlo error’ for the current sampler. Both the recorded value and the rerun carry Monte Carlo error, so their difference has a standard deviation of about \(\sqrt{2}\) MCSE. Under normality, three MCSE then flags a pure change of draws about 3.4 percent of the time, which is the price of a second tier that still catches a real change.

The second tier applies only to a quantity that has Monte Carlo error, such as a bootstrap bound or a simulation summary. The Cox hazard ratio and its confidence limits in test-baseline-cohort.R are deterministic functions of the data, so only the first tier applies to them, and compare_to_baseline() with mcse = NULL reduces to it. The precision of the table follows the same distinction. Its four decimals are there for the seed-exact tier. As a statistical result, an MCSE of 0.015 supports reporting the bound as 29.1 under either sampler, by the rule the next section gives.

TipVerification

To see this hold on your own machine: generate the raw extract (Rscript analysis/scripts/simulate_cohort.R from the project root), then run RENV_CONFIG_AUTOLOADER_ENABLED=FALSE Rscript -e 'pkgload::load_all("."); tinytest::run_test_dir("inst/tinytest")' and confirm test-baseline-cohort.R reports five passes (habit 4). To make it fail on purpose, open inst/baseline/cohort_baseline.rds, read it with readRDS(), change hr by more than 1e-8 relative, write it back, and rerun. The comparison above, run with sample.kind = 'Rejection' against a baseline recorded under 'Rounding', is this exact failure, executed in the chapter itself.

A baseline test is what L3 requires. The top rung, as Chapter 3 defines it, asks that recorded outputs be shown to regenerate within a stated tolerance from a clean rebuild. The baseline test supplies the recorded outputs and the tolerance; running it inside an environment rebuilt from its recipe supplies the clean rebuild, and Chapter 12 assembles the two in continuous integration. A suite with no baseline test can show that the code runs and that its functions are correct on known inputs. However, it cannot show that a recorded result regenerates, so it does not reach L3 however thorough it is. In the zzcollab composition, zzc verify --full applies this rule. It stamps L3 only when the image rebuilds, the test suite passes, and at least one inst/tinytest/test-baseline*.R file exists and is named in the test log. Otherwise it reports the project as rebuilt and tested, at L2, with no baseline comparison. A baseline whose guard exits early is still named in the log, so the guard must not fire in the image.

11.8 Monte Carlo error and parallel streams

Table 11.2 fixed the seed and varied only the sampler (the default that R changed in version 3.6.0), which isolated one source of change in the bootstrap bound. A second source is always present and has nothing to do with R versions. A bootstrap estimate is itself a Monte Carlo quantity, built from a finite number of resamples. It carries its own sampling variability on top of the statistical sampling variability of the original cohort. Two bootstrap runs at different seeds, same data, same number of replicates B, will not return exactly the same bound. The standard deviation of the bound across seeds is its Monte Carlo standard error (MCSE), distinct from the standard error the bootstrap is estimating.

But how much does such a bound move in practice? The practical diagnostic needs no extra theory: rerun the same call at many seeds for a given B, and take the standard deviation of the result; then raise B and watch it shrink. The chunk below does this with bootstrap_ci() on the HbA1c odds ratio, at B = 250 and at the package default B = 1000, with 20 seeds each.

cohort <- clean_cohort(
  load_cohort('../../analysis/data/raw_data/cohort_raw.csv')
)
bounds_at <- function(B) {
  t(vapply(1:20, function(s) bootstrap_ci(cohort, B = B, seed = s),
           numeric(2)))
}
b250 <- bounds_at(250)
b1000 <- bounds_at(1000)
mcse250 <- apply(b250, 2, sd)
mcse1000 <- apply(b1000, 2, sd)
ratio <- mcse250 / mcse1000
# With 20 seeds the log of a ratio of two SDs has a standard
# error near sqrt(2 / 38); the prose calls the ratios
# consistent with 2 only while they are within two of them.
stopifnot(all(abs(log(ratio / 2)) < 2 * sqrt(2 / 38)))
d_upper <- digits_supported(mcse1000['upper'])
d_lower <- digits_supported(mcse1000['lower'])
stopifnot(d_upper < 2)
ci_default <- bootstrap_ci(cohort)
# Replicates for the MCSE to fall below half a unit of one
# more decimal, scaling the observed MCSE by 1/sqrt(B).
B_for <- function(mcse, d, B = 1000) {
  ceiling(B * (mcse / (0.5 * 10^-d))^2)
}
B_upper_next <- B_for(mcse1000['upper'], d_upper + 1)

At B = 250 the MCSE of the upper bound is 0.43 and that of the lower bound is 0.072. At B = 1000 they are 0.22 and 0.040, on bounds near 5.5 and 1.5. An MCSE that scales as \(1/\sqrt{B}\) halves when B is quadrupled, and the observed ratios are 2.0 for the upper bound and 1.8 for the lower. With 20 seeds, each estimated standard deviation is itself uncertain by about 16 percent, so these ratios are consistent with that scaling but do not pin it closely.

The standard guidance for simulation studies is to report the MCSE of every Monte Carlo estimate and to choose the number of repetitions from the precision required (Morris et al., 2019). Applied to reported digits, it gives a rule for the number of replicates. Choose B so that the MCSE of each reported quantity is below about half a unit of its last reported digit, or report only the digits the MCSE supports. At the default B = 1000 the rule allows the lower bound 1 decimal place and the upper bound 0, so the default call’s interval should be reported as 1.5 to 5. Reporting the upper bound to one more decimal would need about 19,000 replicates. A two-decimal interval from bootstrap_ci() at its default therefore prints digits that are Monte Carlo noise. The baseline test’s 1e-8 in Table 11.2 is a numerical tolerance for one fixed sequence of draws, and it says nothing about this. The MCSE is estimated once, when choosing B, and recorded with the baseline as the scale of the second tier; the test suite does not re-estimate it on every run.

Resampling at the B a real analysis needs is often parallelized across cores, and the random streams then need as much care as the seed. Naive parallelization fails first by irreproducibility. Under the default Mersenne-Twister generator and the default mc.set.seed = TRUE, parallel::mclapply() discards the inherited generator state in each child process, which then seeds itself from the clock and its process id. A set.seed() before the call therefore does not fix the result, and two runs differ. With mc.set.seed = FALSE the children inherit the parent’s state, and the workers draw identical streams, so supposedly independent replicates are duplicated. The L’Ecuyer-CMRG generator, which RNGkind('L\'Ecuyer-CMRG') selects, was built to provide many independent streams. Under it, mclapply() gives each child its own stream, and parallel::clusterSetRNGStream() gives one to each worker of a cluster. Per-worker streams are independent, but the result still depends on the number of workers. The replicates are divided among the workers in chunks, so a given replicate draws from a different stream when the worker count changes.

Invariance to the worker count needs per-task streams: one stream for each replicate, fixed before any worker starts. parallel::nextRNGStream() can precompute them, the future.apply package does so with future.seed = TRUE, and the doRNG package does so for a foreach loop. The book’s package provides replicate_streams(), which precomputes the streams with nextRNGStream() and runs the replicates with mclapply(). The chunk below runs the same 2000 bootstrap means of BMI on one worker and on two. Forking is unavailable on Windows, where both runs are sequential and the comparison is trivially true.

boot_mean <- function(i) mean(sample(bmi, replace = TRUE))
one_worker <- replicate_streams(2000, boot_mean, seed = 2024,
                                cores = 1)
two_workers <- replicate_streams(2000, boot_mean, seed = 2024,
                                 cores = 2)
identical(one_worker, two_workers)
#> [1] TRUE

The two runs are identical (TRUE), replicate by replicate, because replicate i draws from stream i wherever it runs. The same seed discipline that makes a serial bootstrap reproducible extends to a parallel one only in this per-task form.

A bootstrap bound changes when rerun with a different seed at the same B. It also changes when rerun with the same seed after an R version upgrade changes the default sampler (Table 11.2). Are these the same problem, and does increasing B fix both?

In size, largely yes; for the seed-exact check, no. The seed-to-seed change at fixed B is Monte Carlo error, and raising B shrinks it as \(1/\sqrt{B}\). A new sampler gives a given seed different resamples, which in effect draws the bound at a new seed, so raising B shrinks that change at the same rate. The two samplers still disagree in the last digits at any B, so the seed-exact tier keeps failing. Pinning the sampler restores seed-exact agreement; the second tier, within three MCSE, confirms that the statistical result reproduced without it.

Each of the four kinds catches defects the others do not see, and Figure 11.2 sets four representative defects against them. Start with the bottom row. Re-execution alone, with no test of any kind, catches none of the four, because every one of these defects is deterministic and will reproduce perfectly. That row is the reproducibility paradox in tabular form.

library(ggplot2)
library(tibble)

defects <- c('dropped rows change\nthe sample size',
             'ages encoded\nin months',
             'variance returned\nfor a standard deviation',
             'seeded bootstrap moves\nafter an R upgrade')
kinds <- c('Re-execution alone',
           'Reproducibility test',
           'Data-validation test',
           'Integration test',
           'Unit test')

cover <- expand.grid(defect = defects, kind = kinds,
                     stringsAsFactors = FALSE)
cover$defect <- factor(cover$defect, levels = defects)
cover$kind <- factor(cover$kind, levels = kinds)

hits <- c('Unit test|variance returned\nfor a standard deviation',
          'Integration test|dropped rows change\nthe sample size',
          'Data-validation test|dropped rows change\nthe sample size',
          'Data-validation test|ages encoded\nin months',
          'Reproducibility test|seeded bootstrap moves\nafter an R upgrade')
cover$catches <- paste(cover$kind, cover$defect, sep = '|') %in% hits

ggplot(cover, aes(x = defect, y = kind)) +
  geom_tile(aes(fill = catches), color = 'white', linewidth = 2) +
  scale_fill_manual(values = c('TRUE' = '#14746f',
                               'FALSE' = '#eef2f2'),
                    guide = 'none') +
  labs(x = NULL, y = NULL) +
  theme_minimal(base_size = 9) +
  theme(
    panel.grid = element_blank(),
    axis.text.x = element_text(size = 7.6, lineheight = 0.95),
    axis.text.y = element_text(face = 'bold', size = 8.4)
  )
A grid with four defect columns and five test-kind rows. The columns are: dropped rows change the sample size; ages encoded in months; a function returns variance instead of standard deviation; a seeded bootstrap changes after an R upgrade. The rows are unit test, integration test, data-validation test, reproducibility test, and re-execution alone. Filled cells run roughly diagonally: the unit test catches the variance error; the integration test and the data-validation test both catch the dropped rows; the data-validation test catches the ages in months; the reproducibility test catches the changed bootstrap. The re-execution row is entirely empty.
Figure 11.2: Four representative defects against the four kinds of test, with a fifth row for re-execution alone. A filled cell marks the kind of test most likely to catch that defect. The matrix is close to diagonal, which is the point: the kinds are complements rather than alternatives, and a suite missing one kind is blind to a whole class of defect. The bottom row is empty because every defect shown is deterministic, so re-running the analysis reproduces it faithfully.

For each defect, name the kind of test most likely to catch it. (a) A colleague refactors clean_cohort and accidentally drops the rows with a missing outcome, changing the sample size. (b) An upstream export encodes age in months rather than years, so ages read as up to fourteen hundred. (c) A function meant to return a standard deviation returns a variance. (d) After an R upgrade, a seeded bootstrap returns different draws than the manuscript reported.

  1. A data-validation test asserting the expected sample size, or an integration test that asserts the cleaned row count. The integration test shown earlier checks only the shape of the model summary, so it would need that line added. The book’s baseline test would also catch it, because it compares the cleaned row count with the recorded one, though it was not written for that purpose. (b) A data-validation test on the age range.
  2. A unit test against a hand-computed standard deviation. (d) A baseline test comparing the seeded result with a value recorded before the upgrade and committed, which is the comparison L3 requires; a same-session test would miss it. Note that the unit and data-validation tests catch defects that reproduction alone would happily propagate.

11.9 Test behavior, not implementation

A test is worth writing only if it survives changes that ought not to break it and fails on changes that ought to. The way to secure this is to test what a function does, its observable behavior, rather than how it does it, its internal implementation. The distinction is easiest to see in a pair of tests for the same function:

# fragile: pins an implementation detail
expect_true('filter' %in% all.names(body(my_clean)))

# robust: pins the behavior
result <- my_clean(data.frame(x = c(1, NA, 3)))
expect_false(anyNA(result))

The first test asserts that my_clean uses a function called filter internally. If the analyst later rewrites my_clean in base R, removing the filter call while leaving its behavior unchanged, the first test fails. It fails for no good reason: the function still does exactly what it promised. Such a test punishes honest refactoring and trains the analyst to distrust the suite. The second test asserts only that the cleaned data carries no missing values, which is the behavior that actually matters. It passes under any correct implementation. As a general rule, a test should fail when, and only when, the function’s contract is broken. Tests that fail for other reasons are a tax on maintenance. Wilson and colleagues, in their best-practices and good-enough-practices guidance, make behavioral testing a recurring recommendation (Wilson et al., 2014; Wilson et al., 2017).

A corollary concerns how much to test. A test that merely confirms a result is a data frame, without checking any value in it, tests too little and gives false comfort. A test that pins every incidental property tests too much and breaks on every revision. Deciding what to pin is the second of the analyst’s contributions named above, and no tool makes that decision.

expect_equal(environment(my_clean)$helper_count, 3) checks that an internal counter variable inside my_clean equals three. Is this a behavioral test or an implementation test, and what happens to it the first time my_clean is honestly rewritten?

It is an implementation test: it pins a detail of how my_clean is built, an internal counter, rather than what it does. A rewrite that preserves the function’s behavior but removes or renames that counter breaks the test for no good reason, training the analyst to distrust or delete it.

11.10 Code quality as a validation feature

Testing confirms that code is correct. A second family of tools confirms that code is legible and consistent. Although these tools do not check correctness directly, they belong to the same validation register and serve the same reproducibility end. The argument is one of mechanism. Code that a reader can follow is code whose errors a reader can find. Consistent formatting removes the visual noise that hides a transposed argument or a misplaced parenthesis. Moreover, a single house style across a compendium means that a difference in the code is always a difference in meaning, never merely a difference in whitespace. Wilson and colleagues, in both of their practice papers, treat readable, conventional code as a component of reproducible work rather than an aesthetic preference (Wilson et al., 2014; Wilson et al., 2017). Sandve and colleagues, in their ten simple rules, make the companion point that the provenance and the recorded results of an analysis must themselves be preserved (Sandve et al., 2013).

Two tools mechanize this, and a third enforces them.

Linting is static analysis: lintr reads the source without running it and flags stylistic deviations and likely errors. Examples are an assignment with = where the house style is <-, a line past the width limit, a variable assigned and never used, and a call to a function that does not exist. Because it does not execute the code, it catches a class of mistake that tests, which only exercise the paths they call, may miss.

Styling is automatic reformatting: styler rewrites the source into a consistent layout, fixing indentation, spacing, and alignment, so that formatting is never a matter for human attention or for a code review. The two are complementary; styler fixes the formatting that lintr would otherwise complain about.

In the framework, both run inside the project container through Makefile targets, so that no host R installation is required and every collaborator runs the identical checks:

$ make style   # reformat R code with styler
$ make lint    # analyze R code with lintr

A pre-commit hook enforces them. A hook is a script that git runs automatically at a defined moment; a pre-commit hook runs before a commit is recorded, and can refuse the commit if a check fails. Configuring the linter and styler as a pre-commit hook means that unstyled or unlinted code cannot be committed from a clone where the hook is installed, unless the analyst bypasses it with git commit --no-verify. That is a stronger guarantee than a convention that relies on memory, but a clone does not carry the hook with it. The configuration lives in a .pre-commit-config.yaml file at the project root:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/lorenzwalthert/precommit
    rev: v0.4.3
    hooks:
      - id: style-files
      - id: lintr
      - id: readme-rmd-rendered
      - id: no-browser-statement

With this in place, an attempt to commit code that styler would reformat, or that lintr flags, or that contains a stray browser() left over from debugging, is stopped before the commit completes. The analyst is then prompted to fix it. The hook is the fast local gate. A collaborator who has not installed it is caught only if continuous integration (Chapter 12) runs the same checks on the server. The workflows in that chapter do not, so a project that relies on the hook should add a step running lintr::lint_dir() there.

Classify each of the following as belonging to the capture register or the validation register, and say what would be lost by removing it. (a) The renv.lock lockfile. (b) A unit test of calculate_mean. (c) The .pre-commit-config.yaml hook. (d) The Dockerfile.

  1. Capture: it fixes the package versions; removing it drops the analysis from L1 to L0. (b) Validation: it checks that a captured function is correct; removing it leaves the analysis at the same level but now trusted rather than checked. (c) Validation: it checks that code is consistently formatted; removing it captures nothing less but permits legibility to decay. (d) Capture: it fixes the environment; removing it drops the analysis from L2 to L1. The rule of thumb is that removing a capture feature lowers the level, while removing a validation feature leaves the level unchanged and only stops the checking. There is one exception: the baseline test is the evidence L3 requires, so removing it drops an analysis from L3 to L2.

11.11 Validation in the regulated sense

Every use of ‘validation’ in this chapter has been the statistical and software-engineering sense: checking that code is correct, that data meet a constraint, that a result regenerates. In pharmaceutical, medical-device, and other FDA- or EMA-regulated work, ‘validation’ names a broader and more procedural idea. Chapter 3’s warning box introduces the distinction; this section develops the three practices that fill it out for testing specifically.

Computer system validation. Under 21 CFR 11.10(a), a closed system that creates, modifies, maintains, or transmits regulated electronic records must be validated ‘to ensure accuracy, reliability, consistent intended performance, and the ability to discern invalid or altered records’ (21 CFR Part 11, n.d.). The European Medicines Agency sets parallel expectations for computerized systems in clinical trials, and names statistical software among them (European Medicines Agency, GCP Inspectors Working Group, 2023). This is validation of the system across its lifecycle, documented once and maintained by a change-control process, not a one-time test run. The regulation does not say how that validation is shown. Installation, operational, and performance qualification (IQ, OQ, and PQ) come from qualification practice, in which a system qualification typically executes all three (Nicholls et al., 2020), not from the text of 11.10. The requirement’s reach is also narrower than its wording suggests. FDA’s 2003 guidance on the scope and application of Part 11 states that the agency will exercise enforcement discretion over the Part 11 validation requirement, while the predicate rules, the underlying requirements of the Act and of FDA’s other regulations, still apply, their own validation requirements included (U.S. Food and Drug Administration, 2003). The R Validation Hub concludes that Part 11 is not relevant or mandatory for statistical analysis software itself, though elements of it apply when R is used as part of a validated system (Nicholls et al., 2020). What an inspector asks in practice is whether the statistical computing environment is in a documented, validated state under the sponsor’s standard operating procedures and the predicate rules. A tinytest suite that passes is evidence toward that state. It is not, by itself, that state, which also rests on qualification records and a record of who approved each change and when.

Double programming. Independent of any code-based test, a regulated submission is commonly checked by double programming. Two analysts, or one analyst and an independent reviewer, implement the same analysis separately, from the same specification, usually in isolation from each other’s code. The two results are then compared. This is a different reproducibility practice from everything else in this chapter. A baseline test reruns the same code and checks that it still returns the same answer; double programming runs two different implementations of the same specification and checks that they agree. Consider a bug shared by both implementations, because both analysts misread the same ambiguous protocol sentence. It would pass a baseline test (the code is unchanged) and would also pass double programming (the two analysts made the same mistake). What double programming catches is a bug particular to one implementation, the error a single analyst’s code review is least likely to find in their own work.

Package risk assessment. Pinning package versions, as Chapter 6 teaches, fixes which code runs. It says nothing about how much to trust that code. But how is that trust to be judged? The R Validation Hub, an R Consortium working group formed for exactly this question, proposes a risk-based answer (Nicholls et al., 2020). The first step is to classify each package as either ‘intended for use’ (called directly by the analysis) or an unexamined dependency reached only through another package. The second is, for each intended-for-use package, to assess its purpose, maintenance practice, community usage, and existing test coverage. The third is to have a qualified reviewer record a documented risk judgment. Higher risk earns additional targeted testing. Base R and the recommended packages are treated as minimal risk. For other widely used packages, such as the tidyverse, the white paper says only that there is a reasonable case to treat their maintainers as a trusted resource, and leaves that decision to each organization. Whatever the risk score, tests are still required for the statistical modeling functions of an intended-for-use package. The riskmetric package operationalizes several of these metrics so the assessment does not start from nothing. This is a judgment about trustworthiness layered on top of the pinning Chapter 6 already does, not a replacement for it.

A compendium has a committed renv.lock, a passing tinytest suite, and a documented sign-off that dplyr 1.1.4 has been risk-assessed as trusted. An inspector asks whether the statistical computing environment is in a documented, validated state under the sponsor’s standard operating procedures. What is missing from the three items given?

All three are real evidence, but none is computer system validation by itself, and the three together are not automatically enough either. The lockfile is capture; the test suite is software validation in this chapter’s sense; the risk assessment is a package-level trust judgment. A validated state is defined by the sponsor’s procedures and required by the predicate rules, not by these artifacts. It concerns the environment as a whole across its lifecycle, maintained through a change-control process. Under qualification practice that means installation, operational, and performance qualification records, and sign-off on each subsequent change; 11.10(a) itself names none of these. It is not met by a snapshot alone of the code, the tests, and one package’s status at a single point in time.

11.12 Reviewing analysis code

Double programming is costly, because it implements every output it covers twice. Most analyses, regulated or not, are checked instead by review: a second analyst reads a proposed change, usually as a pull request (Chapter 4), and approves it before it merges. The two practices find different errors. A reviewer reads one implementation against the plan, and is well placed to see a wrong exclusion, a covariate the plan does not name, or a seed set in the wrong place. A reviewer is poorly placed to see an arithmetic slip in code that reads correctly, which is the error a second, independent implementation exposes. The choice between them is therefore a judgment about risk, made per output and recorded. One defensible arrangement is to double-program the outputs a decision rests on, such as the primary estimate, and to review the rest.

A reviewer needs a short list rather than a general impression. For a pull request that changes an analysis, we suggest six questions.

  1. Does every output the change affects, whether a table, a figure, or a number in the prose, trace to committed code, with no value typed by hand?
  2. Are the inputs read only from the raw data, never from a hand-edited intermediate file or an object left in a session?
  3. Are the exclusions and derived variables the ones the analysis plan states, with the same thresholds and the same handling of missing values?
  4. Is each seed set at the call that uses it, in the analysis code, rather than in .Rprofile or nowhere?
  5. Did the tests change alongside the code, and does the pull request say why? An expected value edited to match new output needs the oracle that justifies the edit.
  6. Does the log of the render or the test run show warnings nobody expected? A clean log is one in which every warning has been read and then removed or explained.

The list draws on the book’s gates, but it is not a gate. It asks for the judgments that continuous integration cannot make (Chapter 12): whether a threshold matches the plan, and whether a changed test was changed for a good reason. The approval on the pull request then records who checked the change and when, the same kind of record the validated state described above rests on. Questions 3 and 5 take the longest, because answering them means opening the plan and the test diff rather than reading the code alone.

11.13 Testing across the archetypes

The four kinds of test are the same in every compendium, but their relative weight shifts with the archetype. Table 5.2 in Chapter 5 is the reference. We state the shift here so that a student who has met testing only in the context of R packages does not wrongly conclude that a data analysis has nothing to test. Testing is native to the R-package archetype, whose suite is dominated by unit tests that R CMD check runs as a matter of course. The data-analysis and simulation archetypes, by contrast, may have few reusable functions and no exported interface. In these the data-validation and reproducibility tests carry the weight instead. A simulation in particular leans on a baseline test that fixes the seed and compares the result with a committed value, since its entire output is a function of that seed. Chapter 12 develops the point at length. The manuscript archetype adds a test that the report renders and that its headline numbers match a baseline. The blog archetype tests the functions and example computations its posts depend on. That is a reproducibility obligation to readers no smaller than a manuscript’s to its reviewers.

A data-analysis compendium with no exported functions and no package structure still needs tests. Which two kinds of test carry most of that weight, and why?

Data-validation tests and reproducibility (baseline) tests. With few reusable functions, the unit-test surface is small. The risk therefore shifts to the two places such a compendium still has: the data it reads, which may be malformed, and the result it reports, which must regenerate. Those are exactly what data-validation and baseline tests check.

11.14 Worked example

We now make the paradox concrete and executable. The following chunk defines the calculate_mean function in two forms, one correct and one carrying the bug of the orientation, and checks each against three inputs whose means are known by hand. Rather than call the tinytest package, which may be absent from a bare R session, we hand-roll a one-line expectation. That is all a bare check really is: a comparison of an actual value against an expected one.

# the function under test, in a correct and a
# buggy form
calculate_mean <- function(x, na.rm = FALSE) {
  if (na.rm) x <- x[!is.na(x)]
  sum(x) / length(x)
}

calculate_mean_buggy <- function(x) {
  sum(x)                 # bug: division dropped
}

# a minimal hand-rolled expectation: no testing
# package required
expect <- function(actual, expected, tol = 1e-8) {
  isTRUE(abs(actual - expected) < tol)
}

# three inputs whose means are known independently
cases <- list(c(1, 2, 3),
              c(10, 20, 30, 40),
              c(5, 5, 5, 5, 5))
oracle <- c(2, 25, 5)

results <- data.frame(
  input = vapply(cases,
                 function(x) paste(x, collapse = ', '),
                 character(1)),
  expected = oracle,
  correct = vapply(cases, calculate_mean, numeric(1)),
  buggy = vapply(cases, calculate_mean_buggy,
                 numeric(1))
)
results$correct_passes <-
  mapply(expect, results$correct, oracle)
results$buggy_passes <-
  mapply(expect, results$buggy, oracle)

knitr::kable(results)
Table 11.3: A correct function and a buggy variant checked against three inputs with hand-computed means. The buggy variant returns the same wrong value on every input and would return it on every machine and every rerun: it is perfectly reproducible and still wrong.
input expected correct buggy correct_passes buggy_passes
1, 2, 3 2 2 6 TRUE FALSE
10, 20, 30, 40 25 25 100 TRUE FALSE
5, 5, 5, 5, 5 5 5 25 TRUE FALSE

The correct function passes every check; the buggy function fails every check, and it fails them identically. Were we to rerun this chunk a thousand times, on a thousand machines, the buggy column would hold the same wrong values every time. The bug is not a source of variability, which reproducibility tooling might eventually surface. It produces the same error on every run, and only a comparison against an independent oracle can catch it. That comparison is what a test is.

In a real compendium, the hand-rolled expect gives way to the tinytest expectations, and the checks live in a file under inst/tinytest/. The following is that file, shown as a static listing that is not run here. It is exactly the unit test that would have caught the bug before it reached a result:

# inst/tinytest/test-calculate_mean.R
expect_equal(calculate_mean(c(1, 2, 3)), 2)
expect_equal(calculate_mean(c(10, 20, 30, 40)), 25)
expect_equal(calculate_mean(rep(5, 5)), 5)
expect_true(is.nan(calculate_mean(numeric(0))))
expect_equal(calculate_mean(c(1, NA, 3),
                            na.rm = TRUE), 2)

The runner tests/tinytest.R that R CMD check invokes is a three-line file:

# tests/tinytest.R
if (requireNamespace('tinytest', quietly = TRUE)) {
  tinytest::test_package('mycompendium')
}

and the pre-commit configuration shown earlier completes the picture. The linter and styler guard the code’s legibility on every commit, and the tinytest suite guards its correctness. Continuous integration re-runs both on a clean machine, so that neither guarantee depends on the author’s local setup.

In Table 11.3, why is oracle <- c(2, 25, 5) written by hand in the chunk rather than computed by calling calculate_mean() on the three cases?

Because an oracle has to come from outside the code under test. Computing the expected values by calling calculate_mean() would make the check compare the function against itself. It would pass identically whether calculate_mean were correct or carried the dropped-division bug, since both forms are internally consistent with their own output. The hand-written vector is what lets the table show the buggy column failing every row.

11.15 Collaborating with an LLM

A language model is a capable assistant for writing tests, and a dangerous one for a specific reason: it can readily produce tests that pass, which is not the same as tests that are correct. The analyst’s verification duty is heaviest exactly where the model is most fluent.

ImportantRestricted data

One rule comes first (habit 3). Never paste or upload protected health information, identifiable records, or data held under a data use agreement, nor outputs derived from them, into a model or a coding agent unless three conditions all hold. The institution has approved that endpoint for that class of data. The IRB protocol and the consent permit the processing. The data use agreement, or the provider’s terms, permits it. If any of the three is unclear, the answer is no.

The third condition is the one most often missed. Institutional approval of a tool does not override the terms under which a data provider released the data. NIH, for example, treats sharing controlled-access data through prompts with public generative AI tools, which it glosses as third-party tools, as a breach of the data use certification. It treats a model trained on such data as a data derivative, to be shared only with other approved users and not kept after the project closes (National Institutes of Health, 2025).

Describe the structure instead: the column names and types, or a small synthetic example generated from the data dictionary, never edited from real rows. A coding agent sees whatever files are in its working tree, so run it in a checkout or container where the real data is not mounted.

An agent also acts, so confine it to a branch and keep merges, pushes, and deposits for yourself (Section 2.10.1).

Tests sit close to the data, so the derived outputs the rule covers include a failing test’s printout of offending rows. The book’s simulated cohort is a suitable synthetic example to describe instead.

Prompt. ‘Here is my calculate_mean function. Write tinytest unit tests for it, including edge cases.’

Watch for. Check whether each expected value was derived by running the function it was given, rather than by independent computation. If the function is buggy, an oracle derived that way matches the bug, producing tests that pass against wrong behavior and thereby certifying the error. Check also for tests of only the paths the code obviously takes, which miss the empty vector or the missing value.

Verification. Check every expected value by hand or from an independent formula, not by running the function, since the oracle must come from outside the code under test. Confirm that the edge cases you care about, the empty input, the single value, the missing value, each have a test.

Prompt. ‘Here is the data dictionary, not the data, for my cohort: age_at_enroll (integer, years), sex (character, coded F or M after cleaning), bmi (numeric, kg/m^2), incident_dm (integer, 0 or 1). Write tinytest data-validation tests for a data frame with these columns.’

Watch for. Check for generic checks, such as non-missing identifiers and numeric outcomes, that are reasonable but do not encode your domain. The dictionary gives types and codes. It does not give the protocol’s eligibility ages, the plausible range of BMI in your population, or how much outcome missingness the analysis can tolerate, so any such bound in the answer was supplied by the model rather than by your protocol.

Verification. Supply the domain constraints yourself, or check that the generated ranges match your data dictionary. A validation test asserting a plausible-looking but wrong range is worse than no test, because it lends false authority.

Prompt. ‘Explain why this test is failing.’

Watch for. The model may recommend changing the test to match the current output, which makes the failure disappear without establishing whether the code or the test was wrong. A failing test is sometimes correct: it may be reporting a real bug.

Verification. Decide first, from the oracle, whether the code or the expectation is at fault, and only then change the one that is wrong. Never silence a failing test by editing its expected value to whatever the code currently returns; that converts a test into a tautology.

A better prompt. ‘The attached tinytest file fails on the line shown, with this message. The expected value was computed by hand from the analysis plan. Do not edit the test file. List the possible causes in the function, and for each give an input I can run, with its hand-computed answer, that would confirm or rule it out.’

A coding agent with write access need not stop at a recommendation: asked to make the suite pass, it can make the edit itself, and the change arrives as an ordinary commit. Treat inst/tinytest/ and inst/baseline/ as protected paths, and read every line of any diff that touches them before it merges, as question 5 of Section 11.12 asks. What an agent may change and what stays with the analyst is set out in Section 2.10.1.

11.16 The cohort, continued

The chapter has introduced four kinds of test. This section applies all four to the running cohort compendium, using the canonical Cox analysis as the analysis under test.

The canonical analysis. The primary question is: does BMI predict time to incident diabetes after adjusting for age and sex? The model is a Cox proportional-hazards model fit to the cleaned extract, with BMI expressed per 5 kg/m^2 so that the hazard ratio has a clinically natural scale. The analysis runs clean only after clean_cohort() has repaired the four planted defects.

library(survival)
cohort_path <- file.path(
  '..', '..', 'analysis', 'data', 'raw_data',
  'cohort_raw.csv')
raw    <- load_cohort(cohort_path)
cohort <- clean_cohort(raw)
fit    <- fit_cox(cohort)
res    <- summarise_cox(fit)

The hazard ratio for BMI per 5 kg/m^2 is 1.83 (95% CI 1.32–2.52), based on 48 events from 337 cleaned rows. Uncleaned, the analysis would still run, but it would fit a different model to the 338-row raw extract, with the duplicate and the miscoded sex included.

Unit tests of fit_cox(). A unit test checks the contract of one function in isolation. For fit_cox() the contract is about which rows the model uses. coxph() drops every row with a missing value on any variable it reads, so the model uses the complete cases on followup_days, incident_dm, bmi, age_at_enroll, and sex, and its event count is the number of events among those cases, not the number of non-missing outcomes. The oracle is computed from the data directly rather than from the fit, so a change to the exclusion rule fails the test. The assertions below are in inst/tinytest/test-fit-cox.R.

model_vars <- c('followup_days', 'incident_dm', 'bmi',
                'age_at_enroll', 'sex')
complete <- stats::complete.cases(cohort[, model_vars])

expect_equal(fit$n, sum(complete),
  info = 'the model uses the complete cases on its variables')
expect_equal(fit$nevent,
             sum(cohort$incident_dm[complete] == 1),
  info = 'the event count is the events among complete cases')
expect_true(fit$nevent < sum(!is.na(cohort$incident_dm)),
  info = 'events are fewer than non-missing outcomes')

On the cleaned cohort the model uses 305 of the 337 rows, and its 48 events are the rows among them with incident_dm equal to 1. A count of non-missing outcomes, here 305, would count the zeros as events too. The same file also removes the age of one event row and checks that exactly one event and one row leave the model.

Integration test of the pipeline. An integration test checks that the pipeline stages compose correctly end to end: load_cohort feeds clean_cohort feeds fit_cox feeds summarise_cox, and the final result is a list with the expected names and no NA values. This listing and the two after it are illustrative and abridged, and the book’s suite does not run them as written.

result <- run_canonical_analysis(cohort_path)
expect_true(all(c('n_rows', 'hr', 'lower',
                   'upper', 'n_events')
                %in% names(result)),
  info = 'canonical result has expected names')
expect_true(all(!is.na(unlist(result))),
  info = 'canonical result contains no NA values')

Data-validation tests. The planted pathologies give the data-validation suite something real to catch. The test file checks each defect before and after cleaning, so a change to clean_cohort() that accidentally fixes or breaks a defect fails a test instead of silently altering the baseline.

# Duplicate before cleaning, gone after
expect_equal(sum(duplicated(raw$mrn)), 1,
  info = 'raw extract has exactly one duplicated MRN')
expect_equal(sum(duplicated(cohort$mrn)), 0,
  info = 'cleaned cohort has no duplicated MRN')

# Implausible age before cleaning
expect_true(any(raw$age_at_enroll > 105, na.rm = TRUE),
  info = 'raw extract has at least one implausible age')
expect_true(all(is.na(cohort$age_at_enroll) |
                cohort$age_at_enroll <= 105),
  info = 'cleaned cohort has no age above 105')

Reproducibility (baseline) test. The baseline test is the L3 step. It reruns run_canonical_analysis() and compares the hazard ratio and confidence limits with values in inst/baseline/cohort_baseline.rds, within a relative tolerance of 1e-8.

baseline <- readRDS(
  system.file('baseline', 'cohort_baseline.rds',
              package = 'reproducibility'))
result   <- run_canonical_analysis(cohort_path)

expect_equal(result$hr, baseline$hr,
  tolerance = 1e-8,
  info = 'Cox HR matches committed baseline')
expect_equal(result$lower, baseline$lower,
  tolerance = 1e-8,
  info = 'lower CI bound matches committed baseline')

A failure here means either the extract changed (check shasum -a 256 cohort_raw.csv), the survival package version changed (check renv.lock), the R version changed (check sessionInfo()), or the model specification changed (check the diff of R/cohort.R). A baseline that also records a Monte Carlo quantity, such as a bootstrap bound, has a fifth cause: the random-number generator changed (check RNGkind() against the recorded kind). The question there is whether the difference is within Monte Carlo error, which the second tier of compare_to_baseline() answers. Each cause has a different fix; the baseline test names the symptom and the diagnosis follows from the audit.

11.17 Exercises

  1. Take a function you have written for a current analysis and write three tinytest unit tests for it: one ordinary case with a hand-computed oracle, and two edge cases. State, for each, where the expected value came from.

  2. Introduce the calculate_mean bug of this chapter into a copy of one of your own summary functions, and confirm that the analysis still runs and produces a stable result. Then write the single unit test that would have caught it. Reflect in a sentence on why reproduction did not catch it.

  3. Write a data-validation test file for a dataset you use, encoding at least four domain constraints. These are an expected set of columns, a plausible range for a continuous variable, an admissible set of labels for a categorical variable, and a bound on the missing-data rate. Justify each range from a data dictionary or from clinical knowledge.

  4. Classify each test you wrote in exercises 1 and 3 as belonging to the capture register or the validation register, and state what removing it would and would not change about the analysis’s position on the L0 to L3 ladder.

  5. Write a reproducibility test of the baseline form: run a small analysis, save its result with saveRDS, and write a test that recomputes the result and compares it against the saved baseline with an appropriate tolerance. Then change one line of the analysis and observe the test fail.

  6. Install the precommit package, add the .pre-commit-config.yaml of this chapter to a project, and attempt to commit deliberately misformatted R code. Report what the hook does, and contrast it with what continuous integration would do in Chapter 12 if the hook were absent.

  7. On the running cohort compendium, run load_cohort() and clean_cohort() from R/cohort.R against analysis/data/raw_data/cohort_raw.csv, then write a data-validation test file that checks, against the real cleaned cohort, this chapter’s four constraints: the expected columns, the enrollment age range, the two coded sex values, and a bound on outcome missingness. Report three numbers a grading script could check without fitting any model: the raw row count, the cleaned row count, and the number of rows removed as duplicates.

  8. This exercise returns to the committing discipline of the version-control chapter and habit 1, ‘commit before you compute.’ The baseline test of this chapter depends on a committed file, inst/baseline/cohort_baseline.rds, which is itself a recorded result. State which must be committed first, the analysis code that the recording script runs or the baseline file that running it produces, and explain what goes wrong for a collaborator who pulls the repository if the order is reversed.

  9. Paired review. Work with a partner, each on your own copy of the cohort compendium. Treat the following as the cohort’s analysis plan for cleaning: keep one row per mrn; parse both date formats in enroll_date; code sex as F or M; set ages above 105, the upper enrollment limit, to missing rather than dropping the row; and keep rows with a missing outcome. Without consulting clean_cohort(), each of you writes a cleaning function to this plan on a branch and opens a pull request. Then review your partner’s pull request against the six questions in Section 11.12, leaving each finding as a review comment, and answer the comments on your own. Finally, run both functions on cohort_raw.csv and compare the two cleaned data frames. Report one discrepancy that review found and one that only the comparison found, or say that there was none.

11.18 Solutions

Attempt an exercise before expanding its solution. You perform these exercises on your own functions and data, so the code will vary; the boxes record the form the answer should take and the reasoning behind it.

The form of the answer matters more than the function chosen. For a hypothetical summarize_visits():

# inst/tinytest/test-summarize_visits.R
expect_equal(summarize_visits(c(2, 4, 6)), 4)
expect_true(is.nan(summarize_visits(numeric(0))))
expect_error(summarize_visits('a'))

Where each expected value came from is the substance of the exercise, and the three answers should be different.

The ordinary case comes from arithmetic done by hand, independently of the function. This is what makes it an oracle. If the expected value were computed by running the function, the test would assert only that the function agrees with itself, which is the commonest way a test suite becomes worthless.

The empty-input case comes from a decision about the contract. There is no arithmetic to do; the analyst must decide what the function should do with no data and then encode the decision. The test is as much a specification as a check.

The wrong-type case comes from the same contract, asserting that the function refuses input it cannot handle rather than returning something misleading.

If you cannot state where an expected value came from, you have usually written a change-detector rather than a test.

The bug is the dropped division, so the function returns a sum where a mean is intended:

summarize_visits <- function(x) sum(x)   # bug

The analysis still runs, produces a number, and produces the same number on every machine and every rerun. If the project is containerized and its packages pinned, that number is perfectly reproducible.

The catching test is the ordinary case from exercise 1:

expect_equal(summarize_visits(c(2, 4, 6)), 4)

It fails immediately, returning 12 against an expected 4.

Why reproduction did not catch it. Because reproduction asks whether the same inputs give the same outputs, and they do. The buggy function is deterministic; nothing about it varies between machines or over time. Reproducibility machinery captures determinants and propagates whatever the code computes with mechanical fidelity, including the wrong thing. The only way to detect the error is to compare the output against a value obtained independently of the code. That is what the hand-computed oracle supplies, and what no amount of environment capture can.

The four constraints, written for the running cohort in the form the chapter uses; your own dataset will have its own names and bounds:

# inst/tinytest/test-data-cohort.R
raw <- load_cohort('../../analysis/data/raw_data/cohort_raw.csv')
cohort <- clean_cohort(raw)

expect_true(
  all(c('mrn', 'age_at_enroll', 'sex', 'incident_dm') %in%
        names(cohort)),
  info = 'the expected columns are present'
)
expect_true(
  all(cohort$age_at_enroll >= 18 &
        cohort$age_at_enroll <= 105, na.rm = TRUE),
  info = 'age lies within the enrollment criteria'
)
expect_true(
  all(cohort$sex %in% c('F', 'M')),
  info = 'sex takes only its two coded levels'
)
expect_true(
  mean(is.na(cohort$incident_dm)) < 0.15,
  info = 'outcome missingness stays under fifteen percent'
)

Each justification should come from outside the data.

The column set comes from the data dictionary: these are the variables the analysis consumes, and a missing one means an upstream export changed. The age range comes from the protocol’s eligibility criteria, not from the observed minimum and maximum. Deriving the bound from the data would make the test unable to detect the very excursion it exists to catch. The label set comes from the coding sheet. The missingness bound comes from what the analysis can tolerate before its conclusions become unsafe, which is a judgment the analyst must make and record.

All of them belong to the validation register. Unit tests and data-validation tests alike pin no determinant: they add no package to the lockfile and no library to the image. They confirm that what was captured behaves as intended.

What removing them would change. Nothing about the position on the L0 to L3 ladder. An analysis at L2 with a full test suite and the same analysis at L2 with no tests sit on the same rung. The rung grades how far down the determinant stack the result is locked, and tests lock nothing.

What removing them would change instead is whether anyone has reason to believe the result is right. The analysis becomes trusted rather than checked, and the class of error the tests were catching, a wrong function and an out-of-range dataset, returns to being invisible.

One qualification concerns L3. L3 requires that recorded outputs be shown to regenerate within a stated tolerance from a clean rebuild, and the evidence for that is a baseline test. The unit and data-validation tests of exercises 1 and 3 are not baseline tests, so removing them leaves L3 where it was. Removing the baseline test, by contrast, would drop a project from L3 to L2, because the one piece of evidence the rung requires would be gone. It is the verification role that matters there, not the testing role.

# once, from the project root, to record the baseline
cohort <- clean_cohort(
  load_cohort('analysis/data/raw_data/cohort_raw.csv')
)
saveRDS(coef(fit_model(cohort)),
        'inst/tinytest/baseline-coefs.rds')
# inst/tinytest/test-baseline-coefs.R
data_file <- '../../analysis/data/raw_data/cohort_raw.csv'
if (!file.exists(data_file)) exit_file('no analysis data')
baseline <- readRDS('baseline-coefs.rds')
cohort <- clean_cohort(load_cohort(data_file))
current <- coef(fit_model(cohort))
expect_equal(current, baseline, tolerance = 1e-8,
             info = 'model coefficients match the baseline')

The model fit draws no random numbers, so no seed is needed; a stochastic quantity would take its seed as an argument, as bootstrap_ci() does in the chapter. The file name begins test-baseline, the convention by which a verification step finds baseline tests. Changing one line of the analysis makes the test fail, reporting the coefficients that moved.

The tolerance is the part of this exercise that needs thought. An exact comparison is the strongest check, and it will produce failures that mean nothing. Floating-point summation reorders under a different BLAS or a different thread count, as Chapter 7 discusses, and the last digits move without the analysis changing. Two different tolerances are in play, and you should name each. The numerical tolerance, 1e-8 relative here, accepts that floating-point noise and still catches any change in the fitted model large enough to alter a reported figure. The reporting tolerance is the precision at which results are published; for a Monte Carlo quantity it is set by the MCSE, as the second tier of the chapter’s baseline check is. Because these coefficients draw no random numbers, only the numerical tolerance applies to them. A numerical tolerance set loosely, near the reporting precision, stops detecting real changes.

What this test is and is not. It is the kind of baseline test L3 requires, and it is a change detector. It asserts that today’s answer matches the recorded one, not that either is correct, because the baseline was produced by the same code it now guards. If the code was wrong when the baseline was recorded, this test will faithfully defend the wrong answer. That is the paradox reappearing one level up. The unit tests of exercise 1 are what establish correctness; this test establishes stability.

What the hook does. The commit is refused. The formatting hook rewrites the offending files in place and aborts, so the commit does not complete and the working tree now contains the reformatted code, which must be staged and committed again. The effect is that misformatted code cannot enter the history at all.

The contrast with continuous integration. CI would let the commit through. The code enters the history, the push succeeds, and the check runs afterward on the server, reporting a failure against a commit that already exists. Fixing it requires a second commit, so the history now records both the violation and its repair.

The differences worth naming are three. Timing: the hook acts before the record is written, CI after. Locality: the hook runs on the analyst’s machine and can therefore be skipped, with --no-verify or by a collaborator who never installed it, whereas CI runs on the server and cannot. Scope: the hook must be fast enough to sit inside a commit, so it does cheap things like formatting and linting, whereas CI can rebuild the container and run the full suite.

The two are complements rather than alternatives, and the chapter’s framing explains why. The hook is a convenience that keeps trivia out of the history. CI is the gate that actually guarantees anything, because it is the one nobody can skip.

The solution for this exercise is in the instructor guide, which is distributed separately to course instructors. Self-study readers who want to check their answer can compare the three autogradable values against those reported in the chapter’s own ‘The cohort, continued’ section.

The code must be committed first, then the recording script is run against that committed code, and the baseline file it writes is committed second, in its own commit. This is habit 1 applied to a derived artifact rather than to the primary analysis: a committed file must always be traceable to the committed code that produced it.

Reversing the order breaks that traceability. Suppose the baseline is regenerated and committed while the code change that motivated it is still uncommitted. A collaborator who pulls the repository then receives the new baseline paired with the old code, because the code change never reached the shared history. Their copy of the baseline test then fails immediately, not because anything is wrong with their checkout, but because the repository itself never recorded a consistent pair of code and baseline. Worse, if the local, uncommitted code change is later lost, the committed baseline becomes orphaned. It is then a recorded result with no committed code that anyone can point to as having produced it, which is exactly the provenance gap habit 1 exists to close.

Your functions and findings will differ; the answer takes the form of the review comments, the replies, and the comparison. Four places in this plan repay a reviewer’s attention, because each is where your two functions can diverge while both run without error.

Ages. The plan sets an implausible age to missing. A function that drops the row instead changes the row count, and the change appears only if you count rows before and after. This is question 3.

Missing outcomes. The plan keeps rows whose outcome is missing, because dropping them is an analytic decision that the model step makes. A cleaning function that drops them has moved that decision into the wrong place.

Duplicates. ‘One row per mrn’ does not say which copy to keep. Check whether the duplicated rows agree on every column. If they do not, the plan is ambiguous, and the fix belongs in the plan, recorded and dated, not in whichever choice each of you happened to code.

Coding and parsing. Look at every distinct value of sex and every format in enroll_date in the raw file before deciding that the plan’s rules cover them. A value the rules do not name should become missing by a stated rule, not by accident.

The comparison is double programming in miniature. Use all.equal() on the two cleaned data frames, after sorting both by mrn, and read every difference it reports. Review is suited to the departures from the plan that are visible in the code, such as a dropped row; the comparison is suited to the slips that read correctly, such as a date format parsed in the wrong order. If the two functions agree, that is evidence for both, but not proof: you may have read the same sentence of the plan the same wrong way, as the section on double programming warns.

11.19 Cheat sheet

Command or rule What it does
expect_equal(a, e) Compares a to oracle e with all.equal()’s default tolerance.
expect_equal(a, e, tolerance = 1e-8) The baseline-test comparison: a stated, not default, tolerance (habit 5).
expect_true(), expect_false(), expect_identical(), expect_error(), expect_inherits() The common tinytest expectations beyond equality.
tinytest::test_package('pkg') Runs the suite under inst/tinytest/ via the tests/tinytest.R runner.
exit_file('message') Exits a test file early, for a test that needs data R CMD check does not install.
make lint / make style Run lintr / styler inside the project container.
test-baseline*.R naming The convention zzc verify --full looks for to certify L3.
Rule of thumb Test behavior, not implementation; a test should fail only when the contract breaks.
Rule of thumb An oracle must be computed independently of the code under test, never by calling it.
Habit 4 Make every gate able to fail, and watch it fail once.
Habit 5 Compare against a recorded baseline, with a stated tolerance.
Code review, six questions (Section 11.12) Output traces to code; raw inputs; the plan’s exclusions; seed at the call; tests changed with a reason; clean log

11.20 Further reading

The single most important reference for this chapter is the short note by Leek and Peng, that reproducible research can still be wrong (Leek & Peng, 2015). It states the paradox that motivates every test in a compendium, and it argues for a prevention-oriented rather than a correction-oriented stance toward analytical error.

For the testing tool itself, van der Loo’s documentation of tinytest (Loo, 2020) explains the design rationale for a zero-dependency, bare-expectation framework. It is worth reading for the argument that a testing framework should not itself be a heavy dependency.

For the practice of testing and code quality as components of scientific computing, the two papers by Wilson and colleagues are the standard references. They are the best-practices paper (Wilson et al., 2014) and the good-enough-practices paper (Wilson et al., 2017), the latter pitched deliberately at researchers without a software background. Sandve and colleagues’ ten simple rules (Sandve et al., 2013) cover the same ground in a compact, rule-by-rule form, and situate version control and recorded results within a general reproducibility discipline.

The connection between testing and the highest rung of the ladder is taken up in Chapter 12, where the reproducibility test of this chapter reappears at the scale of the whole compendium as output verification. Chapter 3 gives the capture-and-validation distinction that places testing in the validation register. Peng’s early account of reproducibility as a spectrum (Peng, 2011) remains the clearest short statement of why reproducibility can serve as a minimum scientific standard when full independent replication is not possible.