Provide Public Access to Scripts, Runs, and Results.
Sandve, Nekrutenko, Taylor, and Hovig, Ten Simple Rules for Reproducible Computational Research, Rule 10 (2013)
A lockfile that has not been rebuilt since it was committed, and a Dockerfile that has not been rebuilt since it was written, are untested claims. By the end of this chapter you will be able to look at a green build and say exactly what it has shown: which of three automated gates ran, what each one would have caught had it been absent, and what a missing baseline test still lets through unnoticed when every gate is green. You will also know how to turn each gate red on purpose, because a check that has never been seen to fail has not been shown to work.
NoteBefore you start
Reading time. About 70 minutes.
Prerequisites. Chapter 11 (Chapter 11) supplies tinytest, the four kinds of test, and the baseline test this chapter’s verification gate runs. Chapter 3 (Chapter 3) supplies the L0-L3 ladder and the capture-versus-validation vocabulary this chapter makes operational. Chapter 6 (Chapter 6) supplies the dependency invariant the dependency-manifest gate automates, and Chapter 7 (Chapter 7) supplies the Dockerfile the verify job rebuilds. The R knowledge assumed is writing and calling a function, together with enough comfort reading YAML to follow a workflow file line by line; no YAML is written from scratch here.
Objectives. By the end of this chapter, can you:
explain why a CI runner, carrying none of the author’s undeclared state, catches a ‘works on my machine’ failure that no test run on the author’s own machine can?
name the three automated gates a CI workflow runs, state what each one proves, and demonstrate how to make each one fail on purpose?
say why verification (L3) compares computational outputs against a committed baseline, within a stated tolerance, rather than comparing rendered PDFs or HTML?
read and extend a checksum manifest that guards the integrity of raw input data?
match the three gates, plus verification, to a compendium’s archetype, so that a package, a manuscript, and a simulation each run only the checks appropriate to them?
Outline. We begin with where CI sits on the stack, a captured environment that nobody has yet rebuilt, and the analyst’s three irreplaceable judgments. We then turn to the machine that is not yours, to the three gates and where their bugs hide, to secrets, tokens, and pinned actions, and to two CI dialects. The middle of the chapter treats verification and level L3, the checksum manifest, the framework’s own capture test matrix, and how CI adapts to five archetypes. We then consider independent external verification, a worked example, debugging a failed reproduction, and a gate demonstrated on this book’s own cohort compendium. Lastly, we discuss collaborating with an LLM, followed by exercises, a cheat sheet, and further reading.
NoteCheck your understanding: from earlier chapters
Two colleagues rerun a compendium in the same digest-pinned image, built from a complete lockfile. Colleague A mounts the project without its .Rprofile, where the author had put RNGkind(sample.kind = 'Rounding') and set.seed(2024), and the bootstrap interval moves. Colleague B, on a different processor, obtains regression coefficients that agree with the author’s except in their last few digits. For each discrepancy, name the determinant layer behind it, say whether any rung of the ladder closes it, and give the earlier remedy.
NoteAnswer
Of the two discrepancies in this warm-up, colleague A’s lies in session configuration. The R release fixes the default sampler at L2, but the .Rprofile overrode it, and an override is fixed only by an explicit RNGkind() call in the analysis code (Chapter 3). The seed belongs there too, set at the call that draws the resamples, not in .Rprofile (Chapter 11). Colleague B’s lies in the numerical libraries, the BLAS and LAPACK beneath crossprod() and solve(). No rung guarantees that layer: an image determines which BLAS is installed, but the thread count and the processor still vary (Chapter 3). The remedy is a baseline test that compares at a stated tolerance, wide enough to absorb reordered floating-point sums and too narrow for a changed coefficient to pass (Chapter 11).
12.1 Where this sits on the stack
Continuous integration pins nothing, and Figure 12.1 marks that explicitly. The three layers its gates touch (package versions, analysis code, and input data) are validated here, not captured, because the lockfile, the scripts, and the raw extract were already pinned or written by the chapters that came before. This chapter’s whole contribution is confirming, on a machine that is not the author’s, that those captures still hold.
plot_determinant_stack( validates =c('Package versions', 'Analysis code', 'Input data'), title ='CI captures nothing; it validates three layers already captured')
Figure 12.1: The determinant stack of Chapter 3, with package versions, analysis code, and input data marked as validated rather than captured. CI pins no new layer: the dependency-manifest gate confirms packages are still pinned, R CMD check and the render gate confirm the analysis code still runs, and the checksum manifest confirms the input data is unchanged, which applies the capture-versus-validation distinction to the three gates of this chapter.
12.2 Orientation
Consider a registry study whose analysis is finished, whose manuscript is written, and whose lockfile and container recipe are committed to version control. The author has done, to all appearances, everything this book has asked. The packages are pinned (Chapter 6), the environment is fixed in an image (Chapter 7), the report is literate and regenerates its own figures (Chapter 9). The compendium sits, by the author’s account, at level L2, a pinned environment. But how does the author know that the captured environment actually rebuilds, on a machine that is not the one where the work was done? Until something tests it, the author does not know. A lockfile can reference a package version that has been pulled from its repository. A Dockerfile can depend on a base image whose digest has drifted. A script can silently rely on a file that lives only in the author’s home directory and was never committed. None of these announce themselves on the machine where the work already runs, because on that machine the missing piece is present.
In this chapter we take up the instrument that closes the gap between a reproducibility claim and its demonstration. Continuous integration, abbreviated CI, is a service that rebuilds the compendium from its committed inputs on every change. It works in a fresh environment that carries none of the developer’s accumulated and undeclared state, and runs a battery of checks against the result. We shall see that CI captures no new layer of the determinant stack. Its whole contribution is to validate that the layers already captured actually suffice. The distinction between capturing reproducibility and validating it was drawn in Chapter 1 and carried through every chapter. It reaches its operational form here: pinning is capture, and CI, together with the verification command it runs, is validation.
We then take the ladder to its top rung. Verification, the L2-to-L3 transition, confirms what has been captured by regenerating the computational outputs. It compares them, number for number and within a stated tolerance, against a recorded baseline. The subtlety, which we shall press, is in what gets compared. We compare a derived dataset, a fitted model object, or a results table, and emphatically not the rendered PDF or HTML, whose bytes carry timestamps and font identifiers that defeat a naive comparison. We close with the schemes by which reproducibility is checked from outside the author’s own project entirely, during peer review.
NoteCheck your understanding: what CI does not know
An author claims their compendium is at level L2 because a Dockerfile is committed and builds on their laptop. Does a green CI run of that same Dockerfile raise the compendium’s level? What, precisely, does the green run add?
NoteAnswer
No. The level is set by what is captured, not by who rebuilds it, and a CI run captures nothing. Whether the compendium is at L2 at all depends on what else is committed: the rungs are cumulative, so a Dockerfile reaches L2 only beside a lockfile, and without one the compendium is at L0. The green CI run adds assurance rather than capture. That is, it shows the same Dockerfile also builds on a machine that carries none of the author’s undeclared state, which the laptop build alone cannot show. A build that only ever succeeds on the author’s own machine is exactly the ‘works on my machine’ pattern this chapter opened with.
12.3 The analyst’s contribution
Before turning to the machinery, we note what it cannot do. Continuous integration is highly automated, but copying a workflow file into a repository and forgetting it leaves three decisions unmade. They remain the analyst’s, and no tool makes them.
The analyst decides what counts as a correct output, and records the baseline. Verification can compare a regenerated result against a recorded one, but it cannot know which results matter or what tolerance is acceptable. The baseline may be a derived dataset, a coefficient table, or a single summary statistic. The agreement required may be to the last bit or to a stated numerical tolerance. Both are scientific choices the analyst must make and defend.
The analyst decides which gates the compendium’s archetype warrants. A CI workflow can run R CMD check, a render gate, and a dependency-manifest gate, but not every archetype needs all three. An R-package archetype has no report to render; a simulation archetype may run a reduced pipeline under CI and the full one elsewhere. Matching the gates to the archetype is the analyst’s call. The aim is that CI tests what actually needs testing, without imposing a render gate on a project that has nothing to render.
The analyst reads a CI failure as evidence. A red build is CI doing its job: it has found, on a clean machine, a dependency the author’s machine supplied silently. The discipline is to treat the failure as a true report about the compendium’s reproducibility and to fix the captured inputs rather than the check.
NoteThe vocabulary of this chapter
Continuous integration (CI). A hosted service that, on every committed change to a repository, checks out the code onto a fresh virtual machine, rebuilds the project, and runs a defined set of checks, reporting success or failure automatically.
Workflow. The file, written in YAML, that tells the CI service what to do: which events trigger it, what environment to run in, and what commands to execute. On GitHub these files live under .github/workflows/; on GitLab the single file is .gitlab-ci.yml.
Runner. The fresh, ephemeral machine (usually a virtual machine or a container) on which the CI service executes the workflow. It is discarded after the run, so nothing persists between runs except what the workflow explicitly restores.
R CMD check. R’s own quality gate for a package, which among many checks verifies that every dependency declared in DESCRIPTION is installed and findable. A research compendium built as an R package inherits this gate for free, for its package skin only: analysis/ is excluded from the build and never checked.
Render gate. A CI check that re-renders the compendium’s report (an R Markdown or Quarto document) from source and fails the build if the render errors.
Dependency-manifest gate. A CI check that enforces the dependency invariant of Chapter 6. In its strict, R-package form it requires every package the code uses to be declared in DESCRIPTION and pinned in renv.lock. A research compendium’s DESCRIPTION declares only the API exported from R/, while its analysis scripts legitimately draw on packages present in renv.lock but not in DESCRIPTION. For a compendium, therefore, the gate enforces the weaker inclusion that every package the code uses is pinned in renv.lock.
Verification. The act of rebuilding the compendium’s environment from its recipe and re-running the analysis in it. The regenerated computational outputs are then compared with a committed baseline, within a stated tolerance, by means of a baseline test (Chapter 11); passing it is what the ladder calls level L3.
Checksum. A short fingerprint, computed from a file’s or an object’s bytes by a hash function such as SHA-256, with the property that any change to the input, however small, changes the fingerprint. A stored checksum lets a later run detect that an input has been altered.
Forge. The platform that hosts the repository and runs its CI, such as GitHub or GitLab. The forge governs where the code lives and which CI dialect the workflow is written in; it pins nothing about the computational environment.
NoteCheck your understanding: the analyst’s irreplaceable judgments
Name the one of the three analyst judgments that a workflow file, however carefully written, cannot make for itself, and say why.
NoteAnswer
Any of the three qualifies, but the clearest is the first: deciding what counts as a correct output and recording the baseline. A workflow can run a comparison and report its result. However, it has no way to decide which quantity matters, which outputs to compare, or what tolerance distinguishes a real change from floating-point noise. That is a scientific judgment about the analysis, and the CI configuration cannot make it.
12.4 The machine that is not yours
We begin with the reason a second machine is needed at all. An analysis reproduces on the machine where it was written partly because the machine is generous. It supplies, without being asked, a package installed months ago and never recorded, or a system library that some other project needed. It may equally supply a data file sitting in a home directory, or an environment variable set once in a shell profile. Every one of these is undeclared state, and every one of them is invisible precisely because it is present. The ‘works on my machine’ report is true: it describes a machine that happens to carry more than the compendium declares.
Continuous integration removes the generosity. When a change is pushed, the CI service provisions a fresh runner, a virtual machine or container that has just been created and carries nothing but a clean operating system. It checks out exactly what is under version control, no more, and attempts to rebuild the compendium from those committed inputs alone. If the analysis depended on anything the author never declared, the rebuild fails. It fails here, rather than in a reviewer’s hands or a colleague’s re-analysis a year on. The CI machine catches the ‘works on my machine’ failure at its source for a reason that is almost tautological: the CI machine is, by construction, not the author’s. It cannot supply the author’s undeclared state because it does not have it.
Continuous integration does not pin a package version, fix an operating system, or record a session option; those are the work of the lockfile and the container, done before CI runs. What CI adds is the repeated, adversarial test that the capture already performed is sufficient. A captured-but-never-retested compendium suffers a slow decay. In this failure mode a declared environment silently stops rebuilding as its external references drift, and no one notices because no one has tried to rebuild it. CI is the standing test that would notice. Removing CI from a repository lowers no level on the ladder; the compendium is exactly as reconstructable as before. It simply stops being checked, and becomes trusted rather than tested.
NoteCheck your understanding: why the runner is generous nowhere
Explain, in one or two sentences, why a freshly provisioned CI runner cannot supply the undeclared state that makes an analysis ‘work on my machine’, and why this is close to a tautology rather than a clever design feature.
NoteAnswer
The runner cannot supply undeclared state because it has none to supply: it was created moments ago from a clean image and has never run anything else. It is almost a tautology because the property that makes it useful, carrying nothing of the author’s, is simply what ‘fresh’ means. No engineering insight is required to make a runner forget what it never knew.
12.5 What the checks are: the three gates
A CI run on a research compendium executes, in the general case, three gates. We take them in turn, because the division of labor among them is exact.
The first is R CMD check. Because a research compendium is structured as an R package (Chapter 5), it inherits R’s own quality gate at no additional cost. Among its many checks, the one that matters most for reproducibility is early and simple. It verifies that every package the code in R/ uses is declared in DESCRIPTION, and that every declared package is installed and findable. On a fresh runner this is a real test, because the runner has only what the lockfile declared.
Its scope is narrower than it looks. The check runs on the package that R CMD build makes, and a compendium’s .Rbuildignore excludes analysis/. The analysis scripts are therefore never parsed, never run, and never scanned for packages (Chapter 5). R CMD check validates the package skin, R/, the tests, and DESCRIPTION, and nothing else. Within that scope there is a second trap. An undeclared package used in R/ is reported as a WARNING, not an ERROR, so the gate fails on it only if it is told to fail on warnings. A workflow that passes error_on = 'error' to rcmdcheck, as many hand-written workflows do, lets the undeclared dependency through with a green build.
The second is the render gate. Where the compendium carries a report, an R Markdown or Quarto document that regenerates its own figures and tables, the workflow re-renders it from source on the runner. A render that errors, because a chunk references an object that no longer exists or a package that was not restored, fails the build. Of the three gates it is the one that executes the analysis code: a report that sources the scripts under analysis/, or calls the functions they define, runs them on the runner, against the restored library and nothing else. The render gate is the literate document’s validation counterpart: Chapter 9 seals the prose to the code, and the render gate confirms, on a clean machine, that the seal holds. It needs the data on the runner. The cohort this book uses is synthetic and committed for that reason. Where the real data are restricted, CI renders against a synthetic stand-in, and the real render runs inside the environment approved for the data. Committing the data to turn a red render gate green is never the fix.
The third is the dependency-manifest gate, the automated form of the dependency invariant developed in Chapter 6. In its strict form, appropriate to the R-package archetype, the invariant is a double inclusion. The set of packages the code actually uses must be a subset of the set declared in DESCRIPTION, which must in turn be a subset of the set pinned in renv.lock. A research compendium is looser. Its DESCRIPTION declares only the API exported from R/, while its analysis scripts legitimately use packages that appear in renv.lock but never in DESCRIPTION. For a compendium the operative invariant is therefore the single inclusion that every package the code uses is pinned in renv.lock, and the strict DESCRIPTION-inclusion gate does not apply. The gate scans the source for library(), require(), and pkg::fun() calls. It then reads DESCRIPTION and renv.lock, and stops with an error if a used package is not pinned. Because it reads the analysis scripts, it is the gate that covers what R CMD check cannot see. Its core needs no special tool; renv itself can find the packages the code uses. The minimal version below scans only analysis/, the part R CMD check never reads:
A companion tool, zzrenvcheck, does the same scan with more care (tests, vignettes, version constraints, and the stricter DESCRIPTION inclusion for the package archetype). Its error_on_fail argument decides whether a failed check stops the step with a nonzero exit status or only prints a report. The default is !interactive(), so a call from Rscript already stops, but CI should pass error_on_fail = TRUE explicitly so that the gate does not depend on how R was started. It is not on CRAN, so the workflow installs it from its source repository, pinned to a commit. Either form catches the most common drift: a package installed interactively during analysis, used in the code, and never snapshotted, so that the lockfile is silently incomplete.
How do the three gates fit together in practice? A condensed GitHub Actions workflow that runs them on every push and pull request illustrates the shape of the thing. It uses the R community’s r-lib/actions for the setup and the check, at the major versions current when this chapter was revised in November 2026 (actions/checkout@v7, r-lib/actions/*@v2). We show it as a static file. Its YAML was checked for syntax, and the command in each gate was run against a toy compendium. However, the workflow as a whole is executed by the CI service, never by this book.
name: compendiumon:push:branches:[main]pull_request:branches:[main]permissions:contents: readjobs:gates:runs-on: ubuntu-24.04env:GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}steps:-uses: actions/checkout@v7 # Install the R version recorded in renv.lock.-uses: r-lib/actions/setup-r@v2with:r-version: renvuse-public-rspm:true-uses: r-lib/actions/setup-pandoc@v2 # Restore renv.lock into the project library. A # stock renv .Rprofile activates that library in # every later R step run from the repository root.-uses: r-lib/actions/setup-renv@v2 # The gate tools are not analysis dependencies, so # they are not in renv.lock. zzrenvcheck comes from # a personal repository and is pinned to a commit.-name: Install the gate toolsshell: Rscript {0} run: | renv::install(c('rcmdcheck', 'remotes', 'pkgload')) remotes::install_github( 'rgt47/zzrenvcheck', ref = '6f53235668cb4048c1ae793d64b10d6505bf69a2') # Gate 1: R CMD check of the package skin. Any # WARNING, such as an undeclared import, fails.-uses: r-lib/actions/check-r-package@v2with:args:'c("--no-manual")'error-on:'"warning"' # Gate 2: every package the code uses, analysis/ # included, is declared and pinned.-name: Dependency-manifest gateshell: Rscript {0} run: | zzrenvcheck::check_packages(strict = TRUE, error_on_fail = TRUE) # Gate 3: the report re-renders from source.-name: Render gateshell: Rscript {0} run: | rmarkdown::render('analysis/report/report.Rmd', output_format = 'html_document', output_dir = tempdir())
Three choices in the file merit a word. First, error-on is set to "warning" explicitly, although that is also the action’s default. The setting is the whole difference between a gate that catches an undeclared import and one that does not. Second, the render gate renders to HTML into a temporary folder. It tests that every chunk runs, and leaves PDF typesetting, which needs a LaTeX installation, to a separate step if the deliverable is a PDF. Third, nothing in the file is marked continue-on-error; a step that is allowed to fail without failing the job is a report, not a gate.
The house rule of this book is that a gate is not shown to work until it has been seen to fail. Each of the three can be turned red on purpose, in a branch, in a few minutes:
Gate
Deliberate breakage
What the run reports
R CMD check
Call sandwich::vcovHC() in a function in R/ without adding sandwich to DESCRIPTION
A WARNING, ‘:: or ::: import not declared from’, which error-on: '"warning"' turns into a failed step
We ran each breakage against a toy compendium (R 4.6.1, rcmdcheck 1.4.0, zzrenvcheck from its source repository in October 2026), and each command exited with a nonzero status. The first breakage also shows why the setting matters: with error_on = 'error' the same check returned normally, the WARNING printed and ignored. Placing library(sandwich) in analysis/ instead of R/ shows the scope of the first gate. In that case R CMD check passes and the dependency gate goes red; the render gate goes red too if the report runs that script.
12.5.1 Where CI bugs hide: cross-boundary state
The workflow above takes care over where the pinned library is restored and from which directory each R step starts. That care avoids a subtle bug whose cause generalizes well beyond the particular case.
The trouble is that a CI run is not one continuous process. It is a sequence of separate steps, each a fresh shell, and within a step the check itself runs in a further separate subprocess. State does not automatically cross these boundaries. In one documented case, the step that restored the pinned packages installed them correctly. The very next step, running R CMD check, then reported a required package as ‘not available’. The package was on disk; the check simply could not see where. Two boundaries conspired. The restore step adjusted the library path in its own shell, but the next step is a fresh shell and inherited none of that. And even within a step, R CMD check runs in a child subprocess whose library path must be passed to it explicitly. Adjusting the path in the calling session does not propagate to the child. The fix is to launch the check from a session in which the restored library is already on the library path, and to make sure that path travels to the child.
In the workflow above, three pieces combine to do this. setup-renv restores into the project library. Every later R step starts in the repository root, where the project’s .Rprofile activates that library. And rcmdcheck() hands the calling session’s .libPaths() to the check subprocess, which is the default of its libpath argument. A workflow that restores into some other folder must pass that folder to libpath by hand.
The second piece is an assumption, and it is the same cross-boundary bug in waiting. It holds for the stock .Rprofile that renv::init() writes, which sources renv/activate.R in every session. It does not hold for every compendium. The .Rprofile that zzcollab scaffolds activates renv only when ZZCOLLAB_CONTAINER is set to true. On a CI host it skips renv, and later steps see the system library rather than the restored one. Such a compendium must activate the library explicitly on the runner: set that variable in the job’s environment, or begin each R step with source('renv/activate.R').
But why did the bug stay hidden? The workflow had passed for several repositories, because each of them imported only packages that the generic container base image already provided; the restored library was never actually needed. The defect surfaced only when a repository imported a specialized modeling package absent from the base image. The broader literature on why deposited analyses fail to re-run (Trisovic et al., 2022) documents how often re-execution simply fails. The particular fault here is cross-boundary state, the state that must pass from one step, shell, or process to the next and silently does not. It is where CI bugs hide. A dormant defect of this kind needs an adversarial input, a dependency outside the comfortable base image, to bring it to light. A check that passes tells us only that nothing has yet exercised it hard enough to fail.
NoteCheck your understanding: capture versus validation
A colleague proposes to save CI minutes by deleting the continuous-integration workflow from a compendium that already has a committed renv.lock and Dockerfile. They argue that since the lockfile and the container ‘contain the reproducibility’, removing CI cannot lower the compendium’s reproducibility level. Are they correct about the level? Are they correct that nothing is lost?
NoteAnswer
They are correct about the level and wrong about the loss. The reproducibility level is set by what is captured: the lockfile pins the packages (L1) and the container pins the environment (L2). Continuous integration captures nothing, so removing it leaves the compendium at exactly the level it held before, L2. What is lost is not level but assurance. Without CI, no one tests, on a clean machine, that the captured environment still rebuilds. The compendium becomes trusted rather than tested. The silent decay of an external reference (a package pulled from its repository, a base image whose digest drifts) will go unnoticed until someone tries a rebuild by hand.
12.6 Secrets, tokens, and pinned actions
Before going further with verification, we pause on a matter of hygiene. A CI runner holds credentials the author’s own laptop never needed, to comment on a pull request, push a built image, or deploy a rendered report. A leaked credential is a worse incident than a failed reproducibility check. It warrants the same discipline this chapter applies elsewhere: nothing should reach the runner, or leave it, undeclared.
The first rule is never to let a secret reach a log or a commit. GitHub’s guidance on securing Actions workflows warns that sensitive values ‘should never be stored as plaintext in workflow files’ (GitHub, n.d.-b). A workflow instead references a stored secret through the secrets context, ${{ secrets.NAME }}. The runner substitutes the value at execution time, and GitHub attempts to redact it from the log. The redaction is best effort: ‘automatic redaction is not guaranteed’, and the documentation recommends auditing how a workflow handles secrets to confirm one is never explicitly printed (GitHub, n.d.-b). The condensed workflow earlier in this chapter already follows the pattern. Its line GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }} passes a credential into the job’s environment without ever writing its value into the file.
The second rule is least privilege. The default permissions of the token GitHub Actions issues a job come from a repository, organization, or enterprise setting, which may be permissive. A workflow with no permissions block inherits that setting and can run with more access than most jobs need. GitHub’s hardening guidance calls a default of read-only repository contents ‘good security practice’, with individual jobs granted more only where they need it (GitHub, n.d.-b). The chapter’s own example already does this: it declares permissions: contents: read at the top of the workflow.
The third rule has a direct parallel already established in this book. Chapter 7 showed that pinning a base image by tag is not enough. A tag is a mutable pointer that a maintainer can repoint to different bytes with no change to the Dockerfile that names it; pinning by digest is what actually fixes the bytes a build receives. A third-party GitHub Action carries the same hazard. A version tag such as @v7 can be moved to a different commit by its maintainer, or by anyone who compromises the maintainer’s account. The workflow that trusts the tag then runs whatever that commit contains on its next trigger, with no diff in the consuming repository to review. GitHub’s own hardening guidance recommends pinning a third-party action to a full-length commit SHA rather than a tag. It states the rationale plainly: a tag ‘can be moved or deleted’ by anyone with write access to the action’s repository, while forging a given commit SHA would need an attacker to produce a SHA-1 collision (GitHub, n.d.-b):
The trailing comment records which release the SHA was taken from, since forty hex characters say nothing to a human reader. A SHA pin does not move on its own, so it also does not pick up a security fix. Dependabot or Renovate can watch the pinned actions and open a pull request that replaces the SHA and its comment when a new release appears, so that each update is a reviewed commit rather than a silent change.
The chapter’s own workflow examples fall short of their own advice. They should be read as a teaching simplification, not a model to copy unchanged. The actions above (actions/checkout@v7 and r-lib/actions/*@v2) and the GitLab example’s rocker/verse:4.6.1 all pin by tag. The CRAN gate tools are installed at whatever version is current. Only zzrenvcheck, which comes from a personal repository, is pinned to a commit. This book’s Dockerfile explains a recipe by tag before showing the digest-pinned version, and for the same reason: a tag is legible in a worked example in a way forty hex characters are not. A compendium’s actual CI configuration should pin every third-party action to a commit SHA, exactly as its Dockerfile pins its base image by digest. A dependency that can change with no commit to the project’s own history is undeclared state, whether it lives in a package repository, a container registry, or an action’s git history.
NoteCheck your understanding: what a version tag on an action actually promises
A teammate argues that pinning an action with uses: some-org/some-action@v3 is already safe, since the tag is version-controlled and anyone can see which commit v3 points to on GitHub. What is wrong with this argument?
NoteAnswer
Visible is not the same as fixed. A tag is a label the action’s maintainer, or anyone with write access to that repository, can repoint to a different commit at any time. The workflow records only the label it asked for, not which commit it ran last time. Checking what v3 resolves to now answers only ‘what would run if I checked right now’. It does not answer what ran last week, or what will run if the tag moves before the next trigger. A commit SHA fixes the content; a tag, however carefully maintained, is a pointer that can be moved. This is exactly the distinction this book already draws between a Docker tag and a Docker digest.
12.7 Two dialects: GitHub Actions and GitLab CI
We turn now from what the checks are to where they run. Continuous integration is a general idea with several implementations, and a health researcher should know at least two, because the choice is often not theirs to make. GitHub Actions and GitLab CI are two dialects of the same underlying activity, and the difference between them is one of syntax and hosting, not of principle.
The forge, the platform hosting the repository, matters for research in a way it does not for general software development. Research compendia are disproportionately authored on institutional, often self-hosted, GitLab. Many universities and laboratories run their own GitLab servers and cannot, or will not, push a repository to github.com. This is particularly so in Europe and wherever data-governance rules discourage commercial cloud hosting. Consider a clinical or registry study whose data-handling agreement forbids external hosting. For it, the ability to run the same reproducibility checks on institutional GitLab is a precondition. A framework that assumed GitHub would simply be unavailable to a large part of its intended audience.
The two dialects differ structurally. GitHub Actions uses one or more workflow files under .github/workflows/, each organized into jobs and, within a job, a sequence of named steps. GitLab CI uses a single .gitlab-ci.yml organized into stages, each stage running a script. The same three gates, expressed in the GitLab dialect, look like this. We again show a static file that GitLab executes and this book does not:
stages:- gates- verifygates:stage: gates # The tag must name the R version recorded in renv.lock.image: rocker/verse:4.6.1script:- Rscript -e 'renv::restore(prompt = FALSE)'- Rscript -e 'renv::install(c("rcmdcheck", "remotes","pkgload"))'- Rscript -e 'remotes::install_github("rgt47/zzrenvcheck", ref = "6f53235668cb4048c1ae793d64b10d6505bf69a2")' # Gate 1: R CMD check of the package skin- Rscript -e 'rcmdcheck::rcmdcheck(args = "--no-manual", error_on = "warning")' # Gate 2: the dependency manifest, analysis/ included- Rscript -e 'zzrenvcheck::check_packages(strict = TRUE, error_on_fail = TRUE)' # Gate 3: the report re-renders from source- Rscript -e 'rmarkdown::render("analysis/report/report.Rmd", output_format = "html_document", output_dir = tempdir())'verify:stage: verify # A shell-executor runner with Docker installed; the # tag is whatever the institution assigns to it.tags:[docker-shell]script:- docker build -t compendium:ci .- docker run --rm -e RENV_CONFIG_AUTOLOADER_ENABLED=FALSE -v "$PWD":/work -w /work compendium:ci Rscript ci/run-tests.R
The two files say the same thing in different words. Each runs on the R version the lockfile records, restores the pinned library into the project library, and installs the gate tools beside it. Each then runs the three gates from the repository root, so that every R step, and the check subprocess it launches, sees the restored library; the .Rprofile assumption of ‘Where CI bugs hide’ applies to both. The image tag above names R 4.6.1, and this book’s own lockfile records R 4.6.1; a compendium on another release names its own. The rocker/verse image is used on GitLab because it ships pandoc, which the render gate needs; on GitHub the setup-pandoc action supplies it. Each script line is a separate shell command, and GitLab fails the job at the first one that exits with a nonzero status. Every gate here can therefore fail the pipeline.
The second job, verify, rebuilds the project image and runs the baseline tests inside it; it is the subject of the next section. On GitLab it needs a runner that can build images, which is an institutional decision. The common alternative, Docker-in-Docker, requires a privileged runner that many institutions do not allow.
On the plus side, maintaining a repository on whichever forge an institution mandates is then a matter of translating the dialect, and the checks carry over. On the minus side, a workflow written twice, once per dialect, is a workflow maintained twice. Every future change must be made in two incompatible formats, a standing divergence hazard. The resolution a mature framework adopts is to single-source the logic in a neutral place, the project’s own Makefile. It holds one target per gate (say check, deps, and render), each carrying the commands shown above. Each forge’s CI file then shrinks to a thin shim that checks out the code and calls make. The dialect differs only in its outermost ten lines, and the checking behavior is defined once.
A further design point is that a workflow need not be configured by hand: it can adapt itself to what the repository contains. A feature is treated as present exactly when its defining file exists. If renv.lock is present, the workflow restores the pinned library; otherwise it installs from DESCRIPTION. If a Dockerfile or a Nix flake.nix is present, the workflow builds and runs inside that environment; otherwise it runs on the host runner. Such a workflow survives a project’s toggling of renv or of its container backend with no edit, and one file can serve every archetype and every capture level. The workflows in this chapter are not written that way. They assume a compendium with both a lockfile and a Dockerfile, so that every step is visible. A presence-driven workflow wraps each step in a condition, in GitHub’s dialect if: hashFiles('Dockerfile') != ''. That convenience has a cost to weigh. A condition is also a way for a gate to disappear without notice, since a misnamed file switches the step off and the run stays green. A workflow that adapts should therefore print which gates it ran and which it skipped.
NoteCheck your understanding: the presence-driven trap
A presence-driven workflow skips the render gate when if: hashFiles('report.Rmd') != '' evaluates to false. Describe a concrete way this condition can silently disable the gate without anyone intending it to, and what single habit of the workflow catches the problem.
NoteAnswer
A rename, for instance moving report.Rmd to analysis-report.Rmd during a reorganization, or a typo in the condition itself, makes hashFiles() return an empty string. The condition then evaluates to false and the step is skipped. The job still reports success, because a skipped step is not a failed one. Nothing in the build output distinguishes ‘rendered successfully’ from ‘did not try to render’. The habit that catches it is having the workflow print, for every conditional step, whether it ran or was skipped. A silent skip is then visible in the log, even though it does not turn the build red.
12.8 Verification and level L3
The first three gates confirm that the compendium builds, checks, and renders on a clean machine. They do not confirm that it produces the same numbers. That is the work of verification, the L2-to-L3 transition and the top of the ladder, which confirms what has been captured.
L3, as Chapter 3 defines it, requires that recorded outputs be shown to regenerate within a stated tolerance from a clean rebuild. Each part of that sentence names a mechanism. The recorded outputs and the tolerance live in a baseline test (Chapter 11): a committed file of key estimates is recorded once, and a test file named test-baseline-*.R recomputes them and compares with expect_equal(..., tolerance = ...). The clean rebuild is the project’s own image, built from its Dockerfile on a machine that has never seen it. Verification is running the first inside the second, and it fails if either fails. It fails if the image does not rebuild, if any test fails, or if no baseline test runs at all. A test suite without a baseline test can pass in a rebuilt image and show that the environment is sound, which is L2 confirmed. However, it has compared nothing with a recorded result, so it is not L3.
In continuous integration this is a second job in the same workflow, run after the gates pass:
verify:needs: gatesruns-on: ubuntu-24.04steps:-uses: actions/checkout@v7 # L2: the project's own Dockerfile must rebuild.-name: Rebuild the project imagerun: docker build -t compendium:ci . # L3: the test suite, with at least one baseline # test, must pass inside the rebuilt image.-name: Run the tests in the image run: > docker run --rm -e RENV_CONFIG_AUTOLOADER_ENABLED=FALSE -v "$PWD":/work -w /work compendium:ci Rscript ci/run-tests.R
The test step runs a short script from the repository. The obvious one-line call, tinytest::run_test_dir('inst/tinytest'), returns its results and exits with status 0 even when an expectation fails; we confirmed this with a deliberately failing test. The script turns failures into a nonzero exit. It also refuses to pass if no baseline test produced a result, which is the case when the baseline file is missing or when its guard found no data and exited early:
# ci/run-tests.Rpkgload::load_all('.', quiet =TRUE)res<-tinytest::run_test_dir('inst/tinytest')print(res)ran<-unique(as.data.frame(res)$file)if(!any(grepl('^test-baseline', ran))){stop('No baseline test ran, so this is not an L3 check.')}if(!tinytest::all_pass(res))quit(status =1)
Two assumptions are built in and should be checked against the image in use. The image must contain tinytest and pkgload, which means they must be in the lockfile it restores. And because RENV_CONFIG_AUTOLOADER_ENABLED=FALSE stops the mounted project from activating its own library, the image must already have the pinned packages on R’s default path, for example in the site library. The house rule applies here as well. Changing one value in the committed baseline file by more than the tolerance turns the job red at the baseline expectation. Deleting or renaming the test-baseline-*.R file turns it red at the ‘no baseline test ran’ check. Breaking a RUN line in the Dockerfile turns it red at the build. Against the toy compendium, outside a container, we ran the first two. We also ran a baseline file whose guard found no data and exited early. Each run of the script exited with status 1, and the unbroken run exited with 0. The image build and the job as a whole were not run for this chapter.
In the framework this book uses, the same composition is the command zzc verify, which runs in two tiers. A default coherence tier requires no rebuild. It checks that the captured artifacts agree with each other and that the R version named in the lockfile matches the one recorded when the Dockerfile was generated. It also checks that the backend is unambiguous, and that the raw data still matches its recorded checksums. A --fullreproduction tier rebuilds the image and runs the test suite inside it. It stamps L3 into the project’s state record only when three things hold: the image rebuilt, the tests passed, and at least one inst/tinytest/test-baseline*.R file exists and is named in the test log. The log names a file even when its guard exits early with no tests, so ci/run-tests.R, which counts results, is the stricter check. Without a baseline test it reports the project as rebuilt and tested, at L2, with no baseline comparison, and does not stamp L3. The stamp is removed when zzcollab regenerates the Dockerfile or when a later --full run falls short of L3. A change made outside zzcollab, such as renv::snapshot() or an edited script, leaves it in place, so a stamp older than the last such change is stale. In regulated work, the stamp is not a validation record: it carries no signer, no stated meaning of a signature, and no quality review, which a controlled record of validation status requires.
But what, precisely, should be compared? Verification compares computational outputs, such as the row count of a derived dataset, a fitted coefficient, or a bootstrap bound, and not the rendered PDF or HTML. The reason is that a rendered document is not a deterministic function of the analysis. As Chapter 9 discussed, a PDF embeds the time of its creation and the identifiers of the fonts the rendering machine happened to have. Two renders of an unchanged analysis, minutes apart, produce PDFs whose bytes differ, even though every number in them is identical. Comparing PDFs would therefore report a difference on every render and catch nothing real. Comparing the estimates themselves, within a tolerance, reports a difference exactly when a number changed by more than floating-point noise, which is what we want to detect. A hash is the right instrument for a file whose bytes should not change, such as a raw data extract (the next section). It is the wrong one for an estimate, whose last binary digit can move under a different BLAS without the analysis changing. There exist techniques to normalize a document’s embedded timestamps, by setting a fixed SOURCE_DATE_EPOCH, and to fix its build path. With enough care a byte-identical PDF is attainable. But the more general practice is to verify the computational outputs directly and leave the document to the render gate.
ImportantThe honest level
The three gates do not reach L3; only the verify job can, by running a baseline test in a rebuilt image. This chapter ran the gate logic and the test script, but not the image build or the job as a whole. A verified stamp older than the last change to the lockfile, the Dockerfile, or the code is stale, not an L3 claim. The gates alone say nothing about whether the raw data changed; the checksum manifest of the next section does. In regulated work, an L3 stamp is also not a validation record (Chapter 11). Nor is a hosted run a lasting record: GitHub deletes workflow logs and artifacts after 90 days by default, a period an organization can change within limits (GitHub, n.d.-a). A log or artifact that supports a claim must therefore be downloaded and kept with the records it supports, for as long as they are kept.
NoteCheck your understanding: trusting a stamp
A compendium was stamped verified three months ago. Last week its renv.lock was updated to pin a new package version, and CI has not run since. Is the verified stamp still an honest L3 claim? Why or why not?
NoteAnswer
No. The stamp certifies that a specific, now-superseded environment reproduced the baseline, not that the environment as it currently stands does. Because the lockfile changed after the last successful verification run, the claim is stale: the thing the stamp describes no longer exists. The honest statement is that the compendium was L3 as of the old lockfile and is unverified now. This is why a change to the environment should drop the stamp automatically.
12.9 Data integrity: the checksum manifest
We now turn to an input that sits outside everything the lockfile and the container pin: the raw data itself. A container fixes the software that reads the data, and a lockfile fixes the packages. However, neither notices if the data file on disk is silently altered between the analysis and its re-execution (a row edited, a value corrected, a column recoded). An altered input produces a changed result with no error and no warning.
The guard is a checksum manifest, a small file, conventionally data-manifest.sha256, that records the SHA-256 checksum of each raw data file. Verification reads the manifest, recomputes the checksum of each file as it now stands, and reports a mismatch. Because any change to a file’s bytes changes its checksum, a silently altered input is caught the next time verification runs.
To illustrate, consider a small synthetic example. We write a tiny data file to disk, record its checksum, write the same values again from a different kind of R object, write a third copy with a single value altered, and hash each file. The checksum is computed from the file’s bytes, not from the R object in memory. The point to observe is that a one-cell change, a shift of a single outcome from 11.1 to 11.2, produces a completely different digest, so the alteration cannot pass unnoticed.
Table 12.1: A single altered value changes the digest of a data file. The three CSV files are hashed by their bytes on disk. The second holds the same values as the first, written from a plain data frame rather than a tibble, and its digest matches. The third differs in exactly one cell, and its SHA-256 digest differs from the first character, so a checksum manifest detects the change.
File
SHA-256 (first 16 chars)
Matches baseline
cohort.csv (baseline)
cc03e329ae1b1114
yes
same values, from a data.frame
cc03e329ae1b1114
yes
one cell altered
632857fa64a59ad4
no
The altered file’s digest differs from the first character, although the files differ in a single cell. That sensitivity to any change is what makes a checksum a reliable integrity guard. Suppose a verification run recomputes these digests and finds the third where it expected the first. It has then caught an altered input, and has done so without needing to know what the correct value was, only that it has changed since the manifest was recorded. The second row shows why the file, and not the object, is what gets hashed. Hashing the tibble and the data frame themselves with digest() would serialize each object, class attributes included, and give two different digests for identical values: here digest(cohort) == digest(as.data.frame(cohort)) is FALSE. The bytes on disk carry only the values.
A manifest line pairs a digest with a path, in the format shasum -a 256 writes and shasum -a 256 -c checks. For this book’s raw extract it reads:
A student proposes to verify their manuscript compendium by recording the SHA-256 checksum of the rendered report.pdf and re-checking it on every rebuild. They reason that if the PDF is byte-identical, the whole analysis must have reproduced. Identify the flaw, and say what they should hash instead.
NoteAnswer
The flaw is that a rendered PDF is not a deterministic function of the analysis. It embeds the time of rendering and the identifiers of the fonts on the rendering machine, so two renders of an unchanged analysis produce PDFs with different bytes and therefore different checksums. The student’s check would report a failure on every single rebuild, catching nothing real and quickly training them to ignore it. They should instead check the computational outputs that feed the report: the estimates and the key counts should be compared with a committed baseline within a stated tolerance by a baseline test. Where a derived file’s bytes should not change, its checksum should be checked as well. The render gate, separately, confirms that the document still builds from those outputs. Byte-identical PDFs are attainable with timestamp normalization and font freezing, but that is a fragile route to a result the baseline test gives directly.
12.10 Proving the framework itself: the capture test matrix
One rung below the compendium’s own verification sits an easily overlooked question: how does the framework know that a project configured for, say, L1 actually reaches L1? A claim that a given combination of toggles produces a given level is itself a claim that can be tested, and a responsible framework tests it rather than asserting it.
The instrument is a capture test matrix. It scaffolds a throwaway compendium for each combination of the capture axes: the package backend present or absent, and the container environment present or absent. For each, it asserts that the result passes R CMD check, is a loadable workspace, and sits at its stated level. The matrix has the following shape. Its axes are the capture axes; CI is the harness that runs the matrix, not a dimension of it.
Backend (renv)
Environment (Docker)
Level
Assertion
off
off
L0
check passes; valid workspace
on
off
L1
check passes; renv activates on host
off
on
L0
check passes; image installs from DESCRIPTION, so the environment is fixed but the packages are unpinned
on
on
L2
check passes; image restores from lockfile
The third row sits at L0 although it builds an image, because the rungs are cumulative. L2 presupposes the lockfile of L1, and an image that installs from DESCRIPTION pins no package version.
Two points about the matrix deserve emphasis. First, Nix, the alternative backend of Chapter 8, is verified as a substitution. It replaces the renv-plus-Docker combination and must reach L2 on its own, so it is checked as a peer of that combination rather than added as a fourth column. Second, the configuration that renders a report with renv on but Docker off is, in practice, the most error-prone, and it is exercised explicitly. Because container builds under emulation are expensive, the matrix is tiered. The host-only rows run on every pull request, and the rows that build an image run on merge and on a nightly schedule.
NoteCheck your understanding: testing the claim, not just the tool
Why does a responsible framework test the claim ‘this combination of toggles reaches level L1’ rather than simply asserting it in documentation?
NoteAnswer
Because the claim is itself a factual statement about behavior, not a design intention. Design intentions drift from implementation the same way an undeclared dependency drifts from a lockfile. That is, a later change to the scaffolding code can silently break the L1 configuration while the documentation still describes the old behavior. Asserting the claim in prose is exactly the kind of trust-without-test this whole chapter argues against. The capture test matrix applies the chapter’s own standard to the framework that makes the claim.
12.11 How continuous integration adapts to the archetype
Continuous integration adapts its gates to the archetype; Table 5.2 in Chapter 5 is the reference. Of the three gates of this chapter, R CMD check and the dependency-manifest gate are universal, run by every archetype. The latter, however, runs in an archetype-conditional form. It is the strict DESCRIPTION-inclusion check for an R package, and the weaker \(\mathrm{Code} \subseteq \mathrm{renv.lock}\) inclusion for a compendium, as the section on the three gates established. The render gate is the conditional one. It is present wherever a report exists, and absent from an R-package archetype, which has none.
The case most worth stating here is the simulation archetype, which raises a distinct concern of cost. A full run can take hours or days. The idiomatic arrangement therefore runs a reduced pipeline under CI: a handful of iterations with a fixed seed, enough to confirm the pipeline executes and the outputs have the right shape. The full pipeline runs elsewhere, on a cluster or a scheduled job, and is verified against a baseline separately. The seeding discipline of Chapter 11 is what makes the reduced run meaningful. A fixed seed makes even a short run reproducible, so CI checks reproducibility rather than merely execution. The presence-driven workflow discovers the archetype from what the repository contains. The analyst’s contribution is therefore to ensure the archetype and its gates are matched.
NoteCheck your understanding: when a package does need the render gate
An R-package archetype is said to have no report, so no render gate. Under what condition does the render gate apply to a package archetype after all, and what does it render in that case?
NoteAnswer
The render gate applies whenever the package ships a vignette, which is itself a rendered document. The gate re-renders vignettes/intro.Rmd (or whichever vignette files exist) on every push. It confirms that the vignette’s code still runs against the restored library, exactly the check a manuscript’s report receives. Only a package with neither a report nor a vignette has nothing for the render gate to do.
12.12 Independent external verification
The checking we have described so far is the author’s own, run on the author’s forge against the author’s baseline. It is necessary and it is not sufficient. An author who has misunderstood their own reproducibility can build a CI pipeline that faithfully confirms the misunderstanding. The complement is verification from outside the project entirely, by someone with no stake in its passing.
The most developed such scheme is CODECHECK, described by Nust and Eglen (Nüst & Eglen, 2021). In it an independent codechecker re-executes an analysis during peer review, on their own machine, and issues a certificate recording what was run and what was reproduced. The value of the exercise lies exactly in its independence. The codechecker carries none of the author’s undeclared state; this is the same property that makes a CI runner effective, here held by a person instead of a virtual machine. In a parallel development, major computing venues have formalized artifact review and badging. Under it, a submission’s computational artifacts are evaluated and, if they reproduce, marked with a badge on the published paper (Association for Computing Machinery, 2020). Stodden and colleagues have documented the persistence of the reproducibility problem. They have also documented the effectiveness, and limits, of journal policies that attempt to compel deposition and checking (Stodden et al., 2018). The ten rules, for their part, ask for automated steps and public access to scripts, runs, and results, which is what makes independent re-execution possible (Sandve et al., 2013).
These schemes validate and capture nothing: they add no lockfile and no container, only a second, independent execution of the analysis. Continuous integration is a machine that is not yours, and independent verification is a person who is not you. Both catch the reproducibility failures that the author’s own generous machine had hidden, at the last responsible moment before a result enters the scientific record.
NoteCheck your understanding: what independence adds beyond CI
An author’s CI pipeline passes every gate, including verification against a baseline. Why might a CODECHECK reviewer still find something CI did not, given that both are described as ‘a machine, or a person, that is not yours’?
NoteAnswer
CI is configured by the author, so it tests what the author thought to test, against a baseline the author recorded, in an environment the author specified. If the author misunderstood what the analysis was supposed to produce, or recorded a baseline after a bug had already entered it, CI will faithfully and indefinitely confirm the misunderstanding. A codechecker reads the paper and forms an independent judgment of what the analysis should produce. The codechecker then checks the artifacts against that judgment rather than against the author’s own configuration, which is a check CI structurally cannot perform on itself.
12.13 Worked example
To make the machinery concrete, consider a plausible sequence of events in a cohort-study compendium, of the kind that recurs in practice.
The analyst is refining a model and, mid-session, reaches for a package that computes robust standard errors, sandwich. They install it interactively with install.packages(), and add library(sandwich) to analysis/scripts/02_model.R, which the report sources. They obtain the result they wanted, and commit. On their own machine everything runs: the package is installed, the script loads it, the report renders, the numbers appear. The compendium looks, to its author, finished.
The push triggers CI, and the fresh runner tells a different story, though not at the gate a reader might expect. The runner checks out the committed files and restores the pinned library from renv.lock. This lockfile was never updated, because the interactive install was never snapshotted. Gate 1, R CMD check, passes. It is not at fault: the script lives in analysis/, which the build excludes, so the check never reads it. Gate 2, the dependency-manifest gate, reads analysis/ and goes red. It names sandwich as a package the code uses and the lockfile does not pin. Had that gate been absent, gate 3 would have failed the same run from the other direction. The report sources the script, the script calls library(sandwich), and the restored library has no such package, so the render stops at that chunk. Either way, CI has found on a clean machine precisely the undeclared state the author’s machine supplied silently. Had the workflow run only R CMD check, as many compendium workflows do, the build would have been green.
The fix is not to weaken the check but to repair the capture. The analyst records the new package in the lockfile by snapshotting the environment, commits the updated renv.lock, and pushes again. This time the runner restores a library that contains the package. The dependency gate passes, the report renders, and the gates go green. The verify job then rebuilds the image from the new lockfile and runs the baseline test. This is where the analyst learns whether the change moved a recorded result, such as a confidence limit. If it did, the baseline is re-recorded deliberately, in a commit that says why, not loosened until it passes.
NoteCheck your understanding: which gate, and why
In the worked example, R CMD check passes even though sandwich is in fact unpinned. Why does this gate miss the defect that the dependency-manifest gate catches, and is that a flaw in R CMD check?
NoteAnswer
It is not a flaw; it is a scope limit stated earlier in the chapter. R CMD check runs on the package R CMD build produces, and .Rbuildignore excludes analysis/, so the check never parses, runs, or scans the script that calls library(sandwich). The gate is doing what it is specified to do, checking the package skin. The defect lives one directory outside that skin, which is why a compendium needs a second gate that does read analysis/.
NoteCheck your understanding: complete the workflow file
The worked example narrates a run rather than printing a workflow file, so this check uses the chapter’s main GitHub Actions example, the condensed compendium workflow of the three gates. It is reprinted verbatim except for four placeholders, two of which stand for a short block rather than a single line. Write each one. For (2), write the line as Section 12.6 says a compendium should pin it, not as the condensed file prints it.
name: compendium# (1) ____: the trigger, every push to main and every# pull request into mainpermissions:contents: readjobs:gates:runs-on: ubuntu-24.04env:GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}steps: # (2) ____: check out the commit, pinned by SHA # Install the R version recorded in renv.lock.-uses: r-lib/actions/setup-r@v2with:r-version: renvuse-public-rspm:true-uses: r-lib/actions/setup-pandoc@v2 # Restore renv.lock into the project library. A # stock renv .Rprofile activates that library in # every later R step run from the repository root. # (3) ____: restore the lockfile # The gate tools are not analysis dependencies, so # they are not in renv.lock. zzrenvcheck comes from # a personal repository and is pinned to a commit.-name: Install the gate toolsshell: Rscript {0} run: | renv::install(c('rcmdcheck', 'remotes', 'pkgload')) remotes::install_github( 'rgt47/zzrenvcheck', ref = '6f53235668cb4048c1ae793d64b10d6505bf69a2') # Gate 1: R CMD check of the package skin. Any # WARNING, such as an undeclared import, fails.-uses: r-lib/actions/check-r-package@v2with:args:'c("--no-manual")'error-on:'"warning"' # Gate 2: every package the code uses, analysis/ # included, is declared and pinned.-name: Dependency-manifest gateshell: Rscript {0} run: | # (4) ____: the call that fails this step when # a package used in analysis/ is not pinned # Gate 3: the report re-renders from source.-name: Render gateshell: Rscript {0} run: | rmarkdown::render('analysis/report/report.Rmd', output_format = 'html_document', output_dir = tempdir())
NoteAnswer
The four completed parts of the condensed compendium workflow are these. Placeholder (1) is the trigger block:
Without pull_request a change is tested only after it reaches main; without push a commit made directly to main is never tested. Placeholder (2) is the checkout:
The condensed file prints actions/checkout@v7, a teaching simplification. A tag can be moved to another commit, whereas a full commit SHA fixes the code that runs, as a digest fixes a base image. Placeholder (3) is - uses: r-lib/actions/setup-renv@v2. A step that ran install.packages() instead would install whatever versions are current, so the gates would test a library the lockfile does not describe. Placeholder (4) is the dependency-manifest call:
This is the gate that went red on sandwich in this chapter’s worked example, because it reads analysis/. Passing error_on_fail = TRUE explicitly keeps the failure from depending on how R was started; with FALSE the step would print the missing package and still pass.
12.14 Debugging a failed reproduction
The worked example above is the easy case: CI catches the defect the same day, on the same commit that introduced it. The harder case arrives later. A compendium that rendered cleanly in March fails in October, when a reviewer clones it, a collaborator resumes it, or the author themselves reruns it for a revision. Nothing in this section is generic R debugging, which is out of scope for this book. It is a diagnostic sequence specific to one question: it worked before, and it does not now, so what changed?
The first move is to localize the failure in time, not in code. Suppose the history between the last known-good state and the present failure holds more than a handful of commits. Reading each one is then slower than letting git find the boundary. git bisect does this by binary search. Told one commit that is good and one that is bad, it checks out the midpoint and asks for a verdict, halving the remaining range at each step.
git bisect start <bad-commit><good-commit>git bisect run Rscript -e'rmarkdown::render("report.Rmd")'git bisect reset
git bisect run automates the verdict. It checks out each candidate commit and runs the given command, taking a nonzero exit status as ‘bad’ and zero as ‘good’. It continues until a single first-bad commit remains. We verified the mechanism on a disposable four-commit toy repository, under git 2.55.0. The first commit established a check that passed. The second, an unrelated-looking refactor, silently changed a value the check depended on, and two further commits added only comments. We marked the fourth commit bad and the first good with git bisect start, then ran git bisect run Rscript check.R. It tested the midpoint, found it still bad, and tested the commit below that. It then named the second commit as the first bad one, in two bisection steps rather than three linear checks. The command handed to bisect run need not be a toy. It can be the render gate or a baseline test from Chapter 11, which is exactly what lets the search run unattended across dozens of commits.
Once the boundary commit is known, the next question is whether the commit changed code, or whether something outside the commit changed under it, because git bisect can only blame a commit if the environment at each checkout is otherwise identical. Consider a pipeline that reruns the same pinned commit weeks apart and gets a different answer. It did not fail because of anything in the diff, and the fix is to compare environments, not history. sessionInfo() (Chapter 6 and the glossary) is the cheapest comparison. It is run in the working environment and the failing one, and the two are diffed by eye for an R version, platform, or package version that moved. renv::status() goes further, comparing the project library with the lockfile and reporting packages that are missing or installed at a different version. It is the first thing to run when the suspicion is drift in the restored library rather than in the analysis code. When the suspicion lands on the lockfile itself, git diff <good-sha> <bad-sha> -- renv.lock shows exactly which package or version entries changed between the two commits. This narrows ‘something in the environment changed’ to the one or two packages responsible.
If the analysis is a targets pipeline (Chapter 10), the diagnostic moves from history into the dependency graph. tar_outdated() lists which targets the pipeline considers stale and therefore due to rerun. tar_visnetwork() draws the same information as a graph. The analyst can then see where in the chain of targets the invalidation begins, rather than at which commit it was introduced. This is the pipeline-native counterpart to bisecting, since targets already tracks what depends on what. The question ‘what broke, and what does it affect’ can therefore be answered from the dependency graph instead of from commit history. This holds provided the pipeline’s metadata store, _targets/, survived the problem. On a fresh CI runner it is absent unless it was committed or the workflow restores it, and a pipeline run there without it rebuilds every target.
The last diagnostic is reading a CI log on its own terms. This chapter has already named the three gates and the discipline of reproducing a failure exactly as CI ran it, not as is convenient to invoke by hand (‘Where CI bugs hide’, above). Applied to a failed reproduction, that discipline has three parts. First, identify which of the three gates reported red: R CMD check, the dependency manifest, or the render. Second, read what it printed, not just that it failed, since the WARNING text or the stop() message names the missing package or the broken chunk directly. Third, reproduce that exact step locally, from a session with the same restored library CI used. This is preferable to re-running the whole render and hoping the same failure reappears in a more informative form.
None of these five moves is a substitute for the others, and the order matters more than any single step. Bisection is cheap to run and narrows the search space before anything else is attempted. Session-info and lockfile comparisons cost nothing once the boundary commit is known. They settle, quickly, whether the defect is in the diff or in the environment around it. targets diagnostics apply only when a pipeline is involved. Reading the CI log carefully is usually the last step, because by the time it is needed the first four moves have already said which gate to expect it from.
NoteCheck your understanding: bisecting an environment problem
A pipeline’s render step fails on a Tuesday rerun of a commit that rendered cleanly the previous Friday. The commit has not changed. Why will git bisect be useless here, and what should be compared instead?
NoteAnswer
git bisect localizes a failure to a commit by checking out different revisions and testing each one, which only works when the thing that changed is in the history it walks. Here the commit is fixed and identical on both days, so bisection has nothing to vary and nothing to find; every checkout it tries is the same code. The defect lies outside the commit, most likely in the restored environment: an unpinned dependency that resolved to a different version between Friday and Tuesday, or a base image whose tag moved. The right comparison is sessionInfo() or renv::status() run on both days. If the lockfile itself is suspect, it is a diff of renv.lock against the copy captured on Friday. It is not a bisection across commits that never changed.
12.15 Demonstrating a gate on the cohort compendium
The logic that runs on a CI runner also runs, in miniature, on this book’s own cohort compendium, without any CI infrastructure at all. Below, a short script mirrors the dependency-manifest gate of the earlier section. It finds every package R/cohort.R actually uses, compares that list against what this compendium currently declares as pinned, and reports whether the gate passes.
used<-unique(renv::dependencies('../../R/cohort.R')$Package)#> Finding R package dependencies ... Done!locked<-names(renv::lockfile_read('../../renv.lock')$Packages)base_pkgs<-rownames(installed.packages(priority ='base'))gate_check<-function(missing){if(length(missing)>0){stop('Used but not pinned in renv.lock: ', toString(missing))}'gate passed'}missing_now<-setdiff(used, c(locked, base_pkgs))cat('Packages R/cohort.R uses:', toString(used), '\n')#> Packages R/cohort.R uses: parallel, stats, survival, tibble, utils, withrcat('Missing from the pinned set:',if(length(missing_now))toString(missing_now)else'(none)', '\n')#> Missing from the pinned set: (none)cat('Result:', gate_check(missing_now), '\n')#> Result: gate passed
The gate passes. R/cohort.R uses parallel, stats, survival, tibble, utils, and withr, and each is either pinned in renv.lock or part of base R. The house rule of this book applies here as well: a gate shown only to pass has not been shown to work. We therefore run the same script against a used list with the chapter’s own running example of an unpinned dependency added. It shows what a real CI log would print the moment someone adds library(sandwich) to a cohort script without snapshotting it:
used_broken<-c(used, 'sandwich')missing_broken<-setdiff(used_broken, c(locked, base_pkgs))result<-tryCatch(gate_check(missing_broken), error =function(e)conditionMessage(e))cat('Missing from the pinned set:', toString(missing_broken), '\n')#> Missing from the pinned set: sandwichcat('Result:', result, '\n')#> Result: Used but not pinned in renv.lock: sandwich
The injected package is named, and the step would stop with a nonzero exit status in real CI. The failure points at the undeclared dependency, which is what the dependency-manifest gate is for. Restoring used to its real value (dropping the injected sandwich) returns the gate to green, as reverting the offending commit would on a real runner.
NoteCheck your understanding: what this demonstration does and does not show
This section ran the dependency-gate logic locally, inside the rendered report, rather than on an actual CI runner. What does that leave unverified about the compendium’s real CI setup, if one exists?
NoteAnswer
Running the logic locally confirms the gate’s logic is correct: it finds the right packages and reports the right ones as missing. However, it says nothing about whether a real workflow file invokes that logic correctly, or restores the right library before running it. Nor does it show that the job actually stops on failure rather than printing a warning and continuing. Those are the cross-boundary and error_on pitfalls the chapter described earlier, and only an actual CI run, on an actual fresh runner, can confirm them.
12.16 Collaborating with an LLM
A large language model is a capable assistant for CI work, which is heavy on boilerplate YAML. It is also a hazardous one, because CI failures are subtle and a plausible-looking workflow can still be wrong. We give three prompt patterns, each with what to watch for and what the analyst must independently verify.
ImportantRestricted data
One rule comes first (habit 3). Never paste or upload protected health information, identifiable records, or data held under a data use agreement, nor outputs derived from them, into a model or a coding agent unless three conditions all hold. The institution has approved that endpoint for that class of data. The IRB protocol and the consent permit the processing. The data use agreement, or the provider’s terms, permits it. If any of the three is unclear, the answer is no.
The third condition is the one most often missed. Institutional approval of a tool does not override the terms under which a data provider released the data. NIH, for example, treats sharing controlled-access data through prompts with public generative AI tools, which it glosses as third-party tools, as a breach of the data use certification. It treats a model trained on such data as a data derivative, to be shared only with other approved users and not kept after the project closes (National Institutes of Health, 2025).
Describe the structure instead: the column names and types, or a small synthetic example generated from the data dictionary, never edited from real rows. A coding agent sees whatever files are in its working tree, so run it in a checkout or container where the real data is not mounted.
An agent also acts, so confine it to a branch and keep merges, pushes, and deposits for yourself (Section 2.10.1).
CI touches data in ways that are easy to miss. A CI log that prints rows of a failing render or test is a derived output in the sense of the rule above, so a log excerpt is pasted only after those rows are removed. For CI work the structure to describe is the workflow file and the error message.
Prompt. ‘Write a GitHub Actions workflow that runs R CMD check on my compendium inside the rocker/tidyverse container and restores packages from renv.lock.’
Watch for. A workflow that restores the library in one step and runs the check in another without ensuring the restored library reaches the check subprocess. As we saw, adjusting the library path in one step does not carry to the next, and R CMD check runs in a child process that sees only the library path it is handed. Check for the naive two-step form, which passes only when every dependency happens to be in the base image. Watch also for error_on = 'error', a dependency step that reports without failing, and continue-on-error: true. Each of these leaves a gate that cannot go red. Remember, too, that R CMD check alone never looks at analysis/.
Verification. Do not trust the workflow because it is syntactically valid. Confirm it fails correctly by introducing a deliberate gap: import a specialized package absent from the base image, do not restore it, and check that the build goes red. A CI check that has never been seen to fail has not been shown to work. Seeing means reading the run log yourself: a coding agent’s report that ‘CI is green’ is a claim about the run, not an observation of it.
An agent asked to make a failing build pass can reach green by the patterns this triple lists, adding continue-on-error: true or loosening error_on, and the change arrives as an ordinary commit. Treat .github/workflows/ as a protected path and read every line of any diff that touches it before it merges. CI is also where embedded instructions matter most. An agent that reads a log, an issue, or a downloaded file can be steered by text planted in it, and in CI it may run with the repository’s secrets. Grant its jobs the least privilege of Section 12.6, and keep pushes and merges with the analyst (Section 2.10.1).
Prompt. ‘Add a verification step that checks my analysis reproduces, by hashing the output and comparing it to a saved value.’
Watch for. A step that hashes the rendered PDF or HTML. The report is the most visible output and the wrong thing to hash, because its bytes carry timestamps and font identifiers and change on every render.
Verification. Insist that the target is a computational output. Estimates should be compared with a committed baseline within a stated tolerance, in a test-baseline-*.R file, while hashes are kept for files whose bytes should not change. Confirm by rendering the report twice without changing the analysis: the report’s own bytes should differ while the baseline test still passes. If the model’s chosen target changes between the two renders, it is the wrong target.
Prompt. ‘My compendium is a simulation study that takes six hours to run. Write CI for it.’
Watch for. A workflow that attempts the full run on every push, which will time out or consume the CI budget, or one that runs so few iterations that it no longer tests anything meaningful.
Verification. Confirm the workflow runs a reduced, fixed-seed pipeline under CI and defers the full run elsewhere. Check that the reduced run is seeded so that its short output is itself reproducible. Verify that the number of iterations in the reduced run is large enough to exercise the code paths that matter, not merely one.
A better prompt. ‘My compendium’s simulation runs <B> replicates in about six hours; R/ holds the functions and inst/tinytest/ the tests. Write a GitHub Actions workflow that, on each push, runs the tests and a fixed-seed run of <b> replicates within 20 minutes, pins each action by commit SHA, and never sets continue-on-error. Say where the full run should happen instead, and what deliberate change I should make to confirm that each step can fail.’
In each case a syntactically valid workflow is the easy part to check. The semantics, what the check actually proves and whether it can fail for the right reason, are what the analyst must verify and cannot delegate.
12.17 Exercises
Capture versus validation. In your own words, explain why removing a continuous-integration workflow from a compendium does not lower its position on the reproducibility ladder, and state precisely what is lost when CI is removed. Give one concrete failure that CI would have caught and that its absence now permits to go unnoticed.
The three gates. For each of the three CI gates (R CMD check, the render gate, the dependency-manifest gate), describe a specific defect in a compendium that the gate catches and the other two do not. Then name one reproducibility failure that none of the three catches, and say which gate, verification or data integrity, addresses it.
What a digest can and cannot tell you. Take the code in Table 12.1 as a starting point. Extend it to hash the coefficients of a small linear model fitted to the synthetic cohort data with lm(outcome ~ arm, data = cohort), record the digest, refit the identical model in the same session, and confirm the digest is unchanged. Then alter one outcome value, refit, and confirm the digest changes. Explain why this hash reliably detects that something changed. Then explain why it is still the wrong tool for the question a baseline test answers, whether a reported estimate has drifted. Say what would make the digest an unreliable guard even when the coefficients themselves are unchanged, and why a tolerance-based comparison is the correct instrument for a floating-point result.
Two dialects. Take the condensed GitHub Actions workflow shown in this chapter and translate it into the equivalent .gitlab-ci.yml. Identify the two structural differences between the dialects (jobs-and-steps versus stages-and-scripts, and per-file versus single-file configuration), and explain why single-sourcing the check logic in a Makefile would let both files shrink to a thin shim.
Archetype and gates. For each of the five archetypes (data analysis, manuscript, R package, simulation, blog), state which of the three CI gates apply and why. For the simulation archetype, describe specifically how you would arrange a reduced pipeline under CI and a full pipeline elsewhere, and what role a fixed seed plays in making the reduced run a genuine reproducibility test.
Independent verification. Read the CODECHECK description of Nust and Eglen (Nüst & Eglen, 2021). Explain in what respect an independent codechecker plays the same role, for a submitted analysis, that a CI runner plays for a committed change. Then identify one class of reproducibility failure that an independent human codechecker would catch and that the author’s own CI, however well configured, cannot.
Cumulative: gate the cohort compendium. Using the pattern of ‘Demonstrating a gate on the cohort compendium’, run the following from the repository root (not from analysis/report/), so that the pinned set is read from the real lockfile rather than typed: script <- 'R/cohort.R', used <- unique(renv::dependencies(script)$Package), locked <- names(renv::lockfile_read('renv.lock')$Packages), and missing <- setdiff(used, c(locked, rownames(installed.packages(priority = 'base')))). Report the length of missing (an autogradable answer: a single integer). Then copy R/cohort.R to a new file, add the line library(sandwich) to the copy, set script to the copy’s path, re-run the used and missing lines, and report the exact character vector missing now contains (a second autogradable answer: the package names, spelled exactly as they would appear in a real CI log).
Spacing: commit before you compute, revisited. This exercise returns to the commit discipline of Chapter 4 (Chapter 4). A CI workflow triggers on push, so it can only test what has actually been committed. Any undeclared state that exists solely in an uncommitted file or an author’s working directory is invisible to it, in exactly the way it is invisible to a colleague who clones the repository. Explain, in your own words, why this makes ‘commit before you compute’ (habit 1) a precondition for CI to mean anything at all, and give one concrete example of an uncommitted change that would let a build pass locally and fail the moment it is pushed.
12.18 Solutions
Attempt an exercise before expanding its solution. Exercises 3 and 4 are hands-on and their outputs will vary; the boxes record the expected result and the reasoning.
NoteExercise 1
The ladder grades capture: how far down the determinant stack a result is locked. CI locks nothing. It installs no package into the lockfile and no library into the image, so removing it leaves every pinned layer exactly as pinned as before. A compendium at L2 with CI and the same compendium at L2 without it sit on the same rung.
What is lost is assurance rather than level. Without CI nobody rebuilds the captured environment on a clean machine and re-runs the analysis in it. The compendium becomes trusted rather than tested, and the distance between ‘this should rebuild’ and ‘this does rebuild’ stops being measured.
A concrete failure. A base image is repointed by its maintainer and the Dockerfile pins it by tag rather than digest, so the environment changes underneath the project. With CI, the next push rebuilds and the check fails, naming the day it broke. Without CI, the compendium looks healthy on the author’s machine, where an old image is cached locally. It then fails for the first reader who builds it from scratch, possibly years later, with no indication of when or why it broke. Equally good answers include a package pulled from CRAN and no longer restorable, or a system library the analysis silently depended on and never declared. A chunk that only ever ran because of state left in the author’s session is another.
NoteExercise 2
R CMD check catches a defect in the package skin as code. One example is a function in R/ calling a package that is used but not declared in DESCRIPTION (a WARNING, so the gate must fail on warnings). Others are a syntax error in a file under R/ that nothing currently calls, or a documented argument that no longer exists. The render gate would not see it if the report does not touch that function. The manifest gate checks declarations rather than whether the code loads. The same defects under analysis/ are invisible to R CMD check, which never sees that directory.
The render gate catches a defect that appears only when the document is built. Examples are a chunk that fails on a clean machine because it read an uncommitted file, a figure whose code errors, or a cross-reference to a label that no longer exists. R CMD check never renders, so it sees none of this.
The dependency-manifest gate catches drift between the code and the manifests: a package used in a script under analysis/ and missing from renv.lock, which R CMD check never sees, or, for the package archetype, a package declared in DESCRIPTION and missing from renv.lock. The code may load and the document may render perfectly on a machine where the package happens to be installed, so neither other gate fires.
A failure none of the three catches: a change in the data. A raw extract silently replaced with a corrected version leaves the code valid, the document renderable, and the manifests consistent. Meanwhile every number in the report changes. All three gates pass. The data-integrity gate addresses it, by hashing the inputs and derived outputs and comparing against a recorded manifest.
The verification gate addresses the adjacent case in which the code changes a result without any of the three noticing. It does so by comparing computed outputs against recorded baselines.
NoteExercise 3
library(digest)m1<-lm(outcome~arm, data =cohort)d1<-digest(coef(m1), algo ='sha256')m2<-lm(outcome~arm, data =cohort)same_refit<-digest(coef(m2), algo ='sha256')==d1same_refit#> [1] TRUEcohort_alt<-cohortcohort_alt$outcome[1]<-cohort_alt$outcome[1]+1m3<-lm(outcome~arm, data =cohort_alt)same_altered<-digest(coef(m3), algo ='sha256')==d1same_altered#> [1] FALSE
Refitting the identical model to identical data, in the same session, gives an identical digest; altering one outcome value changes it. The chunk above, run on R 4.6.0 when this book was rendered, returned TRUE and then FALSE.
Why the digest is sensitive to any change. The coefficients are a deterministic function of the data and the model specification. Nothing about the time of fitting, the machine’s fonts, or the session’s history enters them, so within one session and platform the digest changes exactly when a number changes and not otherwise.
Why it is still the wrong tool for verification. A digest compares serialized bytes, not numeric values. R’s own documentation for serialize() (the mechanism digest() and saveRDS() both rely on) states that the default serialization format changed from version 2 to version 3 in R 3.6.0. It also warns explicitly that ‘the serialization format may change in future versions of R, so this interface should not be used for long-term storage of R objects’ (?serialize, inspected). Nothing guarantees that two R versions, or two platforms, serialize an identical numeric vector to identical bytes. An added attribute, a changed internal representation, or a future format bump can flip the digest while every coefficient is still correct to the last bit. That makes a coefficient digest a plausible false-positive generator across upgrades. This is exactly the failure mode this book’s baseline test is built to avoid, by comparing values with expect_equal(..., tolerance = ...) rather than comparing serialized bytes with ==. The digests in Table 12.1 are defensible because they hash the bytes of a data file on disk, not a serialized R object, and byte-for-byte identity of a raw data file is the actual requirement. Reused on a fitted model’s coefficients, it answers a different, stricter question (‘are the bytes identical’) than the one a reproducibility check needs answered (‘is the value the same within tolerance’). It will occasionally answer it wrong in a way that looks like a false alarm.
One practical refinement the exercise invites. Digesting the whole lm object rather than coef() is fragile, because the object stores the call and a terms environment, which can carry incidental references. Hashing the coefficient vector, or a named list of the quantities actually reported, is more robust against incidental differences. However, it does not remove the cross-version risk above. A baseline test with a stated tolerance remains the right tool for verifying that an estimate has not drifted.
Why hashing a plot would not be sound. A rendered plot is an image file. Image files embed creation timestamps, are affected by device version and font availability, and can differ in their compression between runs. Two plots of the identical model can differ byte for byte while depicting exactly the same thing. The check would therefore fail on every rebuild and carry no information. The rule this generalizes to is that verification hashes computational outputs, not rendered artifacts; it is the same rule that governs the PDF in Chapter 9.
NoteExercise 4
# .gitlab-ci.ymlstages:[gates]gates:stage: gatesimage: rocker/verse:4.6.1script:- Rscript -e 'renv::restore(prompt = FALSE)'- Rscript -e 'renv::install(c("rcmdcheck", "remotes"))'- Rscript -e 'remotes::install_github("rgt47/zzrenvcheck", ref = "6f53235668cb4048c1ae793d64b10d6505bf69a2")' # Gate 1: fails on a WARNING, as error-on does on GitHub- Rscript -e 'rcmdcheck::rcmdcheck(args = "--no-manual", error_on = "warning")' # Gate 2: the target that modifies nothing- make check-renv-no-fix # Gate 3: render inside this image, without Docker- Rscript -e 'rmarkdown::render("analysis/report/report.Rmd", output_format = "html_document", output_dir = tempdir())'
The two structural differences.
Jobs and steps versus stages and scripts. GitHub Actions composes a job from reusable uses: steps, each a versioned action maintained elsewhere. The workflow therefore reads as an assembly of components. GitLab CI defines stages containing jobs whose bodies are shell scripts, so the pipeline reads as commands in an image. The GitHub form is more declarative and depends on an ecosystem of actions; the GitLab form is more direct and depends on nothing but the image.
Per-file versus single-file configuration. GitHub reads every file under .github/workflows/, so a project naturally accumulates several workflows with separate triggers. GitLab reads a single .gitlab-ci.yml at the root, so everything is expressed in one file with rules selecting what runs when.
Why a Makefile shrinks both. The differences above are entirely about orchestration: what runs, in what order, on what trigger. The check logic itself, run R CMD check, render the report, verify the manifests, is identical in both and belongs in neither file. Putting it behind named targets leaves each CI file a thin shim naming the image and calling a short list of them. The logic is then single-sourced, testable locally with the same command CI runs, and portable to a third forge for the cost of one more shim.
A zzcollab-scaffolded compendium’s Makefile already ships targets for this, but not every target is a gate, and the file above calls only one of them. Three cautions apply. First, check-renv runs the dependency check with auto_fix = TRUE and is meant to be run before a commit. On a runner it repairs renv.lock in place, so it reports success for the very omission the gate exists to catch. Its sibling check-renv-no-fix modifies nothing and exits nonzero on failure, and only that target is a gate. Second, check runs R CMD check --as-cran, which exits nonzero on an ERROR but not on a WARNING, so an undeclared import passes it; the file therefore calls rcmdcheck with error_on = "warning" directly. Third, the render targets (docker-render, docker-render-qmd, docker-render-book) start a container with docker run. A job whose image: is rocker/verse has no Docker daemon, so the file renders inside the image itself, after the renv::restore() that the body’s GitLab file also performs. Rendering through the project’s own image belongs in a job on a runner that can run Docker, as the verify job above does. If you adapt this exercise to your own compendium, name the targets that match it, and confirm that each one can fail.
NoteExercise 5
Data analysis:R CMD check and the manifest gate always; the render gate if a report is produced. Manuscript: all three, since the rendered document is the deliverable. R package:R CMD check and the manifest gate; the render gate only if a vignette exists. Simulation:R CMD check and the manifest gate; the render gate if a report is produced, plus a seed-determinism check. Blog: the render gate above all, since the deliverable is the site. R CMD check and the manifest gate apply to any code the posts run.
Arranging the simulation. CI runs a reduced pipeline: the same driver, the same functions, the same seed. The replicate count is set by a parameter to something small, perhaps a hundred replicates in place of a hundred thousand. The full pipeline runs elsewhere, on a cluster, and its outputs are recorded and deposited.
What the fixed seed contributes. It is what makes the reduced run a genuine reproducibility test rather than a smoke test. With the seed fixed, the first hundred replicates of the reduced run are the same hundred replicates the full run begins with. Their summary is therefore a deterministic value that can be recorded and compared exactly. Without the seed, the reduced run could only be checked against a tolerance derived from Monte Carlo error. That is a much weaker statement, and one that passes even when the generator has changed.
The limit is that the reduced run verifies that the machinery reproduces, not that the full study’s estimates are correct, since a hundred replicates estimate the target quantity poorly.
NoteExercise 6
The shared role. Both are an independent re-execution of the analysis in an environment the author does not control, performed by a party who has no stake in it succeeding. A CI runner does this for every committed change, cheaply and mechanically; a codechecker does it once for a submitted manuscript, expensively and with judgment. In both cases the value comes from the same property. The analysis runs somewhere that carries none of the author’s undeclared state, so anything the author depended on without knowing it is exposed.
What a human codechecker catches that the author’s own CI cannot. The class of failure in which the CI configuration itself encodes the author’s assumptions. CI is written by the author, so it tests what the author thought to test, in an environment the author specified, against outputs the author chose to compare. Consider a workflow that installs a system library the Dockerfile omits, or that silently skips the expensive analysis and checks only the cheap one. Consider also one that compares against a baseline the author generated after the bug was introduced. Each passes indefinitely while the compendium is not reproducible at all.
A codechecker reads the paper, forms an independent view of what the analysis is supposed to produce, and checks that the artifacts produce it. That is a judgment about whether the right thing was verified. No amount of automation supplies it, because the automation is the thing in question.
The compact statement: CI answers ‘does the analysis still do what it did’, and a codechecker answers ‘does the analysis do what the paper says’.
NoteExercise 7
The solution for this exercise is in the instructor guide, which is distributed separately to course instructors. Self-study readers can check both autogradable values against the chunk below, which runs the exercise’s own lines against this repository’s lockfile at render time.
A CI runner tests only what is checked out from version control, which is exactly what is committed, no more. An uncommitted change exists solely on the author’s own machine, so from the runner’s point of view it does not exist at all. The runner cannot fail on a problem it never sees, and it cannot confirm a fix that was never pushed either. ‘Commit before you compute’ is therefore what makes CI’s test meaningful in the first place. Without it, a green build certifies a state of the repository that nobody’s actual analysis ran against.
A concrete example. An analyst fixes a bug in analysis/scripts/02_model.R, reruns the report locally, confirms the figure now looks right, and pushes. However, the analyst forgets to git add a second file, a helper function in R/model_utils.R the fix also depended on. Locally, everything passes, because the uncommitted helper is still on disk. On push, the runner checks out only what was committed, and the helper function is missing or stale. The render gate fails at the first chunk that calls it. The author cannot reproduce this failure by looking at their own unchanged working directory until they notice the file was never staged.
Dependency-manifest gate; passing error_on_fail = TRUE explicitly keeps the step’s failure from depending on how R was started.
uses: actions/checkout@<full SHA> # v7.0.1
Pins an action to a commit; Dependabot or Renovate can propose updates as pull requests.
renv::dependencies('analysis')
Scans source for packages actually used; the core of a hand-written dependency gate.
rmarkdown::render(...) in a CI step
The render gate: fails the job if any chunk errors on a clean restore.
docker build -t compendium:ci .
Rebuilds the project image from a clean machine, the L2 rebuild zzc verify --full requires.
test-baseline-*.R naming
What zzc verify --full and ci/run-tests.R look for to certify L3.
digest(file = path, algo = 'sha256')
Checksums a file’s bytes that should not change (raw data); the wrong tool for a floating-point estimate.
.Rbuildignore entry for analysis/
Why R CMD check never sees analysis scripts or their dependencies.
Rule of thumb
Verify computational outputs within a tolerance; never hash rendered PDF or HTML bytes.
Habit 4
Make every gate able to fail, and watch it fail once.
Habit 6
Rebuild from clean, on a machine that is not yours.
12.20 Further reading
The distinction between capturing reproducibility and validating it organizes this chapter. It is drawn out across the introduction of this book (Chapter 1), and it rests on the general framing of computational reproducibility as a spectrum in Peng (2011).
On independent, external verification, the CODECHECK initiative described by Nust and Eglen (Nüst & Eglen, 2021) is the most concrete model of re-execution during peer review. It merits reading for its account of what a codechecker does and does not certify. The ACM’s artifact review and badging policy (Association for Computing Machinery, 2020) documents the badge scheme now common at computing venues.
On the persistence of the problem that all this machinery addresses, Stodden and colleagues (Stodden et al., 2018) analyze the effectiveness of journal reproducibility policies. They find, soberingly, that a policy on paper is not a reproduction in practice. Trisovic and colleagues (Trisovic et al., 2022), in re-running thousands of deposited R files, quantify how often re-execution simply fails, which is the empirical case for the standing checks this chapter describes.
For the practices that surround the tooling, the ten simple rules of Sandve and colleagues (Sandve et al., 2013) ask for automated steps and public access to scripts, runs, and results, the preconditions of independent re-execution. They also connect the mechanics of CI to the habits that make it worthwhile.
Nüst, D., & Eglen, S. J. (2021). CODECHECK: An open science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility. F1000Research, 10, 253. https://doi.org/10.12688/f1000research.51738.2
Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013). Ten simple rules for reproducible computational research. PLOS Computational Biology, 9(10), e1003285. https://doi.org/10.1371/journal.pcbi.1003285
Stodden, V., Seiler, J., & Ma, Z. (2018). An empirical analysis of journal policy effectiveness for computational reproducibility. Proceedings of the National Academy of Sciences, 115(11), 2584–2589. https://doi.org/10.1073/pnas.1708290115
Trisovic, A., Lau, M. K., Pasquier, T., & Crosas, M. (2022). A large-scale study on research code quality and execution. Scientific Data, 9(1), 60. https://doi.org/10.1038/s41597-022-01143-6