4  Version Control with Git and GitHub

Maintaining a high degree of transparency in scientific reporting is essential not just for gaining trust and credibility within the scientific community but also for facilitating the development of new ideas.

Karthik Ram, Git Can Facilitate Greater Reproducibility and Increased Transparency in Science (2013)

This chapter has three practical aims. By its end you will be able to write a commit history that answers ‘what changed and why’ for any analysis you run. You will be able to build a .gitignore that keeps protected health information out of a repository before the first commit rather than after a breach. Finally, you will be able to state exactly which single rung of the reproducibility ladder a GitHub URL has actually earned, and which it has not.

We begin by placing version control on the determinant stack and naming the failure it cures. We then set out the analyst’s three judgments, the git model, and the reading of a history, before turning to what must never be committed. Branches, the undoing of mistakes and merge conflicts, and remotes follow, and we close the exposition with what git does not do. A worked example, a note on collaborating with an LLM, and the exercises complete the chapter.

NoteBefore you start

Reading time. About 65 minutes.

Prerequisites. Chapter 2 and Chapter 3, for the determinant stack and the L0 rung that this chapter’s practice secures; no other chapter is assumed. Statistically, nothing is required. R knowledge assumed: none beyond reading a short script. The git and shell commands that make up most of this chapter are independent of R, and the two R chunks that appear are read for their output, not written by the reader. Git itself must be installed and configured; the preflight in the setup appendix (Git preflight) checks both in a few minutes.

By the end of this chapter you will be able to answer:

  • Can you describe the failure mode version control cures, the drawer of analysis_final_v2_FINAL.R files, and explain why the cure is a different kind of object, a history, rather than a tidier naming scheme?
  • Can you explain the git model, the repository, the commit as an atomic, message-bearing unit, and the history as a tamper-evident chain, together with the limits of that record as evidence?
  • Can you perform the everyday git operations at the shell, staging and committing, reading the log and the diff, and writing an archetype-appropriate .gitignore?
  • Can you state what must never be committed in health research, and explain why deleting such a file in a later commit does not remove it from the history?
  • Can you place version control on the reproducibility ladder, name the one rung it secures, and list the layers of the determinant stack it leaves exposed?

Outline.

  • Where this sits on the stack.
  • Orientation: the drawer of FINAL files, and the object that replaces it.
  • The analyst’s contribution: three judgments no tool automates.
  • What version control records, and what it cures.
  • The git model: repository, commit, history.
  • Reading the history: the log and the diff.
  • What must never be committed: .gitignore and PHI.
  • Branches, lightly.
  • Undoing things, and merge conflicts.
  • The remote: GitHub, GitLab, and collaboration.
  • What git does not do.
  • Worked example: an untracked folder to a tagged repository.
  • Collaborating with an LLM.
  • Exercises, a cheat sheet, and further reading.

A collaborator’s repository holds analysis.R and a Dockerfile whose build installs packages from the current CRAN release; it has no renv.lock. Its README claims L2. A year later the image is rebuilt. One call now fails because a package changed its interface. Inside the same rebuilt image, a regression coefficient computed on the cluster differs from the laptop’s in the sixth decimal place. Using Chapter 2 and Chapter 3, which rung has the repository earned? For each discrepancy, which layer is responsible, and which tool or rung addresses it?

The collaborator’s repository, with a Dockerfile and no lockfile, has earned L0, locatable, and no more. The rungs are cumulative (Chapter 3): L1 requires a lockfile, and L2 requires a container recipe as well, so a Dockerfile without a lockfile is L0 whatever its README says. The failing call is package churn in the package-versions layer, the code rot of Chapter 2. A lockfile, renv.lock, closes it and marks L1; restoring that lockfile inside the image then reaches L2. The sixth-decimal difference is the signature of the numerical-library layer. No rung guarantees it: the image determines which BLAS is installed, but the thread count and the processor still vary between machines (Chapter 2, Chapter 3). Fixing the thread count narrows the gap, and a stated tolerance in an L3 baseline test absorbs what remains.

4.1 Where this sits on the stack

Version control is this book’s clearest case of pure capture at the very top of the determinant stack. Figure 4.1 marks the layer it secures. Before a repository exists, the analysis code is not merely unpinned; it is not reliably even locatable by anyone but the analyst who wrote it. Placing it under git, and pushing it to a remote, is the single act that lifts that top layer from nothing to L0. The input-data layer directly beneath it is deliberately left unhighlighted here, because this chapter’s central judgment runs the other way. Health data, unlike code, often must be kept out of the very history that pins everything else. That is the subject of the section on .gitignore below.

plot_determinant_stack(
  captures = c('Analysis code'),
  title = 'Version control captures the analysis code; later chapters capture the rest')
A horizontal bar chart of the nine-layer determinant stack, hardware at the bottom and intrinsic non-determinism at the top, each bar colored by, and labeled with, the rung that secures it across the book. This chapter captures Analysis code; those layers are drawn at full color, labeled as captured or validated here, and, where validated, with a heavy outline. The remaining 8 layers are shown faded, for context, as secured elsewhere in the book. From the base, the rungs are: Hardware, unpinned by any rung; Numerical libraries (BLAS, LAPACK), unpinned by any rung; Operating system, system libraries, L2 pinned environment; R runtime, L2 pinned environment; Package versions, L1 pinned packages; Session configuration, L2 partially pinned; Analysis code, L0 locatable; Input data, L0 locatable; Intrinsic non-determinism, unpinned by any rung.
Figure 4.1: The determinant stack of Chapter 3, with the analysis-code layer marked as captured here. Version control is what first pins the code, lifting it from unlocatable to L0; the input-data layer immediately below is left unhighlighted on purpose, because this chapter spends as much effort keeping some data out of the history as it spends getting the code into it.

4.2 Orientation

If we open the folder of almost any analysis that has run for more than a month without version control, the same artifact greets us. There is analysis.R, and beside it analysis_v2.R, analysis_final.R, analysis_final_v2.R, and, inevitably, analysis_final_v2_FINAL.R. A dated copy or two, analysis_2025-03-14.R, sits among them, and a analysis_reviewer.R from the week the referee reports came back. Every file is a manual snapshot, taken because the analyst feared to overwrite a version that worked. The scheme fails in three ways at once. It does not say which file produced the figure in the submitted manuscript. It does not say what differs between any two of them, so recovering a change means reading both side by side. And it does not say why any change was made, because the reason lived in the analyst’s head and is now gone.

This is the failure that version control exists to cure. What, then, is the cure? A tidier naming scheme will not do; the cure is a different object. A version-control system keeps a single working copy of each file. It records, as a separate and complete history, every state that copy has passed through, together with a note on each change saying what it was and why it was made. The drawer of copies collapses back into one analysis.R, and each copy becomes an entry in a history that can be read, searched, and, if need be, restored. The tool that has won this role, across the sciences and across software of every kind, is git. The service that most often hosts a git history on the network is GitHub, with GitLab its principal institutional sibling. This chapter teaches both from first principles. Version control is the single most universally taught element of a modern reproducibility course, and it is the foundation on which the rest of this book stands.

Of the three things the FINAL-file drawer loses, the identity of the working version, the content of each change, and the reason for each change, which one is recoverable, laboriously, by reading the files side by side, and which is not recoverable at all once the analyst forgets or leaves?

The content of a change, what differs between two snapshot files, is recoverable by diffing them line by line, though at real cost in effort. The reason for a change is not recoverable from the files at all. It lived only in the person who made the change, and once they forget it or leave the project, it is gone for good. Version control is useful precisely because the reason is the one loss that no amount of careful reading of the files themselves can undo.

It is also the foundation in a precise, technical sense. The next chapter, Chapter 5, makes the compendium the unit of reproducible work, and Chapter 3 placed the bare compendium at the bottom rung of the ladder, L0 locatable. The artifact that marks L0 is exactly a version-controlled repository. That is, an analysis is locatable when, and only when, it is under version control and can therefore be found, read, and referred to by a second person. Everything the later chapters add is installed on top of a version-controlled history: the lockfile of Chapter 6, the container of Chapter 7, the continuous integration of Chapter 12.

4.3 The analyst’s contribution

Version control automates the mechanics of history perfectly and automates none of the judgment that makes a history useful and safe. Three decisions belong to the analyst and are the subject of this chapter as much as the commands are.

  1. Deciding what must never be committed. A commit is, by design, permanent and copied to every clone of the repository. The judgment of what may safely be made permanent and public, and what must be kept out of the history entirely, is the analyst’s. In health research it is a judgment with legal and ethical force. Protected health information, identifiable data, and credentials must never enter the history. As we shall see, deleting them in a later commit does not remove them from it. This is the first and gravest of the three judgments.

  2. Choosing the granularity and the message of a commit. A commit is a unit of change. How much change it should contain, and what its message should say, are matters of craft that no tool decides. A history of one enormous commit reading ‘did the analysis’ is almost as useless as no history at all. A history of small, coherent commits whose messages say why each change was made is a readable record of decisions.

  3. Deciding when to branch and when to tag. Whether a piece of work warrants a branch of its own, and whether a particular state of the analysis warrants a permanent tag, are design decisions about the shape of the history. They are cheap to make and expensive to omit. The state of the code at the moment a manuscript was submitted is worth a tag, and no tool will place it for the analyst.

Of the three judgments listed above, which one carries legal and ethical force specific to health research rather than being craft judgment that applies to any software project, and why?

The first, deciding what must never be committed. Choosing a commit’s granularity and message, and deciding when to branch or tag, are matters of craft that apply to any software project, health-related or not. The judgment that protected health information and identifiable data must never enter the history carries legal and ethical force specific to working with human-subjects data. It is also the one judgment in this chapter whose consequence, once realized, no git command can undo.

NoteThe vocabulary of this chapter

New terms are defined again on first use; they are collected here for reference.

  • Repository (repo). A directory whose contents and full history git tracks, held in a hidden .git subdirectory. A clone is a complete copy of a repository, history and all.
  • Working tree. The ordinary files in the repository directory as they currently exist on disk, as opposed to the versions recorded in the history.
  • Commit. A recorded snapshot of the tracked files at one moment, carrying an author, a timestamp, a message, and an immutable identifier (a hash). The atomic unit of history.
  • Staging area (index). The set of changes marked, with git add, to go into the next commit. Staging lets a commit hold a chosen subset of the current changes.
  • .gitignore. A file listing path patterns that git should not track, used to keep derived outputs, large data, and, above all, sensitive files out of the history.
  • Branch. A movable name for a line of development. The default branch is conventionally called main, although git itself still names it master unless told otherwise, as with git init -b main.
  • Remote. A copy of the repository hosted elsewhere, typically on GitHub or GitLab, used for backup, distribution, and collaboration. origin is the conventional name of the primary remote.
  • Tag. A permanent, human-readable name for one specific commit, used to mark a milestone such as a manuscript submission.
  • Pull request (merge request). A proposal, made on the remote, to merge one branch or fork into another, reviewed before it is accepted.

4.4 What version control records, and what it cures

The value of a version-control system is easiest to state as the negation of the FINAL-file failure. That failure loses three things, and version control restores each.

It loses the identity of the working version. With a drawer of dated copies, no file is authoritative. With git, there is one analysis.R, and its history holds every prior state, so the question ‘which version is current’ cannot arise. It loses the content of each change. Two snapshot files differ in ways that can be recovered only by comparing them line by line by hand. A git history presents each change as a diff, a precise statement of the lines added and removed. Git computes the diff on demand from the snapshots it stores, since git keeps full snapshots, not diffs. Lastly, it loses the reason for each change, the single most valuable and most perishable piece of information about an analysis. Why was the outlier-exclusion threshold changed from three standard deviations to two? A dated file cannot say. A commit whose message reads ‘Tighten outlier threshold to 2 SD per reviewer 2’s comment’ says exactly why, and the record stays with the code.

The recorded reason is what makes version control an instrument of scientific accountability. A computational result is the end of a chain of decisions, each of which could have gone otherwise. In a regulated analysis, a clinical trial subject to audit, a study destined for a guideline, the chain of decisions is itself an object of scrutiny. ‘Why did the estimate change between the interim and the final analysis’ is a question an auditor is entitled to ask and the analyst is obliged to answer. A commit history in which each change carries its reason is most of the answer, already written, dated, and attributed. It is, however, a record of what the committers asserted, not a regulated audit trail. The next section explains the difference.

A commit message reads ‘Tighten outlier threshold to 2 SD per reviewer 2’s comment.’ Which of the three things the FINAL-file drawer lost does this message, by itself, restore, and which two does it leave to the rest of git’s model?

It restores the reason for the change, naming the reviewer’s request as the cause. It does not, by itself, restore the identity of the working version, which comes from git keeping one tracked analysis.R rather than a drawer of copies. Nor does it restore the precise content of the change, which comes from the diff the commit carries alongside its message.

4.5 The git model: repository, commit, history

Git is often taught as a list of commands to memorize, which is why it is so often found bewildering. It is far better understood as a small and coherent data model, after which the commands follow naturally. There are three ideas.

The first is the repository. A git repository is an ordinary directory, the analyst’s project, that git has been told to track. Git then maintains, in a hidden .git subdirectory, the complete history of everything under that directory. The ordinary files the analyst edits are the working tree; the .git directory is the history. Creating a repository is a single command run once in the project directory.

cd diabetes-cohort
git init -b main

The -b main names the first branch main; without it, git as of version 2.56 names it master, and the git push -u origin main commands later in this chapter would then fail. The git preflight of the setup appendix (Git preflight) sets main as the default once per machine, together with the name and e-mail address every commit records. It is worth running before this chapter’s first command.

The second idea, and the central one, is the commit. A commit is a snapshot of the tracked files at one moment, recorded together with four things: who made it, when, a message describing it, and a hash that names it. That hash is a forty-character SHA-1 string, usually shown in an abbreviated form such as a1b2c3d. It is computed over the whole commit object, and the snapshot alone does not determine it. The commit object references the snapshot, folds in the metadata, and records the hash of the commit before it. (Git also supports repositories that use the longer SHA-256 hash of sixty-four characters, though SHA-1 remains the default.) Because each commit’s hash incorporates its parent’s, the history is a chain. No past entry can be altered without changing its hash and the hash of every commit after it. The hashes are, for practical purposes, unique, and git guards against the known SHA-1 collision attacks with built-in collision detection. The chain makes a history tamper-evident in a narrow sense. Anyone who recorded a commit’s hash, in a tag, a manuscript, or a colleague’s clone, can detect that the history leading to it was later changed.

But how much evidence is tamper-evidence? It is weaker than it sounds, and anyone headed for regulated work should know exactly how much weaker. The author’s name and e-mail address, and both dates a commit carries, are whatever the committer’s own git client supplies. Setting GIT_AUTHOR_DATE and GIT_COMMITTER_DATE and passing --author produces a commit under another person’s name dated years in the past, which git accepts without complaint.

GIT_AUTHOR_DATE='2019-03-01T09:00:00' \
GIT_COMMITTER_DATE='2019-03-01T09:00:00' \
  git commit --author='Someone Else <someone@example.org>' \
    -m 'Backdated commit'

History is also rewritable by design. git commit --amend, git rebase, and git filter-repo all replace commits with new ones, and in a repository the analyst controls nothing prevents it. The hash chain detects a rewrite only for someone holding an earlier hash to compare against. Assurance is added from outside the commit. One source is commits signed with a key bound to a verified identity. Another is a forge configured to protect the main branch against force-pushes and to log pushes, reviews, and merges on its own servers with its own clock. Even then, a git history is not, by itself, the audit trail that FDA’s regulations require of electronic records in the studies they cover. That trail must be secure, computer-generated, and time-stamped, and must record operator actions independently (21 CFR Part 11, n.d.). Within those limits, a well-kept history is an honest and very useful record of how an analysis was built. The rest of this chapter treats it as that.

The third idea is the history itself, the chain of commits, each pointing back to its parent, running from the first commit to the most recent. Reading that chain is reading the story of how the analysis came to its current state. A branch, which we treat below, is nothing more than a movable name pointing at one commit in this structure.

The everyday cycle of work has just two steps against this model. They exist because git deliberately separates choosing what goes into a commit from making the commit. First one stages the changes to be recorded, with git add; then one commits the staged changes, with git commit, attaching the message. The staging step is not ceremony. It lets a single commit hold a coherent subset of the work in progress, so that an afternoon’s editing across several concerns can be recorded as several focused commits.

# edit analysis.R, then see what has changed
git status

# stage the file for the next commit
git add analysis.R

# record the staged change with a message
git commit -m "Add Cox model for 30-day readmission"

The message is the part of a commit that carries the scientific value. The convention, which merits adoption from the first day, is a short imperative summary line of about fifty characters. It is optionally followed by a blank line and a paragraph giving the why. ‘Fix bug’ is a wasted message. ‘Correct person-time denominator to exclude pre-enrollment follow-up’ is a message that will still be intelligible, and still exculpatory, at an audit three years later.

A colleague argues that the staging area is needless friction: since one almost always wants to commit everything one has changed, git should simply commit the whole working tree and spare the analyst a git add. What does the staging step buy that a commit-everything model would lose?

It buys the ability to make a commit an atomic unit of one idea rather than a snapshot of whatever happened to be on disk. In an afternoon an analyst may fix a bug in the cleaning script, add an unrelated figure, and jot a half-finished note in a third file. Committing all three together produces a history entry that cannot be described in one honest message and cannot be reverted without disturbing unrelated work. Staging lets the analyst record the bug fix as one commit, with its own message and reason, and the figure as another, leaving the unfinished note uncommitted. The history is only as useful a record as its individual commits are coherent, and staging is the mechanism that coherence requires. (For the genuine everyday case where one does want everything, git commit -a stages all tracked changes in one step, so the friction is opt-out, not mandatory.)

4.6 Reading the history: the log and the diff

Once a history exists, how do we read it? A history that cannot be read is of little use, and git provides two complementary views of it. The log is the list of commits, most recent first, each with its hash, author, date, and message. It is the table of contents of the analysis’s development.

# the full log
git log

# a compact one-line-per-commit view
git log --oneline

# which commits touched analysis/scripts/
git log --oneline -- analysis/scripts/

In a scratch copy of the worked example below, three commits in, the compact view reads as follows. Each line is the abbreviated hash and the summary line of the message, most recent first.

# output abridged, from git 2.56.0
git log --oneline
#> c7d4711 Tighten outlier threshold to 2 SD per reviewer 2
#> a9aa55f Add cleaning script for registry extract
#> 936e050 Initialize compendium and .gitignore

The diff is the complementary view. Where the log lists the changes, the diff gives the content of one. git diff shows the changes in the working tree that have not yet been staged. git show displays the full diff of any single commit named by its hash, and git diff between two commit hashes shows everything that changed between them.

# what have I changed but not yet staged?
git diff

# show one commit's changes in full
git show a1b2c3d

# what changed between two points in the history?
git diff a1b2c3d f4e5d6c

Here is the diff from the same scratch repository, taken just before the third commit, when one line of the cleaning script had been edited. A line beginning - is the committed version and a line beginning + is its replacement. The unmarked lines are context, and the @@ header gives their position.

# output abridged, from git 2.56.0
git diff
#> diff --git a/analysis/scripts/01-clean.R b/analysis/scripts/01-clean.R
#> --- a/analysis/scripts/01-clean.R
#> +++ b/analysis/scripts/01-clean.R
#> @@ -1,3 +1,3 @@
#>  cohort <- read.csv('analysis/data/raw_data/extract.csv')
#>  cohort <- subset(cohort, age >= 18)
#> -sd_cut <- 3
#> +sd_cut <- 2

Together the log and the diff answer the two questions the FINAL-file drawer could not. The log tells us when and why a thing changed, and the diff tells us what exactly the change consisted of. To illustrate, consider the commit history of the first weeks of an incident-diabetes cohort study. It is rendered below as it might appear to an auditor reviewing how the reported hazard ratio came to be. The table is generated from synthetic data, so that we may display a realistic history without any real project’s contents.

library(tibble)
library(dplyr)
library(knitr)

commit_history <- tibble::tibble(
  hash = c('a1b2c3d', 'e4f5a6b', 'c7d8e9f',
           'b0a1c2d', 'd3e4f5a', 'f6a7b8c',
           'a9b0c1d'),
  date = as.Date(c('2026-02-03', '2026-02-05',
                   '2026-02-09', '2026-02-12',
                   '2026-02-18', '2026-02-24',
                   '2026-03-02')),
  author = c('analyst.a', 'analyst.a', 'analyst.a',
             'analyst.b', 'analyst.a', 'analyst.b',
             'analyst.a'),
  message = c(
    'Initialize compendium and .gitignore',
    'Add cleaning script for registry extract',
    'Build analysis-ready cohort table',
    'Fit Cox model for 30-day readmission',
    'Tighten outlier threshold to 2 SD per reviewer 2',
    'Add Kaplan-Meier figure to report',
    'Correct person-time denominator'),
  files_changed = c(2L, 1L, 3L, 1L, 1L, 2L, 1L)
)

commit_history |>
  dplyr::mutate(date = format(date, '%Y-%m-%d')) |>
  knitr::kable(
    col.names = c('Hash', 'Date', 'Author',
                  'Message', 'Files'),
    align = c('l', 'l', 'l', 'l', 'r'))
Table 4.1: A synthetic commit history for the first weeks of a cohort analysis, read as a record of decisions. Each row is one atomic, message-bearing commit; together they record not only what changed but, in the message, why. Note the reviewer-driven change to the outlier threshold, the kind of decision an auditor is entitled to trace.
Hash Date Author Message Files
a1b2c3d 2026-02-03 analyst.a Initialize compendium and .gitignore 2
e4f5a6b 2026-02-05 analyst.a Add cleaning script for registry extract 1
c7d8e9f 2026-02-09 analyst.a Build analysis-ready cohort table 3
b0a1c2d 2026-02-12 analyst.b Fit Cox model for 30-day readmission 1
d3e4f5a 2026-02-18 analyst.a Tighten outlier threshold to 2 SD per reviewer 2 1
f6a7b8c 2026-02-24 analyst.b Add Kaplan-Meier figure to report 2
a9b0c1d 2026-03-02 analyst.a Correct person-time denominator 1

Two features of this history could not come from a bare list of files. The fifth commit records that the outlier threshold changed in response to a reviewer. The final commit records a correction to the person-time denominator, the sort of methodological fix whose provenance an auditor will want to establish.

An auditor asks two questions about Table 4.1: ‘why did the outlier threshold change’ and ‘exactly which lines changed when it did.’ Which git view, the log or the diff, answers each question?

The log answers the first question: git log lists each commit’s message, and the fifth commit’s message in Table 4.1 names the reviewer’s request directly. The diff answers the second: git show on that commit’s hash displays exactly which lines were added and removed. Neither replaces the other; the log without the diff says why but not what, and the diff without the log says what but not why.

4.7 What must never be committed: .gitignore and PHI

We come now to the gravest of the three judgments. A commit records the files git is tracking, and by default git offers to track every file in the working tree. Two categories of file must be kept out of that net. The mechanism for keeping them out is the .gitignore file, a plain-text list of path patterns that git will refuse to track.

The first category is the merely unwanted. These are files that can be committed but should not be, because they are regenerable or bulky and would only clutter the history. Rendered outputs (a compiled PDF, an .html report, the figures a script regenerates on every run) are the clearest case. Committing them couples the history to products that the code already determines. Large raw data files bloat every clone. Editor and operating-system cruft (.Rhistory, .DS_Store, .Rproj.user/) is pure noise. None of these is dangerous; all of them are better ignored.

The second category makes this section a matter of ethics and law. It comprises files that must never enter the history, because making them permanent and shareable would be a breach. In health research three kinds dominate.

  • Protected health information and identifiable data. A file containing patient identifiers, dates of birth, medical-record numbers, free-text clinical notes, or any data from which an individual could be re-identified must never be committed. This is the first rule of version control in a health setting, and it admits no exception.
  • Credentials. Database passwords, API tokens, the contents of a .Renviron holding a secret, private keys: a credential committed to a repository, and worse pushed to a remote, is a credential to be treated as compromised.
  • Restricted or embargoed data more generally, under a data-use agreement that forbids redistribution, even when not directly identifying.
ImportantRestricted data

Protected health information, identifiable data, and anything held under a data-use agreement that forbids redistribution must never go into a git history. The rule admits no exception: not even briefly on a branch that is never merged, and not even in a local repository that will ever be pushed. A commit is permanent by design and, once pushed, copied to every clone. The gravest and least intuitive fact about git is that its ordinary commands only ever add to the history. Suppose a file containing protected health information is committed, and the mistake is noticed and the file deleted in a later commit. The file is gone from the working tree but remains in full in the history. It is recoverable by anyone with a clone from the commit that introduced it. The deletion is itself just another commit layered on top; it conceals nothing. Removing the file requires rewriting the history. That rewrite changes every subsequent commit hash, cannot reach copies already made, and is error-prone precisely when it matters most. If the offending commit was ever pushed to a remote, one must assume it has been cloned and cached beyond recall. The only sound response is then to treat the exposed data as disclosed and follow the applicable incident procedure. The remedy is therefore prevention. That means a correct .gitignore written before the first commit, and the habit of reading git status before every commit to confirm that nothing sensitive has been staged.

When prevention has failed, the order of the response matters more than the commands, and the commands are the last step.

  1. Stop and report. Push nothing further. If the commit reached any remote, a disclosure has occurred. Notify the data steward, privacy office, or IRB under the institution’s incident procedure, because that obligation is not a technical question and does not wait for the cleanup.

  2. Rotate credentials first. If the file held a password, token, or key, revoke or rotate it at once. A rotated secret is harmless wherever copies survive; a rewritten history is not a substitute (GitHub, n.d.).

  3. Rewrite the history on a fresh clone. git filter-repo removes the file from every commit on every branch and tag. Its --sensitive-data-removal option also fetches all refs first and keeps the remote configured so that the result can be pushed.

    git clone https://github.com/lab/diabetes-cohort.git
    cd diabetes-cohort
    git filter-repo --sensitive-data-removal --invert-paths \
      --path analysis/data/raw_data/extract.csv
    # expect no output: the file is in no remaining commit
    git log --all --oneline -- analysis/data/raw_data/extract.csv
    git push --force --mirror origin

    The force-push may require branch protection to be lifted for the purpose and restored afterwards.

    WarningRegulated records

    In a repository that holds records under GxP or another regulated quality system, a history rewrite is a controlled change, not a cleanup. Before running it, open a deviation or incident, obtain QA approval under the organization’s SOPs, document what is being removed and why, and keep whatever record of the pre-rewrite state the quality system requires. The audit-trail provision of 21 CFR Part 11 states that ‘Record changes shall not obscure previously recorded information’ (21 CFR Part 11, n.d., Section 11.10(e)). A rewrite obscures that information by design, which is why the deviation record must carry what the history no longer does. Better still, keep regulated records out of a rewritable forge in the first place, so that the question never arises.

  4. Clean what the forge holds. On GitHub the purged commits can remain reachable through cached views and pull-request references, which the owner cannot push over. Removing them requires a request to GitHub Support. Forks are not touched by the rewrite at all, and their owners must be asked to delete them (GitHub, n.d.).

  5. Replace every clone. Every collaborator’s clone still holds the file, and a collaborator who pulls and pushes from an old clone puts it back. Each one must delete the clone and clone afresh, and any copy outside the team’s reach, a backup, a CI cache, a stranger’s clone, is beyond recall (GitHub, n.d.).

In a scratch repository, a clone made before the rewrite still lists the offending commit after the remote has been cleaned; the rewrite reaches only the copies it is run on and pushed to.

A .gitignore is a small file, and its content depends on the archetype of the compendium, a classification the next chapter, Chapter 5, develops in full. For now two archetypes suffice: a data-analysis compendium, whose deliverable is a set of results from data, and an R-package compendium, whose deliverable is reusable code. The reproducibility machinery is identical across archetypes; the list of files to ignore is not. A data-analysis compendium ignores its derived outputs and its raw data. These are the regenerable tables and figures, and the large or sensitive registry extract that must stay out of the history entirely. An R-package compendium has no analysis/ outputs to ignore but must ignore its build artifacts. These are the .Rcheck directory, the built .tar.gz and binary archives, and the src/ object files a compiled package leaves behind. The two .gitignore files barely overlap, and writing the right one is part of naming the archetype.

# .gitignore for a data-analysis compendium
# --- sensitive: never commit ---
analysis/data/raw_data/*
.Renviron
*.pem

# --- regenerable outputs ---
analysis/figures/*
analysis/tables/*
analysis/report/*.pdf
analysis/report/*.html

# --- editor and OS cruft ---
.Rhistory
.RData
.Rproj.user/
.DS_Store
# .gitignore for an R-package compendium
# --- build artifacts ---
*.tar.gz
*.Rcheck/
src/*.o
src/*.so
# generated pkgdown site; a comment must be on its own
# line, since text after a pattern is part of the pattern
docs/

# --- editor and OS cruft ---
.Rhistory
.RData
.DS_Store
TipVerification

A .gitignore that has never been watched blocking a real file is untested. To see it hold, create a file the pattern should catch and confirm git refuses it: touch analysis/data/raw_data/scratch.csv && git status should not offer to add the new file, because analysis/data/raw_data/* already matches it. To make it fail on purpose, comment out that one line in .gitignore, repeat the touch and git status, and confirm the file now appears as untracked and ready to be staged. Then restore the line before doing anything else (habit 4).

Two refinements warrant mention. A leading ! re-includes a pattern otherwise ignored. This is how one ignores a whole data directory but keeps a small, non-sensitive example file or a data dictionary under version control (analysis/data/raw_data/* followed by !analysis/data/raw_data/data_dictionary.csv). Second, a committed-by-accident file, once added to .gitignore, is still tracked until explicitly untracked with git rm --cached. Ignoring a file does not retroactively remove an already-committed copy, which returns us to the warning above.

This book’s own repository applies the rule to its running cohort. The extract analysis/data/raw_data/cohort_raw.csv is synthetic and describes no real person, yet it is not committed. What is committed is the script that writes it, analysis/scripts/simulate_cohort.R, which regenerates the extract from stated parameters; a reader with a fresh clone runs it once before the examples. The .gitignore excludes the whole raw_data directory rather than the one filename, so that a second extract saved there under any name is excluded too. The check below confirms that the generated extract carries identifying-looking fields, that the generator is present, and that the directory pattern is in force.

raw <- load_cohort('../data/raw_data/cohort_raw.csv')
nrow(raw)
#> [1] 338
intersect(c('mrn', 'patient_name', 'dob'), names(raw))
#> [1] "mrn"          "patient_name" "dob"

gitignore <- readLines('../../.gitignore')
repo_state <- c(
  generator_present = file.exists('../scripts/simulate_cohort.R'),
  raw_dir_ignored = 'analysis/data/raw_data/*' %in% gitignore
)
repo_state
#> generator_present   raw_dir_ignored 
#>              TRUE              TRUE

The generated extract carries 3 of the three columns this section warns against, by name. The synthetic cohort invents an identifier, a name, and a date of birth so that the risk appears in a real file instead of only in prose. The generator is present, and the pattern analysis/data/raw_data/* is in the .gitignore, which is the arrangement a real project should copy: the code that produces or fetches the extract under version control, the extract itself never.

A .gitignore guards paths, however, and nothing else. The tests in inst/tinytest/ need a copy of the cohort, and an earlier version of this repository met that need by copying the extract to inst/testdata/cohort_raw.csv and committing it. No ignore pattern covered that path, so the identifier columns entered the history by a second route while the first stayed closed. The fixture now at that path is derived from the synthetic extract. It keeps the rows and the duplicate structure the tests need, drops the name and date-of-birth columns, and replaces each record number with a pseudonymous ID-0001-style id.

fixture <- load_cohort('../../inst/testdata/cohort_raw.csv')
intersect(c('mrn', 'patient_name', 'dob'), names(fixture))
#> [1] "mrn"
all(grepl('^ID-[0-9]{4}$', fixture$mrn))
#> [1] TRUE

Replacing the file only added a commit, and the earlier copy is still in the history. Two commands find it. The first lists every commit, on any branch, that touched the path; the second prints the header of the file as the oldest of those commits recorded it.

git log --all --oneline -- inst/testdata/cohort_raw.csv
git show <oldest-hash>:inst/testdata/cohort_raw.csv | head -1
#>    commits_touching_path old_copy_has_identifiers 
#>                        2                        1 
#>      old_copy_same_bytes 
#>                        1

Run on this repository when the page was rendered, the log listed 2 commits touching the path. The oldest, 0bec92e, still holds a copy whose header names patient_name and dob, and its bytes are identical to the generated extract. Anyone holding a clone, made at any time since, can read it. The data are synthetic, so nothing real has leaked, but the pattern is the one by which real data leak: an extract protected at one path and copied, for a convenience such as a test, to another. Removing the copy from the history would take the rewrite described above, with every cost listed there; we keep it as this worked example. Deriving the new fixture from the extract is acceptable here only because the extract is synthetic. Rows of real data with the name columns dropped still carry dates, ages, and measurements that can re-identify, so a fixture for a real project is generated or built from the data dictionary, and a commit that adds a data file under any path warrants the question of where its rows came from.

A report.pdf sits at the top of one analyst’s ignore list and is deliberately committed by another. A data_dictionary.csv describing a restricted registry is committed, while the registry extract it describes is ignored. Explain both choices.

The rendered report.pdf is a regenerable output. In a data-analysis or manuscript compendium it is a product the code reproduces on render, so committing it couples the history to a derivative and is usually avoided. An analyst who nonetheless commits it, perhaps to preserve the exact submitted PDF as a fixed reference, is making a defensible different judgment about what the artifact is for. It is a judgment, not a rule. The data_dictionary.csv and the registry extract sit on opposite sides of the one line that is not a matter of taste. The dictionary carries variable names and codings, which are metadata and safe to share, and belongs in the history so a reader can interpret the analysis. The extract carries the protected health information itself and must never be committed, whatever its convenience. The ! re-inclusion pattern exists precisely to keep the safe metadata while ignoring the unsafe data beside it.

4.8 Branches, lightly

With the history and its exclusions in place, we turn to its shape. For a solo analyst working linearly, a single line of commits on the main branch is often the whole story, and it is honest to say so rather than to impose ceremony. But two situations make branches worth the small investment of learning them. The first is speculative work: trying an alternative model specification, or a reviewer’s proposed reanalysis, without disturbing the main line that produced the current results. The second is collaboration, treated in the next section. There each contributor works on a branch of their own, and the branches are merged only when ready.

A branch is created and switched to in one command, and merged back with another.

# create and switch to a branch for a reanalysis
git switch -c sensitivity-analysis

# ... make commits on the branch ...

# return to main and merge the branch in
git switch main
git merge sensitivity-analysis

The mental model is the one given above: a branch is a movable name for a line of development, and main is merely the name given to the first branch, here by git init -b main. Nothing about branching changes the commit model; it only lets two lines of commits coexist and later rejoin. We keep the treatment light here. We shall return to the collaborative use of branches, which is where they earn their keep, when we reach pull requests.

Figure 4.2 draws the shape. The branch does not copy the history. It names a new commit whose parent is a commit on main, and the merge names a commit with two parents. Everything before the branch point is shared, and no hash anywhere in the diagram changes when the branch is created or merged.

library(ggplot2)
library(tibble)

nodes <- tibble(
  id    = c('A', 'B', 'C', 'M', 'D', 'S1', 'S2'),
  x     = c(1, 2, 3, 4.6, 5.7, 3.1, 4.0),
  y     = c(0, 0, 0, 0, 0, 0.8, 0.8),
  shade = c(rep('main', 5), 'branch', 'branch')
)

edges <- tibble(
  x    = c(1, 2, 3, 4.6, 2, 3.1, 4.0),
  xend = c(2, 3, 4.6, 5.7, 3.1, 4.0, 4.6),
  y    = c(0, 0, 0, 0, 0, 0.8, 0.8),
  yend = c(0, 0, 0, 0, 0.8, 0.8, 0)
)

ggplot() +
  geom_segment(data = edges,
               aes(x = x, y = y, xend = xend, yend = yend),
               color = 'grey45', linewidth = 0.45) +
  geom_point(data = nodes, aes(x = x, y = y, fill = shade),
             shape = 21, size = 9, stroke = 0.5,
             color = 'grey30', show.legend = FALSE) +
  geom_text(data = nodes, aes(x = x, y = y, label = id),
            size = 3, fontface = 'bold') +
  scale_fill_manual(values = c(main = '#c3ddda',
                               branch = '#f0e2c4')) +
  annotate('text', x = 0.55, y = 0, label = 'main',
           hjust = 1, fontface = 'bold', size = 3.2) +
  annotate('text', x = 2.55, y = 0.8, label = 'branch',
           hjust = 1, fontface = 'bold', size = 3.2) +
  annotate('text', x = 4.6, y = -0.52,
           label = 'M: two parents', size = 2.9,
           color = 'grey25') +
  annotate('text', x = 2.0, y = -0.52,
           label = 'branch point', size = 2.9,
           color = 'grey25') +
  coord_cartesian(xlim = c(-0.1, 6.1), ylim = c(-0.8, 1.1)) +
  theme_void()
A commit graph with two horizontal rows of circles joined by lines, with time running left to right. The lower row, labeled main, runs A, B, C, then the merge commit M, then D. The upper row, labeled a sensitivity-analysis branch, holds two commits S1 and S2. A line rises from B to S1, marking the branch point, and a line descends from S2 to M, the merge. M is annotated as the only commit with two parents. Commits A, B, and C lie on the shared portion of the graph rather than being copied onto the branch.
Figure 4.2: A branch and its merge, drawn as the commit graph git actually stores. The lower line is main; the upper line is a sensitivity-analysis branch created at commit B and merged back at M. Commits A through C exist once and are shared by both lines: the branch adds commits rather than copying them. The merge commit M is the only commit in the diagram with two parents, which is what records that the two lines rejoined and when.

After creating and merging the sensitivity-analysis branch of the example above, do any of the commit hashes on main from before the branch was created change?

No. A branch adds new commits whose parent chain points backward to an existing commit on main; it never rewrites or recopies what came before the branch point. The merge adds one further commit with two parents rather than replacing anything. Every hash that existed before the branch was created, on main or anywhere else, is exactly as it was. That is what Figure 4.2’s shared commits A through C illustrate.

4.9 Undoing things, and merge conflicts

Two kinds of trouble are common enough to deserve their own section: a change that needs to be undone, and a merge that git cannot finish by itself.

Undoing. Git has three distinct commands for undoing work, and the right one depends on where the unwanted change lives.

git restore <file> discards unstaged edits in the working tree, putting the file back to its last committed (or staged) content. It never touches history; there is no commit to undo, only a draft to throw away.

# an edit no one has staged or committed
echo 'threshold <- 99' > analysis.R
git restore analysis.R
# analysis.R is back to its last committed content

git reset moves what HEAD points to. That is, it moves the branch itself to an earlier (or different) commit. The three flags differ only in how much else they touch along the way:

  • --soft moves HEAD and leaves everything else alone: the undone commit’s changes reappear staged, ready to be recommitted differently.
  • --mixed, the default, additionally unstages those changes, so they sit in the working tree, edited but not staged.
  • --hard additionally overwrites the working tree, discarding the changes outright.
# move HEAD back one commit; --mixed is the default
git reset HEAD~1
# the undone commit's edits are now unstaged, not gone

git reset --hard HEAD~1
# the undone commit's edits are discarded entirely

A reset, like the amend and rebase discussed earlier in this chapter, is a rewrite of what the branch name points to, not an addition to history. The same caution applies. A reset is harmless on commits that exist only in the analyst’s own clone. It is a serious act, needing the collaborators’ agreement and a force-push, once those commits have been pushed or pulled elsewhere. A local reset that moves past commits can usually be undone. The discarded commits are not immediately deleted, and git reflog lists where HEAD pointed before the reset, so git reset --hard <that-hash> restores them. The reflog records only commits, however. Edits that were never committed are not in it, and --hard overwrites them in the working tree. Unstaged edits are then gone permanently; staged ones survive only as unnamed objects that git fsck --lost-found can sometimes recover, file by file and without their names. Before any --hard, therefore, read git status, and commit or git stash whatever uncommitted work should survive.

git revert <commit> takes the opposite approach. Instead of moving HEAD backward, it makes a new commit whose changes are the exact opposite of the one named, applied on top of the current history. The reverted commit is left untouched, in place, for anyone to read later. Because revert only ever adds a commit, it is the one of the three that is safe on a branch other people have already pulled; reset and amend are not.

git revert HEAD
# creates a new commit, e.g. "Revert 'Add note line'",
# that undoes the previous commit's changes; the
# reverted commit itself still exists in the history

The choice can be stated briefly. For an edit that is not yet staged, use restore. For a commit that exists only on a private branch, reset is simplest. For a commit that others may already have, use revert and let history show both the mistake and its correction.

Merge conflicts. A merge like the one in Figure 4.2 goes through cleanly whenever the two branches changed different lines. When both branches change the same lines of the same file, git has no way to choose between them, and the merge stops partway, leaving the decision to the analyst.

git switch main
git merge sensitivity-analysis
# Auto-merging analysis.R
# CONFLICT (content): Merge conflict in analysis.R
# Automatic merge failed; fix conflicts and then
# commit the result.

git status reports the file as having ‘unmerged paths’, and the file itself now contains both versions, separated by conflict markers that git inserts directly into the text:

<<<<<<< HEAD
threshold <- 4
=======
threshold <- 5
>>>>>>> sensitivity-analysis

Everything between <<<<<<< HEAD and the ======= divider is the current branch’s version; everything between the divider and the final marker is the version being merged in. Resolving the conflict means editing the file by hand down to the single version that should survive. That may be one side, the other, or a combination of both. The analyst then deletes the markers themselves; nothing removes them automatically. Once the file reads the way it should, the usual two-step cycle finishes the merge. One runs git add on the resolved file to tell git the conflict is settled, then git commit to record the merge. (git merge --abort backs out of the whole attempt instead, if the conflict turns out to need more thought than a mid-merge edit allows.)

# edit analysis.R to resolve the conflict, then:
git add analysis.R
git commit -m "Merge sensitivity-analysis: keep threshold 4"

A conflict is not git malfunctioning; it is git correctly refusing to guess at a scientific decision, such as which of two threshold values the analysis should use, that only the analyst can make.

A collaborator has three problems: an edit in analysis.R that was never staged and should be thrown away; a commit made on a private branch, pushed nowhere, that turns out to be wrong; and a commit on main that was already pulled by two other collaborators and turns out to be wrong. Which command fits each case?

The unstaged edit: git restore analysis.R, since nothing has been committed yet. The private, unpushed commit: git reset, most simply --hard if the change should be discarded outright, since no one else has a copy of that commit to lose synchronization with. The already-shared commit: git revert. Reset or amend would rewrite a commit other clones already hold. Reverting instead adds a new commit that undoes it without touching what collaborators already pulled.

4.10 The remote: GitHub, GitLab, and collaboration

Everything so far is local. The repository, its history, and its branches all live in the .git directory on one machine, and if that machine is lost, so is the work. A remote is a copy of the repository hosted elsewhere, on GitHub, on GitLab, or on an institution’s own server. Connecting a local repository to a remote serves three distinct purposes that are worth separating.

The first is backup. A remote holds a complete copy of the history, so that a lost laptop costs at most the commits made since the last push. The second is distribution. A public remote is the mechanism by which a compendium becomes locatable to the world. It is a URL a reviewer can be given, a repository a reader can clone, the concrete form of the L0 rung of the ladder. The third is collaboration, the purpose that shapes the rest of this section.

The commands that connect to and synchronize with a remote are few. One names the remote once, then pushes local commits to it and pulls others’ commits from it.

# connect the local repo to a remote named origin
git remote add origin https://github.com/lab/diabetes-cohort.git

# push the main branch and remember the link
git push -u origin main

# later: send new local commits up
git push

# and bring collaborators' commits down
git pull

Consider a team working across sites, a clinical trial with several participating centers, a multi-institution consortium. Such a team works in one of two models on GitHub and GitLab. In the shared-repository model, every contributor has write access to one repository, works on a branch of it, and the main branch is protected so that nothing reaches it without review. This is the usual arrangement for a private trial repository, where forking is often disabled. In the fork-and-pull-request model, a contributor without write access forks the canonical repository, making a personal copy on the remote, and works on a branch of that fork. In either model the contributor opens a pull request (GitLab calls it a merge request, the same thing under another name) proposing that the branch be merged into main. The pull request is a unit of review. The maintainers see exactly the diff being proposed, discuss it inline, request changes, and merge it only when satisfied. In a multi-site analysis, the pull request is a documented, attributable, reviewable gate through which every change to the shared analysis passes. Its record, who proposed what, who reviewed it, what was said, and when it was merged, is held on the forge’s servers rather than asserted by a client. It therefore complements the commit history with a record of the review of each change that is harder to backdate. That record is review evidence, not by itself a regulated signature. An approval on a forge does not, by default, carry the printed name, the date and time, and the meaning of the signature that 21 CFR 11.50(a) requires of a signed electronic record (21 CFR Part 11, n.d.). Whether it counts as a controlled approval depends on the organization’s validated configuration of the forge and on its SOPs.

The same gate serves when the contributor is a coding agent. The agent works on a branch of its own and opens a pull request; the analyst reads the diff, reviews it, and alone decides whether to merge. The agent’s account of its change is a description to check against the diff, not a substitute for reading it (Section 2.10.1).

The last piece of the remote’s vocabulary is the pair tags and releases, and they matter to reproducibility specifically. A tag is a permanent, human-readable name for one commit. git tag -a v1.0-submission -m "State at journal submission" marks, forever and unambiguously, the exact state of the analysis at the moment a manuscript was submitted. A release on GitHub or GitLab is a tag dressed with notes and downloadable archives. The reproducibility payoff is that a reviewer, or the analyst a year later, can recover the precise state that produced the submitted numbers by naming the tag. There is no need to guess which commit was current on the submission date. In Chapter 13 we shall see the same tagged state deposited to a public archive with a citable identifier. The tag is the local anchor to which that archival identity attaches.

A single analyst, working alone, keeps a clean commit history with careful messages. A reviewer asks why, if the history already records each decision, a multi-site trial should bother with the additional machinery of forks and pull requests. What does the pull request record that the commit history does not?

The commit history records what changed and, in its messages, why. It does not record who reviewed the change before it entered the shared analysis, or what they said. In a solo project there is no reviewer, so nothing is lost. In a multi-site trial, a change to the shared analysis by one center affects every center’s results. There the review is itself a controlled step, and the pull request is the artifact that records it: the proposed diff, the discussion, the requested changes, the approval, and the merge, each attributable and timestamped. The commit history answers ‘what was changed and why’; the pull request answers ‘who checked it, and did they agree’. For a regulated analysis both questions have auditors.

4.11 What git does not do

Having made version control the foundation, we must be exact about the boundary of what it provides. The whole discipline of this book is matching a claim to what has actually been secured. Git versions text and tracks its history. It records every state of the files under it and the reason each committer gave for each change. That is the entirety of what it does, and it is the L0 rung of the ladder. An analysis under version control is locatable, findable and readable by a second person, and no more.

In particular, git does not pin the package versions an analysis uses. A repository cloned a year later contains the same analysis.R, but running it installs whatever package versions are current. These may not be the versions the analyst used, so the code that ran may no longer run, or may run and return a different number. Git also does not pin the computational environment. The R version, the operating system, the system libraries a package links against are nowhere in the history, and a clone is at the mercy of whatever the next machine provides. These are exactly the layers of the determinant stack that Chapter 3 enumerated as unpinned at L0. Closing them is the work of the two chapters that follow. The lockfile of Chapter 6 pins the package layer to reach L1, and the container of Chapter 7 pins the environment layer to reach L2. Version control is necessary for all of this, since the lockfile and the Dockerfile are themselves files whose history git tracks. It is sufficient for none of it.

A repository has a full, well-written commit history, a tag marking the submitted manuscript, and is hosted on GitHub. Has it reached L1?

No. Everything described pins the analysis code and makes it locatable, which is exactly L0. Reaching L1 requires a lockfile recording the exact version of every package the analysis used, and no amount of git history, however complete, records that on its own. A repository can be a model of version-control discipline and still sit at L0 if no lockfile exists.

ImportantThe honest level

This chapter’s own practice reaches L0, locatable, by placing the repository under version control and pushing it to a remote. The cohort extract is not part of that history; its generator is, and the extract is regenerated from it. The practice leaves every layer beneath the analysis code, the packages, the environment, and the rest of the determinant stack, exposed and unpinned.

4.12 Worked example

To make the chapter concrete, we take the incident-diabetes cohort of the running example from an untracked folder to a tagged, remotely hosted repository. We stay at the level of the commands and the judgment behind them. The shell sessions below are not executed when the book renders. The outputs shown with #> were captured once, on 2026-11-08, from a scratch copy of the example with a synthetic identity, and are abridged.

The analyst begins with a folder of scripts and a registry extract. The very first act, before a single commit, is to write the .gitignore, because the extract contains protected health information that must never enter the history.

cd diabetes-cohort
git init -b main
# write .gitignore FIRST, ignoring raw_data and credentials
git status --untracked-files=all   # is the extract listed?

The flag matters. Plain git status collapses a directory that holds no tracked files into one line, here analysis/, so it cannot show whether the extract inside is ignored. Run on a scratch copy of this example, the full listing names the extract before the .gitignore is written and omits it after, and git check-ignore -v names the rule responsible.

# output abridged, from git 2.56.0
# before .gitignore
git status --untracked-files=all
#> Untracked files:
#>   DESCRIPTION
#>   analysis/data/raw_data/extract.csv
#>   analysis/scripts/01-clean.R

# after .gitignore
git status --untracked-files=all
#> Untracked files:
#>   .gitignore
#>   DESCRIPTION
#>   analysis/scripts/01-clean.R

git check-ignore -v analysis/data/raw_data/extract.csv
#> .gitignore:1:analysis/data/raw_data/  analysis/data/raw_data/extract.csv

Only once git status confirms that the sensitive extract is ignored does the analyst make the first commit, the one that lifts the folder to L0, locatable. Its output names the branch, marks the commit as the root of the history, and lists the files it records.

git add .gitignore DESCRIPTION
git commit -m "Initialize compendium and .gitignore"
# output abridged, from git 2.56.0
#> [main (root-commit) 936e050] Initialize compendium and .gitignore
#>  2 files changed, 4 insertions(+)
#>  create mode 100644 .gitignore
#>  create mode 100644 DESCRIPTION

A commit that stops instead with ‘Please tell me who you are’ means git has no identity on this machine. The Git preflight sets it, and When git refuses lists this and the other refusals a first push can meet.

Work proceeds as a sequence of small, message-bearing commits: the cleaning script, the derived cohort table, the Cox model. Each is its own commit with its own reason, exactly the history tabulated in Table 4.1. When a reviewer later asks why the outlier threshold changed, the answer is one git log away.

To try a reviewer’s proposed reanalysis without disturbing the results already reported, the analyst branches.

git switch -c reviewer-sensitivity
# ... reanalysis commits ...
git switch main            # main is untouched and still runs

The work is backed up and made locatable by connecting a remote and pushing.

git remote add origin https://github.com/lab/diabetes-cohort.git
git push -u origin main
# output abridged, from git 2.56.0 (local stand-in remote)
#>  * [new branch]      main -> main
#> branch 'main' set up to track 'origin/main'.
git status
#> On branch main
#> Your branch is up to date with 'origin/main'.

The last line of the push is the one to look for: the local main now tracks the remote’s, so a later bare git push or git pull knows where to go. The scratch run pushed to a local bare repository rather than to a forge, so its output omits the forge’s own progress lines. A push that is refused instead is diagnosed from When git refuses.

A collaborator at a second site forks the repository, adds a site-specific adjustment on a branch of their fork, and opens a pull request. The analyst reviews the diff inline, requests one change, and merges it, leaving a reviewable record of the exchange. Finally, at journal submission, the analyst marks the exact state.

git tag -a v1.0-submission -m "State at journal submission"
git push origin v1.0-submission

The folder is now a version-controlled, remotely hosted, tagged repository. It is firmly at L0, with a clean history, a reviewed collaboration record, and a permanent anchor to the submitted state. It is not yet L1 or L2, because nothing has pinned the packages or the environment, and the honest reproducibility statement says exactly that. The lockfile of Chapter 6 is the next commit.

At the point the worked example ends, tagged, pushed, and with a reviewed pull request merged, what single artifact would need to be added for the compendium to reach L1, and is adding it itself a version-control operation?

A lockfile, such as renv.lock, recording the exact version of every package the analysis uses. Writing it is not a version-control operation in the sense this chapter teaches; it belongs to Chapter 6. Version control will track the lockfile once it exists, exactly as it tracks any other file, but producing it is a different act from committing, branching, or tagging.

4.13 Collaborating with an LLM

A language model is a fluent assistant for git, which is a domain rich in memorized command recipes. Precisely that fluency is the hazard. Check every command it proposes for operations beyond what was asked, irreversible ones above all; the model does not know which of the analyst’s files are sensitive. A coding agent that works in the repository can read those files, run git itself, and push. It is therefore run only in projects whose data it is allowed to see, and never with credentials that let it push to a remote unreviewed.

ImportantRestricted data

One rule comes first (habit 3). Never paste or upload protected health information, identifiable records, or data held under a data use agreement, nor outputs derived from them, into a model or a coding agent unless three conditions all hold. The institution has approved that endpoint for that class of data. The IRB protocol and the consent permit the processing. The data use agreement, or the provider’s terms, permits it. If any of the three is unclear, the answer is no.

The third condition is the one most often missed. Institutional approval of a tool does not override the terms under which a data provider released the data. NIH, for example, treats sharing controlled-access data through prompts with public generative AI tools, which it glosses as third-party tools, as a breach of the data use certification. It treats a model trained on such data as a data derivative, to be shared only with other approved users and not kept after the project closes (National Institutes of Health, 2025).

Describe the structure instead: the column names and types, or a small synthetic example generated from the data dictionary, never edited from real rows. A coding agent sees whatever files are in its working tree, so run it in a checkout or container where the real data is not mounted.

An agent also acts, so confine it to a branch and keep merges, pushes, and deposits for yourself (Section 2.10.1).

For git, the structure a prompt describes is the repository’s: its paths and file types, not the contents of the files. The three triples below mark the division of labor.

Prompt. ‘I accidentally committed a file of patient data three commits ago. Give me the git commands to remove it.’

Watch for. A history-rewriting recipe (git filter-repo, or the older filter-branch, or a reset) presented as a clean fix. The model cannot know whether the offending commit was ever pushed to a remote. That is the fact that determines whether removal is even possible, so check whether the answer raises the breach question at all.

Verification. Treat the exposure, not the git command, as the primary problem. Establish first whether the commit was pushed. If it was, assume the data has been cloned and cached and follow your institution’s incident procedure, because no git command can recall it. Only then apply a rewriting tool, on a fresh clone and in the order the five-step procedure earlier in this chapter gives, and in a repository of regulated records only as the approved, documented change its third step requires. Check that the recipe covers the forge’s caches, forks, and every collaborator’s clone. A coding agent asked the same question does not stop at a recipe: it can run the rewrite and the force-push itself. The breach assessment, the decision to rewrite, and the force-push stay with the analyst. At most the agent prepares the commands on a fresh clone, for the analyst to read and run (Section 2.10.1).

Prompt. ‘Write a .gitignore for my R health-data analysis project.’

Watch for. A generic R .gitignore (the standard .Rhistory, .RData, .Rproj.user/ list) that omits the one category that matters most, the raw data and credentials specific to your own project. A chat model has not seen your directory. A coding agent may have listed it, but a file’s name does not say whether it holds PHI. An agent that opened the files to find out has already read the data, which is why it belongs only in projects it is cleared to see.

Verification. Take the generic list as a starting point, then add, by hand, the explicit paths of every raw-data directory, every .Renviron or credential file, and every restricted dataset. Confirm with git status that each is untracked before the first commit. The generic patterns are safe to accept; the project-specific sensitive paths are yours to supply.

A better prompt. ‘Below is the directory tree of my R analysis compendium, file names only, with no file opened. Write a .gitignore that excludes analysis/data/raw_data/, .Renviron and any other credential file, and regenerable outputs, but keeps the code, renv.lock, and the synthetic cohort tracked. Label each line generic or project-specific, and list the commands that would show me each sensitive path is ignored.’

Prompt. ‘Write good commit messages for these changes.’

Watch for. Messages that fluently describe what changed, which the model can read from the diff, but invent or omit the why. The why is the scientifically load-bearing part, and the model cannot know it. A plausible-sounding ‘Update outlier threshold’ hides the reviewer request that the record must carry.

Verification. Supply the reason yourself. Let the model draft the imperative summary line from the diff, then edit in the motivation, the reviewer comment, the corrected denominator, the protocol amendment, that turns a description into an accountable record.

4.14 Exercises

  1. Take a folder of your own currently versioned by the FINAL-file method, count the manual snapshot files it contains, and for two of them state what changed between them and why. Note which of those two questions you can still answer, and use the difficulty as a measure of what a commit history would have preserved.

  2. Initialize a git repository in a scratch directory, make three commits that each change one file, and then use git log --oneline and git show to recover, for the middle commit, both what changed and the message you wrote. Confirm that the commit hashes are unchanged by a later commit.

  3. Write a .gitignore for a data-analysis compendium and a second for an R-package compendium, and list the patterns that appear in one but not the other. Explain each difference in terms of what the archetype produces.

  4. Explain, to a colleague who has just deleted a PHI-containing file in a new commit and believes the problem solved, why the file is still in the history, what determines whether it can be removed, and what to do if the offending commit was already pushed to GitHub.

  5. Construct the sequence of commands by which a collaborator at a second trial site would contribute a change through a fork and a pull request, and state, for each step, what part of the project’s record it adds that a direct push to main would not.

  6. An analyst pushes a version-controlled compendium to GitHub and writes in the methods section that the analysis is ‘fully reproducible’. State the rung the analysis has actually reached, name the two layers of the determinant stack that remain unpinned, and rewrite the sentence so that it claims exactly what version control secures and no more.

  7. In a fresh clone of this book’s repository, run Rscript analysis/scripts/simulate_cohort.R from the top directory, then raw <- load_cohort('analysis/data/raw_data/cohort_raw.csv'). Report nrow(raw) and the three columns among names(raw) that would, in a real project, require the extract to sit behind the .gitignore pattern analysis/data/raw_data/*, and use git check-ignore -v analysis/data/raw_data/cohort_raw.csv to name the line that excludes it. Then run git log --all --oneline -- inst/testdata/cohort_raw.csv and state what the oldest commit it lists shows about whether that pattern kept the identifier columns out of the history.

  8. This exercise returns to the determinant stack and the L0-to-L3 ladder of Chapter 3 (Chapter 3). A colleague deletes the .git directory of a well-tested, fully containerized analysis and runs git init again to ‘start clean’ before pushing, so the repository now hosted on GitHub shows a single commit from yesterday. Using what this chapter and Chapter 3 together establish, state the rung the repository occupies today, and explain what the colleague’s own prior discipline, the lockfile and the container, can and cannot restore of what the deleted history held.

4.15 Solutions

Attempt an exercise before expanding its solution. The first two ask you to work in your own directories, so what you produce will vary; the boxes say what a good answer establishes.

No single correct answer. What a good answer establishes is the asymmetry between the two questions, because the asymmetry is the argument for version control.

Almost every reader can answer what changed between two snapshot files, because the files are still there and a diff tool will show it. Almost no reader can answer why, because the reason was never written down anywhere. It lived in the analyst’s head, or in an email, or in a conversation after a meeting, and it is gone. Readers commonly find they can reconstruct the what for files a year old and cannot reconstruct the why for files a month old.

That gap is precisely what a commit history preserves. The diff is recoverable from the files themselves, so it is the cheaper half; the message is not recoverable from anything, so it is the half that decays. This is why the chapter insists that a commit is an atomic unit of one idea with a message attached. The message is the only part of the record that cannot be reconstructed later.

If you find that you can answer why, because you named the files carefully, you have demonstrated the FINAL method working at its best. Notice, then, how much of the reasoning fits in a filename.

The mechanics are straightforward, so the substance is the last clause.

git init
# edit file_a.txt
git add file_a.txt && git commit -m 'Add the cohort filter'
# edit file_b.txt
git add file_b.txt && git commit -m 'Raise the outlier threshold'
# edit file_c.txt
git add file_c.txt && git commit -m 'Add the summary table'
git log --oneline
git show <middle-hash>

git log --oneline lists the three commits newest first, each with an abbreviated hash and its subject line. git show on the middle hash prints both halves of the record: the message you wrote, and the diff of what changed.

On the hashes: recording the middle commit’s hash before making the third commit and comparing afterward shows it unchanged. A commit hash is computed over the commit’s content and its parent’s hash, so it fixes the entire history leading to it. Adding a commit on top cannot alter it, because the new commit points backward and nothing points forward. That is the property that makes the history tamper-evident to anyone who recorded a hash.

The honest qualification, which a careful reader may raise: hashes are stable under adding history, not under rewriting it. An amend or a rebase produces new commits with new hashes, and the old ones become unreferenced. A rewrite is detectable only by someone who kept an old hash, and the author and dates inside each commit were set by the committer’s own client in the first place. The record is trustworthy for a history that is appended to and whose hashes are held elsewhere, in tags, in collaborators’ clones, on a protected forge. That is why rewriting shared history is treated as a serious act.

Your own files will differ in detail. Taking the two samples in this chapter as the model answer, they barely overlap, and the difference follows from what each archetype produces.

Common to both: the editor and operating-system cruft, .Rhistory, .RData, and .DS_Store.

In the data-analysis file but not the package one: the raw-data extract, analysis/data/raw_data/*, which carries the protected information; the regenerable outputs, analysis/figures/*, analysis/tables/*, and the rendered *.pdf and *.html reports, since the report is a product the code regenerates; and .Renviron, *.pem, and .Rproj.user/.

In the R-package file but not the analysis one: the build artifacts, *.tar.gz and *.Rcheck/; the compiled object files src/*.o and src/*.so, which exist only if the package carries compiled code and are rebuilt on every install; and docs/, the generated pkgdown site.

The last three analysis-only lines are differences of the abridged samples, not of the archetypes. Credentials and editor state can appear in any project, so a package repository should ignore .Renviron, *.pem, and .Rproj.user/ as well. A good answer says so.

The explanation to give for each remaining difference has the same shape. An archetype ignores what it generates and commits what it is. A data-analysis compendium generates a report and so ignores it. A package generates a tarball, check output, compiled objects, and a website, and so ignores those. Neither ignores its own source.

The one line that is not a matter of archetype is the data carrying protected health information. It is ignored in every archetype, always, and is the subject of the next exercise.

The colleague’s mental model is that a repository holds the current state of the files. It holds their whole history, and that is the point of it.

Why the file is still there. Deleting a file in a new commit records the deletion as a change. The commit that added the file is untouched, and the file’s contents are retrievable from it by anyone who clones the repository. The deletion is a new fact appended to the record, not an erasure of the old one.

What determines whether it can be removed. Whether the history has been shared. Removing content means rewriting every commit from the offending one forward, with git filter-repo or an equivalent, which produces new hashes for all of them. If the offending commit exists only in the local repository, this is a local operation and costs nothing but care. If it has been pushed, every clone and fork holds the old objects, and rewriting the local copy does not touch them.

What to do if it was already pushed to GitHub. Treat the data as disclosed and act in that order. Notify whoever is responsible for the data, an IRB or a privacy office, because a disclosure has occurred and the notification obligation is not a technical question. Rotate anything rotatable. Then rewrite the history and force-push, and ask GitHub Support to purge cached views and any dangling objects. Require every collaborator to re-clone rather than pull, since a pull onto a rewritten history reintroduces the old objects. Note the ordering: the cleanup is the last step and the least important one, because it does not undo the exposure.

This is why the .gitignore for protected data is written before the first commit rather than after the first mistake.

A serviceable sequence, with what each step adds to the record.

# 1. Fork the canonical repository on the forge.
git clone https://github.com/<site2>/<study>.git
cd <study>
git remote add upstream https://github.com/<coord>/<study>.git

# 2. Branch from an up-to-date main.
git fetch upstream
git switch -c fix/outlier-threshold upstream/main

# 3. Commit the change.
git add R/clean_cohort.R
git commit -m 'Raise outlier threshold to 4 SD per SAP amendment 2'

# 4. Push to the fork and open a pull request.
git push origin fix/outlier-threshold

What each step records that a direct push would not. The fork lets a contributor without write access to the canonical analysis propose a change, which then cannot enter without someone else’s action. (A fork does not show that the contributor lacked write access; anyone may fork. On a private trial repository with forking disabled, a branch of the shared repository with a protected main plays the same role.) The branch separates the proposal from the accepted state, so the history of main contains only what was agreed. The commit is the same in either workflow and records what and why. The pull request is the artifact that carries everything a direct push destroys. It holds the proposed diff before it was accepted, the discussion, any requested changes and the revisions answering them, the identity of the approver, and the timestamp of the approval. The merge records that the approval preceded the entry into main.

The compressed version of the answer: the commit history answers ‘what changed and why’; the pull request answers ‘who checked it, and did they agree’. A direct push to main answers only the first, and in a multi-site trial the second question has auditors too. In a regulated trial, the approval is review evidence rather than, by itself, a signed record carrying the printed name, the date and time, and the meaning that 21 CFR 11.50(a) requires; whether it serves as one depends on the sponsor’s validated configuration and SOPs. The analysis change behind the commit is itself approved through the sponsor’s own process, here an amendment to the statistical analysis plan, not through the pull request.

The rung reached is L0. Version control makes the analysis locatable and gives it a history. It pins the analysis code, which is a real achievement and the precondition for everything above it. However, the ladder grades how far down the determinant stack a result is locked, and the code sits at the top of that stack.

Two layers that remain unpinned. The first is the package versions, since nothing in a git repository records which version of dplyr produced the numbers. The second is the environment beneath them, the R runtime, the operating system, and the system libraries. You may equally name the numerical libraries or the session configuration and be right, since the repository touches neither. The common wrong answer is to name the data, which version control can pin for a small committed dataset, though usually does not for a large or protected one.

A rewritten sentence. ‘The analysis code is version controlled and deposited at [location], so the exact code that produced these results can be retrieved. The software environment was not recorded; re-execution may therefore require package versions differing from those used here, and we have not verified that the results regenerate.’

The three clauses map onto the three claims: what is secured, what is not, and what has not been checked. The last clause is the one authors most often omit, and it is the difference between claiming L0 honestly and claiming L3 by implication.

The solution for this exercise is in the instructor guide, which is distributed separately to course instructors. Self-study readers who want to check their answer can compare it with the executed output of the cohort checks in the .gitignore section of this chapter: the row count and identifier columns of the generated extract, the directory pattern, and the history of inst/testdata/cohort_raw.csv. The point of the last step is that git check-ignore succeeds on the extract while the oldest copy of the fixture, on the render of this page, still carries the identifier columns: the pattern held at its own path, and the copy entered the history by another.

The rung is L2, unchanged by the deletion. The repository that is hosted on GitHub today is still locatable, so L0 holds and a reviewer can find and read it. This is so even though its visible history begins with yesterday’s single commit. Nothing about git init run fresh touches the lockfile or the container. Those files, if they still sit in the working tree, are tracked again by the new history and hold the analysis at L1 and L2 exactly as before. Package pinning and environment pinning are properties of files, not of the commit chain that records them. If the analysis had reached L3, its baseline test is a file too and survives, but the CI runs that showed it passing were attached to commits that no longer exist. L3 is reached only by verification, so it must be shown again by a fresh run of that test after a clean rebuild.

What is lost, and what is not. The lockfile and the container restore the analysis’s position on the capture ladder, L1 and L2. renv.lock and the Dockerfile are ordinary files that a fresh git init tracks exactly as well as the old history did. What they cannot restore is the history itself: the sequence of decisions, the reviewer-driven changes, the dated and attributed record Table 4.1 exemplifies, and any tag that marked a past submission. That record is not a determinant of the computational result at all, so no rung of the L0 to L3 ladder grades its loss. It is a loss of scientific provenance and accountability, which is this chapter’s subject rather than the ladder’s. What the colleague erased is the one part of the record a baseline test or a container rebuild was never designed to protect.

4.16 Cheat sheet

Command or rule What it does
git init -b main Starts tracking the current directory as a repository, with the first branch named main.
git add <file> Stages a change for the next commit.
git commit -m '...' Records the staged changes with a message.
git status Shows staged, unstaged, and untracked files; read it before every commit (habit 3).
git log --oneline Lists commits, most recent first, by hash and message.
git show <hash> Displays one commit’s full diff.
git switch -c <name> Creates and switches to a new branch.
git remote add origin <url>; git push -u origin main Connects to, and first pushes to, a remote.
git tag -a v1.0 -m '...' Marks one commit permanently, e.g. at submission (Chapter 13 anchors to this).
!pattern in .gitignore Re-includes a path otherwise ignored, e.g. a data dictionary beside an ignored extract.
git check-ignore -v <path> Names the .gitignore rule that ignores a path, or prints nothing if none does.
git rm --cached <file> Untracks a file already committed; does not erase it from the history.
Git refuses Look up the message in When git refuses; run the Git preflight once per machine.
git filter-repo --sensitive-data-removal Rewrites every commit to remove a file; only on a fresh clone, only before re-pushing. Where records are regulated, only as an approved, documented controlled change.
Rule of thumb PHI, credentials, and restricted data are never committed; deleting them later does not remove them from the history.
Rule of thumb Version control reaches L0 only; it pins the analysis code, not the packages or the environment.
Habit 1 Commit before you compute.

4.17 Further reading

  • Ram (2013) is the direct argument for git as an instrument of reproducibility and transparency in science. It is the natural first reading for a researcher who has met git only as a software tool, and it makes the case for the commit history as a record of scientific provenance.
  • Bryan (2018) is the standard orientation to version control written for data scientists rather than software engineers. It motivates GitHub as the collaboration substrate with the working analyst’s concerns foremost. The longer online book by Bryan and colleagues, Happy Git and GitHub for the useR (Bryan et al., n.d.), is the companion practical manual. The setup appendix points to its chapters on installing git, identity, and credentials.
  • Wilson et al. (2017) and Sandve et al. (2013) place version control among the good practices of reproducible computational research. The former treats it as one of a set of habits for scientific computing, and the latter places it among its rules for reproducible analysis. Read them to see version control situated alongside the other pillars this book develops.
  • The layers that version control does not pin are the subject of Chapter 6 and Chapter 7. The continuous integration and collaboration built upon a git history are developed in Chapter 12 and in Chapter 13; this chapter is best read as the foundation those four presuppose.