notebook-ci-pipeline-author

Composes the three notebook testing tools into one GitHub Actions pipeline: papermill executes parameterized notebooks, nbval validates output regression, testbook runs function-level unit tests, and nbstripout gates committed output. Per-tool depth lives in references/papermill.md (parameterized execution) and references/nbval.md (output regression); this SKILL.md covers the wiring and integration decisions, plus the PR review checklist for notebook changes.

When to use

Teams using all three tools but assembling the pipeline by hand: no consistent artifact naming, no shared caching, duplicate install steps, no HTML report on failure. Also: any PR that adds or modifies .ipynb files and needs the review checklist below.

Hard-reject conditions

Do not proceed if any of the following apply:

• The repository has no parameters-tagged notebook cell. Papermill injection silently uses notebook defaults when the tag is absent per the Papermill execute docs; the pipeline will report green against stale values.
• nbval ground-truth outputs are absent from the committed .ipynb files. Re-running cells with no stored outputs produces no diff, so regressions are invisible.
• testbook tests use execute=True per-function without a scope="module" fixture. Each test re-executes the kernel; CI timeouts follow per the testbook docs.

State the blocker to the user and stop.

Step 1 - Install nbstripout as a pre-commit filter

Install once per clone so committed notebooks carry no output noise per the nbstripout README:

pip install nbstripout
nbstripout --install                    # writes .git/config filter entry
nbstripout --install --attributes .gitattributes  # repo-wide via .gitattributes

Add to .gitattributes:

*.ipynb filter=nbstripout

For pull-request verification without modifying files, use the kynan/nbstripout action:

- name: Verify notebooks are stripped
  uses: kynan/nbstripout@main
  with:
    paths: '**/*.ipynb'

The action runs a dry-run check and fails if any notebook carries uncommitted output.

Step 2 - Install dependencies with pip caching

Per GitHub Actions: Building and Testing Python, the setup-python action accepts cache: 'pip' and locates requirements.txt automatically:

- uses: actions/setup-python@v5
  with:
    python-version: '3.11'
    cache: 'pip'

- name: Install dependencies
  run: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt
    pip install papermill nbval pytest testbook nbconvert

Keep papermill nbval pytest testbook nbconvert pinned in requirements.txt so the cache key (hashFiles('**/requirements.txt')) reflects version changes.

Step 3 - Stage 1: papermill parameterized execution

Papermill executes the notebook with injected parameters and writes a fully-rendered output notebook:

- name: Execute notebook (papermill)
  run: |
    papermill notebooks/analysis.ipynb \
      artifacts/analysis-executed.ipynb \
      -p seed 42 \
      -p n_samples 1000

Use -p for numeric/boolean parameters and -r for string parameters to prevent type-coercion surprises per the Papermill execute docs. Full papermill depth - parameter flags, Python API, matrix sweeps, regression-test wiring - is in references/papermill.md. Store the output path (artifacts/analysis-executed.ipynb) in an env var shared across stages:

env:
  EXECUTED_NB: artifacts/analysis-executed.ipynb

Step 4 - Stage 2: nbval output regression

Run nbval in lax mode on the executed notebook. Strict mode fails on every non-deterministic output; lax mode fails only on errors unless cells carry #NBVAL_CHECK_OUTPUT per the nbval docs:

- name: Output regression (nbval-lax)
  run: |
    pytest --nbval-lax $EXECUTED_NB \
      --sanitize-with sanitize.cfg \
      -v

sanitize.cfg example for timestamps and memory addresses:

[regex1]
regex: \d{1,2}/\d{1,2}/\d{2,4}
replace: DATE-STAMP

[regex2]
regex: 0x[0-9a-fA-F]+
replace: MEMORY-ADDR

Pin per-cell markers on cells that emit timestamps or large floats: # NBVAL_IGNORE_OUTPUT. Use # NBVAL_RAISES_EXCEPTION to validate expected error paths. Full nbval depth - strict vs lax mode, all per-cell controls, sanitize patterns, discovery - is in references/nbval.md.

Step 5 - Stage 3: testbook function unit tests

Run testbook tests against the source notebook (not the executed artifact) using a module-scoped fixture so the kernel executes once per pytest session per the testbook docs:

- name: Unit tests (testbook)
  run: pytest tests/test_notebook_functions.py -v

The scope="module" fixture is the load-bearing wiring decision - it stops each test re-executing the kernel:

@pytest.fixture(scope="module")
def tb():
    with testbook("notebooks/analysis.ipynb", execute=True) as tb:
        yield tb

The full tests/test_notebook_functions.py, with per-function tb.ref() assertions, is in references/notebook-ci-pipeline.md.

Step 6 - Stage 4: HTML report via nbconvert

Convert the executed notebook to a self-contained HTML report per the nbconvert docs:

- name: Convert to HTML
  if: always()
  run: |
    jupyter nbconvert --to html \
      --template lab \
      --embed-images \
      $EXECUTED_NB \
      --output artifacts/analysis-report.html

if: always() per GitHub Actions expressions ensures the report generates even when nbval or testbook failed; the HTML is the primary debugging artifact.

Step 7 - Artifact upload with failure-aware retention

Upload both the executed notebook and the HTML report. Use if: always() so artifacts surface on failure per actions/upload-artifact@v4:

- name: Upload artifacts
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: notebook-ci-${{ github.run_id }}
    path: |
      artifacts/analysis-executed.ipynb
      artifacts/analysis-report.html
    if-no-files-found: warn
    retention-days: 14

Set retention-days within the 1-90 day range allowed by actions/upload-artifact@v4; 14 days covers sprint cycles without excessive storage.

Step 8 - Complete workflow

Steps 1-7 assemble into one workflow file. The full assembled YAML is in references/notebook-ci-pipeline.md; paste it to .github/workflows/notebook-ci.yml and adjust the notebook path, papermill parameters, and test path to match the repo.

Step 9 - Review checklist for notebook PRs

When a PR adds or modifies .ipynb files, walk each notebook through six checks and emit a finding table with a verdict. Every finding must trace to an observable file pattern or a cited source, never intuition.

1. Untested notebooks. Cross-reference conftest.py, pytest.ini, CI workflow YAML, and tests/. Each notebook needs at least one of: an --nbval / --nbval-lax CI step, a testbook test file referencing it, or a papermill execution step. No coverage = UNTESTED.
2. --nbval-lax misuse. --nbval-lax "collects notebooks and runs them, failing if there is an error" but skips output comparison unless cells carry #NBVAL_CHECK_OUTPUT (nbval docs). Flag when the notebook is a regression test (its purpose is validating outputs) AND no cell carries the marker; fix is strict --nbval or adding markers to load-bearing cells.
3. Hardcoded credentials. Scan source cells for API-key assignments (api_key =, token =, password =, secret =), connection strings (://user:pass@), and long base64 blobs in string literals - and scan cell outputs too (secrets echoed in stream / execute_result blocks are equally dangerous and often missed). A match is a match; do not judge intent.
4. Non-deterministic output cells. Stored outputs containing timestamps, memory addresses, UUIDs, or floats with > 4 significant figures, with no #NBVAL_IGNORE_OUTPUT marker and no --sanitize-with config in CI, produce spurious diffs or false failures.
5. parameters tag. For papermill-executed notebooks, exactly one cell must carry the parameters tag; with no tag, papermill inserts injected-parameters at the top, potentially before imports (Papermill parameterize docs). Flag zero tags (MISSING-PARAMS-TAG) and multiple tags (AMBIGUOUS-PARAMS-TAG).
6. Committed outputs. Any code cell with a non-empty outputs array, in a repo without an nbstripout hook, bloats diffs and causes merge conflicts per nbstripout README - flag COMMITTED-OUTPUTS. With the hook configured, the flag means the PR bypassed it.

Severity: BLOCK = credentials, untested; WARN = lax misuse, committed outputs, non-deterministic outputs; INFO = params-tag findings. Verdict is BLOCK if any BLOCK-severity finding is present, PASS otherwise.

Anti-patterns

Anti-pattern	Why it fails	Fix
Run nbval on the source notebook before papermill	nbval re-executes from scratch; parameter injection never happens	Run nbval on the papermill output notebook (Stage 2)
Run testbook tests against the executed artifact	testbook needs the source notebook to resolve cell tags; .ipynb with injected-parameters cell confuses selective execution	Point testbook at the source notebook, not the artifact
Omit nbstripout --install from onboarding	Developers commit outputs; nbval diffs against stale ground truth in CI	Document nbstripout --install in CONTRIBUTING.md; enforce via the kynan/nbstripout action (Step 1)
Upload artifacts only on success	Failures produce no HTML; engineers cannot inspect which cell errored	Use if: always() on the convert and upload steps (Steps 6-7)
Module-scope fixture missing from testbook tests	Each test re-executes the full notebook kernel; multi-minute CI runs per test	Add @pytest.fixture(scope="module") (Step 5)

Limitations

• Papermill, nbval, and testbook each spawn a separate kernel session. For very large notebooks (> 5 min execution), total CI time triples. Use selective cell execution in testbook (pass a list of tags to execute) and split long notebooks into composable units.
• nbval and testbook conflict when run against the same notebook in the same pytest session per the testbook docs. This workflow avoids the conflict by pointing each tool at a different target (nbval on the executed artifact, testbook on the source).
• The kynan/nbstripout action checks output presence but does not enforce kernel metadata stripping; add --extra-keys flags locally if kernel version metadata causes diff noise.

References

• references/papermill.md - papermill in tool depth: parameters tag, CLI + Python API, sweeps, regression wiring
• references/nbval.md - nbval in tool depth: strict/lax, per-cell controls, sanitize config, discovery
• Papermill execute docs - Python API, CLI flags, parameter types, papermill_description TQDM integration
• nbval docs - strict/lax modes, per-cell markers, sanitize config
• testbook docs - @testbook decorator, tb.ref(), tb.inject(), tb.patch(), module-scoped fixture
• nbstripout README - install, git filter, --install --attributes, --verify flag
• kynan/nbstripout action - GitHub Actions dry-run check
• nbconvert docs - --to html, --template, --embed-images flags
• GitHub Actions: Building and Testing Python - setup-python cache: 'pip' pattern
• actions/upload-artifact@v4 - name, path, if-no-files-found, retention-days parameters
• GitHub Actions expressions - always() and failure() status check functions in step if conditions