Skip to content

Add mktestdocs to catch docs/code drift#1096

Open
cianc wants to merge 1 commit intomasterfrom
test/add-mktestdocs
Open

Add mktestdocs to catch docs/code drift#1096
cianc wants to merge 1 commit intomasterfrom
test/add-mktestdocs

Conversation

@cianc
Copy link
Contributor

@cianc cianc commented Mar 3, 2026

Description

Adds automated testing of Python code examples in the documentation to prevent examples from drifting out of sync with the library.

  • Add mktestdocs and pytest to the doc dependency group in pyproject.toml so they are available alongside the other doc-build tools without pulling in the full dev group.

  • Add scripts/check-docs-drift.py: a pytest-based script that uses mktestdocs.grab_code_blocks() to collect every ```python fenced block under docs/, skips any block whose first line is `# skip`, and executes the rest via `exec_python()`. A new taskipy task `docs-check-drift` runs it with `pytest scripts/check-docs-drift.py -v`.

  • Fix all ```python code blocks across docs/ so they are correctly picked up by mktestdocs:

    • Remove the stray space in python fences (changed topython) so that mktestdocs can identify them (it matches on the exact string "python" immediately after the backticks).
    • Add save_to_file=False, log_level="error" to EmissionsTracker and OfflineEmissionsTracker instantiations to avoid creating CSV files or noisy output during CI runs.
    • Add # skip as the first line of blocks that cannot run in CI because they depend on external services or optional heavy dependencies (TensorFlow, Prometheus, Logfire, Google Cloud, Comet ML, live CodeCarbon API).
    • Correct a pip install command that was incorrectly fenced as python in `comet.md`; changed to console.
  • Update .github/workflows/build-docs.yml to run docs-check-drift as a step before the docs build, triggered on changes to docs/**, mkdocs.yml, or scripts/check-docs-drift.py.

  • Document the drift check and the # skip convention in CONTRIBUTING.md under the "Build Documentation" section.

Related Issue

Please link to the issue this PR resolves: #1083

Motivation and Context

Helps prevent drift of code blocks in documentation

How Has This Been Tested?

Ran all tests including new mktestdocs tests.

Screenshots (if appropriate):

Types of changes

What types of changes does your code introduce? Put an x in all the boxes that apply:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Go over all the following points, and put an x in all the boxes that apply.

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING.md document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

Adds automated testing of Python code examples in the documentation
to prevent examples from drifting out of sync with the library.

Changes:

- Add `mktestdocs` and `pytest` to the `doc` dependency group in
  `pyproject.toml` so they are available alongside the other doc-build
  tools without pulling in the full `dev` group.

- Add `scripts/check-docs-drift.py`: a pytest-based script that uses
  `mktestdocs.grab_code_blocks()` to collect every ```python fenced
  block under `docs/`, skips any block whose first line is `# skip`,
  and executes the rest via `exec_python()`. A new taskipy task
  `docs-check-drift` runs it with `pytest scripts/check-docs-drift.py -v`.

- Fix all ```python code blocks across `docs/` so they are correctly
  picked up by mktestdocs:
  - Remove the stray space in ``` python fences (changed to ```python)
    so that mktestdocs can identify them (it matches on the exact string
    "python" immediately after the backticks).
  - Add `save_to_file=False, log_level="error"` to `EmissionsTracker`
    and `OfflineEmissionsTracker` instantiations to avoid creating CSV
    files or noisy output during CI runs.
  - Add `# skip` as the first line of blocks that cannot run in CI
    because they depend on external services or optional heavy
    dependencies (TensorFlow, Prometheus, Logfire, Google Cloud,
    Comet ML, live CodeCarbon API).
  - Correct a `pip install` command that was incorrectly fenced as
    ```python in `comet.md`; changed to ```console.

- Update `.github/workflows/build-docs.yml` to run `docs-check-drift`
  as a step before the docs build, triggered on changes to `docs/**`,
  `mkdocs.yml`, or `scripts/check-docs-drift.py`.

- Document the drift check and the `# skip` convention in
  `CONTRIBUTING.md` under the "Build Documentation" section.
@cianc cianc marked this pull request as ready for review March 3, 2026 21:51
@cianc cianc requested a review from a team as a code owner March 3, 2026 21:51
@codecov
Copy link

codecov bot commented Mar 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.22%. Comparing base (9be333a) to head (9057de3).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1096   +/-   ##
=======================================
  Coverage   78.22%   78.22%           
=======================================
  Files          38       38           
  Lines        3632     3632           
=======================================
  Hits         2841     2841           
  Misses        791      791           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add docstring examples and tests for mkdocstrings and mktestdocs

1 participant