Open
Conversation
Adds automated testing of Python code examples in the documentation
to prevent examples from drifting out of sync with the library.
Changes:
- Add `mktestdocs` and `pytest` to the `doc` dependency group in
`pyproject.toml` so they are available alongside the other doc-build
tools without pulling in the full `dev` group.
- Add `scripts/check-docs-drift.py`: a pytest-based script that uses
`mktestdocs.grab_code_blocks()` to collect every ```python fenced
block under `docs/`, skips any block whose first line is `# skip`,
and executes the rest via `exec_python()`. A new taskipy task
`docs-check-drift` runs it with `pytest scripts/check-docs-drift.py -v`.
- Fix all ```python code blocks across `docs/` so they are correctly
picked up by mktestdocs:
- Remove the stray space in ``` python fences (changed to ```python)
so that mktestdocs can identify them (it matches on the exact string
"python" immediately after the backticks).
- Add `save_to_file=False, log_level="error"` to `EmissionsTracker`
and `OfflineEmissionsTracker` instantiations to avoid creating CSV
files or noisy output during CI runs.
- Add `# skip` as the first line of blocks that cannot run in CI
because they depend on external services or optional heavy
dependencies (TensorFlow, Prometheus, Logfire, Google Cloud,
Comet ML, live CodeCarbon API).
- Correct a `pip install` command that was incorrectly fenced as
```python in `comet.md`; changed to ```console.
- Update `.github/workflows/build-docs.yml` to run `docs-check-drift`
as a step before the docs build, triggered on changes to `docs/**`,
`mkdocs.yml`, or `scripts/check-docs-drift.py`.
- Document the drift check and the `# skip` convention in
`CONTRIBUTING.md` under the "Build Documentation" section.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1096 +/- ##
=======================================
Coverage 78.22% 78.22%
=======================================
Files 38 38
Lines 3632 3632
=======================================
Hits 2841 2841
Misses 791 791 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds automated testing of Python code examples in the documentation to prevent examples from drifting out of sync with the library.
Add
mktestdocsandpytestto thedocdependency group inpyproject.tomlso they are available alongside the other doc-build tools without pulling in the fulldevgroup.Add
scripts/check-docs-drift.py: a pytest-based script that usesmktestdocs.grab_code_blocks()to collect every ```python fenced block underdocs/, skips any block whose first line is `# skip`, and executes the rest via `exec_python()`. A new taskipy task `docs-check-drift` runs it with `pytest scripts/check-docs-drift.py -v`.Fix all ```python code blocks across
docs/so they are correctly picked up by mktestdocs:python fences (changed topython) so that mktestdocs can identify them (it matches on the exact string "python" immediately after the backticks).save_to_file=False, log_level="error"toEmissionsTrackerandOfflineEmissionsTrackerinstantiations to avoid creating CSV files or noisy output during CI runs.# skipas the first line of blocks that cannot run in CI because they depend on external services or optional heavy dependencies (TensorFlow, Prometheus, Logfire, Google Cloud, Comet ML, live CodeCarbon API).pip installcommand that was incorrectly fenced aspython in `comet.md`; changed toconsole.Update
.github/workflows/build-docs.ymlto rundocs-check-driftas a step before the docs build, triggered on changes todocs/**,mkdocs.yml, orscripts/check-docs-drift.py.Document the drift check and the
# skipconvention inCONTRIBUTING.mdunder the "Build Documentation" section.Related Issue
Please link to the issue this PR resolves: #1083
Motivation and Context
Helps prevent drift of code blocks in documentation
How Has This Been Tested?
Ran all tests including new mktestdocs tests.
Screenshots (if appropriate):
Types of changes
What types of changes does your code introduce? Put an
xin all the boxes that apply:Checklist:
Go over all the following points, and put an
xin all the boxes that apply.