Python Build System by CavRiley · Pull Request #302 · InsightSoftwareConsortium/ITKPythonPackage

CavRiley · 2026-03-29T01:46:27Z

Python-based build system for generating Python wheels

This refactoring transitions build logic entirely to Python scripts while maintaining the shell script entry points for cached builds either locally or in the GitHub Action environment. A couple of the build improvements are outlined below:

Consolidated build dependencies into pixi environments. A pixi.toml manifest replaces requirements-dev.txt and implicit system tool assumptions. All dependencies are specified with version constraints and composed into per-platform, per-Python-version environments (e.g. manylinux228-py310, macosx-py311).
Uses a Python class hierarchy: BuildPythonInstanceBase defines the shared build pipeline, with LinuxBuildPythonInstance, MacOSBuildPythonInstance, and WindowsBuildPythonInstance overriding only platform-specific behavior (wheel repair, environment setup, tarball creation).

Some developer improvements:

pre-commit hooks, linters/formatters/shellchecks (run locally and via GitHub actions)
GitHub actions workflow for generating sphinx documentation
Issue/PR templates
Moves to conventional commits (this can be changed if the format should not be changed)
Adds scripts to publish wheels and cached ITK builds

Migration Guide for GitHub Actions

To migrate the GitHub actions work flows to use this new build infrastructure, new caches builds need to be created with this build system (build paths and names changed to be consistent across environment).

Once the cache files are uploaded to the ITKPythonBuilds repository the actions can use the specific Git tag the caches are uploaded under. To use this infrastructure you would also need to specify the branch this addition would go under. For example:

uses: InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction/.github/workflows/build-test-package-python.yml@main
    with:
      itk-wheel-tag: 'v6.0b02.post1' # an example tag
      itk-python-package-tag: python-build-system # an example branch

Backwards compatibility

The *-download-cache-and-build-module-wheels.sh / .ps1 entry points are preserved so existing remote module CI configurations and ITKRemoteModuleBuildTestPackageAction workflows continue to work without modification (assuming the build uses the new cache)
Environment variables (ITK_PACKAGE_VERSION, CMAKE_OPTIONS, ITK_MODULE_PREQ, MANYLINUX_VERSION, etc.) remain the same and are still accepted.

I'd appreciate any feedback you could provide regarding improvements or issues

CavRiley · 2026-03-29T02:08:19Z

Also may close #127

dzenanz

I had a somewhat quick look.

docs/Quick_start_guide.rst

thewtex · 2026-03-31T21:23:34Z

@CavRiley amazing effort!! 🥇 🔥

With a cursory review, I noted that we do not need to support manylinux2014 for ITK 6.

A few preliminary questions (I have not taken a detailed look):

Does it use pixi-build? Could we use pixi-build to build / re-use a C++ package? The motivation is to reduce build times and improve itk conda / conda-forge package compatibility.
Can we make it re-use the itk C++ conda-forge package?
Are the Linux packages compatible with pypi/manylinux?

See https://www.perplexity.ai/search/we-have-a-python-package-itk-t-MLAY4xnZRCeU9SGHhyJHQQ

CavRiley · 2026-04-01T15:15:37Z

Thank you, @thewtex!

With a cursory review, I noted that we do not need to support manylinux2014 for ITK 6.

Okay, I will remove this!

Does it use pixi-build? Could we use pixi-build to build / re-use a C++ package? The motivation is to reduce build times and improve itk conda / conda-forge package compatibility.

This does not use pixi-build, but I think it would be possible to use pixi-build to build the remote modules if not the base ITK modules as well. However, I would need to investigate this further to confirm it can handle ITK's CMake/SWIG wrapping.

Can we make it re-use the itk C++ conda-forge package?

Possibly, but I think the itk C++ conda-forge package would need to be built with the python wrapping (I'm not sure if this is currently the case) in order for it to be re-used here.

Are the Linux packages compatible with pypi/manylinux?

The linux-py* pixi environments produce wheels tagged linux_x86_64 which I don't think are uploadable to PyPI. For PyPI-compatible wheels, the manylinux228-py* environments with dockcross containers do make the wheels compatible with pypi/manylinux.

@thewtex

manylinux2014 is not needed for ITK 6. Remove the manylinux2014 image tag resolution, the manylinux1/manylinux2014 pixi-to-pattern renaming entries, and update shell script comments to reference _2_28 and _2_34 as the supported manylinux versions. Addresses review feedback from @thewtex on PR #302. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@thewtex

manylinux2014 is not needed for ITK 6. Remove the manylinux2014 image tag resolution, the manylinux1/manylinux2014 pixi-to-pattern renaming entries, and update shell script comments to reference _2_28 and _2_34 as the supported manylinux versions. Addresses review feedback from @thewtex on PR InsightSoftwareConsortium#302. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add pre-commit hooks in Utilities/Hooks/ matching ITK's directory structure and commit message enforcement: - prepare-commit-msg: Inserts ITK prefix instructions into the commit message editor so developers see valid prefixes while composing. - kw-commit-msg.py: Validates commit messages with the same rules as upstream ITK: prefix check (BUG/COMP/DOC/ENH/PERF/STYLE/WIP), subject line max 78 chars, no leading/trailing whitespace, empty second line, and Merge/Revert exemptions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…n scripts Replaces the primary build orchestration layer with a Python-based system. Adds core abstractions including BuildManager for step-wise build persistence/resumption, CMakeArgumentBuilder for managing CMake definitions, cross-platform venv utilities, subprocess helpers, and pixi environment integration. Establishes platform-specific base classes that Linux, macOS, and Windows implementations extend. Restructures CMake layout into modular BuildWheelsSupport and SuperbuildSupport components. Shell scripts remain where still required. Co-authored-by: Hans J. Johnson <hans-johnson@uiowa.edu>

… Windows Implements platform build scripts using the new Python infrastructure. Linux/manylinux builds use dockcross with auditwheel for manylinux compliance. macOS builds integrate delocate and respect MACOSX_DEPLOYMENT_TARGET. Windows builds use delvewheel for wheel repair. Removes legacy shell scripts superseded by the new Python implementations. Co-authored-by: Hans J. Johnson <hans-johnson@uiowa.edu>

Adds make_tarballs.sh and integrates tarball generation logic into build_python_instance_base.py. Supports creating, downloading, and reusing ITK build caches across Linux, macOS, and Windows, including manylinux tarball support, arch-specific naming, and a --build-itk-tarball-cache CLI option. Adds make_windows_zip.ps1 for streamlined Windows wheel packaging and zip creation.

Removes unused and outdated shell scripts and legacy build utilities superseded by the new Python build system. Adds pre-commit hooks with linting applied across the codebase. Drops Python 3.9 support as it is EOL. Includes README and documentation updates for the new build workflow.

Add a rattler-build recipe to produce a conda package containing ITK C++ libraries with full Python wrapping artifacts (SWIG interfaces, CastXML outputs, compiled Python modules, stubs). This package can be consumed as a host-dependency to skip the expensive ITK C++ build when generating Python wheels for ITK or remote modules. The recipe supports overriding the ITK source repository and tag via ITK_GIT_URL and ITK_GIT_TAG environment variables, enabling builds from feature branches and PRs for testing. Build scripts are provided for both Unix and Windows platforms. CMake flags match conda-forge's libitk-feedstock (using system libraries) plus the additional wrapping options needed for Python wheel generation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

When a pre-built ITK is available in the conda/pixi environment (via the libitk-wrapping package), automatically skip the superbuild and C++ compilation steps and point the wheel builder at the installed ITK CMake config directory. Detection checks CONDA_PREFIX and PIXI_ENVIRONMENT_DIR for an ITK CMake config (lib/cmake/ITK-*/ITKConfig.cmake). When found, steps 01_superbuild_support_components and 02_build_wrapped_itk_cplusplus are replaced with no-op stubs, reducing remote module build times from ~1-2 hours to ~15 minutes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Update the Linux, macOS, and Windows download-cache-and-build scripts to detect a pre-installed ITK from a conda/pixi environment before attempting to download ITKPythonBuilds tarballs. When ITKConfig.cmake is found under CONDA_PREFIX or PIXI_ENVIRONMENT_DIR, the entire download/extract phase is skipped and the build proceeds directly using the conda-provided ITK. This is backward-compatible: when no conda ITK is detected, the scripts behave exactly as before (tarball download path). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add commitizen configuration to pyproject.toml that enforces the ITK commit message convention (PREFIX: Description) with interactive prompts for selecting the change type. Valid prefixes: BUG, COMP, DOC, ENH, PERF, STYLE. The commitizen pre-commit hook in .pre-commit-config.yaml validates commit messages at the commit-msg stage. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add conda-forge-ready recipe files (meta.yaml, build.sh, bld.bat) for the libitk-wrapping package. These are prepared for submission to conda-forge/staged-recipes once the ITK install rule changes are merged upstream and an ITK 6 release is tagged. The recipe builds ITK C++ with Python wrapping using conda-forge system libraries and includes the CMAKE_FIND_ROOT_PATH settings required by conda-build cross-compilation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Document the libitk-wrapping conda package integration, conda ITK auto-detection behavior, and the ITK commit message convention enforced via commitizen. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Document the libitk-wrapping conda package workflow for building remote module wheels without recompiling ITK from source. Includes instructions for building and installing the package, environment variable overrides for custom ITK branches, and cross-references between the ITK and module build docs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remote modules hard-code ITK sub-package version pins in their pyproject.toml (e.g., itk-io == 5.4.*). When building against ITK 6, these pins cause pip install conflicts. Add _update_module_itk_deps() which rewrites exact ITK pins to minimum version constraints (>= 5.4) before invoking scikit-build-core. This runs automatically during build_external_module_python_wheel() when ITK_PACKAGE_VERSION is set, so remote module wheels always have dependency metadata compatible with the ITK version they were built against. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Back up pyproject.toml to pyproject.toml.orig before rewriting ITK dependency pins, then restore the original after the wheel is built. The modified version is saved as pyproject.toml.whl for reference. Uses try/finally to guarantee restore even if the build fails. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Document the plan to migrate from build-time pyproject.toml rewriting (Strategy 1) to a scikit-build-core dynamic metadata provider (Strategy 3) that resolves ITK dependency versions at build time without modifying source files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

When a remote module declares dynamic = ["dependencies"] in its [project] table, skip the pyproject.toml rewrite (Strategy 1) and instead set ITK_PACKAGE_VERSION in the environment for the scikit-build-core metadata provider to pick up (Strategy 3). Modules that have not yet opted into dynamic dependencies continue to use the build-time rewrite fallback, so both approaches coexist during the transition. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Limit the build-time pyproject.toml rewrite to the six ITK base sub-packages (itk-core, itk-numerics, itk-io, itk-filtering, itk-registration, itk-segmentation) whose versions are tied to the ITK release. Remote module cross-deps like itk-meshtopolydata are versioned independently and are not rewritten — a warning is printed instead so they can be reviewed manually. Prevents incorrect rewriting of e.g. itk-meshtopolydata == 0.12.* in BSplineGradient and WebAssemblyInterface. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The dependency floor must match the ITK series the wheel was compiled against, not a hardcoded backward-compatible value. For ITK 6.0.0b2 the rewrite now produces "itk-io >= 6.0" instead of "itk-io >= 5.4". This ensures wheels are only installable alongside the correct ITK release series. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remote modules must support installation with any ITK from v5.4.0 through the latest release. Use a fixed floor of >= 5.4 rather than pinning to the version being built, so wheels are installable across the full supported range. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

A wheel compiled against ITK 6.0 must require >= 6.0 at install time, not >= 5.4. The floor reflects the ITK series the wheel was linked against. The build system supports building against any ITK from v5.4 through the latest — the floor is derived dynamically from ITK_PACKAGE_VERSION at build time. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…main Add shell and Python scripts that clone ITK, ITKPythonPackage, and all remote modules from their latest main branches, then build ITK Python wheels followed by every remote module that has Python wrapping. All wheels are collected into a single /tmp/<timestamp>_LatestITKPython/dist directory. Both scripts parse ITK's Modules/Remote/*.remote.cmake files to discover remote module git repositories, filter to those with wrapping/ and pyproject.toml, and build each module wheel reusing the ITK build from the first step. Usage: ./scripts/build-all-latest-wheels.sh [platform-env] python scripts/build_all_latest_wheels.py --platform-env linux-py311 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add --itk-ref option to both shell and Python build-all-latest-wheels scripts. Defaults to "main" but accepts any git ref (branch name, tag like v6.0b02, or commit hash). The ITK clone is now a full clone (not shallow) to support arbitrary refs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

CavRiley force-pushed the python-build-system branch from 182e8a5 to 4aa0abb Compare March 29, 2026 02:05

CavRiley marked this pull request as ready for review March 29, 2026 22:06

CavRiley force-pushed the python-build-system branch from 1ea2b13 to db39910 Compare March 30, 2026 01:02

dzenanz reviewed Mar 30, 2026

View reviewed changes

docs/Quick_start_guide.rst Show resolved Hide resolved

CavRiley force-pushed the python-build-system branch 3 times, most recently from 4d78860 to 4288703 Compare March 31, 2026 16:15

CavRiley force-pushed the python-build-system branch from 4288703 to 86b1e2f Compare April 1, 2026 21:10

hjmjohnson marked this pull request as draft April 3, 2026 15:35

hjmjohnson force-pushed the python-build-system branch 2 times, most recently from afed60a to 1af5a71 Compare April 3, 2026 15:54

hjmjohnson and others added 14 commits April 3, 2026 15:51

DOC: Update online docs and infrastructure for docs generation

e3a9da8

ENH: Add GitHub Action to generate documentation artifacts

892ef20

DOC: Add CONTRIBUTING.md file

b9859c4

ENH: Add scripts to publish wheels and cached builds

1728684

hjmjohnson and others added 10 commits April 3, 2026 15:54

hjmjohnson force-pushed the python-build-system branch from 03cc291 to 221359d Compare April 3, 2026 20:54

hjmjohnson and others added 3 commits April 3, 2026 16:15

STYLE: Move build-all-latest-wheels scripts to Utilities/scripts

578e438

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python Build System#302

Python Build System#302
CavRiley wants to merge 27 commits intoInsightSoftwareConsortium:mainfrom
BRAINSia:python-build-system

CavRiley commented Mar 29, 2026 •

edited

Loading

Uh oh!

CavRiley commented Mar 29, 2026

Uh oh!

dzenanz left a comment

Uh oh!

Uh oh!

thewtex commented Mar 31, 2026

Uh oh!

CavRiley commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

CavRiley commented Mar 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Python-based build system for generating Python wheels

Migration Guide for GitHub Actions

Backwards compatibility

Uh oh!

CavRiley commented Mar 29, 2026

Uh oh!

dzenanz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

thewtex commented Mar 31, 2026

Uh oh!

CavRiley commented Apr 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CavRiley commented Mar 29, 2026 •

edited

Loading