Skip to content

build(python): publish manylinux_2_28 wheels for broader glibc compat…#1117

Open
phamgialinhlx wants to merge 1 commit intoNVIDIA:mainfrom
phamgialinhlx:feat/wheel-manylinux-2-28
Open

build(python): publish manylinux_2_28 wheels for broader glibc compat…#1117
phamgialinhlx wants to merge 1 commit intoNVIDIA:mainfrom
phamgialinhlx:feat/wheel-manylinux-2-28

Conversation

@phamgialinhlx
Copy link
Copy Markdown

Summary

Build released Linux wheels inside a PyPA manylinux_2_28 container so the binary runs on any Linux distribution shipping glibc >= 2.28 (RHEL 8, Debian 10+, Ubuntu 18.04+). The previous native build on the noble (glibc 2.39) CI image produced manylinux_2_39_x86_64 wheels that uv and pip refuse to install on common LTS hosts such as Ubuntu 22.04 / Debian 11.

Related Issue

Reported via the install path. On Ubuntu 22.04 (glibc 2.35):

Distribution openshell==0.0.36 @ registry+https://pypi.org/simple can't be installed because it doesn't have a source distribution or wheel for the current platform … only has wheels for: manylinux_2_39_aarch64, manylinux_2_39_x86_64, macosx_13_0_arm64

Changes

  • New deploy/docker/Dockerfile.python-wheels-linux — builds the wheel inside quay.io/pypa/manylinux_2_28_{x86_64,aarch64}. Mirrors the layer layout (manifest copy → dummy-source dep build → real source rebuild) used by Dockerfile.python-wheels-macos so cargo cache mounts behave the same way. Maturin is invoked with --compatibility manylinux_2_28 so the resulting wheel is tagged accordingly.
  • New mise tasks in tasks/python.toml:
    • build:python:wheel:linux:docker — driver task; takes TARGETARCH and WHEEL_OUTPUT_DIR.
    • build:python:wheel:linux:{amd64,arm64}:docker — per-arch wrappers.
    • The release-pipeline aliases python:build:linux:{amd64,arm64} are re-pointed at the Docker path.
    • The legacy native build:python:wheel:linux:{amd64,arm64} tasks remain for fast local iteration (still produce a wheel tagged for the host's glibc, which is fine for dev installs).
  • Workflow .github/workflows/release-tag.yml and release-dev.yml, in the build-python-wheels-linux job:
    • Mount /var/run/docker.sock so the in-container mise run step can reach the host docker daemon to launch the manylinux build.
    • Add a setup-buildx step so the Dockerfile's BuildKit cache mounts (--mount=type=cache,target=/root/.cargo/registry etc.) work.
    • Both lines mirror the pre-existing build-python-wheel-macos job, which already uses this exact pattern for its osxcross Docker build.
  • Docs architecture/build-containers.md updated to describe the new portable wheel build path and call out that the native task is now a local-iteration convenience, not the release path.

The pre-existing dead deploy/docker/Dockerfile.python-wheels (no references in any task or workflow) is left untouched to keep this PR additive — happy to remove it in a follow-up if maintainers want.

Alternatives considered

  • maturin build --zig --compatibility manylinux_2_28 (cargo-zigbuild) was the smaller-diff option but carries real risk: bundled-z3 builds Z3 from C++ source via cmake, and Zig as the C++ toolchain on a heavy C++ codebase is unverified for this repo. Switching to the manylinux container keeps Z3 building under gcc-toolset-14, which is what the Z3 project itself tests against.
  • Changing the base of Dockerfile.ci to an older-glibc distro would let the native build produce portable wheels, but it would simultaneously affect every other job that uses the CI image. Out of scope.

Testing

End-to-end validation on Ubuntu 22.04 / glibc 2.35 / Python 3.14:

Step Result
docker buildx build -f deploy/docker/Dockerfile.python-wheels-linux --target wheels --build-arg TARGETARCH=amd64 … exit 0, ~9 min on a warm cache
Wheel produced openshell-0.0.0-py3-none-manylinux_2_28_x86_64.whl (18.8 MB)
uv add <wheel> into a Python 3.14 project that previously failed with the platform-tag error above accepted
import openshell; openshell.SandboxClient works

The build was iterated once during testing: the first attempt failed at dnf install perl-FindBin because that's not a standalone package on AlmaLinux 8 (manylinux_2_28's base). Replaced with perl-core, which provides FindBin and is the canonical "full perl" meta-package on the EL family. The Z3 bundled-z3 build (the highest-risk piece pre-test) worked cleanly under gcc-toolset-14 inside the manylinux container.

A workflow note: the docker socket mount means the linux wheel job, like the macOS wheel job, requires /var/run/docker.sock to be present on the self-hosted build-amd64 / build-arm64 runners. That matches the
contract the macOS job already relies on.

  • mise run pre-commit passes
  • Unit tests added/updated — N/A (build-system change, no library code)
  • E2E tests added/updated — N/A; the existing release pipelines exercise
    the new task end-to-end on the next dev/tag run

Checklist

  • Follows Conventional Commitsbuild(python): publish manylinux_2_28 wheels for broader glibc compatibility
  • Commits are signed off (DCO)
  • Architecture docs updated (architecture/build-containers.md)

…ibility

Build released Linux wheels inside a PyPA manylinux_2_28 container so the
resulting binary runs on any Linux distribution with glibc >= 2.28
(RHEL 8, Debian 10+, Ubuntu 18.04+). The previous native build on the
noble (glibc 2.39) CI image produced manylinux_2_39 wheels that uv
refused to install on common LTS hosts such as Ubuntu 22.04.

- Add deploy/docker/Dockerfile.python-wheels-linux based on
  quay.io/pypa/manylinux_2_28_{x86_64,aarch64}.
- Add build:python:wheel:linux:docker mise task and per-arch aliases.
  Re-point python:build:linux:{amd64,arm64} (used by release workflows)
  to the Docker-based path. The legacy native build:python:wheel:linux
  task remains for fast local iteration.
- Mount the host docker socket and set up buildx in the
  build-python-wheels-linux job in release-tag.yml and release-dev.yml,
  mirroring the existing macOS wheel job.
- Update architecture/build-containers.md to describe the new flow.

Signed-off-by: phamgialinhlx <phamgialinhlx2@gmail.com>
@phamgialinhlx phamgialinhlx requested a review from a team as a code owner May 1, 2026 16:20
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 1, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 1, 2026

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@phamgialinhlx
Copy link
Copy Markdown
Author

I have read the DCO document and I hereby sign the DCO.

@phamgialinhlx
Copy link
Copy Markdown
Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant