Skip to content

Ensure reproducible builds on Linux#1731

Open
curious-rabbit wants to merge 3 commits into
veracrypt:masterfrom
curious-rabbit:master
Open

Ensure reproducible builds on Linux#1731
curious-rabbit wants to merge 3 commits into
veracrypt:masterfrom
curious-rabbit:master

Conversation

@curious-rabbit
Copy link
Copy Markdown
Contributor

Make the Linux build and packaging pipeline deterministic so identical
sources yield byte-identical binaries and installer archives regardless
of build path, host, user or wall-clock time. Every change degrades
safely: each flag/tool feature is probed before use, so older toolchains
and the FreeBSD/macOS build paths are unaffected.

src/Makefile

  • Honour SOURCE_DATE_EPOCH (derived from the HEAD commit when unset,
    fixed constant for git-less tarball builds).
  • Add -ffile-prefix-map (fallback -fdebug-prefix-map) and
    -fno-record-gcc-switches, each gated behind a cc-option probe so
    compilers that lack them (GCC < 8) still build.
  • Pin -Wl,--build-id=sha1 on Linux, also probed.

src/Build/Include/Makefile.inc

  • Build static libraries with ar's 'D' modifier and 'ranlib -D',
    selected via functional probes; old binutils falls back to a normal
    archive. Remove the stale archive first so rebuilds cannot merge
    leftover members.

src/Main/Main.make

  • strip with --enable-deterministic-archives when supported (probed).
  • Linux 'prepare': clamp mtimes of the staged tree to
    SOURCE_DATE_EPOCH so checkout-time stamps do not leak into archives.
  • Linux 'package': build the tarball with sorted members, pinned
    mtime, numeric 0/0 ownership and 'gzip -n', and drive makeself with
    --packaging-date / --tar-extra.

Note that other builds like Windows are not touched and probably impossible to build reproducible. Not like there is any point to this on closed source platforms anyway

@idrassi
Copy link
Copy Markdown
Member

idrassi commented May 15, 2026

Thank you for working on reproducible Linux builds. This is a useful goal, but I can't merge this as-is.

The PR description says that "each flag/tool feature is probed before use, so older toolchains and the FreeBSD/macOS
build paths are unaffected". This isn't accurate as written.

Some compiler/linker/archive features are probed, but several new packaging requirements are used unconditionally:
touch --no-dereference --date=@..., tar --sort=name --mtime=@... --pax-option=..., gzip -n, and makeself --packaging-date --tar-extra. In particular, --tar-extra requires Makeself 2.3.1 or newer, so older build hosts would fail instead of degrading safely.

FreeBSD/macOS are also not completely unaffected: the new reproducibility flags in src/Makefile and the static archive changes in src/Build/Include/Makefile.inc are global. They may be acceptable, but they still need gating/verification or the PR scope should be described more narrowly.

There is also a scope issue. The current Linux .deb packaging path uses src/Build/build_cmake_deb.sh and CPack.
Since SOURCE_DATE_EPOCH is set inside make, later cmake/cpack invocations won't inherit it unless the caller already exported it. So this doesn't yet make the full Linux packaging pipeline deterministic.

Before this can be considered for merge, please:

  • either narrow the PR to the legacy Makefile tar.gz/makeself artifacts, or cover the DEB/RPM/AppImage paths as well
  • add functional probes or explicit minimum-version checks for the packaging tools
  • provide a reproducibility test/log building the same commit from two different source paths and comparing the resulting binary, static libraries, and produced artifacts.

@curious-rabbit
Copy link
Copy Markdown
Contributor Author

here is the log:

Date:       2026-05-15 16:34:59 UTC
Host:       Linux 6.18.5 x86_64 / Ubuntu 24.04.4 LTS
Toolchain:  gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
            GNU ar (GNU Binutils for Ubuntu) 2.42
            tar (GNU tar) 1.35
            gzip 1.12
            Makeself version 2.5.0
            GNU Make 4.3


Build A:  path=/tmp/repro/srcA    umask=022  HOME=/tmp/repro/homeA
Build B:  path=/var/build/path-B  umask=027  HOME=/var/build/homeB

Both invoked under:
  env -i PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
    HOME=... SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) LC_ALL=C TZ=UTC \
    make NOGUI=1 -j1 && make NOGUI=1 -j1 package

Artifact comparison (sha256)
  veracrypt binary          IDENTICAL  70b6542b3d8e011813faca05125fd7112b64ca9505509abf1a0266c82ee597bc
  Volume.a static archive   IDENTICAL  ecf60f438a0a2a6fdbecf23ff81db95d9d0989a7cacebda1a85a80a7337a47d7
  console tar.gz            IDENTICAL  09324dd355f2fdbe362d2d2904aca250f67f0762d4db32b19f11537e89b9518f
  console self-extractor    IDENTICAL  d60a0cfcec2b8ef7c22115549203dc1dc0c8d2b432abcf55ea389690e90b99ab

Note that the ci failure seems unrelated to the code.

@idrassi
Copy link
Copy Markdown
Member

idrassi commented May 16, 2026

@curious-rabbit
Thank you very much for the update and the reproducibility log. The new changes move things in the right direction.

A few items still need another revision before I can merge. Below is a list (sorry for the verbosity, I wanted to give as much details as possible because this feature is important)

1. install(CODE ...) in src/Build/CMakeLists.txt needs to be reworked

The new block is declared at the top of the file, but the actual install(DIRECTORY ...) rules that populate the staging tree are roughly 180 lines below it. CMake runs install() rules in declaration order at install-time, so the find ... -exec touch runs against an empty (or pre-existing) tree before any VeraCrypt files are placed.

There is also a more serious concern: under CPack, this project sets CPACK_PACKAGING_INSTALL_PREFIX to /usr, and the install code can run as find /usr -exec touch --no-dereference --date=@<epoch> {} +, outside the package staging directory. Outside CPack, ${CMAKE_INSTALL_PREFIX} is /usr/local by default or user-specified, which is still a host install tree. As root, either case can rewrite mtimes outside the package staging area.
Please:

  • move this into an install(SCRIPT ...) or equivalent placed after all install(DIRECTORY/FILES) rules
  • make it operate only on the real package staging root, not bare ${CMAKE_INSTALL_PREFIX}
  • if using $ENV{DESTDIR}${CMAKE_INSTALL_PREFIX}, explicitly enable/set the DESTDIR-based CPack staging mode and refuse to run when DESTDIR is empty.

2. The touch ... /dev/null probes aren't safe

Both src/Build/CMakeLists.txt and src/Main/Main.make probe with touch --no-dereference --date=@0 /dev/null. Setting a non-current timestamp requires file ownership (or CAP_FOWNER), so for any normal unprivileged user the probe fails with EPERM and the deterministic path is silently disabled: exactly the case it is supposed to detect. When run as root the probe instead succeeds and rewrites /dev/null's mtime. Please use a private mktemp target (symlink if --no-dereference is part of what you want to probe) and clean it up.

3. makeself_repro_finalize.py recomputes CRCsum incorrectly

Makeself stores the POSIX cksum value in CRCsum, but the script writes zlib.crc32(payload). The algorithms aren't interchangeable: POSIX cksum includes its own finalization/length handling, so zlib.crc32(payload) produces a different value. The archive will still pass --check while MD5 is present, but the cksum fallback path is broken. Two acceptable fixes:

  • compute the same value Makeself does (shell out to cksum, or implement POSIX cksum in the script), or
  • set CRCsum="0000000000", which Makeself's extractor treats as "skip CRC check".

Please pick one of these rather than a third option.

4. The tar probe only checks for "GNU tar"

The recipe relies on --sort=name, --mtime=@..., --pax-option=..., --owner/--group/--numeric-owner, --mode. --sort=name/--mtime=@... are GNU tar ≥ 1.28 (Oct 2014): the current probe accepts older versions and will then fail at recipe time. Please probe the exact option set, e.g. tar --sort=name --mtime=@0 --pax-option=... -cf /dev/null -T /dev/null, or enforce/document a minimum GNU tar version.

5. Minor: strip --enable-deterministic-archives -V probe

-V prints version, but unknown long options can fail before that depending on getopt parsing. Please align this with the other probes by running it against a small $(mktemp) artifact.

6. Reproducibility log only covers the legacy Makefile path

The log you provided exercises only:

make NOGUI=1 -j1 && make NOGUI=1 -j1 package

the console tar.gz + makeself path. The PR now also modifies CMakeLists.txt, build_cmake_deb.sh, build_cmake_rpm.sh, build_cmake_opensuse.sh, and the appimage target. Please either narrow the PR back to the legacy tar.gz/Makeself artifacts, or provide reproducibility logs for the DEB, RPM, openSUSE and AppImage paths as well, built from two distinct source paths and compared by SHA-256, like your existing log.

All in all, the direction is good and I'd like to see this land. Once the items above (especially #1, #2 and #3, which are correctness issues, not style) are addressed I'll take another pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants