Live bootstrap to guix

[vxtls]

No description provided.

[Googulator]

During the ISO build, I got an error complaining about a missing aux-files/logo.txt file. It's in gnu/installer in the tarball & channel repo, so before building the ISO, one needs to test -d /usr/share/guile/site/3.0/gnu/installer/aux-files || cp -r /var/lib/guix/local-channels/guix/gnu/installer/aux-files /usr/share/guile/site/3.0/gnu/installer/

[Googulator]

In the "build-native-utils-earlier" patch, the procedure mesboot0-package needs to read ˙(supported-systems '("i686-linux" "x86_64-linux")) instead of just (supported-systems '("i686-linux")), otherwise ISO generation will fail, thinking that there is no way to build 64-bit packages on the bootstrapped system.

It might also be necessary to change the target in the guix pull script to x86_64-linux, to avoid running the Guix bootstrap twice.

EDIT: Oops...

[Googulator]

And... bitrot strikes.

Guix tries to download 0001-unordered-fix-copy-assign.patch from boost.org, and then check it against a specific hash. But Boost decided to "clean up" their copy of this patch, removing trailing whitespace - breaking the hash.

The original one with the right hash can still be found @ https://cdn.jsdelivr.net/npm/@openziti/ziti-sdk-nodejs@0.14.2/vcpkg/ports/boost-unordered/0001-unordered-fix-copy-assign.patch - manually copying it to /gnu/store/chzch29xi2gzdxbzk77sb432klkddra1-boost-0001-unordered-fix-copy-assign.patch seems to work (EDIT: it doesn't).

[vxtls]

The original one with the right hash can still be found @ https://cdn.jsdelivr.net/npm/@openziti/ziti-sdk-nodejs@0.14.2/vcpkg/ports/boost-unordered/0001-unordered-fix-copy-assign.patch - manually copying it to /gnu/store/chzch29xi2gzdxbzk77sb432klkddra1-boost-0001-unordered-fix-copy-assign.patch seems to work.

Should we patch Guix to fix this problem? Just like the bootstrap binary, use a local web server to serve this file

[Googulator]

A local web server is probably a good idea, yes.

[vxtls]

Wait a minute, what do you mean by "it doesn't", you mean, the file didn't match or we can't copy it to /gnu/store?

[Googulator]

Copying to the store directly didn't work. Modifying the package definition to use the alternative URL did.

But now, I'm facing a new problem, where one of the "module-import-compiled" packages fails to build, apparently because contents of the host Guile (3.0.11) leak into the Guix environment, where the local Guile (3.0.9) can't execute them. I'm still working on solving this.

[vxtls]

But now, I'm facing a new problem, where one of the "module-import-compiled" packages fails to build, apparently because contents of the host Guile (3.0.11) leak into the Guix environment, where the local Guile (3.0.9) can't execute them. I'm still working on solving this.

What's the status of this issue? One thing that puzzles me is, aren't all Guix builds supposed to run in a chroot or namespace environment?
EDIT: can i have the log?
EDIT2: Currently I am running ISO build, see if I can reproduce the problem.

[vxtls]

Copying to the store directly didn't work. Modifying the package definition to use the alternative URL did.

But now, I'm facing a new problem, where one of the "module-import-compiled" packages fails to build, apparently because contents of the host Guile (3.0.11) leak into the Guix environment, where the local Guile (3.0.9) can't execute them. I'm still working on solving this.

  GEN      etc/init.d/guix-daemon
  GEN      etc/guix-daemon.conf
  GEN      etc/guix-publish.conf
make[2]: Leaving directory '/tmp/guix-build-guix-1.5.0rc1.drv-0/source'
make[1]: Leaving directory '/tmp/guix-build-guix-1.5.0rc1.drv-0/source'
phase `build' succeeded after 4432.3 seconds
starting phase `copy-bootstrap-guile'
accepted connection from pid 8350, user nixbld
accepted connection from pid 8378, user nixbld
Backtrace:
In ice-9/boot-9.scm:
  1752:10 10 (with-exception-handler _ _ #:unwind? _ # _)
In unknown file:
           9 (apply-smob/0 #<thunk 7ffff77352a0>)
In ice-9/boot-9.scm:
    724:2  8 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
In ice-9/eval.scm:
    619:8  7 (_ #(#(#<directory (guile-user) 7ffff773ac80>)))
In ice-9/command-line.scm:
   185:19  6 (_ #<input: string 7ffff7734850>)
In unknown file:
           5 (eval (begin (use-modules (guix)) (with-store store …)) #)
In ice-9/boot-9.scm:
  1752:10  4 (with-exception-handler _ _ #:unwind? _ # _)
In guix/store.scm:
   690:37  3 (thunk)
In ice-9/eval.scm:
    619:8  2 (_ #(#(#(#(#<directory (guile-user) 7ffff773a…>) …) …) …))
In unknown file:
           1 (symlink "/tmp/guix-tests/store/r26xz0l0wb2x0kil3c539f…" …)
In ice-9/boot-9.scm:
  1685:16  0 (raise-exception _ #:continuable? _)

ice-9/boot-9.scm:1685:16: In procedure raise-exception:
In procedure symlink: File exists
error: in phase 'copy-bootstrap-guile': uncaught exception:
%exception #<&invoke-error program: "./test-env" arguments: ("guile" "-c" "(begin (use-modules (guix)) (with-store store (let* ((item (add-to-store store \"guile-static-stripped-2.0.9-i686-linux.tar.xz\" #f
phase `copy-bootstrap-guile' failed after 0.6 seconds
command "./test-env" "guile" "-c" "(begin (use-modules (guix)) (with-store store (let* ((item (add-to-store store \"guile-static-stripped-2.0.9-i686-linux.tar.xz\" #f \"sha256\" \"/gnu/store/29dkam4a6sdz1pn81
build process 12 exited with status 256
bash-5.3# ls  /tmp/guix-tests
ls: cannot access '/tmp/guix-tests': No such file or directory

is it something like this?

[Googulator]

No, it was a different issue. Sourcing the Guix profile (/var/guix/profiles/per-user/root/current-guix/etc/profile) solved that, but it causes a new, strange issue where the ISO build will fail instantly when called by the main build script, or manually in the primary TTY1 terminal. Switching to TTY2 and calling it there makes it work.

I did encounter the issue you describe - I solved it differently, by making the 2nd copy operation conditional on the i686 and x86_64 bootstrap guile paths being different.

Now, I'm facing another issue, where tests fail for the "guix-1.5.0rc1.drv" derivation. The cause seems to be that it's trying to download the official bootstrap binaries to run the tests with, but is failing to do so. IMO the right way to fix this is to inject the patches we use when building our Guix into this derivation, so it too uses the locally built bootstrap binaries instead.

[vxtls]

I did encounter the issue you describe - I solved it differently, by making the 2nd copy operation conditional on the i686 and x86_64 bootstrap guile paths being different.

Where I am doing is to copy the original Bootstrap Guile directory, then add i686.txt and x86_64.txt to it, ensuring that their hash values are different. This way, guix won't fail during the copy operation because a file already exists at the destination (currently being tested; it just failed because the patch wasn't written correctly).

Now, I'm facing another issue, where tests fail for the "guix-1.5.0rc1.drv" derivation. The cause seems to be that it's trying to download the official bootstrap binaries to run the tests with, but is failing to do so. IMO the right way to fix this is to inject the patches we use when building our Guix into this derivation, so it too uses the locally built bootstrap binaries instead.

Where is the definition file for this?

And another question: Do you think we should make the generated ISO to be installed offline? (Because the default ISO image downloads external substitutes during the installation process, which undermines our assumption of a fully source-code-based build.)

[Googulator]

The definition file is gnu/packages/package-management.scm - I've attached the patch I used for the double-bootstrap-guile bug. The issue where tests fail due to trying to access the official binaries is also in the same package definition.

IMO we should keep the ISO's ability to use substitutes, but have it default to using the bootstrap machine as its sole substitute source. The packages there are properly bootstrapped, and therefore fine to use as substitutes.

[Googulator]

Currently running a bare metal test with this diff on top of the latest commit:

$ git diff
diff --git a/steps-guix/improve/guix-daemon-and-pull.sh b/steps-guix/improve/guix-daemon-and-pull.sh
index c7625474..f61b5d9f 100644
--- a/steps-guix/improve/guix-daemon-and-pull.sh
+++ b/steps-guix/improve/guix-daemon-and-pull.sh
@@ -218,6 +218,9 @@ mkdir -p /proc /sys /dev "${guix_localstate_dir}/daemon-socket" /var/lib/guix /r
 mount | grep ' on /proc ' >/dev/null 2>&1 || mount -t proc proc /proc
 mount | grep ' on /sys ' >/dev/null 2>&1 || mount -t sysfs sysfs /sys
 mount | grep ' on /dev ' >/dev/null 2>&1 || mount -t devtmpfs devtmpfs /dev
+# tmpfs must be unmounted to avoid overfilling memory
+mount | grep ' on /tmp ' >/dev/null 2>&1 && umount /tmp
+test -f /swapfile && swapon /swapfile
 if ! mount | grep ' on /dev/pts ' >/dev/null 2>&1; then
     mkdir -p /dev/pts
     mount -t devpts devpts /dev/pts
diff --git a/steps-guix/jump/linux64.sh b/steps-guix/jump/linux64.sh
index 1bb6ba93..f942c2a1 100644
--- a/steps-guix/jump/linux64.sh
+++ b/steps-guix/jump/linux64.sh
@@ -37,4 +37,8 @@ else
         --append="console=ttyS0 root=/dev/sda1 init=/init rw rootwait consoleblank=0"
 fi
 quiesce_filesystem_for_kexec
-kexec -e
+if [ "${BARE_METAL}" = True ]; then
+    echo b > /proc/sysrq-trigger || true
+else
+    kexec -e
+fi

I disabled kexec for bare metal, because I couldn't get the framebuffer to work reliably after kexecing from 32-bit 4.14-openela to 64-bit 6.12-gnu. The other change activates the swapfile, and unmounts /tmp before starting the Guix bootstrap - this was needed to avoid going OOM on my 8GiB RAM bootstrap rig.

[vxtls]

I’m also thinking about running a bare-metal setup. Is using Coreboot + SeaBIOS on a modern motherboard a viable option?

[Googulator]

If you can run Coreboot on your board, it should be, especially since qemu's default BIOS implementation is SeaBIOS.

Note that I had to edit that patch in that previous comment, as I got Bash's conditional syntax wrong (it's if...fi, not if...endif).

[vxtls]

Great, I’ll add the patch shortly.
I’m currently experimenting with an offline ISO image (I’ll keep the default build using substitutes in the project), but I’ll include a configuration file that allows for building an image suitable for offline installation. I’m running into some issues at the moment.

[vxtls]

There is currently no reliable method for including Closure in an ISO image and performing a guided installation.

[Googulator]

The new patch is malformed: @@ -211,11 +211,8 @@ should be @@ -211,11 +211,4 @@.

With that fixed, I had a test failure in wmin69qfszbjz0mflj2sya8sm2r5c7bs-glib-2.83.3.drv (GLib 2.83.3), that went away on retry. Maybe we should disable tests for it?

[Googulator]

OK, after lots of wrangling...

I have an ISO.

More to come tomorrow. I've just came home, and am quite tired. And it did took quite some wrangling to get here.