Skip to content

Improve test-matrix coverage #186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jserv merged 1 commit into from
Jul 9, 2026
Merged

Improve test-matrix coverage #186

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 39 additions & 8 deletions docs/testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,17 @@ make clean
What they do:

- `make elfuse`: build and sign `build/elfuse`
- `make check`: the recommended pre-commit gate. Runs, in order:
- `make check`: fast elfuse-internal gate. Runs, in order:
- `scripts/check-syscall-coverage.py` so any new `dispatch.tbl`
entry without a direct or aliased test reference fails the build
- the unit suite from `tests/manifest.txt`
- the unit suite from `tests/manifest.txt` -- deliberately narrow: only
tests that assert elfuse-internal implementation details with no real
Linux counterpart (the EL1 shim fast-path suite, `test-mremap-infra`,
`test-oom-proc`), plus whatever `mk/tests.mk`'s `SANITIZER_SECTIONS`
needs for the `check-{asan,ubsan,tsan}` lanes. Everything that is
meaningful to cross-check against a real Linux kernel lives exclusively
in `tests/test-matrix.sh`'s `run_unit_tests` instead (see Test Matrix
below) -- `make check` alone is *not* a substitute for it
- the TLBI RVAE1IS encoder unit test
- the proctitle argv-tail and low-stack regressions
- the BusyBox applet smoke suite (auto-resolved from
Expand All @@ -115,20 +122,28 @@ What they do:

## Quick Iteration

For normal code changes:
For normal code changes touching syscall or runtime logic:

```sh
make elfuse
make check
make test-matrix-elfuse-aarch64
```

For changes that touch procfs, path handling, `/dev`, FUSE, networking, dynamic
linking, or guest process semantics, run the matrix as well:
`make check` alone only covers elfuse-internal plumbing and the sanitizer
subset now; `test-matrix-elfuse-aarch64` is what actually exercises the full
unit-test surface against `build/elfuse` (no qemu boot needed, so it is about
as fast to iterate with as `make check` was before the split). For changes
that touch procfs, path handling, `/dev`, FUSE, networking, dynamic linking,
or guest process semantics, also cross-check against the qemu reference
kernel:

```sh
make test-matrix
make test-matrix-qemu-aarch64
```

or run all matrix modes back-to-back with `make test-matrix`.

`make check` already runs the BusyBox applet suite as a second stage, so a
green `make check` covers BusyBox validation. Use `make test-busybox` to
iterate on a single applet failure without rerunning the unit suite.
Expand All @@ -147,6 +162,22 @@ execution modes:
The goal is not to compare performance. The goal is to compare guest-observable
behavior against a ground-truth Linux AArch64 environment so that any divergence
in syscall translation, procfs emulation, or process semantics is caught early.

`run_unit_tests` in `tests/test-matrix.sh` is the full aarch64 unit-test
surface -- every binary that is meaningful to run against a real kernel, which
is almost everything. It deliberately excludes only the handful of tests that
assert elfuse-internal implementation details with no meaningful counterpart
on a real kernel (the EL1 shim fast-path suite, `test-mremap-infra`,
`test-oom-proc` -- these live solely in `tests/manifest.txt` / `make check`,
see that file's header for the full split rationale). There is no separate
"core" vs "extended" test set inside the matrix; a test that has a real,
understood divergence from the qemu reference kernel is listed in
`QEMU_SKIP` with a comment explaining why instead -- see that variable in
`tests/test-matrix.sh` for the current list and rationale. `run_unit_tests`
runs in both `elfuse-aarch64` and `qemu-aarch64` modes, so most tests are
exercised twice per matrix run: once against `build/elfuse`, once against the
real kernel.

The x86_64 mode is narrower: it aggregates the Rosetta-specific acceptance
scripts and their per-binary summaries into the same matrix runner, including
the Rosetta thread/signal audit smoke, the LuaJIT guest-JIT probe, and the
Expand Down Expand Up @@ -327,8 +358,8 @@ Suggested minimum validation:
| Change area | Recommended validation |
|-------------|------------------------|
| CLI, logging, docs-only build rules | `make elfuse` |
| General syscall or runtime logic | `make elfuse && make check` |
| `/proc`, `/dev`, path, or BusyBox-sensitive behavior | `make elfuse && make check` |
| General syscall or runtime logic | `make elfuse && make check && make test-matrix-elfuse-aarch64` |
| `/proc`, `/dev`, path, or BusyBox-sensitive behavior | `make elfuse && make check && make test-matrix-elfuse-aarch64` |
| Rosetta hosting, x86_64 dispatch, VZ ioctls, AOT cache | `make elfuse && make test-rosetta-all` |
| Broad behavioral changes | `make elfuse && make check && make test-matrix` |
| Debugger or ptrace flow | `make elfuse && make test-gdbstub` |
128 changes: 20 additions & 108 deletions tests/manifest.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,21 @@
#
# BINARY is resolved relative to TESTDIR (default: build/).
# Lines starting with # are comments. Blank lines are ignored.
#
# Scope: this file only lists tests that make check needs directly --
# elfuse-internal implementation tests with no meaningful counterpart on a
# real kernel (they can only ever be exercised against elfuse itself), plus
# whatever mk/tests.mk's SANITIZER_SECTIONS regex selects for the
# check-{asan,ubsan,tsan} lanes (those need a *sanitizer-instrumented elfuse
# binary*, which only driver.sh -e can point at; qemu-aarch64 runs an
# uninstrumented real kernel and cannot substitute for them). Every other
# aarch64 test -- anything that is meaningful to cross-check against a real
# Linux kernel -- lives exclusively in tests/test-matrix.sh's run_unit_tests,
# which is a superset: it runs every one of these binaries too (via
# elfuse-aarch64 mode), plus everything that used to be duplicated here.
# Don't add a portable/cross-checkable test here; add it to test-matrix.sh
# instead. Only add a test here if it is genuinely elfuse-internal, or if a
# SANITIZER_SECTIONS section needs it.

[section] Assembly tests
test-hello
Expand All @@ -28,39 +43,19 @@ test-ls tests/
test-roundtrip
test-comprehensive

[section] Process tests
test-exec $TESTDIR/echo-test exec-works
test-exec-limits
test-fork
test-session
test-pidfd # diff=skip

[section] Signal tests
test-signal
test-kill-broadcast
test-kill-pgroup
test-sigio # diff=skip

[section] Socket tests
test-socket

[section] Syscall coverage tests
test-file-ops
test-flock
test-sysinfo
test-times
test-io-opt
[section] elfuse-internal implementation tests
# These probe elfuse's own EL1 shim fast paths (identity cache, urandom
# ring, shim_data privilege) and procfs sendfile/copy_file_range
# interception -- plumbing with no Linux-kernel counterpart, so they are
# never part of test-matrix.sh (see each file's own header comment).
test-oom-proc
test-syscall-smoke
test-vdso
test-shim-identity
test-shim-identity-attention
test-shim-verbose-trace
test-shim-data-el1
test-shim-urandom-smp
test-shim-urandom-toctou
test-shim-urandom-wrap
test-poll # diff=skip

[section] I/O subsystem tests
test-eventfd
Expand All @@ -81,15 +76,6 @@ test-pty
test-ioctl-fioasync
test-getdents-refcount

[section] /proc and /dev emulation tests
test-proc
test-sysfs-cpu

[section] Network tests
test-net
test-netstat
test-netlink

[section] Threading tests
test-thread # diff=skip
test-pthread
Expand All @@ -100,9 +86,6 @@ test-simd-clone # diff=skip
test-stress # diff=skip
test-mprotect-mt # diff=skip

[section] Negative / error-path tests
test-negative # diff=skip

[section] Signal + thread tests
test-signal-thread
test-fault-signal-mt # diff=skip
Expand All @@ -118,23 +101,10 @@ test-fork-lowbase
test-cow-fork
test-fork-synthetic-fd

[section] O_CLOEXEC tests
test-cloexec

[section] O_PATH semantics tests
test-opath

[section] xattr semantics tests
test-xattr

[section] Guard page / mmap edge cases
test-guard-page
test-mmap-hint

[section] Low-base ET_EXEC memory regression tests
test-lowbase-mem-200000
test-lowbase-mem-300000

[section] mremap tests
test-mremap
test-mremap-infra
Expand All @@ -152,81 +122,23 @@ test-cross-fork-mapshared # diff=skip
[section] madvise MADV_DONTNEED tests
test-madvise

[section] Scatter-gather I/O tests
test-readv-writev

[section] inotify emulation tests
test-inotify

[section] PI futex + EINTR regression tests
test-futex-pi # diff=skip

[section] futex_waitv (SYS 449) tests
test-futex-waitv # diff=skip

[section] SIGILL / null guard tests
test-sigill

[section] X11 raw protocol tests
test-x11

[section] Robust futex tests
test-robust-futex

[section] /proc completeness tests
test-procfs
test-procfs-exec
test-proc-limits

[section] FD table race tests
test-fd-race

[section] FD lifecycle tests
test-fd-lifecycle

[section] Multithreaded fork tests
test-mt-fork

[section] Exit-group teardown reachability tests
test-exit-group-teardown wait # expected_rc=42
test-exit-group-teardown stop # expected_rc=42
test-exit-group-teardown fork # expected_rc=42

[section] SysV shared memory tests
test-sysv-shm

[section] Credential/identity emulation tests
test-credentials # diff=skip

[section] Scheduler policy stub tests
test-sched-policy

[section] membarrier tests
test-membarrier

[section] rseq registration tests
test-rseq # diff=skip

[section] Abstract Unix socket tests
test-abstract-socket

[section] Ancillary data (SCM_RIGHTS/MSG_CTRUNC) tests
test-ancillary

[section] Tier A compatibility tests
test-tier-a # diff=skip

[section] /proc fidelity tests
test-proc-fidelity # diff=skip

[section] Linux syscall fidelity tests
test-syscall-fidelity # diff=skip

[section] fd-family tests
test-fd-family # diff=skip

[section] SCM_CREDENTIALS tests
test-scm-creds # diff=skip

[section] Virtual chown overlay tests
test-chown-overlay # diff=skip
Loading
Loading