Fix openat2 ENOTDIR/ENOENT errno mismatches

[Max042004]

Summary

• dirfd_guest_base_path() returned EBADF instead of ENOTDIR for a valid but non-directory dirfd with no host-nameable path (pipes, sockets, etc.), surfacing through openat2()'s RESOLVE_NO_XDEV precheck.
• sc_openat2() let an empty pathname succeed under RESOLVE_IN_ROOT, since path_openat2_normalize_in_root() substitutes "." for "" and resolves to dirfd itself. openat2() has no AT_EMPTY_PATH equivalent, so empty paths now fail ENOENT up front.

Fixes #133.

Test

Added two regression tests to tests/test-syscall-fidelity.c
make check — 120/120 passed

---

Summary by cubic

Fix errno handling in openat2 path resolution: valid non-directory dirfds now return ENOTDIR (not EBADF), and empty pathnames return ENOENT even under RESOLVE_IN_ROOT. Added two regression tests to confirm Linux-compatible behavior.

^{Written for commit 9b99624. Summary will update on new commits.}

[cubic-dev-ai]

No issues found across 3 files

_{Re-trigger cubic}

[doanbaotrung]

Emulated Test Script (test_openat2_fixes.py)

This script uses the guest's Python 3 interpreter running under elfuse to directly make Linux openat2 system calls (syscall 437) via ctypes and verifies that the correct Linux error numbers (ENOENT and ENOTDIR) are returned.

Save this code as test_openat2_fixes.py:

import ctypes
import os
import struct

# Load guest libc and define __errno_location to read thread-local errno
libc = ctypes.CDLL(None)
libc.__errno_location.restype = ctypes.POINTER(ctypes.c_int)
def get_errno():
    return libc.__errno_location().contents.value

# Linux openat2 syscall & resolve constants
SYS_openat2 = 437
RESOLVE_NO_XDEV = 0x01
RESOLVE_IN_ROOT = 0x10

# Test 1: Empty path with RESOLVE_IN_ROOT should fail with ENOENT (Linux errno 2)
print("Running Test 1: openat2 empty path with RESOLVE_IN_ROOT...")
dirfd = os.open('/tmp', os.O_RDONLY | os.O_DIRECTORY)
how = struct.pack('QQQ', os.O_RDONLY, 0, RESOLVE_IN_ROOT)
res1 = libc.syscall(SYS_openat2, dirfd, b'', how, 24)
err1 = get_errno()
os.close(dirfd)

# Test 2: Relative path with non-directory dirfd (e.g., a pipe) should fail with ENOTDIR (Linux errno 20)
print("Running Test 2: openat2 relative path with pipe fd...")
r_fd, w_fd = os.pipe()
how = struct.pack('QQQ', os.O_RDONLY, 0, RESOLVE_NO_XDEV)
res2 = libc.syscall(SYS_openat2, r_fd, b'relative', how, 24)
err2 = get_errno()
os.close(r_fd)
os.close(w_fd)

# Output results
print(f"Test 1 (Empty path)       -> Result: {res1}, Errno: {err1} (Expected: -1, 2 [ENOENT])")
print(f"Test 2 (Non-directory fd) -> Result: {res2}, Errno: {err2} (Expected: -1, 20 [ENOTDIR])")

if res1 == -1 and err1 == 2 and res2 == -1 and err2 == 20:
    print("RESULT: PASS")
else:
    print("RESULT: FAIL")

---

Test Execution Command

Run the test script directly using the standalone elfuse binary and your guest sysroot:

elfuse/build/elfuse --sysroot ~/prefixes/ubuntu-arm64/rootfs /usr/bin/python3 test_openat2_fixes.py

---

Output Log

Running Test 1: openat2 empty path with RESOLVE_IN_ROOT...
Running Test 2: openat2 relative path with pipe fd...
Test 1 (Empty path)       -> Result: -1, Errno: 2 (Expected: -1, 2 [ENOENT])
Test 2 (Non-directory fd) -> Result: -1, Errno: 20 (Expected: -1, 20 [ENOTDIR])
RESULT: PASS