linux: improve openat and openat2 handling

[cyphar]

When looking at the usage of openat and openat2 on Linux, I noticed two bugs:

• openat(2) famously does not return -EINVAL for unknown flag bits on Linux, so the fallback code to use O_RESOLVE_BENEATH if defined would result in all open operations with the "secure resolver" becoming insecure. This would require a fairly convoluted build environment to trigger, so this isn't exploitable but if it were triggered it would be a security bug.
• The usage of openat2(2) did not handle -EAGAIN at all, which could lead to spurious errors. openat2(2) can return -EAGAIN when trying to walk into .. during a scoped lookup like RESOLVE_BENEATH and there was a rename or mount operation anywhere else on the system. The reason for this behaviour is a little esoteric but in short I implemented it this way so that userspace could choose their retry policy and (more importantly) the kernel could avoid potentially-unbounded retries that could lead to a DoS. The man page does mention this but it is easy to miss:

  EAGAIN how.resolve contains either RESOLVE_IN_ROOT or RESOLVE_BENEATH, and the kernel could not ensure that a ".." component didn't escape (due to a race condition or potential attack). The caller may choose to retry the openat2() call.
  It is possible for some of the concerns raised in other threads about instability with 3.8.3 were caused by this, but nobody has posted strace logs so it's hard to know for sure, and I haven't had time to test this myself.

---

I also took a quick look at the path_has_dotdot_component usage and I'm not really convinced that it is actually doing anything useful. The comments seem to be Claude-authored so I'm not sure reading them is super helpful, but the upshot of:

	/* Reject any path with a literal ".." component (bare "..",
	 * "../foo", "foo/..", "foo/../bar", "subdir/.."). The previous
	 * substring-based check caught only "../" prefix and "/../"
	 * substring; bare ".." and trailing "/.." escape on the per-
	 * component walk fallback used by NetBSD/OpenBSD/Solaris/Cygwin
	 * and pre-5.6 Linux. RESOLVE_BENEATH on Linux/FreeBSD/macOS
	 * catches some of these in-kernel with EXDEV, but the front
	 * door must reject them consistently with EINVAL across all
	 * platforms so callers can rely on the validation. */

appears to be that .. is problematic and RESOLVE_BENEATH catches "some of these". This is is incorrect -- .. cannot escape the starting point with RESOLVE_BENEATH (and O_RESOLVE_BENEATH as it was modeled after RESOLVE_BENEATH), regardless of where you put it in the path. However, you can walk into .. during a lookup without issue if it remains within the starting root.

Reading the rest of the code, it seems this was actually done because the fallback userspace resolver cannot handle .. and so the decision was to ban it for everything for "consistency" but this doesn't really make sense -- symlinks can contain .. so there will always be an inconsistency between the resolver that rejects all symlinks and those that can handle them. It seems like it would be more helpful to most users to permit .. for RESOLVE_BENEATH and only block it in the resolver.

My final patch does this, but I can drop it if you prefer.

For what it's worth, there actually are safe ways of handling .. in userspace. libpathrs (shameless plug!) does this on Linux using /proc/self/fd/... to check the logical path against the real path in a very solid way, and Go's os.Root has a brute-force approach that in principle works on any platform by restarting the lookup each time they hit .. and re-applying the logical path walked so far (I plan to adapt the approach for libpathrs at some point so it can be cross-platform as well).

I suspect this will also help alleviate some of the issues people have had with regressions on v3.8.3.