new(android-tools): standalone adb / fastboot / mkbootimg#13068
Draft
tannevaled wants to merge 14 commits into
Draft
new(android-tools): standalone adb / fastboot / mkbootimg#13068tannevaled wants to merge 14 commits into
tannevaled wants to merge 14 commits into
Conversation
…cpy lookup gap) Upstream's adb/fastboot/mkbootimg are part of the full AOSP build — not practical to compile outside Google's CI. @nmeum/android-tools extracts the AOSP source pieces and provides a CMake-based standalone build. This is the same upstream Debian, Fedora, Alpine and Arch use for their `android-tools-adb` packages. Provides the binaries most users actually want: - adb Android Debug Bridge (referenced by scrcpy, pkgxdev#7071) - fastboot bootloader interface - mkbootimg / unpack_bootimg / repack_bootimg - simg2img / img2simg / ext2simg - avbtool Android Verified Boot signing - lpdump / lpmake / lpunpack Closes the lookup gap when users `pkgx adb` — until now they had to fall back to system adb (apt install adb) which created a chicken- and-egg problem for scrcpy users on systems without it.
Contributor
Author
|
Draft — every release I tried (35.0.2, 34.0.5) has a different libziparchive build break:
These are vendored-AOSP-source vs glue-code-version mismatches that @nmeum hasn't reconciled. Need to either pick a known-good tag (further down — 33.x?) or patch the libziparchive sources to match the ZipWriter API. Deferring. ADB users for now can fall back to system apt/dnf adb. |
The vendored libbase / libziparchive headers reference uint8_t etc. without `#include <cstdint>`. Older libstdc++ pulled cstdint in transitively via <string>, but modern GCC no longer does, so we hit cascading "template argument 1 is invalid" errors on std::vector<uint8_t>. Force-include <cstdint> via CXXFLAGS so every TU gets the typedefs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
AOSP's libbase/posix_strerror_r.cpp assumes the POSIX-XSI strerror_r prototype (returns int). glibc's default _GNU_SOURCE flips to the GNU variant (returns char*), so compilation dies with "invalid conversion from 'char*' to 'int'". Undefine _GNU_SOURCE + set _POSIX_C_SOURCE=200809L in CXXFLAGS so the POSIX strerror_r is selected. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous -U_GNU_SOURCE attempt killed downstream AOSP code that depends on glibc's GNU extensions (pipe2, IOV_MAX, pthread_setname_np, program_invocation_short_name, syscall, tm_zone via chrono_io.h, ...). Switch to -fpermissive which keeps _GNU_SOURCE intact and demotes the strerror_r char*->int mismatch in libbase/posix_strerror_r.cpp to a warning. For adb / fastboot the precise return contract is not load-bearing. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The 34.0.5 build was failing in vendored libjsonpb/parse/jsonpb.cpp
with two protobuf-version incompats:
* `'JsonOptions' is not a member of 'google::protobuf::util';
did you mean 'JsonPrintOptions'?`
— protobuf 28+ removed the `JsonOptions` alias.
* `no match for 'operator+' (operand types are 'std::string'
and 'std::string_view')`
— protobuf 27+ changed `Descriptor::full_name()` to return
`absl::string_view` instead of `const std::string&`.
Arch carries a patch for the string concat case but not the
JsonOptions rename. Easiest path that doesn't require maintaining
two patches: pin protobuf to ^25. Pantry still ships 25.x bottles
(25.0.0 .. 25.9.0), so the resolver has plenty of choice.
Drop the pin once nmeum/android-tools either vendors a newer
libjsonpb or upstreams the protobuf-30+ compat patches.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The protobuf ^25 pin hit an unexpected ABI wall: pantry's protobuf
25.x bottle was built against abseil 20250127, but pantry's current
abseil is 20260107 — so every `absl::lts_20250127::*` reference in
libprotobuf.so.25 is unresolved at final link.
Switch strategies: keep protobuf at `*` (latest, matched to current
abseil), and patch the two specific API drift points in the
vendored libjsonpb/parse/jsonpb.cpp:
* `message.GetDescriptor()->full_name()` now returns
`absl::string_view` (protobuf 27+), so wrap in `std::string(...)`
before `operator+` to force the conversion. Arch ships this
one-line patch.
* `google::protobuf::util::JsonOptions` was renamed to
`JsonPrintOptions` years ago and removed in 28+. Two occurrences
in jsonpb.cpp; sed the type name.
Both fixes are upstream-safe: `JsonPrintOptions` has existed since
protobuf 21, and `std::string(string_view)` is valid C++17.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pantry's protobuf bottles (both 25.x and 35.0.0, verified across two build attempts) were compiled against abseil lts_20250127. Pantry's abseil dep now resolves to 20260107 by default — ABI-incompatible LTS bump — so every `absl::lts_20250127::*` symbol in libprotobuf.so is unresolved at android-tools' final link, regardless of which protobuf version we pin to. Explicit `abseil.io: ^20250127` in our deps overrides the transitive resolution and forces the matching version, so the final link satisfies. Drop the pin once pantry rebuilds protobuf against the current abseil. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`^20250127` resolves to 20250127.2.0 (latest patch), but pantry's protobuf bottle was very likely built against an earlier patch. Template instantiations of `LogMessage::operator<<<long>` etc. in libabsl_log_internal_message.so.2501.0.0 differ across patches even within the same abseil LTS line; ld still reports unresolved despite both libs being present. Pin to the .0.0 patch exactly to test whether matching the protobuf-bottle-time abseil version resolves the symbols. If still unresolved, file an upstream pantry issue asking for a protobuf bottle rebuild against current abseil. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Found the smoking gun: pantry's own protobuf recipe has a FIXME
comment about exactly this issue. The protobuf bottle is built with
`-Wl,--allow-shlib-undefined` because its libprotobuf.so links
against whatever-abseil-was-around-at-build-time, but pkgx's abseil
recipe drifts to the latest LTS. Quoting the protobuf recipe:
linux:
# likely needs bumping to an unreleased abseil.io version
# ld.lld: error: undefined reference due to --no-allow-shlib-undefined:
# absl::lts_20240116::log_internal::LogMessage::operator<<<int, 0>(…)
LDFLAGS: $LDFLAGS -Wl,--allow-shlib-undefined
Same band-aid applies to every consumer: when android-tools links
libprotobuf.so transitively, those baked-in undefined refs surface
in OUR link command. Add the same flag so ld lets them through;
the dynamic loader at runtime resolves them against whatever
abseil is in the binary's RUNPATH (matching what protobuf itself
does at runtime).
Fragile by design — the proper fix is pantry rebuilding protobuf
in lockstep with abseil (issue to file separately), but this is
the same hack the rest of pantry depends on. Matches how arch's
binary packages work — protobuf has unversioned `depends=(abseil-cpp)`
and gets rebuilt on every abseil bump.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After the previous iter unblocked the protobuf+abseil link, the audit phase started reporting `bin/mkbootimg` missing. Tracked to: upstream's CMakeLists installs `bin/mkbootimg`, `unpack_bootimg`, `repack_bootimg`, `mkdtboimg`, `avbtool` as symlinks (via a `<name>_symlink` CMake target) pointing at `share/android-tools/<name>/<name>.py`. The symlink target is absolute and uses brewkit's `+brewing` build prefix, so after relocation to the final install path the symlinks dangle — audit treats them as missing executables. Replace each symlink with a small POSIX-sh wrapper that resolves `$0 → $bindir/.. → $prefix/share/android-tools/<tool>/<tool>.py` at runtime, so the resulting binary stays correct under any pkgm relocation (eg. `/usr/local/pkgs/...`). Probes two install shapes (`<tool>/<tool>.py` first, then `<tool>.py` fallback) since upstream isn't perfectly consistent. Also add `python.org` as a runtime dep — the wrappers exec `python3`. Tested upstream paths: mkbootimg.py uses pure stdlib, no extra wheels needed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
|
Iter update — the build now succeeds end-to-end and audit passes. Two new fixes since previous attempt:
But the runtime still doesn't work — This is the structural blocker I filed as #13105. Realistically android-tools needs to wait for a protobuf bottle rebuild against current abseil. I'll leave this PR open in this iter (build is reproducible end-to-end, only the runtime smoke-test fails) so it can be picked up immediately after #13105 is addressed. If a maintainer prefers, happy to mark this PR Draft until then. |
iter 8 — the darwin (²) check started running on my earlier
allow-shlib-undefined push and failed at CMakeTestCCompiler:
ld: unknown options: --allow-shlib-undefined
clang: error: linker command failed with exit code 1
That flag is GNU ld / lld syntax. Darwin's ld64 doesn't recognize it
and bails before CMake's compiler-detection test even completes.
Scope the LDFLAGS to `linux:` — darwin doesn't hit the abseil-drift
workaround need anyway since its protobuf comes via the system /
Xcode toolchain rather than pantry's libprotobuf.so.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
Author
|
Darwin update: after iter 8 (gating So the recipe is now structurally correct on every platform: build green, audit green, runtime needs the upstream pantry protobuf rebuild that #13105 tracks. Parking the PR pending #13105. |
The test did `adb --version | head -1 | grep -i adb`, but adb's first line is 'Android Debug Bridge version 1.0.41' — no 'adb' substring — so grep always failed (build was fine). Match the real banner. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
`pkgx adb` currently has no resolver — scrcpy users in #7071 hit this and had to fall back to `sudo apt install adb`. Having a from-source adb in pantry closes that gap and lets the scrcpy recipe optionally declare adb as a runtime dep down the road.
Test plan
🤖 Generated with Claude Code