From f933c70d10f9a7a1338bc6b8642061f88e55fe89 Mon Sep 17 00:00:00 2001 From: Peter Smith Date: Thu, 25 Jun 2026 13:00:26 +1200 Subject: [PATCH 1/6] Add scoped extension-mechanism proposal OpenSpec change proposing a Git-style extension mechanism for lstk: when `lstk ` is not a built-in, lstk resolves and execs an `lstk-` binary, forwarding args/streams/exit code and passing runtime context via a versioned `LSTK_EXT_*` environment contract. Scope (4 capabilities): - extension-framework: bundled-dir + PATH resolution (bundled wins), built-in precedence, leading-only global-flag handling, side-effect-free help listing. - extension-runtime-context: emulator endpoint/type/port, config dir, auth token, and resolved global flags (e.g. LSTK_EXT_NON_INTERACTIVE). - extension-entitlement: lstk passes the token; the extension self-authorizes. The lstk-side signed-grant mechanism is explicitly deferred. - extension-bundling: ship LocalStack-built (closed-source) extensions like lstk-deploy by default, with a release-generated descriptions file for help text and atomic version-matched updates. Deferred: emulator attestation, signed entitlement grants, and user-facing `lstk extension` management commands. --- .../add-extension-mechanism/.openspec.yaml | 2 + .../changes/add-extension-mechanism/design.md | 122 +++++++++++++++++ .../add-extension-mechanism/proposal.md | 41 ++++++ .../specs/extension-bundling/spec.md | 71 ++++++++++ .../specs/extension-entitlement/spec.md | 49 +++++++ .../specs/extension-framework/spec.md | 123 ++++++++++++++++++ .../specs/extension-runtime-context/spec.md | 82 ++++++++++++ .../changes/add-extension-mechanism/tasks.md | 56 ++++++++ 8 files changed, 546 insertions(+) create mode 100644 openspec/changes/add-extension-mechanism/.openspec.yaml create mode 100644 openspec/changes/add-extension-mechanism/design.md create mode 100644 openspec/changes/add-extension-mechanism/proposal.md create mode 100644 openspec/changes/add-extension-mechanism/specs/extension-bundling/spec.md create mode 100644 openspec/changes/add-extension-mechanism/specs/extension-entitlement/spec.md create mode 100644 openspec/changes/add-extension-mechanism/specs/extension-framework/spec.md create mode 100644 openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md create mode 100644 openspec/changes/add-extension-mechanism/tasks.md diff --git a/openspec/changes/add-extension-mechanism/.openspec.yaml b/openspec/changes/add-extension-mechanism/.openspec.yaml new file mode 100644 index 00000000..ff1fbc83 --- /dev/null +++ b/openspec/changes/add-extension-mechanism/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-06-19 diff --git a/openspec/changes/add-extension-mechanism/design.md b/openspec/changes/add-extension-mechanism/design.md new file mode 100644 index 00000000..c51ea1a7 --- /dev/null +++ b/openspec/changes/add-extension-mechanism/design.md @@ -0,0 +1,122 @@ +## Context + +`lstk` today is a single Go binary whose commands are wired in `cmd/` (Cobra) with domain logic in `internal/`. New capabilities must be merged into the open-source core, which blocks proprietary/partner features and couples their release to the core. We want to let internal and external authors add `lstk ` subcommands as independent binaries — open or closed source — placed on the user's `PATH`. lstk's job is dispatch plus handing the extension enough runtime context to be useful; any authorization decision belongs to the extension. + +The closest existing pattern in the codebase is the IaC proxy (`internal/iac//cli/`, wired via `cmd/iac.go`): lstk resolves an external binary, builds a child environment (resolved endpoint, credentials, stripped real-AWS vars), execs it with `DisableFlagParsing: true`, streams stdio, and propagates the exit code. The extension mechanism generalizes this pattern from a fixed set of known tools to a dynamic, user-installable set discovered by name on `PATH`. + +Relevant existing building blocks: +- **Auth**: `internal/auth` resolves the token (keyring → `LOCALSTACK_AUTH_TOKEN` → file fallback). The token is conveyed to the extension; lstk does not validate licenses for extensions. +- **Endpoint resolution**: `internal/endpoint.ResolveHost()` and container discovery (`internal/container`) already produce the emulator URL used by `lstk aws`/IaC proxies. +- **Config dir**: `internal/config` resolves the lstk config directory; extension dispatch passes it through. + +## Goals / Non-Goals + +**Goals:** +- Git-style dispatch: `lstk ` → `lstk-` executable on `PATH` when `` is not built-in, with built-ins always winning. +- A stable, versioned environment-variable contract (`LSTK_EXT_*`) carrying emulator endpoint/type/port, config dir, and auth token. +- Convey the auth token so an extension can authorize the user itself; lstk makes no entitlement decision. +- Parse lstk's own global flags (e.g. `--non-interactive`) before the command name and convey the resolved state to the extension via the `LSTK_EXT_*` contract. +- Bundle LocalStack's own extensions (e.g. a closed-source `lstk-deploy`) alongside `lstk` so they ship by default, resolve ahead of `PATH`, and update atomically with lstk. +- Help listing (bundled dir + PATH scan) so installed extensions are discoverable by name. +- Keep the mechanism in the open-source repo; closed-source extensions ship only as binaries (bundled by LocalStack, or placed on `PATH` by anyone else). + +**Non-Goals (deferred to future work):** +- lstk-side entitlement verification, signed grants / signed entitlement descriptions, and offline grant verification by extensions. +- Emulator genuineness attestation (licensed vs. clone). +- User-facing `lstk extension` management commands (`install`/`remove`/`list`/`info`) and a user-mutable managed extensions directory. (Static, ships-with-lstk bundling is in scope — see Decision 6 — but the user-driven management UX is not.) +- An in-process plugin ABI (Go plugins, shared libraries, WASM) — we deliberately use separate processes for language-agnostic, closed-source friendliness. +- Sandboxing or capability-limiting extension processes — extensions run with the user's full privileges, same as the IaC proxies and any tool on `PATH`. + +## Decisions + +### Decision 1: Separate-process, Git-style extensions (not in-process plugins) + +`lstk ` resolves and execs an `lstk-` binary on `PATH`, forwarding args/stdio/exit code, exactly like Git's `git-` model and lstk's own IaC proxies. + +**Rationale**: This is the only model that cleanly satisfies "open or closed source" and "internal or external authors" — a closed-source extension ships as an opaque binary in any language and never touches the core repo. Go's `plugin` package is Linux/macOS-only, requires identical toolchain/build flags, and exposes a Go ABI (excludes other languages and complicates closed source). WASM would sandbox nicely but can't easily run arbitrary partner toolchains and adds a heavy runtime. Process-per-invocation matches an existing, proven pattern in this codebase. + +**Alternatives considered**: Go `plugin` shared objects (rejected: platform/toolchain coupling, Go-only); embedded scripting/WASM (rejected: language lock-in or runtime weight); MCP-style long-running server processes (rejected: overkill for CLI subcommands, more lifecycle complexity). + +### Decision 2: Dispatch via Cobra's unknown-command hook, built-ins first + +Wire extension dispatch in `cmd/root.go` so that resolution happens only after Cobra fails to match a built-in command/alias. This guarantees built-ins always shadow extensions (spec requirement) and avoids parsing extension flags (`DisableFlagParsing` semantics applied to the synthesized extension command, as the IaC proxies do). + +**Rationale**: Reuses Cobra's existing matching and keeps the change localized. An extension can never override `start`, `snapshot`, etc., which matters because extensions are user-installable. + +**Global flags before the command name**: lstk's global flags (`--non-interactive`, `--config`, and any added later) must be recognized when they precede the extension name but must not be parsed out of the extension's own args. The mechanism is `SetInterspersed(false)` on the root flag set: Cobra parses *leading* flags into `cfg` and stops at the first positional (the command name), handing the dispatch path everything from the command name onward verbatim. This gives Git-style "globals only before the command" for free and, because it reuses the existing persistent-flag definitions, any future global flag is handled with no extension-specific code. This deliberately differs from the IaC proxies, which strip `--non-interactive` from *any* position via `stripGlobalFlags` ([cmd/proxy.go](../../../cmd/proxy.go)); for generic extensions, stripping a flag out of the middle of the extension's args could clobber an extension's own identically-named flag, so we only consume leading globals. The proxy's manual `stripGlobalFlags` remains the fallback if `SetInterspersed(false)` interacts badly with the existing bare-root `start` behavior. + +**Alternatives considered**: Pre-scanning `os.Args[1]` before Cobra runs (rejected: duplicates Cobra's alias/normalization logic and risks divergence). Registering a synthetic Cobra command per discovered extension at startup (we do a lightweight bundled-dir + PATH scan for `--help` listing but keep dispatch in the unknown-command path to avoid eagerly stat-ing the filesystem on every invocation). Reusing `stripGlobalFlags` verbatim (rejected as the primary path: its strip-from-anywhere behavior risks colliding with extension-owned flags). + +### Decision 3: Environment-variable runtime contract (`LSTK_EXT_*`), versioned + +Context flows to the extension purely through environment variables prefixed `LSTK_EXT_`, with `LSTK_EXT_API_VERSION` as an integer contract version. Variables in this change: `LSTK_EXT_API_VERSION`, `LSTK_EXT_EMULATOR_ENDPOINT`, `LSTK_EXT_EMULATOR_TYPE`, `LSTK_EXT_EMULATOR_PORT`, `LSTK_EXT_CONFIG_DIR`, `LSTK_EXT_AUTH_TOKEN`, and `LSTK_EXT_NON_INTERACTIVE`. The host environment is inherited; only `LSTK_EXT_*` is added/overridden. Endpoint vars are omitted when no emulator is running; the auth token is omitted when none is available. + +Resolved global flags travel by env, not argv: `LSTK_EXT_NON_INTERACTIVE` is set when the session is non-interactive (the `--non-interactive` flag was given or stdout is not a TTY — the same condition lstk already computes at [cmd/root.go](../../../cmd/root.go) `isInteractive`). Carrying global state by env (rather than re-forwarding `--non-interactive` on the extension's command line) is what lets the extension own its entire flag space without collision, and each new global flag becomes one additive `LSTK_EXT_*` variable. + +**Rationale**: Mirrors how the IaC proxies already pass context, is language-agnostic (every runtime can read env), and avoids forcing extensions to parse lstk's TOML or re-implement discovery. Versioning makes the contract evolvable: additive within a major version, bump on any removal/repurpose. A future signed-entitlement description would be added additively (e.g. `LSTK_EXT_GRANT` / `LSTK_EXT_PUBLIC_KEY`) under a documented version bump. + +**Alternatives considered**: A JSON context file referenced by a single env var (cleaner for large/structured payloads; reconsider if the contract grows, but env is simpler for the current fields and avoids temp-file lifecycle/cleanup). Passing context as CLI flags (rejected: collides with extension-owned flag space). + +### Decision 4: No lstk-side manifest — extensions are self-describing + +lstk executes any resolvable `lstk-` on `PATH` without reading a manifest. Contract compatibility is checked by the extension reading `LSTK_EXT_API_VERSION`; whether auth is needed is the extension's call (lstk always passes the token when it has one); a human description is shown in help only as the command name (PATH scan). + +**Rationale**: This is the pure Git model (`git-` extensions have no manifest) and it removes a whole class of questions — where the manifest lives in a shared `PATH` bindir, how lstk trusts it, how it stays in sync with the binary. Because lstk no longer makes any entitlement decision (Decision 5), it has no need to read an entitlement name from a manifest either. + +**Alternatives considered**: A co-located `extension.toml` (rejected for now: ambiguous in shared `PATH` directories and unnecessary once lstk does no pre-launch gating). A managed extensions directory that owns manifests (deferred together with `lstk extension` management). + +### Decision 5: Authorization belongs to the extension; the signed mechanism is deferred + +lstk conveys `LSTK_EXT_AUTH_TOKEN` and makes no entitlement or license decision for an extension. An extension that wants to restrict its use authorizes the user itself — typically a server-side check against the LocalStack platform with the conveyed token. + +**Rationale**: Because lstk is open source and rebuildable, any lstk-side gate is a UX speed bump, not a control — the only durable boundary is a check the extension performs against something a modified lstk cannot forge (a server response keyed to the user's token, or a signature the extension verifies). Deferring the signed mechanism lets the host land now with no platform changes; when signing is added, it slots into the `LSTK_EXT_*` contract additively without changing dispatch. + +**Alternatives considered**: lstk-side entitlement gate + per-extension signed grant (the original design; deferred, not rejected — it improves UX and enables offline verification but requires platform work and adds complexity not needed for the first cut). Purely client-side license checks inside lstk (rejected: trivially bypassable, no IP protection). + +### Decision 6: Bundled LocalStack extensions live next to the binary and resolve ahead of PATH + +LocalStack ships its own extensions (e.g. a closed-source `lstk-deploy`) in a fixed directory derived from lstk's own symlink-resolved executable path — the Git `libexec/git-core` model. Resolution order becomes: built-ins → bundled dir → `PATH`, so a bundled extension wins over a same-named `PATH` executable. `internal/update` replaces lstk and its bundled extensions as one atomic, version-matched set. This reuses the `lstk-` naming convention (no manifest) and does not change the authorization model — a bundled premium extension still self-authorizes with the conveyed token. + +**Rationale**: Searching the directory next to the resolved executable is robust where co-located binaries are not on `PATH` — bare tarballs where the user symlinks only `lstk`, and npm/Homebrew layouts where the invoked `lstk` is a shim (hence resolving symlinks to find the real sibling dir). Bundled-wins precedence makes the official `lstk-deploy` deterministic: a stray or malicious `lstk-deploy` on `PATH` cannot hijack it. This is a deliberately narrow re-introduction of what the management capability deferred: a *static, read-only, ships-with-lstk* location and the packaging/update wiring — but **not** the user-mutable managed dir or the `lstk extension install/remove` UX, which stay deferred. The closed-source binaries are built in private CI and injected into the public release artifacts, version-pinned to the lstk release that carries them. + +**Alternatives considered**: Just install bundled binaries onto `PATH` (rejected: zero lstk change but fragile for bare tarballs and pollutes the user's `PATH`). PATH-wins precedence (rejected: lets a stray binary shadow the official extension; bundled-wins is safer for closed-source premium commands). A managed mutable dir under the config dir (deferred with the rest of management; bundling needs only a static dir). + +### Decision 7: One-line help descriptions from a release-generated file, bundled-only + +Bundled extensions get a one-line description in `lstk help` from a static descriptions file generated during the release process and shipped alongside the bundled extensions (a single file mapping command name → description, e.g. `extensions.toml` in the bundled dir). lstk reads it at help time and never executes an extension to obtain text. `PATH`/custom extensions are always name-only. + +**Rationale**: This keeps help rendering side-effect-free — the property we lost in the earlier exec-based approach. Because LocalStack controls the bundled set, their descriptions are known at release time, so there is no need to run anything: no timeout, no cache, no incidental-usage-on-a-typo hazard, no risk of executing untrusted `PATH` binaries during help. It is not a return of the per-extension `extension.toml` manifest (Decision 4): it is one release-owned file covering only the bundled set, generated by CI and version-locked to the binaries it describes, not authored per-extension by third parties. The cost — descriptions are static and only available for bundled extensions — is acceptable: third-party `PATH` extensions being name-only matches Git's `git help -a`. + +**Alternatives considered**: A describe protocol that execs `lstk- lstk:describe` (rejected: turns inert help into code execution, needs timeout/cache/trust-boundary machinery, and a mistyped command rendering Cobra usage could fork extensions). Per-extension manifests (rejected with Decision 4). Name-only for everything (viable and simplest, but the user wants descriptions for bundled extensions like `lstk-deploy`, which this delivers with no exec). + +## Threat Model: hostile lstk rebuild (why authorization lives in the extension) + +Because lstk is and remains open source, the adversary is assumed to have the full lstk source and can rebuild it with any check removed or any value forged. The single question every security claim must pass is: **does this check depend on lstk behaving honestly, or on a response the attacker cannot produce?** Anything in the first category is a UX speed bump, not a control. + +Consequence for this change: lstk deliberately holds **no** authorization logic. It conveys the auth token and dispatches. An extension that needs protection must anchor its enforcement server-side (a LocalStack platform call keyed to the token) or in its own verification — a rebuilt lstk that strips or forges `LSTK_EXT_*` values cannot make such a check pass. This is also why the eventual signed-entitlement description (deferred) is designed to be verified *by the extension* against LocalStack's public key, not gated by lstk. + +Residual risk we explicitly accept: an extension whose value is purely local logic cannot be protected client-side at all — an attacker rebuilds it with the checks removed. The only durable protection is server-side gating of the valuable behavior. This is the standard DRM reality and should be stated plainly in the author guide rather than implied away. + +## Risks / Trade-offs + +- **Auth token exposed to extension processes via env** → Tokens already flow to subprocesses for IaC proxies; only pass it when one is resolved, and document the trust boundary. A future scoped/short-lived token can replace the raw auth token without changing the contract shape. +- **Name collisions / malicious shadowing** → Built-ins always win (dispatch only on unknown commands); bundled LocalStack extensions win over `PATH`, so a stray `lstk-deploy` on `PATH` cannot hijack the official one. Among `PATH` entries, standard first-match-wins resolution applies, same as any `lstk-*` or shell command. +- **Bundled/lstk version skew** → `internal/update` must replace lstk and its bundled extensions atomically; an interrupted update must not leave them mismatched. Treated as a single versioned set. +- **npm/Homebrew shim hides the real binary location** → resolve symlinks to find lstk's real executable before locating the sibling bundled dir; verify on each channel. +- **Untrusted extension binaries run with user privileges** → Same trust model as installing any CLI tool or the IaC binaries lstk already shells out to; no sandboxing is promised. +- **No lstk-side authorization in this cut** → An extension that forgets to authorize is wide open; the author guide must make the self-authorization responsibility explicit and provide a recommended pattern (server-side check with the conveyed token). +- **Contract drift between lstk and extensions** → `LSTK_EXT_API_VERSION` lets extensions detect/require a minimum contract and refuse to run when incompatible. + +## Migration Plan + +- Purely additive: no built-in command behavior changes, so no user migration is required. Existing `terraform-proxy`/`cdk-proxy` specs are untouched. +- Land the host mechanism (dispatch + runtime context) with a reference extension so discovery, dispatch, the `LSTK_EXT_*` contract, and help listing are testable with no platform dependency. +- Rollback: because dispatch only triggers on unknown commands, having no `lstk-*` on `PATH` leaves built-in behavior identical. + +## Open Questions + +- **Token scoping**: Should extensions receive the raw `LOCALSTACK_AUTH_TOKEN` or a derived, audience-/extension-scoped, short-lived token? (Affects the eventual signed mechanism too.) +- **Help-listing cost**: Scanning the bundled dir + all of `PATH` for `lstk-*` on every `--help` has a filesystem cost; is lazy/cached scanning warranted, or is on-demand fine? +- **Closed-source build/release pipeline**: Where do the prebuilt closed-source bundled binaries (e.g. `lstk-deploy`) come from at release time — a private artifact registry, a private release? How are their versions pinned to the lstk release, and how does the public repo's release workflow pull them into the Homebrew formula, npm package, and binary archive without exposing source? +- **Exact bundled-dir layout**: Is it the same directory as the lstk binary, or a dedicated sibling (e.g. `libexec`-style) to avoid mixing with unrelated binaries — and how does each channel (Homebrew Cellar/libexec, npm package dir, tarball root) lay it out so the symlink-resolved lookup finds it? +- **Future signed entitlement**: When the deferred mechanism lands, what is the token format (JWT? which alg/curve), how is LocalStack's public key distributed to extensions, and what is the key-rotation strategy? (Tracked for the follow-up change, not this one.) diff --git a/openspec/changes/add-extension-mechanism/proposal.md b/openspec/changes/add-extension-mechanism/proposal.md new file mode 100644 index 00000000..68775b52 --- /dev/null +++ b/openspec/changes/add-extension-mechanism/proposal.md @@ -0,0 +1,41 @@ +## Why + +`lstk` ships a fixed set of built-in commands, so any new capability — whether built by LocalStack engineers or partners — has to land in the core open-source repository. That blocks closed-source/proprietary features, slows third-party contribution, and couples every new feature's release to the core CLI's release cadence. We want a Git-style extension mechanism so anyone can add `lstk ` subcommands as separate binaries on their `PATH`, with lstk handing each extension enough runtime context (resolved emulator endpoint, config dir, auth token) to do useful work — while leaving any authorization decision to the extension itself. + +## What Changes + +- Introduce a Git-style extension model: when `lstk ` is not a built-in command, lstk resolves and executes an `lstk-` executable found on `PATH`, forwarding all remaining arguments and propagating the child's stdin/stdout/stderr and exit code. +- Define an **extension runtime contract**: lstk passes the resolved emulator endpoint, emulator type, config directory, auth token, and resolved global-flag state to the extension process through a stable, versioned set of `LSTK_EXT_*` environment variables so extensions can talk to the emulator and the platform without re-implementing discovery or config resolution. +- **Honor global flags before the command name**: lstk parses its own global flags (e.g. `--non-interactive`, and any added later) when they precede the extension name, consumes them itself, and conveys the resolved state to the extension via `LSTK_EXT_*` (e.g. `LSTK_EXT_NON_INTERACTIVE`) rather than forwarding them on the extension's command line. +- **Bundle LocalStack's own extensions**: ship LocalStack-built extensions (e.g. a closed-source `lstk-deploy`) by default alongside `lstk` in a directory next to the binary, resolved ahead of `PATH` and updated atomically with `lstk` — with no user-facing install step. +- Establish that **authorization is the extension's responsibility**: lstk conveys the user's auth token and makes no entitlement decision of its own. An extension that needs to restrict its use authorizes the user itself (e.g. by calling the LocalStack platform with the conveyed token). A richer lstk-side mechanism — lstk obtaining a LocalStack-signed entitlement description for the extension to verify offline — is deliberately **deferred** to future work. +- List resolvable extensions in `lstk help` by scanning the bundled directory and `PATH` for `lstk-*` executables; bundled extensions show a one-line description read from a static descriptions file generated during the release process, while `PATH`/custom extensions are name-only. Help rendering never executes an extension. +- Keep the entire mechanism in the open-source repository; closed-source extensions ship only as binaries placed on `PATH` and never require source in the core repo. + +## Capabilities + +### New Capabilities + +- `extension-framework`: Git-style discovery, resolution, and dispatch of `lstk-` extension executables (bundled dir + `PATH`), including built-in precedence, leading-only global-flag handling, forwarding of arguments/streams/exit codes, and side-effect-free help listing (bundled extensions described from a static file, others name-only). No lstk-side manifest — extensions are self-describing and self-validating. +- `extension-runtime-context`: The versioned environment-variable contract lstk establishes for an extension process — resolved emulator endpoint/type/port, config directory, auth token, and resolved global-flag state (e.g. non-interactive) — so extensions can reach the emulator and platform and honor lstk's global flags. +- `extension-entitlement`: The authorization model — lstk conveys the auth token and the extension authorizes itself — plus the explicit deferral of any lstk-side signed-entitlement mechanism and the security rationale (lstk is open source, so authorization cannot depend on it). +- `extension-bundling`: Shipping LocalStack-built (possibly closed-source) extensions by default alongside `lstk` — a read-only bundled directory next to the binary, resolution ahead of `PATH`, a release-generated descriptions file for help text, cross-channel packaging (binary archive, Homebrew, npm), and atomic version-matched updates via `internal/update`. Excludes user-facing management commands and a user-mutable directory. + +### Modified Capabilities + + + +## Impact + +- **New code**: `internal/extension/` (bundled-dir + PATH resolution, runtime context builder, global-flag conveyance, exec), and unknown-command dispatch + help-listing wiring in `cmd/root.go`. +- **Touched code**: `cmd/root.go` (fallthrough to extension dispatch for unknown commands; `SetInterspersed(false)` for leading-only global flags; bundled-dir + PATH scan for help), reuse of `internal/auth` (token resolution), `internal/config`/`internal/endpoint` (config dir and emulator endpoint resolution), `internal/container` (running-emulator discovery for endpoint/type), and `internal/update` (atomic version-matched update of bundled extensions). +- **Packaging/release**: binary archive, Homebrew formula, and npm package must lay out bundled extensions where lstk resolves them; the public release workflow must pull prebuilt closed-source bundled binaries from private CI, version-pinned to the lstk release. +- **External dependencies/services**: None required by this change. Extensions that authorize use the existing LocalStack platform with the conveyed auth token; no new platform or emulator endpoints are needed. +- **Security surface**: lstk passes the auth token into extension processes via env (as it already does for IaC proxies); this defines a local trust boundary to document. Authorization guarantees live in the extension, never in lstk. +- **Docs**: New "Extensions" section in CLAUDE.md and a public extension-author guide (manifest-free contract, `LSTK_EXT_*` variables, the self-authorization model and why it cannot rely on lstk). + +## Deferred (future work) + +- lstk-side entitlement verification and a LocalStack-signed entitlement description (grant) that extensions verify offline against a published public key. +- Emulator genuineness attestation (distinguishing a licensed emulator from a clone). +- User-facing `lstk extension` management commands (`list`/`info`/`install`/`remove`) and a user-mutable managed extensions directory. (Static, ships-with-lstk bundling is **in scope** via the `extension-bundling` capability; only the user-driven install/remove UX is deferred.) diff --git a/openspec/changes/add-extension-mechanism/specs/extension-bundling/spec.md b/openspec/changes/add-extension-mechanism/specs/extension-bundling/spec.md new file mode 100644 index 00000000..49b06952 --- /dev/null +++ b/openspec/changes/add-extension-mechanism/specs/extension-bundling/spec.md @@ -0,0 +1,71 @@ +# extension-bundling Specification + +## Purpose + +Allow LocalStack to ship its own extensions (for example a closed-source `lstk-deploy`) by default alongside `lstk`, so they are available immediately after a standard install with no separate step, are kept in lockstep with the `lstk` version that ships them, and are resolved deterministically ahead of any same-named executable on `PATH`. This capability covers only static, read-only, ships-with-lstk extensions; user-driven install/remove of extensions and a user-mutable managed directory remain out of scope (deferred). + +## ADDED Requirements + +### Requirement: Bundled-extensions directory alongside the executable + +lstk SHALL look for bundled extensions in a fixed directory derived from the location of its own executable, resolving symlinks so the directory is found even when `lstk` is invoked through a symlink or package shim (e.g. an npm `.bin` link). Bundled extension executables follow the same `lstk-` naming convention as any other extension and SHALL NOT require a manifest. This directory is owned by the lstk distribution and is read-only from the user's perspective: lstk SHALL NOT provide commands to add to or remove from it in this change. + +#### Scenario: Bundled directory resolved through a symlink + +- **WHEN** `lstk` is invoked via a symlink or package shim and an `lstk-deploy` is bundled +- **THEN** lstk resolves its real executable location, finds the bundled-extensions directory, and can resolve `lstk deploy` + +#### Scenario: Naming convention identifies bundled extensions + +- **WHEN** the bundled-extensions directory contains an executable named `lstk-deploy` +- **THEN** lstk treats it as the `deploy` extension without reading any manifest + +### Requirement: Bundled extensions are available after a standard install + +A set of extensions MAY be designated as bundled and SHALL be installed alongside `lstk` by the same single installation command across supported distribution channels (binary archive, Homebrew, npm), placed in the bundled-extensions directory, and resolvable immediately as `lstk ` with no separate install step. Packaging SHALL place bundled extensions where lstk resolves them without requiring the user to add them to `PATH`. + +#### Scenario: Bundled extension available immediately + +- **WHEN** a user installs lstk via the standard installation command for any supported channel and `lstk-deploy` is bundled +- **THEN** `lstk deploy` resolves to the bundled extension with no extra install step + +#### Scenario: Bundled extension found without PATH changes + +- **WHEN** a user extracts the binary archive and places only `lstk` on `PATH` +- **THEN** a bundled `lstk-deploy` sibling is still resolved by `lstk deploy` because lstk searches the directory alongside its executable + +### Requirement: Bundled extensions update atomically with lstk + +Updating lstk SHALL update its bundled extensions to the matching version as a single, atomic set, so a running `lstk` and its bundled extensions are never left at mismatched versions. `internal/update` SHALL replace the lstk executable and its bundled extensions together regardless of the install method, or fail without partially updating. + +#### Scenario: Bundled extensions updated with lstk + +- **WHEN** lstk is updated to a new version that ships a newer bundled `lstk-deploy` +- **THEN** the bundled `lstk-deploy` is replaced with the matching version as part of the same update +- **AND** an interrupted update does not leave lstk and the bundled extension at mismatched versions + +### Requirement: Release-generated descriptions file for bundled extensions + +The release process SHALL generate a static descriptions file that maps each bundled extension's command name to a one-line description, and SHALL ship it with the distribution where lstk reads it (alongside the bundled extensions). The file SHALL cover only bundled, LocalStack-controlled extensions; it is not a per-extension manifest authored by third parties. It SHALL be versioned and updated together with the bundled extension set, so descriptions never drift from the binaries that ship. lstk reads this file for help rendering (see the extension-framework capability) and never executes an extension to obtain a description. + +#### Scenario: Descriptions file ships with the bundled set + +- **WHEN** a release bundles `lstk-deploy` +- **THEN** the release process produces a descriptions file entry mapping `deploy` to its one-line description +- **AND** that file is shipped where lstk resolves bundled extensions + +#### Scenario: Descriptions update atomically with the bundled set + +- **WHEN** lstk is updated to a version that bundles a renamed or re-described extension +- **THEN** the descriptions file is updated as part of the same atomic update +- **AND** lstk never shows a description that disagrees with the bundled binaries + +### Requirement: Bundled closed-source extensions still self-authorize + +Bundling SHALL NOT change the authorization model: a bundled extension that gates on entitlement (for example a premium closed-source extension) SHALL perform its own authorization using the conveyed auth token exactly as a separately distributed extension would. lstk SHALL NOT treat a bundled extension as automatically entitled. + +#### Scenario: Bundled premium extension enforces its own entitlement + +- **WHEN** a bundled `lstk-deploy` requires entitlement and an unentitled user runs `lstk deploy` +- **THEN** lstk dispatches to the bundled extension and conveys the token +- **AND** the bundled extension performs its own authorization and refuses the unentitled user diff --git a/openspec/changes/add-extension-mechanism/specs/extension-entitlement/spec.md b/openspec/changes/add-extension-mechanism/specs/extension-entitlement/spec.md new file mode 100644 index 00000000..91489fd0 --- /dev/null +++ b/openspec/changes/add-extension-mechanism/specs/extension-entitlement/spec.md @@ -0,0 +1,49 @@ +# extension-entitlement Specification + +## Purpose + +Establish that authorization for an extension is the extension's own responsibility, and that lstk's role is limited to conveying the user's auth token so the extension can decide for itself whether the user is entitled. A richer lstk-side mechanism (lstk obtaining a LocalStack-signed entitlement description and passing it for offline verification) is deliberately deferred to future work; this capability records that intent and the security rationale behind it. + +## ADDED Requirements + +### Requirement: lstk conveys the auth token; the extension authorizes + +lstk SHALL make the resolved user auth token available to the extension via the runtime context (`LSTK_EXT_AUTH_TOKEN`) and SHALL NOT itself perform any entitlement or license decision for the extension. An extension that wishes to restrict its use SHALL perform its own authorization check — for example, by calling the LocalStack platform with the conveyed token — and SHALL refuse to perform protected work when that check does not pass. + +#### Scenario: Token conveyed for the extension to authorize + +- **WHEN** a user with a resolved auth token invokes an extension +- **THEN** lstk passes the token to the extension via `LSTK_EXT_AUTH_TOKEN` +- **AND** lstk invokes the extension without making any entitlement decision of its own + +#### Scenario: Extension enforces its own authorization + +- **WHEN** an extension that gates on entitlement determines the user is not entitled +- **THEN** the extension refuses to perform its protected work +- **AND** this decision is made by the extension, not by lstk + +#### Scenario: Unauthenticated invocation still dispatches + +- **WHEN** no auth token is available and a user invokes an extension +- **THEN** lstk still resolves and executes the extension (omitting `LSTK_EXT_AUTH_TOKEN`) +- **AND** any requirement for authentication is enforced by the extension itself + +### Requirement: Security rests on the extension, not on lstk + +Because lstk is open source and can be rebuilt with any check removed, no authorization guarantee SHALL depend on lstk behaving honestly. An extension that needs durable protection SHALL anchor its enforcement in something a modified lstk cannot forge — a server-side check against the LocalStack platform using the conveyed token, and/or verification it performs itself — rather than relying on lstk to gate invocation. + +#### Scenario: Modified lstk cannot bypass extension authorization + +- **WHEN** a rebuilt lstk skips conveying the token or alters its behavior +- **THEN** an extension that authorizes server-side (or otherwise verifies independently) still refuses unauthorized work +- **AND** the absence of an lstk-side gate does not weaken the extension's protection + +### Requirement: Signed-entitlement mechanism is deferred + +This change SHALL NOT implement lstk-side entitlement verification, signed grant/entitlement-description issuance, or offline grant verification. These remain future work. lstk SHALL NOT set `LSTK_EXT_GRANT` or `LSTK_EXT_PUBLIC_KEY`; if a future change introduces a LocalStack-signed entitlement description, it will be added as an additive extension to the runtime-context contract under a documented version bump. + +#### Scenario: No grant or public key conveyed + +- **WHEN** lstk invokes any extension +- **THEN** `LSTK_EXT_GRANT` is not set +- **AND** `LSTK_EXT_PUBLIC_KEY` is not set diff --git a/openspec/changes/add-extension-mechanism/specs/extension-framework/spec.md b/openspec/changes/add-extension-mechanism/specs/extension-framework/spec.md new file mode 100644 index 00000000..c700c24f --- /dev/null +++ b/openspec/changes/add-extension-mechanism/specs/extension-framework/spec.md @@ -0,0 +1,123 @@ +# extension-framework Specification + +## Purpose + +Provide a Git-style extension model so that `lstk ` invokes an external `lstk-` executable when `name` is not a built-in command, forwarding arguments, streams, and exit codes, and resolving extensions from the user's `PATH`. + +## ADDED Requirements + +### Requirement: Unknown commands dispatch to extension executables + +When a user runs `lstk [args...]` and `` does not match any built-in command or alias, lstk SHALL attempt to resolve and execute an extension executable named `lstk-`. If no such executable is found, lstk SHALL fail with its standard unknown-command error and a non-zero exit code. + +Built-in commands SHALL always take precedence over extensions: an extension named `lstk-` SHALL NOT shadow or override a built-in command ``. + +#### Scenario: Built-in command takes precedence + +- **WHEN** a user runs `lstk start` and a built-in `start` command exists +- **THEN** the built-in command runs +- **AND** no `lstk-start` executable is searched for or executed + +#### Scenario: Unknown command resolves to an extension + +- **WHEN** a user runs `lstk hello world` and no built-in `hello` command exists but an `lstk-hello` executable is resolvable on `PATH` +- **THEN** lstk executes the `lstk-hello` executable +- **AND** passes `world` as its argument + +#### Scenario: Unknown command with no matching extension + +- **WHEN** a user runs `lstk doesnotexist` and neither a built-in command nor an `lstk-doesnotexist` executable exists +- **THEN** lstk prints an unknown-command error +- **AND** exits with a non-zero status + +### Requirement: Extension resolution order + +lstk SHALL resolve `lstk-` executables by searching, in order: (1) the bundled-extensions directory alongside the lstk executable (see the extension-bundling capability), then (2) the directories on the user's `PATH`, using the platform's standard executable lookup. The first match SHALL be used, so a bundled extension takes precedence over a `PATH` executable of the same name. On Windows, platform executable extensions (e.g. `.exe`, `.cmd`, `.bat`) SHALL be honored when resolving the executable name. + +#### Scenario: Bundled extension wins over PATH + +- **WHEN** an `lstk-deploy` exists both in the bundled-extensions directory and on the user's `PATH` +- **THEN** lstk executes the bundled `lstk-deploy` + +#### Scenario: Resolves from PATH when not bundled + +- **WHEN** an `lstk-hello` executable exists on the user's `PATH` and not in the bundled-extensions directory +- **THEN** lstk executes it when the user runs `lstk hello` + +#### Scenario: Not found anywhere + +- **WHEN** no `lstk-hello` executable exists in the bundled-extensions directory or on the user's `PATH` +- **THEN** lstk reports an unknown-command error and exits non-zero + +### Requirement: Argument, stream, and exit-code forwarding + +When invoking an extension, lstk SHALL forward all arguments that follow `` to the extension executable unmodified, and SHALL NOT attempt to parse or interpret extension-specific flags. lstk's own global flags are recognized only when they appear before ``; everything from `` onward is treated as opaque and forwarded verbatim (see the extension-runtime-context capability for how resolved global flags reach the extension). lstk SHALL pass through the child process's standard input, standard output, and standard error unmodified, and SHALL propagate the child process's exit code as lstk's own exit code. + +#### Scenario: Flags after the command name are forwarded, not parsed by lstk + +- **WHEN** a user runs `lstk hello --verbose --name=foo` +- **THEN** lstk invokes `lstk-hello` with `--verbose --name=foo` +- **AND** lstk does not error on unknown flags + +#### Scenario: Global flags before the command name are consumed by lstk + +- **WHEN** a user runs `lstk --non-interactive hello --verbose` +- **THEN** lstk consumes `--non-interactive` itself and invokes `lstk-hello` with only `--verbose` +- **AND** an extension's own flag of the same name appearing after `` is forwarded unchanged + +#### Scenario: Exit code is propagated + +- **WHEN** the `lstk-hello` extension exits with status 3 +- **THEN** lstk exits with status 3 +- **AND** lstk does not print an additional lstk-level error message + +#### Scenario: Streams are passed through + +- **WHEN** an extension reads from stdin and writes to stdout and stderr +- **THEN** the user's terminal stdin/stdout/stderr are connected to the extension unaltered + +### Requirement: Extensions are self-describing; no lstk-side manifest + +lstk SHALL NOT require a manifest file to discover, validate, or invoke an extension. Any compatibility or requirement checks (for example, a minimum supported contract version, or whether authentication is needed) are the extension's own responsibility: the extension SHALL determine these for itself from the runtime context (notably `LSTK_EXT_API_VERSION`) and SHALL refuse to run when its requirements are not met. lstk SHALL execute any resolvable `lstk-` executable without inspecting metadata about it. + +#### Scenario: No manifest required to run + +- **WHEN** an `lstk-hello` executable exists on `PATH` with no accompanying metadata file +- **THEN** lstk executes it directly without looking for or parsing a manifest + +#### Scenario: Extension self-enforces contract compatibility + +- **WHEN** an extension requires a newer runtime-context contract than `LSTK_EXT_API_VERSION` advertises +- **THEN** the extension detects the mismatch from the environment and refuses to run +- **AND** lstk does not perform this check on the extension's behalf + +### Requirement: Help and discoverability + +lstk SHALL include resolvable extensions in its help output by scanning the bundled-extensions directory and `PATH` for `lstk-*` executables and listing each discovered extension's command name under a distinct "Extensions" grouping, so users can discover installed extensions. When a bundled and a `PATH` extension share a name, the entry SHALL be listed once (the one that would run). Built-in command help SHALL remain unchanged. + +#### Scenario: Extensions listed in help + +- **WHEN** a user runs `lstk --help`, an `lstk-deploy` is bundled, and an `lstk-hello` is on `PATH` +- **THEN** the help output lists both `deploy` and `hello` under an Extensions section + +### Requirement: One-line descriptions from a bundled descriptions file + +lstk SHALL enrich the help listing with a one-line description for bundled extensions by reading a static descriptions file shipped with the distribution (generated during the release process — see the extension-bundling capability), which maps a bundled extension's command name to its description. lstk SHALL NOT execute any extension to obtain help text; help rendering remains side-effect-free. A bundled extension named in the descriptions file SHALL be listed with that description; a bundled extension absent from the file, and every `PATH`/custom extension, SHALL be listed by command name only. A missing or unreadable descriptions file SHALL degrade to name-only listing without error. + +#### Scenario: Bundled extension shows its description + +- **WHEN** the descriptions file maps `deploy` to a one-line description and a user runs `lstk help` +- **THEN** lstk lists `deploy` with that description +- **AND** lstk does not execute `lstk-deploy` to render help + +#### Scenario: PATH and custom extensions are name-only + +- **WHEN** an `lstk-hello` is resolved from `PATH` +- **THEN** lstk lists `hello` by command name with no description +- **AND** lstk does not execute it during help + +#### Scenario: Missing descriptions file degrades gracefully + +- **WHEN** no descriptions file is present (or it cannot be read) +- **THEN** lstk lists all extensions by command name only +- **AND** help rendering does not error diff --git a/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md b/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md new file mode 100644 index 00000000..9465d8f6 --- /dev/null +++ b/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md @@ -0,0 +1,82 @@ +# extension-runtime-context Specification + +## Purpose + +Define the versioned contract by which lstk passes runtime context — resolved emulator endpoint and type, config directory, auth token, and resolved global-flag state — to an extension process, so extensions can talk to the emulator and the LocalStack platform and honor lstk's global flags without re-implementing discovery, config resolution, or auth. + +## ADDED Requirements + +### Requirement: Versioned context contract + +lstk SHALL pass runtime context to an extension exclusively through environment variables prefixed with `LSTK_EXT_`, and SHALL set `LSTK_EXT_API_VERSION` to the integer version of the contract it implements. The contract SHALL be additive within a major version; removing or repurposing a variable SHALL require incrementing `LSTK_EXT_API_VERSION`. lstk SHALL NOT require the extension to parse lstk's own config files. + +#### Scenario: API version is advertised + +- **WHEN** lstk invokes any extension +- **THEN** the extension's environment includes `LSTK_EXT_API_VERSION` set to the current contract version + +#### Scenario: Extension reads context from environment only + +- **WHEN** an extension needs the emulator endpoint and config directory +- **THEN** it can obtain both from `LSTK_EXT_` environment variables without reading lstk's TOML config + +### Requirement: Emulator endpoint and type are provided + +When a LocalStack emulator is running, lstk SHALL resolve the emulator endpoint using the same discovery and host resolution used by built-in commands, and SHALL expose it to the extension as `LSTK_EXT_EMULATOR_ENDPOINT` (a full URL), along with `LSTK_EXT_EMULATOR_TYPE` (e.g. `aws`, `snowflake`, `azure`) and the port. When no emulator is running, lstk SHALL omit the endpoint variable rather than setting an invalid value. + +#### Scenario: Endpoint provided when emulator running + +- **WHEN** an AWS emulator is running and lstk invokes an extension +- **THEN** `LSTK_EXT_EMULATOR_ENDPOINT` is set to the resolved emulator URL +- **AND** `LSTK_EXT_EMULATOR_TYPE` is set to `aws` + +#### Scenario: Endpoint omitted when no emulator running + +- **WHEN** no emulator is running and lstk invokes an extension +- **THEN** `LSTK_EXT_EMULATOR_ENDPOINT` is not set +- **AND** the extension is still executed + +### Requirement: Auth token and config directory are provided + +When the user is authenticated, lstk SHALL pass the resolved auth token to the extension as `LSTK_EXT_AUTH_TOKEN` so the extension can call the emulator and the LocalStack platform on the user's behalf. lstk SHALL pass the resolved lstk config directory as `LSTK_EXT_CONFIG_DIR`. When no auth token is available, lstk SHALL omit `LSTK_EXT_AUTH_TOKEN` rather than setting an empty value. + +#### Scenario: Auth token passed when available + +- **WHEN** the user has a resolved auth token and invokes an extension +- **THEN** `LSTK_EXT_AUTH_TOKEN` is set to that token in the extension's environment + +#### Scenario: Auth token omitted when unauthenticated + +- **WHEN** no auth token can be resolved and lstk invokes an extension +- **THEN** `LSTK_EXT_AUTH_TOKEN` is not set +- **AND** the extension is still executed + +#### Scenario: Config directory always provided + +- **WHEN** lstk invokes any extension +- **THEN** `LSTK_EXT_CONFIG_DIR` is set to the resolved lstk config directory + +### Requirement: Resolved global flags are conveyed + +lstk SHALL parse its own global flags (for example `--non-interactive`) when they appear before the extension command name, resolve them, and convey the resulting state to the extension via `LSTK_EXT_` environment variables rather than forwarding the flags on the extension's command line. In particular, lstk SHALL set `LSTK_EXT_NON_INTERACTIVE` to a truthy value when the session is non-interactive (the user passed `--non-interactive` or the standard output is not a TTY). Each lstk global flag that affects runtime behavior SHALL be conveyed as an `LSTK_EXT_` variable; adding a new global-flag variable is an additive change under `LSTK_EXT_API_VERSION`. lstk SHALL NOT include its global flags in the arguments forwarded to the extension. + +#### Scenario: Non-interactive flag conveyed via environment + +- **WHEN** a user runs `lstk --non-interactive hello --foo` and `lstk-hello` is resolvable +- **THEN** `LSTK_EXT_NON_INTERACTIVE` is set in the extension's environment +- **AND** the extension is invoked with only `--foo` (the `--non-interactive` global flag is not forwarded on its command line) + +#### Scenario: Non-interactive inferred from a non-TTY + +- **WHEN** lstk invokes an extension and standard output is not a terminal +- **THEN** `LSTK_EXT_NON_INTERACTIVE` is set even if `--non-interactive` was not passed + +### Requirement: Host environment is preserved + +lstk SHALL pass the user's existing environment through to the extension and only add or override the `LSTK_EXT_` variables defined by this contract, so extensions inherit the user's `PATH`, locale, and tool configuration. + +#### Scenario: Existing environment inherited + +- **WHEN** the user has `HTTP_PROXY` set and invokes an extension +- **THEN** the extension's environment still contains `HTTP_PROXY` +- **AND** also contains the `LSTK_EXT_` variables diff --git a/openspec/changes/add-extension-mechanism/tasks.md b/openspec/changes/add-extension-mechanism/tasks.md new file mode 100644 index 00000000..402f7f7a --- /dev/null +++ b/openspec/changes/add-extension-mechanism/tasks.md @@ -0,0 +1,56 @@ +## 1. Extension package scaffolding + +- [ ] 1.1 Create `internal/extension/` package with an `Extension` struct (resolved name, executable path) and constructor; use `log.Nop()` in tests +- [ ] 1.2 Define and document the `LSTK_EXT_API_VERSION` integer constant and the full `LSTK_EXT_*` variable contract in a package doc comment +- [ ] 1.3 Unit tests for the package's basic types/helpers + +## 2. Discovery and resolution + +- [ ] 2.1 Implement `Resolve(name)` searching the bundled dir (next to the symlink-resolved lstk executable) first, then `PATH`, for `lstk-`; honor Windows executable extensions; return first match (bundled wins) +- [ ] 2.2 Implement `List()` scanning bundled dir + `PATH` for `lstk-*` executables (names), de-duplicating by command name with bundled-then-PATH precedence +- [ ] 2.3 Implement bundled-dir resolution from `os.Executable()` with symlink resolution (works through npm/Homebrew shims) +- [ ] 2.4 Unit tests for resolution order (bundled wins), PATH fallback, not-found behavior, Windows extension handling, List de-duplication, and symlink-resolved bundled-dir lookup + +## 3. Runtime context contract + +- [ ] 3.1 Implement a builder that produces the `LSTK_EXT_*` environment (API version, emulator endpoint/type/port, config dir, auth token) layered on the inherited host environment +- [ ] 3.2 Wire emulator endpoint/type/port resolution via existing `internal/endpoint` + `internal/container` discovery; omit endpoint vars when no emulator is running +- [ ] 3.3 Include `LSTK_EXT_AUTH_TOKEN` only when a token is resolved; always include `LSTK_EXT_CONFIG_DIR`; do not set `LSTK_EXT_GRANT`/`LSTK_EXT_PUBLIC_KEY` +- [ ] 3.4 Set `LSTK_EXT_NON_INTERACTIVE` from lstk's resolved interactivity (the `isInteractive` condition: `--non-interactive` given or stdout not a TTY); document that future global flags are conveyed as additive `LSTK_EXT_*` vars +- [ ] 3.5 Unit tests asserting variable presence/absence across scenarios (emulator running vs not, authed vs not, non-interactive flag vs non-TTY, host env inherited) + +## 4. Invocation (exec) path + +- [ ] 4.1 Implement `Invoke(extension, args, ctx)` that builds the runtime env, execs the extension with args forwarded unmodified, passes stdin/stdout/stderr through, and propagates the exit code (model on `internal/iac/.../cli/exec.go`) +- [ ] 4.2 Ensure non-zero extension exits propagate without an extra lstk-level error message +- [ ] 4.3 Unit/integration tests for argument forwarding, stream passthrough, and exit-code propagation using a reference extension + +## 5. Command wiring and dispatch + +- [ ] 5.1 Wire unknown-command dispatch in `cmd/root.go`: when Cobra finds no built-in/alias for ``, attempt bundled+PATH resolution and invoke; built-ins always take precedence +- [ ] 5.2 Apply `SetInterspersed(false)` so lstk's global flags are parsed only before the command name and everything from `` onward is forwarded verbatim; verify it doesn't disturb bare-root `start` or built-in subcommand flags (fall back to a `stripGlobalFlags`-style pass if needed) +- [ ] 5.3 Ensure extension args are not parsed by lstk (`DisableFlagParsing` semantics for the synthesized extension path) +- [ ] 5.4 Add extensions to `lstk help` under an "Extensions" grouping by scanning bundled dir + `PATH` for `lstk-*` (de-duplicated, bundled wins) +- [ ] 5.5 Read the bundled descriptions file (name → one-liner) shipped in the bundled dir and attach descriptions to bundled extensions in help; `PATH`/custom extensions and bundled names missing from the file are name-only; a missing/unreadable file degrades to name-only without error; never execute an extension during help +- [ ] 5.6 Wire config initialization only where needed (extension dispatch needs config dir/endpoint); keep side-effect-free paths unaffected +- [ ] 5.7 Integration tests: built-in precedence, unknown→extension, unknown with no extension errors, help listing showing bundled descriptions from the file, PATH extensions name-only (and not executed during help), missing-descriptions-file degrades to name-only, and `lstk --non-interactive ` conveying `LSTK_EXT_NON_INTERACTIVE` while not forwarding the flag + +## 6. Reference extension and end-to-end coverage + +- [ ] 6.1 Add a small reference/example `lstk-*` extension used by tests that echoes the `LSTK_EXT_*` it received +- [ ] 6.2 Integration test: place the reference extension on a test `PATH`, invoke via `lstk `, and assert it received the expected runtime context (endpoint when emulator running, auth token when authed) + +## 7. Bundled LocalStack extensions (distribution + update) + +- [ ] 7.1 Define the bundled-extensions on-disk layout next to the lstk binary (including the descriptions file) and how each channel populates it: binary archive (sibling files), Homebrew (libexec, not symlinked to global bin), npm (package dir resolvable via the symlink-resolved exe path) +- [ ] 7.2 Generate the bundled descriptions file (name → one-liner) during the release process, version-locked to the bundled binaries, and ship it where lstk reads it +- [ ] 7.3 Wire the public release workflow to pull prebuilt closed-source bundled binaries (e.g. `lstk-deploy`) from private CI, version-pinned to the lstk release, without exposing source +- [ ] 7.4 Extend `internal/update` to replace lstk, its bundled extensions, and the descriptions file as one atomic, version-matched set across all install methods; never leave them mismatched on interrupted update +- [ ] 7.5 Add a reference bundled extension (in-tree, for tests) with a descriptions-file entry that echoes the `LSTK_EXT_*` it received and performs a stubbed self-authorization, exercising the bundled path end to end +- [ ] 7.6 Integration tests: bundled extension resolvable immediately, bundled wins over a same-named `PATH` extension, resolvable via a symlinked/shim `lstk`, bundled description shown in help, and bundled premium extension still self-authorizes + +## 8. Documentation and finalize + +- [ ] 8.1 Add an "Extensions" section to CLAUDE.md describing the mechanism, the `LSTK_EXT_*` contract (including `LSTK_EXT_NON_INTERACTIVE`), bundled-dir + PATH resolution, bundled-wins precedence, the release-generated descriptions file (bundled-only help text), and the self-authorization model (lstk passes the token; authorization is the extension's job) +- [ ] 8.2 Write a public extension-author guide: the manifest-free contract, runtime-context variables, global-flag conveyance via env, that help descriptions are bundled-only (custom/PATH extensions are name-only), how to authorize the user with the conveyed token, and the security note that authorization must not rely on lstk (which is open source); note the deferred signed-entitlement mechanism +- [ ] 8.3 Run `make lint`, `make test`, and `make test-integration`; ensure all pass From 6aa4b71a7a057d8f82a68f0e86e1bf91f67adba3 Mon Sep 17 00:00:00 2001 From: Peter Smith Date: Wed, 1 Jul 2026 07:11:49 +1200 Subject: [PATCH 2/6] docs(extensions): refine extension-mechanism spec and split out bundled-extension distribution Incorporate PR #340 review feedback into the OpenSpec artifacts and phase the launch: - Runtime context becomes a versioned JSON object (LSTK_EXT_API_VERSION + LSTK_EXT_CONTEXT) carrying an emulators array, so multiple simultaneously running emulators are representable from day one. - Record extension invocations as lstk-side telemetry; defer trace-context propagation, a shared SDK, and an extension allow-list. - Scope the first release to running extensions + bundled-dir resolution (validated by manual placement); defer automated distribution and atomic co-update to the new add-bundled-extension-distribution change. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../design.md | 27 +++++++ .../proposal.md | 32 +++++++++ .../extension-bundling-distribution/spec.md | 52 ++++++++++++++ .../tasks.md | 22 ++++++ .../changes/add-extension-mechanism/design.md | 65 +++++++++++++---- .../add-extension-mechanism/proposal.md | 25 ++++--- .../specs/extension-bundling/spec.md | 47 ++---------- .../specs/extension-framework/spec.md | 9 ++- .../specs/extension-runtime-context/spec.md | 61 ++++++++++------ .../changes/add-extension-mechanism/tasks.md | 71 ++++++++++--------- 10 files changed, 289 insertions(+), 122 deletions(-) create mode 100644 openspec/changes/add-bundled-extension-distribution/design.md create mode 100644 openspec/changes/add-bundled-extension-distribution/proposal.md create mode 100644 openspec/changes/add-bundled-extension-distribution/specs/extension-bundling-distribution/spec.md create mode 100644 openspec/changes/add-bundled-extension-distribution/tasks.md diff --git a/openspec/changes/add-bundled-extension-distribution/design.md b/openspec/changes/add-bundled-extension-distribution/design.md new file mode 100644 index 00000000..a309435e --- /dev/null +++ b/openspec/changes/add-bundled-extension-distribution/design.md @@ -0,0 +1,27 @@ +## Context + +`add-extension-mechanism` delivers the extension *mechanism* and bundled-directory *resolution*: lstk runs an `lstk-` found next to its binary, ahead of `PATH`. It intentionally defers *distribution* so the first release can validate bundled extensions by manual placement. This change automates getting LocalStack's bundled extensions into the install artifacts and keeping them version-matched. The code for these decisions was prototyped during `add-extension-mechanism` and then removed from that change; this change re-introduces it. + +## Decisions + +### Decision 1: Atomic, version-matched update of the `lstk`/`lstk-*` set + +`internal/update` treats `lstk` and its bundled `lstk-*` set (binaries + the descriptions file) as one unit. For the self-managed binary channel, the extractor stages every new `lstk`/`lstk-*` member next to its destination (`.lstk-new` siblings) and renames each into place, so an interrupted update never leaves `lstk` and a bundled extension at mismatched versions. For Homebrew and npm, the package manager replaces the whole package — and therefore the whole bundled set — atomically. + +**Rationale**: a bundled extension and the `lstk` that conveys its contract are released together; a mismatched pair could violate the `LSTK_EXT_API_VERSION` contract. Staging-then-rename keeps the swap crash-safe within a directory. + +### Decision 2: Hand-authored descriptions file, release-validated by a shell script + +The descriptions file (`lstk-extensions.toml`) is hand-authored in LocalStack's private extensions repository — the same source of truth that builds the closed-source binaries — and shipped as-is. The open-source repo does not generate it. A release-time bash script, `scripts/check-descriptions.sh` (consistent with `scripts/test-integration.sh`), extracts the described command names (the bare left-hand identifiers of the flat `name = "…"` table — values are never parsed) and fails the release if any described name has no corresponding `lstk-` binary in the staged dir. A staged binary with no description is allowed (help degrades to name-only). + +**Validation targets a single, host-native staging dir** — descriptions are os/arch-independent, so the check runs once against one staging dir (the release host's own OS), where binaries are bare `lstk-` with no `.exe`/PATHEXT ambiguity. + +**Rationale**: validating (not generating) keeps one source of truth in the private repo while preserving the version-lock guarantee; key-only parsing keeps the shell check trivially correct. + +**Alternatives considered**: a Go validator reusing the runtime `scanDir` + go-toml (rejected: extra build entrypoint and against the repo's "domain logic in Go, helpers in shell" grain for a small set-difference); generating the file from an in-repo manifest (rejected: duplicates a list the private repo already owns). + +### Decision 3: Cross-channel packaging places bundled files where lstk resolves them + +GoReleaser includes the staged bundled binaries and the descriptions file at each archive root (and in the Homebrew/npm payloads), siblings of `lstk`. The public release workflow pulls the prebuilt closed-source binaries for each `os/arch` from a private artifact location into a `bundled/_/` staging dir, authenticated with a repository/organization secret; only binaries are pulled, never source. The GoReleaser inclusion is gated/commented until the private-CI pull is wired, so the credential-less open-source build never fails on an empty glob. + +**Rationale**: lstk resolves the directory next to its symlink-resolved executable, so every channel must land bundled files there — sibling-at-archive-root for tarballs, libexec for Homebrew, package dir for npm. diff --git a/openspec/changes/add-bundled-extension-distribution/proposal.md b/openspec/changes/add-bundled-extension-distribution/proposal.md new file mode 100644 index 00000000..6cf2c1d9 --- /dev/null +++ b/openspec/changes/add-bundled-extension-distribution/proposal.md @@ -0,0 +1,32 @@ +## Why + +The `add-extension-mechanism` change ships the extension *mechanism* — lstk resolves and runs `lstk-` executables (PATH and a bundled directory next to the binary) and conveys runtime context to them. It deliberately stops short of *distributing* LocalStack's own bundled extensions: the first release is a test bed where a bundled `lstk-` is validated by manual placement. This change closes that loop — it automates packaging LocalStack's (possibly closed-source) bundled extensions into the install artifacts, ships their help descriptions, and keeps the `lstk`/`lstk-*` set version-matched across updates — so bundled extensions like `lstk-deploy` are available immediately after a standard install with no manual step. + +## What Changes + +- **Package bundled extensions into every install channel** (binary archive, Homebrew, npm) so they land in the directory lstk resolves, with no `PATH` change required by the user. +- **Pull the prebuilt closed-source bundled binaries from private CI** into the release build context, version-pinned to the lstk release, without exposing source in the public repository. +- **Ship a hand-authored descriptions file** (`lstk-extensions.toml`) alongside the bundled binaries, owned by LocalStack's private extensions repository, and **validate it at release time** against the staged binaries so a described-but-missing extension is a release-blocking error. +- **Update the `lstk`/`lstk-*` set atomically** in `internal/update`, so a running `lstk` and its bundled extensions are never left at mismatched versions across any install method. + +## Capabilities + +### New Capabilities + +- `extension-bundling-distribution`: Automated distribution and version-matched co-update of LocalStack's bundled extensions — cross-channel packaging (binary archive, Homebrew, npm), the private-CI binary pull, the release-shipped + release-validated descriptions file, and atomic updates of the `lstk`/`lstk-*` set via `internal/update`. Builds on the bundled-directory *resolution* delivered by `extension-bundling` in `add-extension-mechanism`. + +### Modified Capabilities + + + +## Impact + +- **Touched code**: `internal/update` (atomic replacement of the `lstk`/`lstk-*` family + descriptions file), `.goreleaser.yaml` (archive/cask/npm payload inclusion), the public release workflow (private-CI pull, version pinning), and `scripts/check-descriptions.sh` (release-time validation). +- **Packaging/release**: binary archive, Homebrew formula/cask, and npm package lay out bundled extensions where lstk resolves them; the release workflow pulls prebuilt closed-source bundled binaries from private CI, authenticated with a repository/organization secret. +- **Docs**: re-introduce `docs/extensions-bundling.md` (on-disk layout per channel, the release pipeline, atomic update, the descriptions file and its validation). +- **External dependencies/services**: a private artifact location for the prebuilt bundled binaries and a release-time credential to pull them. + +## Deferred (future work) + +- User-facing `lstk extension` management commands (`list`/`info`/`install`/`remove`) and a user-mutable managed extensions directory. +- Internet-download of third-party extensions and any associated allow-listing / signature verification. diff --git a/openspec/changes/add-bundled-extension-distribution/specs/extension-bundling-distribution/spec.md b/openspec/changes/add-bundled-extension-distribution/specs/extension-bundling-distribution/spec.md new file mode 100644 index 00000000..457a6dfb --- /dev/null +++ b/openspec/changes/add-bundled-extension-distribution/specs/extension-bundling-distribution/spec.md @@ -0,0 +1,52 @@ +# extension-bundling-distribution Specification + +## Purpose + +Automate shipping LocalStack's own bundled extensions (for example a closed-source `lstk-deploy`) so they are available immediately after a standard install, carry their help descriptions, and stay version-matched with the `lstk` binary across updates. This builds on the bundled-directory *resolution* delivered by the `extension-bundling` capability (which lets lstk run a bundled extension that is present); here we cover how bundled extensions get *there* and stay correct. + +## ADDED Requirements + +### Requirement: Bundled extensions are available after a standard install + +A set of extensions MAY be designated as bundled and SHALL be installed alongside `lstk` by the same single installation command across supported distribution channels (binary archive, Homebrew, npm), placed in the bundled-extensions directory, and resolvable immediately as `lstk ` with no separate install step. Packaging SHALL place bundled extensions where lstk resolves them without requiring the user to add them to `PATH`. The closed-source bundled binaries SHALL be built in private CI and pulled into the release build context version-pinned to the lstk release, without exposing source in the public repository. + +#### Scenario: Bundled extension available immediately + +- **WHEN** a user installs lstk via the standard installation command for any supported channel and `lstk-deploy` is bundled +- **THEN** `lstk deploy` resolves to the bundled extension with no extra install step + +#### Scenario: Bundled extension found without PATH changes + +- **WHEN** a user extracts the binary archive and places only `lstk` on `PATH` +- **THEN** a bundled `lstk-deploy` sibling is still resolved by `lstk deploy` because lstk searches the directory alongside its executable + +### Requirement: Bundled extensions update atomically with lstk + +Updating lstk SHALL update its bundled extensions to the matching version as a single, atomic set, so a running `lstk` and its bundled extensions are never left at mismatched versions. `internal/update` SHALL replace the lstk executable and its bundled extensions together regardless of the install method, or fail without partially updating. + +#### Scenario: Bundled extensions updated with lstk + +- **WHEN** lstk is updated to a new version that ships a newer bundled `lstk-deploy` +- **THEN** the bundled `lstk-deploy` is replaced with the matching version as part of the same update +- **AND** an interrupted update does not leave lstk and the bundled extension at mismatched versions + +### Requirement: Hand-authored descriptions file, validated at release time + +A static descriptions file that maps each bundled extension's command name to a one-line description SHALL ship with the distribution where lstk reads it (alongside the bundled extensions). The file is hand-authored and owned by LocalStack's private extensions repository — the same source of truth that produces the bundled binaries — and lstk's open-source repository SHALL assume it exists rather than generate it. The file SHALL cover only bundled, LocalStack-controlled extensions; it is not a per-extension manifest authored by third parties. The release process SHALL validate the file against the staged binaries so a name described in the file but with no shipped binary is a release-blocking error; a shipped binary with no description is permitted (it degrades to name-only help). lstk reads this file for help rendering (see the extension-framework capability) and never executes an extension to obtain a description. + +#### Scenario: Descriptions file ships with the bundled set + +- **WHEN** a release bundles `lstk-deploy` +- **THEN** the hand-authored descriptions file contains an entry mapping `deploy` to its one-line description +- **AND** that file is shipped where lstk resolves bundled extensions + +#### Scenario: Release validation rejects a description with no shipped binary + +- **WHEN** the descriptions file describes `deploy` but no `lstk-deploy` binary is present in the staged bundled set +- **THEN** the release-time validation fails, so lstk never ships a help entry for an extension that was not bundled + +#### Scenario: Descriptions update atomically with the bundled set + +- **WHEN** lstk is updated to a version that bundles a renamed or re-described extension +- **THEN** the descriptions file is updated as part of the same atomic update +- **AND** lstk never shows a description that disagrees with the bundled binaries diff --git a/openspec/changes/add-bundled-extension-distribution/tasks.md b/openspec/changes/add-bundled-extension-distribution/tasks.md new file mode 100644 index 00000000..4215d363 --- /dev/null +++ b/openspec/changes/add-bundled-extension-distribution/tasks.md @@ -0,0 +1,22 @@ +## 1. On-disk layout and packaging + +- [ ] 1.1 Define the bundled-extensions on-disk layout next to the lstk binary (including the descriptions file) and how each channel populates it: binary archive (sibling files at the archive root), Homebrew (libexec, not symlinked to global bin), npm (package dir resolvable via the symlink-resolved exe path) — documented in `docs/extensions-bundling.md` (re-introduce this doc) +- [ ] 1.2 Wire GoReleaser archive/cask/npm payload inclusion of the staged `bundled/_/lstk-*` binaries and `lstk-extensions.toml`; keep it gated/commented until the private-CI pull is wired so the credential-less open-source build does not fail on an empty glob + +## 2. Private-CI binary pull + +- [ ] 2.1 Wire the public release workflow to pull prebuilt closed-source bundled binaries (e.g. `lstk-deploy`) from private CI into a `bundled/_/` staging dir, version-pinned to the lstk release, authenticated with a repository/organization secret, without exposing source + +## 3. Descriptions file shipping + validation + +- [ ] 3.1 Re-introduce `scripts/check-descriptions.sh` (bash, consistent with `scripts/test-integration.sh`): extract the described names from the hand-authored `lstk-extensions.toml` and fail the release if any has no corresponding `lstk-` binary in the staged dir (a staged binary with no description is allowed; runs against one host-native staging dir since descriptions are os/arch-independent) +- [ ] 3.2 Wire the release process to pull the hand-authored descriptions file from the private extensions repo into the staging dir and run `scripts/check-descriptions.sh` against the staged binaries + +## 4. Atomic version-matched update + +- [ ] 4.1 Extend `internal/update` (`extract.go`) to replace lstk, its bundled `lstk-*` extensions, and the descriptions file as one atomic, version-matched set across all install methods (stage `.lstk-new` siblings, then rename); never leave them mismatched on an interrupted update — re-introduce `extract_test.go` coverage (`TestExtractAndReplaceUpdatesLstkSet`, `TestExtractAndReplaceAddsNewBundledExtension`) + +## 5. Tests and docs + +- [ ] 5.1 Integration test: bundled extension available immediately after a simulated standard install; bundled set updates atomically with lstk; descriptions file shipped where lstk reads it +- [ ] 5.2 Update `docs/extensions-bundling.md` and the CLAUDE.md Extensions section to document distribution + atomic update once enabled diff --git a/openspec/changes/add-extension-mechanism/design.md b/openspec/changes/add-extension-mechanism/design.md index c51ea1a7..82069bcd 100644 --- a/openspec/changes/add-extension-mechanism/design.md +++ b/openspec/changes/add-extension-mechanism/design.md @@ -47,15 +47,24 @@ Wire extension dispatch in `cmd/root.go` so that resolution happens only after C **Alternatives considered**: Pre-scanning `os.Args[1]` before Cobra runs (rejected: duplicates Cobra's alias/normalization logic and risks divergence). Registering a synthetic Cobra command per discovered extension at startup (we do a lightweight bundled-dir + PATH scan for `--help` listing but keep dispatch in the unknown-command path to avoid eagerly stat-ing the filesystem on every invocation). Reusing `stripGlobalFlags` verbatim (rejected as the primary path: its strip-from-anywhere behavior risks colliding with extension-owned flags). -### Decision 3: Environment-variable runtime contract (`LSTK_EXT_*`), versioned +### Decision 3: Environment-variable runtime contract — a versioned JSON context object -Context flows to the extension purely through environment variables prefixed `LSTK_EXT_`, with `LSTK_EXT_API_VERSION` as an integer contract version. Variables in this change: `LSTK_EXT_API_VERSION`, `LSTK_EXT_EMULATOR_ENDPOINT`, `LSTK_EXT_EMULATOR_TYPE`, `LSTK_EXT_EMULATOR_PORT`, `LSTK_EXT_CONFIG_DIR`, `LSTK_EXT_AUTH_TOKEN`, and `LSTK_EXT_NON_INTERACTIVE`. The host environment is inherited; only `LSTK_EXT_*` is added/overridden. Endpoint vars are omitted when no emulator is running; the auth token is omitted when none is available. +Context flows to the extension through exactly two environment variables: -Resolved global flags travel by env, not argv: `LSTK_EXT_NON_INTERACTIVE` is set when the session is non-interactive (the `--non-interactive` flag was given or stdout is not a TTY — the same condition lstk already computes at [cmd/root.go](../../../cmd/root.go) `isInteractive`). Carrying global state by env (rather than re-forwarding `--non-interactive` on the extension's command line) is what lets the extension own its entire flag space without collision, and each new global flag becomes one additive `LSTK_EXT_*` variable. +- `LSTK_EXT_API_VERSION` — a flat integer contract version, kept outside the payload so an extension can check compatibility *before* parsing anything. +- `LSTK_EXT_CONTEXT` — a single JSON object carrying the whole resolved context: + - `configDir` (string, always present) + - `authToken` (string, **omitted** when no token is available) + - `nonInteractive` (bool) + - `emulators` (array of `{ "type", "endpoint", "port" }`, **`[]` when no emulator is running**) -**Rationale**: Mirrors how the IaC proxies already pass context, is language-agnostic (every runtime can read env), and avoids forcing extensions to parse lstk's TOML or re-implement discovery. Versioning makes the contract evolvable: additive within a major version, bump on any removal/repurpose. A future signed-entitlement description would be added additively (e.g. `LSTK_EXT_GRANT` / `LSTK_EXT_PUBLIC_KEY`) under a documented version bump. +The host environment is inherited; only these two `LSTK_EXT_*` variables are added/overridden (any stray `LSTK_EXT_*` in the user's shell is stripped first so it cannot shadow the resolved values). -**Alternatives considered**: A JSON context file referenced by a single env var (cleaner for large/structured payloads; reconsider if the contract grows, but env is simpler for the current fields and avoids temp-file lifecycle/cleanup). Passing context as CLI flags (rejected: collides with extension-owned flag space). +Resolved global flags travel inside this object, not on argv: `nonInteractive` is true when the session is non-interactive (the `--non-interactive` flag was given or stdout is not a TTY — the same condition lstk computes at [cmd/root.go](../../../cmd/root.go) `isInteractive`). Carrying global state in the context (rather than re-forwarding `--non-interactive` on the extension's command line) lets the extension own its entire flag space without collision, and each new global flag becomes one additive JSON field. + +**Rationale**: The decisive driver is **multi-emulator support** (PR review): lstk can run an AWS, a Snowflake, and an Azure emulator simultaneously, which the original flat `LSTK_EXT_EMULATOR_ENDPOINT/TYPE/PORT` triple cannot represent. Adopting a JSON `emulators` array **from day one** means multi-emulator is expressible without a later breaking change to the contract. Once one field needs structure, a single structured object beats a growing sprawl of flat scalars: every future field is an additive JSON key under the same version rather than a new variable name, and the payload is still language-agnostic (every runtime has a JSON parser) and never forces the extension to read lstk's TOML or re-implement discovery. Keeping `LSTK_EXT_API_VERSION` as a flat scalar preserves clean version negotiation. A future signed-entitlement description slots in as an additive field (e.g. `grant` / `publicKey`) under a documented version bump. + +**Alternatives considered**: Flat per-field variables — the original design; rejected because it cannot represent multiple running emulators and grows a new variable per field. A JSON context *file* referenced by a path env var (rejected: temp-file lifecycle/cleanup, where inline JSON in the env var is simpler and the payload is small). Putting the version *inside* the JSON too (rejected: keeping it a flat scalar lets an extension negotiate compatibility before parsing). Passing context as CLI flags (rejected: collides with extension-owned flag space). ### Decision 4: No lstk-side manifest — extensions are self-describing @@ -73,21 +82,51 @@ lstk conveys `LSTK_EXT_AUTH_TOKEN` and makes no entitlement or license decision **Alternatives considered**: lstk-side entitlement gate + per-extension signed grant (the original design; deferred, not rejected — it improves UX and enables offline verification but requires platform work and adds complexity not needed for the first cut). Purely client-side license checks inside lstk (rejected: trivially bypassable, no IP protection). -### Decision 6: Bundled LocalStack extensions live next to the binary and resolve ahead of PATH +### Decision 6: Bundled extensions resolve from a directory next to the binary, ahead of PATH (distribution + update deferred) + +lstk resolves bundled extensions (e.g. a closed-source `lstk-deploy`) from a fixed directory derived from its own symlink-resolved executable path — the Git `libexec/git-core` model. Resolution order is: built-ins → bundled dir → `PATH`, so a bundled extension wins over a same-named `PATH` executable. This reuses the `lstk-` naming convention (no manifest) and does not change the authorization model — a bundled premium extension still self-authorizes with the conveyed token. + +**Scope split for the first release** (per PR review): the first release ships the *resolution* above — enough that a LocalStack-built `lstk-` placed next to `lstk` runs as `lstk ` — so bundled extensions can be validated by **manual placement** before they are ever shipped automatically. What is **deferred to a separate future change (`add-bundled-extension-distribution`)**: packaging the bundled binaries into the binary-archive / Homebrew / npm payloads, pulling the closed-source binaries from private CI, shipping + release-validating the descriptions file, and the atomic version-matched co-update of the `lstk`/`lstk-*` set in `internal/update`. The first release deliberately does **not** download or upgrade bundled extensions; it is a test bed. + +**Rationale**: Searching the directory next to the resolved executable is robust where co-located binaries are not on `PATH` — bare tarballs where the user symlinks only `lstk`, and npm/Homebrew layouts where the invoked `lstk` is a shim (hence resolving symlinks to find the real sibling dir). Bundled-wins precedence makes the official `lstk-deploy` deterministic: a stray or malicious `lstk-deploy` on `PATH` cannot hijack it. Shipping resolution first, distribution later, lets us de-risk the closed-source bundling pipeline (private-CI pull, atomic update, packaging across three channels) against a manually-staged binary before automating it — without blocking the run-extensions mechanism on that pipeline. + +**Alternatives considered**: Just install bundled binaries onto `PATH` (rejected: zero lstk change but fragile for bare tarballs and pollutes the user's `PATH`). PATH-wins precedence (rejected: lets a stray binary shadow the official extension; bundled-wins is safer for closed-source premium commands). Shipping distribution + atomic update in the first release together with resolution (rejected per PR review: couples the first launch to the full closed-source pipeline; phasing it lets the first release validate bundled extensions manually). A managed mutable dir under the config dir (deferred with the rest of management; bundling needs only a static dir). + +### Decision 7: One-line help descriptions from a hand-authored file owned by the private repo, bundled-only + +Bundled extensions get a one-line description in `lstk help` from a static descriptions file shipped alongside the bundled extensions (a single file mapping command name → description, `lstk-extensions.toml` in the bundled dir). lstk reads it at help time and never executes an extension to obtain text. `PATH`/custom extensions are always name-only. + +The file is **hand-authored and owned by LocalStack's private extensions repository** — the same place the closed-source bundled binaries are built — and the open-source lstk repo simply assumes it exists in the bundled dir. lstk does **not** generate it. The only safety rail kept in this repo is a release-time **validation** step, `scripts/check-descriptions.sh` (a bash script, consistent with the repo's only other script, `scripts/test-integration.sh`): it extracts the described command names from the file (the bare left-hand identifiers of the flat `name = "…"` table — it never needs to parse the description values) and fails the release if any described name has no corresponding `lstk-` executable in the staged bundled dir. A shipped binary with no description is allowed and degrades to name-only. + +**Validation targets a single, host-native staging dir.** The descriptions file is os/arch-independent — `deploy` is `deploy` on every platform — so the check runs once against one staging dir (the release host's own OS, i.e. Linux), where binaries are bare `lstk-` with no `.exe` suffix or Windows executable-bit ambiguity. This keeps the shell check trivially correct (`find … -perm -u+x` + name-prefix strip) and avoids the per-OS `.exe`/PATHEXT handling that lstk's runtime `scanDir` only applies when `GOOS == windows`; pointing any validator at a Windows staging dir from a Linux host would otherwise mis-read `lstk-deploy.exe` as the name `deploy.exe` and false-fail. + +**Rationale**: A shell script keeps the open-source repo's release tooling consistent (one `scripts/` home, no extra top-level `release/` dir for a single tool) and the logic is small and key-only. The validation needs only the described *names*, so there is no TOML value/quote parsing to get wrong, and validating one clean host-native dir removes the only real cross-platform hazard. This preserves the version-locking guarantee the former generator (`gen-descriptions` + `BuildDescriptions` + the `release/bundled-extensions.toml` manifest) enforced by construction, now enforced by the release-blocking check instead. + +**Alternatives considered**: A Go validator in-repo (`extension.ValidateDescriptions` reusing the runtime `scanDir` + go-toml, run via `go run ./...`) — rejected: it is the only thing that *cannot* drift from how lstk resolves extensions at runtime, but it costs an extra build entrypoint, runs against the repo's "domain logic in Go, helpers in shell" grain for what is a 20-line set-difference, and the drift it guards against is negligible here (validation reads only bare-identifier keys against a single clean dir). Acceptable to lose that guarantee while the descriptions file stays a flat name→string table; revisit if the file gains structure (per-extension metadata, min API version) that should share types with `LoadDescriptions`. Generating the file from a manifest in the open-source repo (the original design; rejected: the descriptions belong with the binaries in the private repo, and generation duplicated a list the private repo already owns — validation gives the same version-lock guarantee with one source of truth). A describe protocol that execs `lstk- lstk:describe` (rejected: turns inert help into code execution, needs timeout/cache/trust-boundary machinery, and a mistyped command rendering Cobra usage could fork extensions). Per-extension manifests (rejected with Decision 4). Name-only for everything (viable and simplest, but the user wants descriptions for bundled extensions like `lstk-deploy`, which this delivers with no exec). + +### Decision 8: lstk records extension invocations as telemetry; trace propagation is deferred + +When telemetry is enabled (lstk's existing `LSTK_OTEL` path), lstk SHALL record each extension invocation — the command name, duration, and exit code — through the same OpenTelemetry export the rest of lstk uses. This is **lstk-side only**: lstk does **not** inject trace context (W3C `traceparent`/`tracestate`) into the extension process, so an extension's own spans do not yet nest under lstk's trace. + +**Rationale** (PR review): visibility into which extensions run, how often, and whether they succeed is valuable and cheap — lstk already has an OTEL pipeline, so the invocation is just one more span/event at the dispatch boundary, with no contract surface and no dependency on the extension being instrumented. Full distributed tracing *through* the extension is a larger, optional concern: it only helps extensions that are themselves OTEL-instrumented, and it is purely additive later (inject `traceparent`, or add a `trace` field to `LSTK_EXT_CONTEXT`, under a version bump) — so it is deferred rather than designed now. Recording respects the same opt-in/opt-out as all lstk telemetry; nothing is emitted when telemetry is disabled. + +**Alternatives considered**: Inject standard `TRACEPARENT`/`TRACESTATE` so any instrumented extension auto-continues the trace (deferred, not rejected — clean and standard, but only useful once we have instrumented extensions, and addable without breaking the contract). An `LSTK_EXT_`-prefixed trace variable (rejected for now: non-standard, and the standard names are the natural choice when we do add propagation). No telemetry at all (rejected: the reviewer specifically wants extension observability). + +### Decision 9: No shared TUI/library between lstk and extensions -LocalStack ships its own extensions (e.g. a closed-source `lstk-deploy`) in a fixed directory derived from lstk's own symlink-resolved executable path — the Git `libexec/git-core` model. Resolution order becomes: built-ins → bundled dir → `PATH`, so a bundled extension wins over a same-named `PATH` executable. `internal/update` replaces lstk and its bundled extensions as one atomic, version-matched set. This reuses the `lstk-` naming convention (no manifest) and does not change the authorization model — a bundled premium extension still self-authorizes with the conveyed token. +Extensions do **not** link or share any lstk Go library — UI components, output sinks, or otherwise. The only coupling is the `LSTK_EXT_*` environment contract (Decision 3). An extension that wants a TUI, spinners, or styled output brings its own libraries. -**Rationale**: Searching the directory next to the resolved executable is robust where co-located binaries are not on `PATH` — bare tarballs where the user symlinks only `lstk`, and npm/Homebrew layouts where the invoked `lstk` is a shim (hence resolving symlinks to find the real sibling dir). Bundled-wins precedence makes the official `lstk-deploy` deterministic: a stray or malicious `lstk-deploy` on `PATH` cannot hijack it. This is a deliberately narrow re-introduction of what the management capability deferred: a *static, read-only, ships-with-lstk* location and the packaging/update wiring — but **not** the user-mutable managed dir or the `lstk extension install/remove` UX, which stay deferred. The closed-source binaries are built in private CI and injected into the public release artifacts, version-pinned to the lstk release that carries them. +**Rationale** (PR review): a shared library would bind extension authors to lstk's Go API surface (Bubble Tea, lipgloss, the `output`/`ui` packages), which changes without much external control and would couple every extension's build to lstk's dependency graph — exactly the lock-in the separate-process model (Decision 1) exists to avoid. A narrow, versioned env contract is far less likely to churn than a Go API, and keeps extensions language-agnostic and independently buildable. The cost — each extension re-implements its own presentation — is acceptable and is the same trade Git's extension model makes. -**Alternatives considered**: Just install bundled binaries onto `PATH` (rejected: zero lstk change but fragile for bare tarballs and pollutes the user's `PATH`). PATH-wins precedence (rejected: lets a stray binary shadow the official extension; bundled-wins is safer for closed-source premium commands). A managed mutable dir under the config dir (deferred with the rest of management; bundling needs only a static dir). +**Alternatives considered**: A published `lstk-extension-sdk` Go module with shared UI/output helpers (rejected for now: convenient for Go authors but Go-only, version-coupled, and contradicts the language-agnostic goal; could ship later as an *optional* convenience without changing the contract). -### Decision 7: One-line help descriptions from a release-generated file, bundled-only +### Decision 10: No extension allow-list / trust policing -Bundled extensions get a one-line description in `lstk help` from a static descriptions file generated during the release process and shipped alongside the bundled extensions (a single file mapping command name → description, e.g. `extensions.toml` in the bundled dir). lstk reads it at help time and never executes an extension to obtain text. `PATH`/custom extensions are always name-only. +lstk does not maintain an allow-list, signature check, or any other trust gate over which `lstk-` executables it will run. It resolves and execs whatever is on `PATH` or in the bundled dir, exactly as Git runs `git-`. -**Rationale**: This keeps help rendering side-effect-free — the property we lost in the earlier exec-based approach. Because LocalStack controls the bundled set, their descriptions are known at release time, so there is no need to run anything: no timeout, no cache, no incidental-usage-on-a-typo hazard, no risk of executing untrusted `PATH` binaries during help. It is not a return of the per-extension `extension.toml` manifest (Decision 4): it is one release-owned file covering only the bundled set, generated by CI and version-locked to the binaries it describes, not authored per-extension by third parties. The cost — descriptions are static and only available for bundled extensions — is acceptable: third-party `PATH` extensions being name-only matches Git's `git help -a`. +**Rationale** (PR review): the only way to introduce a malicious `lstk-` is to place an executable on the user's machine — at which point the attacker would target far higher-value binaries (`ls`, `cat`, the user's shell rc) rather than a `lstk-` prefixed one. A whitelist in open-source lstk is also trivially bypassable (Decision 5's threat model: lstk can be rebuilt), so it would be security theater. Bundled LocalStack extensions are not a separate download — they ship inside the `lstk` artifact (Decision 6) and inherit its trust — and lstk deliberately does **not** download extensions from the internet, which is where an allow-list would actually matter. Bundled-wins precedence (Decision 6) already prevents a stray `PATH` binary from shadowing an official bundled command, which is the one hijack worth blocking. -**Alternatives considered**: A describe protocol that execs `lstk- lstk:describe` (rejected: turns inert help into code execution, needs timeout/cache/trust-boundary machinery, and a mistyped command rendering Cobra usage could fork extensions). Per-extension manifests (rejected with Decision 4). Name-only for everything (viable and simplest, but the user wants descriptions for bundled extensions like `lstk-deploy`, which this delivers with no exec). +**Alternatives considered**: An `lstk extension add`/allow-list UX seen in some tools (rejected as overkill: it presumes internet-downloaded extensions, which is out of scope and deferred with the rest of management). Signature verification of extension binaries (rejected: meaningful only with a download channel and a trust root lstk cannot anchor while open source). ## Threat Model: hostile lstk rebuild (why authorization lives in the extension) diff --git a/openspec/changes/add-extension-mechanism/proposal.md b/openspec/changes/add-extension-mechanism/proposal.md index 68775b52..670f2f45 100644 --- a/openspec/changes/add-extension-mechanism/proposal.md +++ b/openspec/changes/add-extension-mechanism/proposal.md @@ -5,11 +5,12 @@ ## What Changes - Introduce a Git-style extension model: when `lstk ` is not a built-in command, lstk resolves and executes an `lstk-` executable found on `PATH`, forwarding all remaining arguments and propagating the child's stdin/stdout/stderr and exit code. -- Define an **extension runtime contract**: lstk passes the resolved emulator endpoint, emulator type, config directory, auth token, and resolved global-flag state to the extension process through a stable, versioned set of `LSTK_EXT_*` environment variables so extensions can talk to the emulator and the platform without re-implementing discovery or config resolution. -- **Honor global flags before the command name**: lstk parses its own global flags (e.g. `--non-interactive`, and any added later) when they precede the extension name, consumes them itself, and conveys the resolved state to the extension via `LSTK_EXT_*` (e.g. `LSTK_EXT_NON_INTERACTIVE`) rather than forwarding them on the extension's command line. -- **Bundle LocalStack's own extensions**: ship LocalStack-built extensions (e.g. a closed-source `lstk-deploy`) by default alongside `lstk` in a directory next to the binary, resolved ahead of `PATH` and updated atomically with `lstk` — with no user-facing install step. +- Define an **extension runtime contract**: lstk passes context to the extension process through two environment variables — `LSTK_EXT_API_VERSION` (a flat integer version) and `LSTK_EXT_CONTEXT` (a single, versioned JSON object carrying the config directory, auth token, resolved global-flag state, and a JSON **array of all running emulators**) — so extensions can talk to every running emulator and the platform without re-implementing discovery or config resolution. The emulator array is JSON from day one so multiple simultaneously-running emulators (e.g. AWS + Snowflake + Azure) are representable without a later breaking change. +- **Honor global flags before the command name**: lstk parses its own global flags (e.g. `--non-interactive`, and any added later) when they precede the extension name, consumes them itself, and conveys the resolved state to the extension as fields of `LSTK_EXT_CONTEXT` (e.g. `nonInteractive`) rather than forwarding them on the extension's command line. +- **Record extension invocations as telemetry**: when telemetry is enabled (the existing `LSTK_OTEL` path), lstk records each extension invocation (command name, duration, exit code) through its OpenTelemetry export. This is lstk-side only; injecting trace context so an extension's own spans nest under lstk's trace is deferred. +- **Resolve bundled extensions from a directory next to the binary** (ahead of `PATH`), so a LocalStack-built `lstk-` placed alongside `lstk` runs as `lstk `. The first release supports *running* such an extension; automated packaging/distribution of LocalStack's bundled extensions and their atomic co-update with `lstk` are **deferred to a later launch** (see Deferred). This first release is a test bed to validate bundled extensions by placing them manually before they ship in the download bundle. - Establish that **authorization is the extension's responsibility**: lstk conveys the user's auth token and makes no entitlement decision of its own. An extension that needs to restrict its use authorizes the user itself (e.g. by calling the LocalStack platform with the conveyed token). A richer lstk-side mechanism — lstk obtaining a LocalStack-signed entitlement description for the extension to verify offline — is deliberately **deferred** to future work. -- List resolvable extensions in `lstk help` by scanning the bundled directory and `PATH` for `lstk-*` executables; bundled extensions show a one-line description read from a static descriptions file generated during the release process, while `PATH`/custom extensions are name-only. Help rendering never executes an extension. +- List resolvable extensions in `lstk help` by scanning the bundled directory and `PATH` for `lstk-*` executables; bundled extensions show a one-line description read from a static descriptions file that ships alongside them — hand-authored in LocalStack's private extensions repository and validated against the shipped binaries at release time — while `PATH`/custom extensions are name-only. Help rendering never executes an extension. - Keep the entire mechanism in the open-source repository; closed-source extensions ship only as binaries placed on `PATH` and never require source in the core repo. ## Capabilities @@ -17,9 +18,9 @@ ### New Capabilities - `extension-framework`: Git-style discovery, resolution, and dispatch of `lstk-` extension executables (bundled dir + `PATH`), including built-in precedence, leading-only global-flag handling, forwarding of arguments/streams/exit codes, and side-effect-free help listing (bundled extensions described from a static file, others name-only). No lstk-side manifest — extensions are self-describing and self-validating. -- `extension-runtime-context`: The versioned environment-variable contract lstk establishes for an extension process — resolved emulator endpoint/type/port, config directory, auth token, and resolved global-flag state (e.g. non-interactive) — so extensions can reach the emulator and platform and honor lstk's global flags. +- `extension-runtime-context`: The versioned contract lstk establishes for an extension process — `LSTK_EXT_API_VERSION` plus a single `LSTK_EXT_CONTEXT` JSON object carrying the config directory, auth token, resolved global-flag state, and a JSON array of all running emulators (type/endpoint/port) — so extensions can reach every running emulator and the platform and honor lstk's global flags. Also covers recording extension invocations as telemetry when enabled. - `extension-entitlement`: The authorization model — lstk conveys the auth token and the extension authorizes itself — plus the explicit deferral of any lstk-side signed-entitlement mechanism and the security rationale (lstk is open source, so authorization cannot depend on it). -- `extension-bundling`: Shipping LocalStack-built (possibly closed-source) extensions by default alongside `lstk` — a read-only bundled directory next to the binary, resolution ahead of `PATH`, a release-generated descriptions file for help text, cross-channel packaging (binary archive, Homebrew, npm), and atomic version-matched updates via `internal/update`. Excludes user-facing management commands and a user-mutable directory. +- `extension-bundling`: The read-only bundled-extensions directory next to the binary and its resolution ahead of `PATH`, plus the rule that a bundled (possibly closed-source) extension still self-authorizes. This first-release capability covers *resolving and running* a bundled extension that is present (e.g. placed manually for validation). **Cross-channel packaging/distribution and atomic version-matched updates are deferred** to a separate future change (`add-bundled-extension-distribution`). Excludes user-facing management commands and a user-mutable directory. ### Modified Capabilities @@ -28,14 +29,18 @@ ## Impact - **New code**: `internal/extension/` (bundled-dir + PATH resolution, runtime context builder, global-flag conveyance, exec), and unknown-command dispatch + help-listing wiring in `cmd/root.go`. -- **Touched code**: `cmd/root.go` (fallthrough to extension dispatch for unknown commands; `SetInterspersed(false)` for leading-only global flags; bundled-dir + PATH scan for help), reuse of `internal/auth` (token resolution), `internal/config`/`internal/endpoint` (config dir and emulator endpoint resolution), `internal/container` (running-emulator discovery for endpoint/type), and `internal/update` (atomic version-matched update of bundled extensions). -- **Packaging/release**: binary archive, Homebrew formula, and npm package must lay out bundled extensions where lstk resolves them; the public release workflow must pull prebuilt closed-source bundled binaries from private CI, version-pinned to the lstk release. +- **Touched code**: `cmd/root.go` (fallthrough to extension dispatch for unknown commands; `SetInterspersed(false)` for leading-only global flags; bundled-dir + PATH scan for help), reuse of `internal/auth` (token resolution), `internal/config`/`internal/endpoint` (config dir and emulator endpoint resolution), `internal/container` (discovery of **all** running emulators for the `emulators` array, not just one), and the existing OTEL/telemetry path (recording extension invocations). +- **Packaging/release**: not in scope for this change — automated bundling into the binary archive / Homebrew / npm payloads and the private-CI pull are deferred to `add-bundled-extension-distribution`. The first release ships no bundled binaries; they are validated by manual placement next to `lstk`. - **External dependencies/services**: None required by this change. Extensions that authorize use the existing LocalStack platform with the conveyed auth token; no new platform or emulator endpoints are needed. - **Security surface**: lstk passes the auth token into extension processes via env (as it already does for IaC proxies); this defines a local trust boundary to document. Authorization guarantees live in the extension, never in lstk. -- **Docs**: New "Extensions" section in CLAUDE.md and a public extension-author guide (manifest-free contract, `LSTK_EXT_*` variables, the self-authorization model and why it cannot rely on lstk). +- **Docs**: New "Extensions" section in CLAUDE.md and a public extension-author guide (manifest-free contract, the `LSTK_EXT_API_VERSION` + `LSTK_EXT_CONTEXT` JSON contract with its `emulators` array, the self-authorization model and why it cannot rely on lstk). ## Deferred (future work) - lstk-side entitlement verification and a LocalStack-signed entitlement description (grant) that extensions verify offline against a published public key. - Emulator genuineness attestation (distinguishing a licensed emulator from a clone). -- User-facing `lstk extension` management commands (`list`/`info`/`install`/`remove`) and a user-mutable managed extensions directory. (Static, ships-with-lstk bundling is **in scope** via the `extension-bundling` capability; only the user-driven install/remove UX is deferred.) +- **Bundled-extension distribution and atomic co-update** — packaging LocalStack's bundled `lstk-*` into the binary archive / Homebrew / npm payloads, pulling the closed-source binaries from private CI, shipping + release-validating the descriptions file, and updating the `lstk`/`lstk-*` set atomically via `internal/update`. Deferred to a separate future change, **`add-bundled-extension-distribution`**. This first release supports running a bundled extension placed manually (the test bed); resolving the bundled dir is in scope, automated shipping/updating is not. +- User-facing `lstk extension` management commands (`list`/`info`/`install`/`remove`) and a user-mutable managed extensions directory. +- **Trace-context propagation into extensions** (injecting W3C `traceparent`/`tracestate` so an extension's own spans nest under lstk's trace). Only lstk-side invocation telemetry is in scope now; propagation is purely additive later. (See design Decision 8.) +- **A shared extension SDK / library** (Go helpers for UI/output). Extensions couple to lstk only through the `LSTK_EXT_*` env contract and bring their own libraries; an optional SDK could ship later without changing the contract. (See design Decision 9.) +- **Extension allow-listing / signature verification.** lstk runs any resolvable `lstk-` like Git, with no trust gate; this only becomes relevant with an internet download channel, which is itself deferred with extension management. (See design Decision 10.) diff --git a/openspec/changes/add-extension-mechanism/specs/extension-bundling/spec.md b/openspec/changes/add-extension-mechanism/specs/extension-bundling/spec.md index 49b06952..4d528d7f 100644 --- a/openspec/changes/add-extension-mechanism/specs/extension-bundling/spec.md +++ b/openspec/changes/add-extension-mechanism/specs/extension-bundling/spec.md @@ -2,17 +2,17 @@ ## Purpose -Allow LocalStack to ship its own extensions (for example a closed-source `lstk-deploy`) by default alongside `lstk`, so they are available immediately after a standard install with no separate step, are kept in lockstep with the `lstk` version that ships them, and are resolved deterministically ahead of any same-named executable on `PATH`. This capability covers only static, read-only, ships-with-lstk extensions; user-driven install/remove of extensions and a user-mutable managed directory remain out of scope (deferred). +Allow lstk to resolve and run LocalStack's own extensions (for example a closed-source `lstk-deploy`) from a read-only directory next to the `lstk` binary, ahead of any same-named executable on `PATH`. This first-release capability covers only *resolving and running* a bundled extension that is present — enough to validate bundled extensions by placing them manually. Automated cross-channel packaging/distribution, the release-shipped descriptions file, and atomic version-matched co-update with `lstk` are out of scope here and are specified by the future `add-bundled-extension-distribution` change. User-driven install/remove of extensions and a user-mutable managed directory also remain deferred. ## ADDED Requirements ### Requirement: Bundled-extensions directory alongside the executable -lstk SHALL look for bundled extensions in a fixed directory derived from the location of its own executable, resolving symlinks so the directory is found even when `lstk` is invoked through a symlink or package shim (e.g. an npm `.bin` link). Bundled extension executables follow the same `lstk-` naming convention as any other extension and SHALL NOT require a manifest. This directory is owned by the lstk distribution and is read-only from the user's perspective: lstk SHALL NOT provide commands to add to or remove from it in this change. +lstk SHALL look for bundled extensions in a fixed directory derived from the location of its own executable, resolving symlinks so the directory is found even when `lstk` is invoked through a symlink or package shim (e.g. an npm `.bin` link). Bundled extension executables follow the same `lstk-` naming convention as any other extension and SHALL NOT require a manifest. A bundled extension SHALL be resolved ahead of a same-named executable on `PATH`. This directory is owned by the lstk distribution and is read-only from the user's perspective: lstk SHALL NOT provide commands to add to or remove from it in this change. #### Scenario: Bundled directory resolved through a symlink -- **WHEN** `lstk` is invoked via a symlink or package shim and an `lstk-deploy` is bundled +- **WHEN** `lstk` is invoked via a symlink or package shim and an `lstk-deploy` is present in the bundled directory - **THEN** lstk resolves its real executable location, finds the bundled-extensions directory, and can resolve `lstk deploy` #### Scenario: Naming convention identifies bundled extensions @@ -20,45 +20,10 @@ lstk SHALL look for bundled extensions in a fixed directory derived from the loc - **WHEN** the bundled-extensions directory contains an executable named `lstk-deploy` - **THEN** lstk treats it as the `deploy` extension without reading any manifest -### Requirement: Bundled extensions are available after a standard install +#### Scenario: Bundled extension wins over a PATH extension of the same name -A set of extensions MAY be designated as bundled and SHALL be installed alongside `lstk` by the same single installation command across supported distribution channels (binary archive, Homebrew, npm), placed in the bundled-extensions directory, and resolvable immediately as `lstk ` with no separate install step. Packaging SHALL place bundled extensions where lstk resolves them without requiring the user to add them to `PATH`. - -#### Scenario: Bundled extension available immediately - -- **WHEN** a user installs lstk via the standard installation command for any supported channel and `lstk-deploy` is bundled -- **THEN** `lstk deploy` resolves to the bundled extension with no extra install step - -#### Scenario: Bundled extension found without PATH changes - -- **WHEN** a user extracts the binary archive and places only `lstk` on `PATH` -- **THEN** a bundled `lstk-deploy` sibling is still resolved by `lstk deploy` because lstk searches the directory alongside its executable - -### Requirement: Bundled extensions update atomically with lstk - -Updating lstk SHALL update its bundled extensions to the matching version as a single, atomic set, so a running `lstk` and its bundled extensions are never left at mismatched versions. `internal/update` SHALL replace the lstk executable and its bundled extensions together regardless of the install method, or fail without partially updating. - -#### Scenario: Bundled extensions updated with lstk - -- **WHEN** lstk is updated to a new version that ships a newer bundled `lstk-deploy` -- **THEN** the bundled `lstk-deploy` is replaced with the matching version as part of the same update -- **AND** an interrupted update does not leave lstk and the bundled extension at mismatched versions - -### Requirement: Release-generated descriptions file for bundled extensions - -The release process SHALL generate a static descriptions file that maps each bundled extension's command name to a one-line description, and SHALL ship it with the distribution where lstk reads it (alongside the bundled extensions). The file SHALL cover only bundled, LocalStack-controlled extensions; it is not a per-extension manifest authored by third parties. It SHALL be versioned and updated together with the bundled extension set, so descriptions never drift from the binaries that ship. lstk reads this file for help rendering (see the extension-framework capability) and never executes an extension to obtain a description. - -#### Scenario: Descriptions file ships with the bundled set - -- **WHEN** a release bundles `lstk-deploy` -- **THEN** the release process produces a descriptions file entry mapping `deploy` to its one-line description -- **AND** that file is shipped where lstk resolves bundled extensions - -#### Scenario: Descriptions update atomically with the bundled set - -- **WHEN** lstk is updated to a version that bundles a renamed or re-described extension -- **THEN** the descriptions file is updated as part of the same atomic update -- **AND** lstk never shows a description that disagrees with the bundled binaries +- **WHEN** both the bundled directory and `PATH` contain an `lstk-deploy` +- **THEN** lstk runs the bundled one for `lstk deploy` ### Requirement: Bundled closed-source extensions still self-authorize diff --git a/openspec/changes/add-extension-mechanism/specs/extension-framework/spec.md b/openspec/changes/add-extension-mechanism/specs/extension-framework/spec.md index c700c24f..02bd92c9 100644 --- a/openspec/changes/add-extension-mechanism/specs/extension-framework/spec.md +++ b/openspec/changes/add-extension-mechanism/specs/extension-framework/spec.md @@ -93,16 +93,21 @@ lstk SHALL NOT require a manifest file to discover, validate, or invoke an exten ### Requirement: Help and discoverability -lstk SHALL include resolvable extensions in its help output by scanning the bundled-extensions directory and `PATH` for `lstk-*` executables and listing each discovered extension's command name under a distinct "Extensions" grouping, so users can discover installed extensions. When a bundled and a `PATH` extension share a name, the entry SHALL be listed once (the one that would run). Built-in command help SHALL remain unchanged. +lstk SHALL include resolvable extensions in its help output by scanning the bundled-extensions directory and `PATH` for `lstk-*` executables and listing each discovered extension's command name under a distinct "Extensions" grouping, so users can discover installed extensions. When a bundled and a `PATH` extension share a name, the entry SHALL be listed once (the one that would run). Built-in command help SHALL remain unchanged. The Extensions section SHALL align its description column with the built-in command/Tools sections, using the same name-padding rule, so the help output reads as one consistent table. #### Scenario: Extensions listed in help - **WHEN** a user runs `lstk --help`, an `lstk-deploy` is bundled, and an `lstk-hello` is on `PATH` - **THEN** the help output lists both `deploy` and `hello` under an Extensions section +#### Scenario: Extension descriptions align with command descriptions + +- **WHEN** a user runs `lstk --help` and a bundled extension with a description is listed +- **THEN** the extension's description begins in the same column as the descriptions of the built-in command sections + ### Requirement: One-line descriptions from a bundled descriptions file -lstk SHALL enrich the help listing with a one-line description for bundled extensions by reading a static descriptions file shipped with the distribution (generated during the release process — see the extension-bundling capability), which maps a bundled extension's command name to its description. lstk SHALL NOT execute any extension to obtain help text; help rendering remains side-effect-free. A bundled extension named in the descriptions file SHALL be listed with that description; a bundled extension absent from the file, and every `PATH`/custom extension, SHALL be listed by command name only. A missing or unreadable descriptions file SHALL degrade to name-only listing without error. +lstk SHALL enrich the help listing with a one-line description for bundled extensions by reading a static descriptions file from the bundled directory when present, which maps a bundled extension's command name to its description. (How that file is hand-authored, shipped, and release-validated is specified by the future `add-bundled-extension-distribution` change; this change covers only reading it when present.) lstk SHALL NOT execute any extension to obtain help text; help rendering remains side-effect-free. A bundled extension named in the descriptions file SHALL be listed with that description; a bundled extension absent from the file, and every `PATH`/custom extension, SHALL be listed by command name only. A missing or unreadable descriptions file SHALL degrade to name-only listing without error. #### Scenario: Bundled extension shows its description diff --git a/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md b/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md index 9465d8f6..a7366d80 100644 --- a/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md +++ b/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md @@ -6,70 +6,74 @@ Define the versioned contract by which lstk passes runtime context — resolved ## ADDED Requirements -### Requirement: Versioned context contract +### Requirement: Versioned JSON context contract -lstk SHALL pass runtime context to an extension exclusively through environment variables prefixed with `LSTK_EXT_`, and SHALL set `LSTK_EXT_API_VERSION` to the integer version of the contract it implements. The contract SHALL be additive within a major version; removing or repurposing a variable SHALL require incrementing `LSTK_EXT_API_VERSION`. lstk SHALL NOT require the extension to parse lstk's own config files. +lstk SHALL pass runtime context to an extension through exactly two environment variables: `LSTK_EXT_API_VERSION`, set to the integer version of the contract it implements, and `LSTK_EXT_CONTEXT`, a single JSON object carrying the resolved context (config directory, auth token, resolved global-flag state, and running emulators). The version is kept as a flat scalar — outside the JSON payload — so an extension can check contract compatibility before parsing the object. The contract SHALL be additive within a major version (new JSON fields may be added); removing or repurposing a field SHALL require incrementing `LSTK_EXT_API_VERSION`. lstk SHALL NOT require the extension to parse lstk's own config files. #### Scenario: API version is advertised - **WHEN** lstk invokes any extension - **THEN** the extension's environment includes `LSTK_EXT_API_VERSION` set to the current contract version -#### Scenario: Extension reads context from environment only +#### Scenario: Context is a single JSON object -- **WHEN** an extension needs the emulator endpoint and config directory -- **THEN** it can obtain both from `LSTK_EXT_` environment variables without reading lstk's TOML config +- **WHEN** lstk invokes any extension +- **THEN** the extension's environment includes `LSTK_EXT_CONTEXT` containing a JSON object the extension can decode to obtain the config directory, auth token (when present), non-interactive state, and the list of running emulators, without reading lstk's TOML config -### Requirement: Emulator endpoint and type are provided +### Requirement: Running emulators are provided as a JSON array -When a LocalStack emulator is running, lstk SHALL resolve the emulator endpoint using the same discovery and host resolution used by built-in commands, and SHALL expose it to the extension as `LSTK_EXT_EMULATOR_ENDPOINT` (a full URL), along with `LSTK_EXT_EMULATOR_TYPE` (e.g. `aws`, `snowflake`, `azure`) and the port. When no emulator is running, lstk SHALL omit the endpoint variable rather than setting an invalid value. +The `LSTK_EXT_CONTEXT` object SHALL include an `emulators` array, with one entry per LocalStack emulator currently running, so an extension can work against every running emulator rather than a single one. Each entry SHALL carry the emulator `type` (e.g. `aws`, `snowflake`, `azure`), the `endpoint` (a full URL resolved with the same discovery and host resolution used by built-in commands), and the `port`. When no emulator is running, `emulators` SHALL be an empty array (`[]`), not omitted, so an extension always decodes a list. -#### Scenario: Endpoint provided when emulator running +#### Scenario: Single emulator provided when one is running - **WHEN** an AWS emulator is running and lstk invokes an extension -- **THEN** `LSTK_EXT_EMULATOR_ENDPOINT` is set to the resolved emulator URL -- **AND** `LSTK_EXT_EMULATOR_TYPE` is set to `aws` +- **THEN** `LSTK_EXT_CONTEXT.emulators` contains one entry with `type` `aws` and `endpoint` set to the resolved emulator URL + +#### Scenario: Multiple emulators provided when several are running + +- **WHEN** an AWS emulator and a Snowflake emulator are both running and lstk invokes an extension +- **THEN** `LSTK_EXT_CONTEXT.emulators` contains an entry for each, distinguished by `type` -#### Scenario: Endpoint omitted when no emulator running +#### Scenario: Empty array when no emulator running - **WHEN** no emulator is running and lstk invokes an extension -- **THEN** `LSTK_EXT_EMULATOR_ENDPOINT` is not set +- **THEN** `LSTK_EXT_CONTEXT.emulators` is an empty array - **AND** the extension is still executed ### Requirement: Auth token and config directory are provided -When the user is authenticated, lstk SHALL pass the resolved auth token to the extension as `LSTK_EXT_AUTH_TOKEN` so the extension can call the emulator and the LocalStack platform on the user's behalf. lstk SHALL pass the resolved lstk config directory as `LSTK_EXT_CONFIG_DIR`. When no auth token is available, lstk SHALL omit `LSTK_EXT_AUTH_TOKEN` rather than setting an empty value. +When the user is authenticated, lstk SHALL include the resolved auth token as the `authToken` field of `LSTK_EXT_CONTEXT` so the extension can call the emulator and the LocalStack platform on the user's behalf. lstk SHALL include the resolved lstk config directory as the `configDir` field. When no auth token is available, lstk SHALL omit the `authToken` field rather than setting an empty value. #### Scenario: Auth token passed when available - **WHEN** the user has a resolved auth token and invokes an extension -- **THEN** `LSTK_EXT_AUTH_TOKEN` is set to that token in the extension's environment +- **THEN** `LSTK_EXT_CONTEXT.authToken` is set to that token #### Scenario: Auth token omitted when unauthenticated - **WHEN** no auth token can be resolved and lstk invokes an extension -- **THEN** `LSTK_EXT_AUTH_TOKEN` is not set +- **THEN** `LSTK_EXT_CONTEXT` has no `authToken` field - **AND** the extension is still executed #### Scenario: Config directory always provided - **WHEN** lstk invokes any extension -- **THEN** `LSTK_EXT_CONFIG_DIR` is set to the resolved lstk config directory +- **THEN** `LSTK_EXT_CONTEXT.configDir` is set to the resolved lstk config directory ### Requirement: Resolved global flags are conveyed -lstk SHALL parse its own global flags (for example `--non-interactive`) when they appear before the extension command name, resolve them, and convey the resulting state to the extension via `LSTK_EXT_` environment variables rather than forwarding the flags on the extension's command line. In particular, lstk SHALL set `LSTK_EXT_NON_INTERACTIVE` to a truthy value when the session is non-interactive (the user passed `--non-interactive` or the standard output is not a TTY). Each lstk global flag that affects runtime behavior SHALL be conveyed as an `LSTK_EXT_` variable; adding a new global-flag variable is an additive change under `LSTK_EXT_API_VERSION`. lstk SHALL NOT include its global flags in the arguments forwarded to the extension. +lstk SHALL parse its own global flags (for example `--non-interactive`) when they appear before the extension command name, resolve them, and convey the resulting state to the extension as fields of `LSTK_EXT_CONTEXT` rather than forwarding the flags on the extension's command line. In particular, lstk SHALL set the `nonInteractive` field to true when the session is non-interactive (the user passed `--non-interactive` or the standard output is not a TTY). Each lstk global flag that affects runtime behavior SHALL be conveyed as a field of the context object; adding a new field is an additive change under `LSTK_EXT_API_VERSION`. lstk SHALL NOT include its global flags in the arguments forwarded to the extension. -#### Scenario: Non-interactive flag conveyed via environment +#### Scenario: Non-interactive flag conveyed via context - **WHEN** a user runs `lstk --non-interactive hello --foo` and `lstk-hello` is resolvable -- **THEN** `LSTK_EXT_NON_INTERACTIVE` is set in the extension's environment +- **THEN** `LSTK_EXT_CONTEXT.nonInteractive` is true - **AND** the extension is invoked with only `--foo` (the `--non-interactive` global flag is not forwarded on its command line) #### Scenario: Non-interactive inferred from a non-TTY - **WHEN** lstk invokes an extension and standard output is not a terminal -- **THEN** `LSTK_EXT_NON_INTERACTIVE` is set even if `--non-interactive` was not passed +- **THEN** `LSTK_EXT_CONTEXT.nonInteractive` is true even if `--non-interactive` was not passed ### Requirement: Host environment is preserved @@ -79,4 +83,19 @@ lstk SHALL pass the user's existing environment through to the extension and onl - **WHEN** the user has `HTTP_PROXY` set and invokes an extension - **THEN** the extension's environment still contains `HTTP_PROXY` -- **AND** also contains the `LSTK_EXT_` variables +- **AND** also contains the `LSTK_EXT_API_VERSION` and `LSTK_EXT_CONTEXT` variables + +### Requirement: Extension invocations are recorded as telemetry + +When lstk telemetry is enabled (the existing `LSTK_OTEL` path), lstk SHALL record each extension invocation — at least the extension command name, the duration, and the exit code — through the same OpenTelemetry export used by the rest of lstk. Recording SHALL respect the same opt-in/opt-out as all lstk telemetry: when telemetry is disabled, lstk SHALL emit nothing for the invocation. lstk SHALL NOT inject trace context into the extension process in this change (an extension's own spans nesting under lstk's trace is deferred and would be an additive change). + +#### Scenario: Invocation recorded when telemetry enabled + +- **WHEN** telemetry is enabled and lstk dispatches to an extension that exits with a status code +- **THEN** lstk records the extension's command name, duration, and exit code via its telemetry export + +#### Scenario: Nothing recorded when telemetry disabled + +- **WHEN** telemetry is disabled and lstk dispatches to an extension +- **THEN** lstk emits no telemetry for the invocation +- **AND** the extension still runs and its exit code still propagates diff --git a/openspec/changes/add-extension-mechanism/tasks.md b/openspec/changes/add-extension-mechanism/tasks.md index 402f7f7a..51727197 100644 --- a/openspec/changes/add-extension-mechanism/tasks.md +++ b/openspec/changes/add-extension-mechanism/tasks.md @@ -1,56 +1,57 @@ ## 1. Extension package scaffolding -- [ ] 1.1 Create `internal/extension/` package with an `Extension` struct (resolved name, executable path) and constructor; use `log.Nop()` in tests -- [ ] 1.2 Define and document the `LSTK_EXT_API_VERSION` integer constant and the full `LSTK_EXT_*` variable contract in a package doc comment -- [ ] 1.3 Unit tests for the package's basic types/helpers +- [x] 1.1 Create `internal/extension/` package with an `Extension` struct (resolved name, executable path) and constructor; use `log.Nop()` in tests +- [x] 1.2 Define and document `LSTK_EXT_API_VERSION` and the `LSTK_EXT_CONTEXT` JSON contract — Go types for the context object (config dir, auth token, non-interactive, `emulators` array of `{type,endpoint,port}`) and a package doc comment (reworked from the flat `LSTK_EXT_*` variable set per Decision 3) +- [x] 1.3 Unit tests for the package's basic types/helpers ## 2. Discovery and resolution -- [ ] 2.1 Implement `Resolve(name)` searching the bundled dir (next to the symlink-resolved lstk executable) first, then `PATH`, for `lstk-`; honor Windows executable extensions; return first match (bundled wins) -- [ ] 2.2 Implement `List()` scanning bundled dir + `PATH` for `lstk-*` executables (names), de-duplicating by command name with bundled-then-PATH precedence -- [ ] 2.3 Implement bundled-dir resolution from `os.Executable()` with symlink resolution (works through npm/Homebrew shims) -- [ ] 2.4 Unit tests for resolution order (bundled wins), PATH fallback, not-found behavior, Windows extension handling, List de-duplication, and symlink-resolved bundled-dir lookup +- [x] 2.1 Implement `Resolve(name)` searching the bundled dir (next to the symlink-resolved lstk executable) first, then `PATH`, for `lstk-`; honor Windows executable extensions; return first match (bundled wins) +- [x] 2.2 Implement `List()` scanning bundled dir + `PATH` for `lstk-*` executables (names), de-duplicating by command name with bundled-then-PATH precedence +- [x] 2.3 Implement bundled-dir resolution from `os.Executable()` with symlink resolution (works through npm/Homebrew shims) +- [x] 2.4 Unit tests for resolution order (bundled wins), PATH fallback, not-found behavior, Windows extension handling, List de-duplication, and symlink-resolved bundled-dir lookup -## 3. Runtime context contract +## 3. Runtime context contract (JSON object — reworked per PR review, Decision 3) -- [ ] 3.1 Implement a builder that produces the `LSTK_EXT_*` environment (API version, emulator endpoint/type/port, config dir, auth token) layered on the inherited host environment -- [ ] 3.2 Wire emulator endpoint/type/port resolution via existing `internal/endpoint` + `internal/container` discovery; omit endpoint vars when no emulator is running -- [ ] 3.3 Include `LSTK_EXT_AUTH_TOKEN` only when a token is resolved; always include `LSTK_EXT_CONFIG_DIR`; do not set `LSTK_EXT_GRANT`/`LSTK_EXT_PUBLIC_KEY` -- [ ] 3.4 Set `LSTK_EXT_NON_INTERACTIVE` from lstk's resolved interactivity (the `isInteractive` condition: `--non-interactive` given or stdout not a TTY); document that future global flags are conveyed as additive `LSTK_EXT_*` vars -- [ ] 3.5 Unit tests asserting variable presence/absence across scenarios (emulator running vs not, authed vs not, non-interactive flag vs non-TTY, host env inherited) +> Reworked from the original flat `LSTK_EXT_EMULATOR_*` variables to a single versioned JSON object so multiple simultaneously-running emulators are representable from day one. The previously-implemented flat-variable builder must be replaced. + +- [x] 3.1 Implement a builder that produces two environment variables layered on the inherited host env: `LSTK_EXT_API_VERSION` (flat integer) and `LSTK_EXT_CONTEXT` (a JSON object with `configDir`, optional `authToken`, `nonInteractive`, and `emulators`); strip any inherited `LSTK_EXT_*` first +- [x] 3.2 Populate `emulators` as a JSON array of `{type, endpoint, port}` for **all** running emulators via `internal/endpoint` + `internal/container` discovery (not just one); use `[]` when none are running (wired at the command boundary in `cmd/extension.go`) +- [x] 3.3 Include `authToken` in the object only when a token is resolved (omit the field otherwise); always include `configDir`; no entitlement/grant fields +- [x] 3.4 Set the `nonInteractive` field from lstk's resolved interactivity (the `isInteractive` condition: `--non-interactive` given or stdout not a TTY); document that future global flags are conveyed as additive JSON fields +- [x] 3.5 Unit tests asserting the JSON shape across scenarios: zero/one/multiple emulators, authed vs not (field present/absent), non-interactive flag vs non-TTY, host env inherited, stray `LSTK_EXT_*` stripped +- [x] 3.6 Record each extension invocation (command name, duration, exit code) via the existing OTEL/telemetry path when telemetry is enabled; emit nothing when disabled; do **not** inject trace context into the child (Decision 8); unit/integration coverage for enabled vs disabled ## 4. Invocation (exec) path -- [ ] 4.1 Implement `Invoke(extension, args, ctx)` that builds the runtime env, execs the extension with args forwarded unmodified, passes stdin/stdout/stderr through, and propagates the exit code (model on `internal/iac/.../cli/exec.go`) -- [ ] 4.2 Ensure non-zero extension exits propagate without an extra lstk-level error message -- [ ] 4.3 Unit/integration tests for argument forwarding, stream passthrough, and exit-code propagation using a reference extension +- [x] 4.1 Implement `Invoke(extension, args, ctx)` that builds the runtime env, execs the extension with args forwarded unmodified, passes stdin/stdout/stderr through, and propagates the exit code (model on `internal/iac/.../cli/exec.go`) +- [x] 4.2 Ensure non-zero extension exits propagate without an extra lstk-level error message +- [x] 4.3 Unit/integration tests for argument forwarding, stream passthrough, and exit-code propagation using a reference extension ## 5. Command wiring and dispatch -- [ ] 5.1 Wire unknown-command dispatch in `cmd/root.go`: when Cobra finds no built-in/alias for ``, attempt bundled+PATH resolution and invoke; built-ins always take precedence -- [ ] 5.2 Apply `SetInterspersed(false)` so lstk's global flags are parsed only before the command name and everything from `` onward is forwarded verbatim; verify it doesn't disturb bare-root `start` or built-in subcommand flags (fall back to a `stripGlobalFlags`-style pass if needed) -- [ ] 5.3 Ensure extension args are not parsed by lstk (`DisableFlagParsing` semantics for the synthesized extension path) -- [ ] 5.4 Add extensions to `lstk help` under an "Extensions" grouping by scanning bundled dir + `PATH` for `lstk-*` (de-duplicated, bundled wins) -- [ ] 5.5 Read the bundled descriptions file (name → one-liner) shipped in the bundled dir and attach descriptions to bundled extensions in help; `PATH`/custom extensions and bundled names missing from the file are name-only; a missing/unreadable file degrades to name-only without error; never execute an extension during help -- [ ] 5.6 Wire config initialization only where needed (extension dispatch needs config dir/endpoint); keep side-effect-free paths unaffected -- [ ] 5.7 Integration tests: built-in precedence, unknown→extension, unknown with no extension errors, help listing showing bundled descriptions from the file, PATH extensions name-only (and not executed during help), missing-descriptions-file degrades to name-only, and `lstk --non-interactive ` conveying `LSTK_EXT_NON_INTERACTIVE` while not forwarding the flag +- [x] 5.1 Wire unknown-command dispatch in `cmd/root.go`: when Cobra finds no built-in/alias for ``, attempt bundled+PATH resolution and invoke; built-ins always take precedence +- [x] 5.2 Apply `SetInterspersed(false)` so lstk's global flags are parsed only before the command name and everything from `` onward is forwarded verbatim; verify it doesn't disturb bare-root `start` or built-in subcommand flags (fall back to a `stripGlobalFlags`-style pass if needed) +- [x] 5.3 Ensure extension args are not parsed by lstk (`SetInterspersed(false)` makes everything from `` onward opaque, giving the same effect as `DisableFlagParsing` for the synthesized path) +- [x] 5.4 Add extensions to `lstk help` under an "Extensions" grouping by scanning bundled dir + `PATH` for `lstk-*` (de-duplicated, bundled wins) +- [x] 5.5 Read the bundled descriptions file (name → one-liner) shipped in the bundled dir and attach descriptions to bundled extensions in help; `PATH`/custom extensions and bundled names missing from the file are name-only; a missing/unreadable file degrades to name-only without error; never execute an extension during help +- [x] 5.6 Wire config initialization only where needed (extension dispatch needs config dir/endpoint); keep side-effect-free paths unaffected +- [x] 5.7 Integration tests: built-in precedence, unknown→extension, unknown with no extension errors, help listing showing bundled descriptions from the file, PATH extensions name-only (and not executed during help), missing-descriptions-file degrades to name-only, and `lstk --non-interactive ` conveying `nonInteractive: true` in `LSTK_EXT_CONTEXT` while not forwarding the flag (update the non-interactive assertion for the JSON contract) ## 6. Reference extension and end-to-end coverage -- [ ] 6.1 Add a small reference/example `lstk-*` extension used by tests that echoes the `LSTK_EXT_*` it received -- [ ] 6.2 Integration test: place the reference extension on a test `PATH`, invoke via `lstk `, and assert it received the expected runtime context (endpoint when emulator running, auth token when authed) +- [x] 6.1 Add a small reference/example `lstk-*` extension used by tests that reads `LSTK_EXT_API_VERSION`, decodes `LSTK_EXT_CONTEXT` JSON, and echoes the decoded fields (config dir, auth token, non-interactive, emulators) so tests can assert on them +- [x] 6.2 Integration test: place the reference extension on a test `PATH`, invoke via `lstk `, and assert it received the expected JSON context (an `emulators` entry when an emulator is running, `[]` when none, `authToken` when authed) + +## 7. Bundled extension resolution (first release) -## 7. Bundled LocalStack extensions (distribution + update) +> Distribution + atomic update (the former 7.1–7.4) are **deferred to the `add-bundled-extension-distribution` change** per the phased launch: the first release resolves and runs a bundled extension placed manually, but does not package or auto-update bundled extensions. The previously-implemented `internal/update` atomic-set logic, `scripts/check-descriptions.sh`, the GoReleaser archive block, and `docs/extensions-bundling.md` were removed from this change and are re-introduced by the future change. -- [ ] 7.1 Define the bundled-extensions on-disk layout next to the lstk binary (including the descriptions file) and how each channel populates it: binary archive (sibling files), Homebrew (libexec, not symlinked to global bin), npm (package dir resolvable via the symlink-resolved exe path) -- [ ] 7.2 Generate the bundled descriptions file (name → one-liner) during the release process, version-locked to the bundled binaries, and ship it where lstk reads it -- [ ] 7.3 Wire the public release workflow to pull prebuilt closed-source bundled binaries (e.g. `lstk-deploy`) from private CI, version-pinned to the lstk release, without exposing source -- [ ] 7.4 Extend `internal/update` to replace lstk, its bundled extensions, and the descriptions file as one atomic, version-matched set across all install methods; never leave them mismatched on interrupted update -- [ ] 7.5 Add a reference bundled extension (in-tree, for tests) with a descriptions-file entry that echoes the `LSTK_EXT_*` it received and performs a stubbed self-authorization, exercising the bundled path end to end -- [ ] 7.6 Integration tests: bundled extension resolvable immediately, bundled wins over a same-named `PATH` extension, resolvable via a symlinked/shim `lstk`, bundled description shown in help, and bundled premium extension still self-authorizes +- [x] 7.5 Add a reference bundled extension (in-tree, for tests) that conveys its received context and performs a stubbed self-authorization, exercising the bundled path end to end. It lives under the test-sample tree, not advertised as a public example — `test/integration/test-samples/extensions/lstk-ref` (the prose author guide in `docs/extensions-authoring.md` replaces the `examples/lstk-ref` showcase) +- [x] 7.6 Integration tests: bundled extension resolvable when present, bundled wins over a same-named `PATH` extension, resolvable via a symlinked/shim `lstk`, bundled description shown in help (when a descriptions file is present), and bundled premium extension still self-authorizes ## 8. Documentation and finalize -- [ ] 8.1 Add an "Extensions" section to CLAUDE.md describing the mechanism, the `LSTK_EXT_*` contract (including `LSTK_EXT_NON_INTERACTIVE`), bundled-dir + PATH resolution, bundled-wins precedence, the release-generated descriptions file (bundled-only help text), and the self-authorization model (lstk passes the token; authorization is the extension's job) -- [ ] 8.2 Write a public extension-author guide: the manifest-free contract, runtime-context variables, global-flag conveyance via env, that help descriptions are bundled-only (custom/PATH extensions are name-only), how to authorize the user with the conveyed token, and the security note that authorization must not rely on lstk (which is open source); note the deferred signed-entitlement mechanism -- [ ] 8.3 Run `make lint`, `make test`, and `make test-integration`; ensure all pass +- [x] 8.1 Add/update the "Extensions" section in CLAUDE.md: the mechanism, the `LSTK_EXT_API_VERSION` + `LSTK_EXT_CONTEXT` JSON contract (config dir, auth token, `nonInteractive`, `emulators` array), bundled-dir + PATH resolution, bundled-wins precedence, the hand-authored descriptions file (bundled-only help text), invocation telemetry, and the self-authorization model +- [x] 8.2 Write/update the public extension-author guide: the manifest-free contract, the `LSTK_EXT_CONTEXT` JSON schema (with the `emulators` array and how to handle multiple/zero emulators), global-flag conveyance via the context object, that help descriptions are bundled-only, how to authorize with the conveyed token, the security note that authorization must not rely on lstk, and the deferred items (signed entitlement, trace propagation, shared SDK) — `docs/extensions-authoring.md` +- [x] 8.3 Re-run `make lint`, `make test`, and `make test-integration` after the JSON-contract + telemetry rework; confirm the extension suite is green (pre-existing token/emulator/`az`-gated failures excepted) From 2dcdb3612ab824fcc09ad2ca5ede6c7744a31d07 Mon Sep 17 00:00:00 2001 From: Peter Smith Date: Wed, 1 Jul 2026 09:34:42 +1200 Subject: [PATCH 3/6] docs(extensions): clarify runtime-context versioning and namespace ownership Resolve two concerns in the JSON runtime-context contract: - Versioning vs feature detection: LSTK_EXT_API_VERSION is bumped only on a breaking change (a field removed or repurposed), never on additions. An extension therefore confirms a field by its presence in LSTK_EXT_CONTEXT, not by the version number, and any field added after v1 must be distinguishable when absent. The author guide now shows presence checks instead of a version floor check, and the version is used only to refuse a contract generation the extension predates. - Namespace ownership: lstk strips inherited LSTK_EXT_* before adding its own, reframed as an ownership invariant (Go's exec dedup already overrides the two vars lstk sets, so the strip exists to keep the LSTK_EXT_ namespace fully owned by this invocation). Updates design.md, the extension-runtime-context spec, and the author guide. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/extensions-authoring.md | 98 +++++++++++++++++++ .../changes/add-extension-mechanism/design.md | 9 +- .../specs/extension-runtime-context/spec.md | 25 ++++- 3 files changed, 130 insertions(+), 2 deletions(-) create mode 100644 docs/extensions-authoring.md diff --git a/docs/extensions-authoring.md b/docs/extensions-authoring.md new file mode 100644 index 00000000..0dd6d6d7 --- /dev/null +++ b/docs/extensions-authoring.md @@ -0,0 +1,98 @@ +# Writing an lstk extension + +lstk supports Git-style extensions. When you run `lstk ` and `` is not a built-in command, lstk looks for an executable called `lstk-` and runs it, forwarding your arguments and the child's input/output and exit code. Anyone can add a command to lstk by putting an `lstk-` executable on their `PATH` — in any language, open or closed source. There is **no manifest** and no registration step. + +## The contract at a glance + +- **Name it `lstk-`** and put it on `PATH`. `lstk ...` will run it; `lstk help` will list it. +- **Your arguments are forwarded verbatim.** Everything after `` is yours — lstk does not parse it. Define and parse your own flags however you like, including flags that happen to share a name with an lstk global flag. +- **lstk's global flags are consumed before the name.** `lstk --non-interactive --foo` runs your extension with just `--foo`; the resolved global state reaches you via environment variables (below), not on your command line. +- **Exit code and streams pass through.** Your exit status becomes lstk's exit status, and your stdin/stdout/stderr are wired straight to the terminal. + +## Runtime context: `LSTK_EXT_API_VERSION` and `LSTK_EXT_CONTEXT` + +lstk passes everything you need through two environment variables, so you never have to read lstk's config files or re-implement emulator discovery. Your existing environment (PATH, locale, proxy settings, …) is inherited unchanged. + +- **`LSTK_EXT_API_VERSION`** — an integer version of this contract, kept as a plain value so you can check it *before* parsing anything. +- **`LSTK_EXT_CONTEXT`** — a single JSON object with the resolved runtime context: + +```json +{ + "configDir": "/home/you/.config/lstk", + "authToken": "ls-...", + "nonInteractive": true, + "emulators": [ + { "type": "aws", "endpoint": "http://localhost.localstack.cloud:4566", "port": "4566" } + ] +} +``` + +| Field | Type | Notes | +| --- | --- | --- | +| `configDir` | string | lstk's resolved config directory. Always present. | +| `authToken` | string | The user's resolved LocalStack auth token. **Omitted** when not authenticated. | +| `nonInteractive` | bool | `true` when the user passed `--non-interactive` or stdout is not a TTY. When true, do not prompt. | +| `emulators` | array | One entry per running LocalStack emulator: `{ "type", "endpoint", "port" }`. An **empty array** `[]` when none are running. | + +`emulators` can hold **more than one** entry — lstk may run an AWS, a Snowflake, and an Azure emulator at the same time. Don't assume a single endpoint: select the one(s) your extension needs by `type`, and handle the empty case. `authToken` is **omitted, not set empty**, when the user is not authenticated — check for its presence. + +### Reading the context + +Use any JSON parser. For example, with `jq` in a shell extension: + +```sh +aws_endpoint=$(printf '%s' "$LSTK_EXT_CONTEXT" | jq -r '.emulators[] | select(.type=="aws") | .endpoint') +token=$(printf '%s' "$LSTK_EXT_CONTEXT" | jq -r '.authToken // empty') +``` + +### Versioning and feature detection + +Two different questions, two different mechanisms — don't use the version to answer the first: + +**"Is the field I need present?" → check the JSON, not the version.** lstk adds new fields to `LSTK_EXT_CONTEXT` *without* changing `LSTK_EXT_API_VERSION`, so the version cannot tell you a newer field exists. An older lstk simply omits a field it doesn't have, so test for it directly and degrade or fail with a clear message: + +```sh +region=$(printf '%s' "$LSTK_EXT_CONTEXT" | jq -r '.region // empty') +if [ -z "$region" ]; then + echo "this command needs a newer lstk (no 'region' in the extension context)" >&2 + exit 1 +fi +``` + +Any field added after the first release is **distinguishable when absent** (a missing key / null), precisely so this check works. + +**"Did a breaking change happen?" → that's all the version tracks.** `LSTK_EXT_API_VERSION` is bumped *only* when a field is removed or repurposed — never for additions. If you were built against version 1, optionally refuse a higher version you don't understand (this guards the one case presence-checks can't catch: a field whose *meaning* changed): + +```sh +if [ "${LSTK_EXT_API_VERSION:-0}" -gt 1 ]; then + echo "this extension was built for an older lstk extension contract" >&2 + exit 1 +fi +``` + +lstk performs **no** compatibility check for you — it runs any resolvable `lstk-`. Confirming the fields you use (by presence) and the contract generation (by version) is your responsibility. + +### Conveyance of global flags + +Each lstk global flag that affects behavior is conveyed as a field of `LSTK_EXT_CONTEXT` rather than being forwarded on your command line (today: `nonInteractive`). This is what lets you own your entire flag namespace without colliding with lstk. As lstk adds global flags, they appear as additional fields — additively, under the same `LSTK_EXT_API_VERSION` major version. + +## Help descriptions + +`lstk --help` lists installed extensions by command name. One-line descriptions are shown **only for extensions LocalStack bundles with lstk**, from a static descriptions file LocalStack ships with them. Third-party and `PATH`-installed extensions are listed by name only (the same as Git's `git help -a`). lstk never executes an extension to render help, so listing is always side-effect-free. + +## Authorizing the user (and why it cannot rely on lstk) + +If your extension is free to use, you do not need to authorize anything — just do your work. + +If your extension must be restricted (for example a paid feature), **authorization is entirely your responsibility.** lstk hands you the user's auth token in `LSTK_EXT_CONTEXT.authToken` and dispatches; it makes no entitlement decision. Authorize by making a server-side check with that token — typically a call to the LocalStack platform that returns whether this user is entitled — and refuse to perform the protected work when it does not pass: + +```sh +token=$(printf '%s' "$LSTK_EXT_CONTEXT" | jq -r '.authToken // empty') +if [ -z "$token" ]; then + echo "this command requires a LocalStack account; run 'lstk login'" >&2 + exit 1 +fi +# Verify entitlement server-side using the token, then proceed or refuse. +``` + +**Security note.** lstk is open source and can be rebuilt with any check removed or any value forged. So no protection can depend on lstk behaving honestly — a check lstk performs is a UX speed bump, not a control. The only durable boundary is something a modified lstk cannot produce: a server-side decision keyed to the user's token, or a signature you verify yourself. Anchor real enforcement there. An extension whose value is purely local logic cannot be protected on the client at all (the standard DRM reality); gate the valuable behavior server-side. diff --git a/openspec/changes/add-extension-mechanism/design.md b/openspec/changes/add-extension-mechanism/design.md index 82069bcd..9985fb93 100644 --- a/openspec/changes/add-extension-mechanism/design.md +++ b/openspec/changes/add-extension-mechanism/design.md @@ -58,10 +58,17 @@ Context flows to the extension through exactly two environment variables: - `nonInteractive` (bool) - `emulators` (array of `{ "type", "endpoint", "port" }`, **`[]` when no emulator is running**) -The host environment is inherited; only these two `LSTK_EXT_*` variables are added/overridden (any stray `LSTK_EXT_*` in the user's shell is stripped first so it cannot shadow the resolved values). +The host environment is inherited; only these two `LSTK_EXT_*` variables are added. Any inherited `LSTK_EXT_*` is stripped first so that **lstk fully owns the `LSTK_EXT_` namespace handed to the child**: every `LSTK_EXT_*` an extension sees came from this lstk invocation, never from the user's shell or a parent process. Note this is an *ownership invariant*, not a correctness fix for the two vars we set — Go's `os/exec` already deduplicates `cmd.Env` keeping the last entry, so appending `LSTK_EXT_API_VERSION`/`LSTK_EXT_CONTEXT` after the inherited env would override a stale value regardless. The strip's residual job is to remove `LSTK_EXT_*` keys lstk does *not* set (a user-exported var, or a field removed in some future breaking version that lingers in a nested `lstk`-from-extension invocation), so a feature-detecting extension never reads a value lstk didn't put there. Resolved global flags travel inside this object, not on argv: `nonInteractive` is true when the session is non-interactive (the `--non-interactive` flag was given or stdout is not a TTY — the same condition lstk computes at [cmd/root.go](../../../cmd/root.go) `isInteractive`). Carrying global state in the context (rather than re-forwarding `--non-interactive` on the extension's command line) lets the extension own its entire flag space without collision, and each new global flag becomes one additive JSON field. +**Versioning and feature detection.** The two jobs — detecting *additive* fields and guarding against *breaking* changes — are handled by different mechanisms, and conflating them is a bug: + +- `LSTK_EXT_API_VERSION` is bumped **only on a breaking change** (a field removed or repurposed). An extension uses it to refuse a contract *generation* it does not understand — i.e. refuse a version *higher* than the one it was built against, since an unknown-higher generation may have changed something it relies on. (The version can never be lower; lstk only moves forward.) +- **Additive fields are detected by presence in the JSON, never by the version number.** Because a new field is added *without* bumping the version (the version still reads `1`), the version cannot tell an extension whether a newer field exists — only the JSON can. An extension that needs a field which may not exist in older lstk checks for that field directly (missing key / null / zero value) and degrades or errors with a clear "needs newer lstk" message. + +This is the payoff of the JSON move: JSON is self-describing, so feature detection is native, and the version shrinks to its one honest job (breaking generations). It follows that **any field added later MUST be distinguishable when absent** — `omitempty`, a pointer, or an explicit null — so an older lstk that omits it is distinguishable from a newer lstk that sends a zero value. The baseline (version 1) fields are: `configDir` (always present), `authToken` (`omitempty`), `nonInteractive` (always present), `emulators` (always present, possibly `[]`). + **Rationale**: The decisive driver is **multi-emulator support** (PR review): lstk can run an AWS, a Snowflake, and an Azure emulator simultaneously, which the original flat `LSTK_EXT_EMULATOR_ENDPOINT/TYPE/PORT` triple cannot represent. Adopting a JSON `emulators` array **from day one** means multi-emulator is expressible without a later breaking change to the contract. Once one field needs structure, a single structured object beats a growing sprawl of flat scalars: every future field is an additive JSON key under the same version rather than a new variable name, and the payload is still language-agnostic (every runtime has a JSON parser) and never forces the extension to read lstk's TOML or re-implement discovery. Keeping `LSTK_EXT_API_VERSION` as a flat scalar preserves clean version negotiation. A future signed-entitlement description slots in as an additive field (e.g. `grant` / `publicKey`) under a documented version bump. **Alternatives considered**: Flat per-field variables — the original design; rejected because it cannot represent multiple running emulators and grows a new variable per field. A JSON context *file* referenced by a path env var (rejected: temp-file lifecycle/cleanup, where inline JSON in the env var is simpler and the payload is small). Putting the version *inside* the JSON too (rejected: keeping it a flat scalar lets an extension negotiate compatibility before parsing). Passing context as CLI flags (rejected: collides with extension-owned flag space). diff --git a/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md b/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md index a7366d80..ccc74992 100644 --- a/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md +++ b/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md @@ -8,7 +8,15 @@ Define the versioned contract by which lstk passes runtime context — resolved ### Requirement: Versioned JSON context contract -lstk SHALL pass runtime context to an extension through exactly two environment variables: `LSTK_EXT_API_VERSION`, set to the integer version of the contract it implements, and `LSTK_EXT_CONTEXT`, a single JSON object carrying the resolved context (config directory, auth token, resolved global-flag state, and running emulators). The version is kept as a flat scalar — outside the JSON payload — so an extension can check contract compatibility before parsing the object. The contract SHALL be additive within a major version (new JSON fields may be added); removing or repurposing a field SHALL require incrementing `LSTK_EXT_API_VERSION`. lstk SHALL NOT require the extension to parse lstk's own config files. +lstk SHALL pass runtime context to an extension through exactly two environment variables: `LSTK_EXT_API_VERSION`, set to the integer version of the contract it implements, and `LSTK_EXT_CONTEXT`, a single JSON object carrying the resolved context (config directory, auth token, resolved global-flag state, and running emulators). The version is kept as a flat scalar — outside the JSON payload — so an extension can read it before parsing the object. + +The two concerns of detecting *additive* fields and guarding against *breaking* changes are handled separately: + +- `LSTK_EXT_API_VERSION` SHALL be incremented **only** when a field is removed or repurposed (a breaking change). Adding a field SHALL NOT increment it. An extension uses the version to refuse a contract generation it does not understand (a version higher than it was built for). +- Additive fields SHALL be detected by an extension through their **presence in the JSON object**, not through the version number — since the version does not change when a field is added, it cannot signal a new field's availability. +- Any field added after version 1 SHALL be distinguishable when absent (omitted, null, or otherwise not a zero value indistinguishable from "not provided"), so an extension running against an older lstk that omits the field can tell it apart from a newer lstk that provides it. + +lstk SHALL NOT require the extension to parse lstk's own config files. #### Scenario: API version is advertised @@ -20,6 +28,21 @@ lstk SHALL pass runtime context to an extension through exactly two environment - **WHEN** lstk invokes any extension - **THEN** the extension's environment includes `LSTK_EXT_CONTEXT` containing a JSON object the extension can decode to obtain the config directory, auth token (when present), non-interactive state, and the list of running emulators, without reading lstk's TOML config +#### Scenario: A newly added field is detected by presence, not version + +- **WHEN** a later lstk adds a new optional field to `LSTK_EXT_CONTEXT` without incrementing `LSTK_EXT_API_VERSION` +- **THEN** an extension determines the field's availability by checking for its presence in the decoded object +- **AND** an older lstk that omits the field is distinguishable from a newer lstk that provides it + +### Requirement: lstk owns the LSTK_EXT_ namespace conveyed to the extension + +lstk SHALL strip any inherited `LSTK_EXT_*` variables from the environment it passes to an extension before adding its own, so every `LSTK_EXT_*` variable an extension observes originates from the current lstk invocation rather than the user's shell or a parent process. This holds even though `LSTK_EXT_API_VERSION` and `LSTK_EXT_CONTEXT` are always set (and would override an inherited value anyway): the guarantee extends to `LSTK_EXT_*` names lstk does not set. + +#### Scenario: Stray inherited contract variables do not reach the extension + +- **WHEN** the environment already contains an `LSTK_EXT_*` variable (e.g. a user-exported `LSTK_EXT_CONTEXT` or a leftover from a nested invocation) +- **THEN** the extension does not observe that inherited value; it sees only the `LSTK_EXT_*` variables lstk resolved for this invocation + ### Requirement: Running emulators are provided as a JSON array The `LSTK_EXT_CONTEXT` object SHALL include an `emulators` array, with one entry per LocalStack emulator currently running, so an extension can work against every running emulator rather than a single one. Each entry SHALL carry the emulator `type` (e.g. `aws`, `snowflake`, `azure`), the `endpoint` (a full URL resolved with the same discovery and host resolution used by built-in commands), and the `port`. When no emulator is running, `emulators` SHALL be an empty array (`[]`), not omitted, so an extension always decodes a list. From df705968da2b10fa2f1cf54c1849df770659d66d Mon Sep 17 00:00:00 2001 From: Peter Smith Date: Wed, 1 Jul 2026 09:37:29 +1200 Subject: [PATCH 4/6] feat(extensions): implement Git-style extension mechanism Dispatch `lstk ` to an external `lstk-` executable when `` is not a built-in, resolved from the bundled directory next to the binary and then PATH (built-ins always win; bundled wins over PATH). Arguments after the name, stdin/stdout/stderr, and the exit code are passed through unchanged. Runtime context is conveyed in two environment variables: LSTK_EXT_API_VERSION (a flat integer) and LSTK_EXT_CONTEXT (a JSON object with the config dir, auth token, non-interactive state, and an array of every running emulator). Each invocation is wrapped in an OpenTelemetry span (name, bundled, exit code) so usage is recorded when LSTK_OTEL is enabled and costs nothing when it is not. Authorization stays with the extension: lstk conveys the token and makes no entitlement decision. Help lists resolvable extensions side-effect-free, with one-line descriptions for bundled ones read from a descriptions file when present, aligned with the built-in command columns. Scope: this first release runs extensions (including a bundled one placed manually); automated bundled-extension distribution and atomic co-update are deferred to the add-bundled-extension-distribution change. Domain logic in internal/extension/; dispatch, help, and context wiring in cmd/extension.go (hooked from cmd/root.go). Covered by unit tests and integration tests using an in-tree reference extension. Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 13 + cmd/extension.go | 175 ++++++++ cmd/help.go | 5 +- cmd/root.go | 22 + internal/extension/context.go | 85 ++++ internal/extension/descriptions.go | 44 ++ internal/extension/descriptions_test.go | 47 +++ internal/extension/exec.go | 67 +++ internal/extension/extension.go | 33 ++ internal/extension/extension_test.go | 307 ++++++++++++++ internal/extension/resolve.go | 198 +++++++++ test/integration/extension_test.go | 395 ++++++++++++++++++ .../test-samples/extensions/lstk-ref/main.go | 124 ++++++ 13 files changed, 1514 insertions(+), 1 deletion(-) create mode 100644 cmd/extension.go create mode 100644 internal/extension/context.go create mode 100644 internal/extension/descriptions.go create mode 100644 internal/extension/descriptions_test.go create mode 100644 internal/extension/exec.go create mode 100644 internal/extension/extension.go create mode 100644 internal/extension/extension_test.go create mode 100644 internal/extension/resolve.go create mode 100644 test/integration/extension_test.go create mode 100644 test/integration/test-samples/extensions/lstk-ref/main.go diff --git a/CLAUDE.md b/CLAUDE.md index 506569ad..295593df 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -90,6 +90,19 @@ Environment variables: lstk proxies third-party IaC tools at the AWS emulator so they run against LocalStack with no `*local` wrapper installed. Each command forwards its args to the real tool after configuring the environment; domain logic lives under `internal/iac//cli/`, wiring in `cmd/.go`, with shared command-boundary helpers in `cmd/iac.go`. Siblings: `lstk terraform` (alias `tf`), `lstk cdk`, `lstk sam`. +# Extensions + +lstk supports Git-style extensions: when `lstk ` is not a built-in command or alias, lstk resolves and execs an external `lstk-` executable, forwarding all arguments after `` verbatim, passing stdin/stdout/stderr through, and propagating the child's exit code. Built-ins always win (dispatch happens only on the unknown-command path). Domain logic lives in `internal/extension/`; the unknown-command dispatch, the help listing, and the runtime-context wiring are in `cmd/extension.go`, hooked from `cmd/root.go`. + +Resolution order is built-ins → bundled dir → `PATH`. The bundled dir is the directory containing the symlink-resolved lstk executable (`filepath.EvalSymlinks(os.Executable())`), so bundled extensions are found through npm/Homebrew shims; a bundled extension wins over a same-named `PATH` executable. Windows executable extensions (`PATHEXT`) are honored. There is no manifest — any resolvable `lstk-` is the `` extension. + +Runtime context is conveyed in two environment variables: `LSTK_EXT_API_VERSION` (a flat integer the extension checks before parsing) and `LSTK_EXT_CONTEXT` (a single JSON object: `configDir`, optional `authToken`, `nonInteractive`, and an `emulators` array of `{type, endpoint, port}` — `[]` when none running, multiple entries when several emulators run at once). The `extension.Context` type and `Environ` builder live in `internal/extension/context.go`; the command boundary (`cmd/extension.go`) discovers all running emulators and populates it. `Invoke` wraps each exec in an OTEL span (extension name, bundled, exit code), so invocations are recorded as telemetry when `LSTK_OTEL` is enabled and cost nothing when it is not. + +Scope: the first release **runs** extensions (PATH and bundled-dir resolution) and conveys context. Automated **distribution and atomic co-update** of LocalStack's bundled extensions are deferred to the `add-bundled-extension-distribution` change — the first release validates bundled extensions by manual placement next to `lstk`. + +See [extensions-authoring.md](docs/extensions-authoring.md) for the author-facing contract. + + # Snapshots `lstk snapshot` captures and restores the running emulator's state. For Snowflake and Azure, snapshot support is still maturing, so these commands surface a friendly heads-up that results may be incomplete. Domain logic lives in `internal/snapshot/`; `cmd/snapshot.go` is wiring + output-mode selection. diff --git a/cmd/extension.go b/cmd/extension.go new file mode 100644 index 00000000..00ca3131 --- /dev/null +++ b/cmd/extension.go @@ -0,0 +1,175 @@ +package cmd + +import ( + "context" + "errors" + "fmt" + "os" + "strings" + + "github.com/localstack/lstk/internal/config" + "github.com/localstack/lstk/internal/container" + "github.com/localstack/lstk/internal/endpoint" + "github.com/localstack/lstk/internal/env" + "github.com/localstack/lstk/internal/extension" + "github.com/localstack/lstk/internal/log" + "github.com/localstack/lstk/internal/output" + "github.com/localstack/lstk/internal/runtime" + "github.com/spf13/cobra" +) + +// dispatchExtension is the unknown-command fallthrough: when Cobra finds no +// built-in command or alias for args[0], lstk resolves an `lstk-` +// executable (bundled dir first, then PATH) and execs it, forwarding the +// remaining args verbatim and conveying runtime context via LSTK_EXT_*. Built-in +// commands never reach here because Cobra routes them to their own command, so +// they always take precedence. When no extension resolves, lstk emits its +// standard unknown-command error and returns a silent, non-zero error. +func dispatchExtension(ctx context.Context, cfg *env.Env, logger log.Logger, args []string) error { + name, extArgs := args[0], args[1:] + sink := output.NewPlainSink(os.Stdout) + + resolver := extension.NewResolver(logger) + ext, err := resolver.Resolve(name) + if err != nil { + if errors.Is(err, extension.ErrNotFound) { + sink.Emit(output.ErrorEvent{ + Title: fmt.Sprintf("unknown command %q for lstk", name), + Actions: []output.ErrorAction{{Label: "See help:", Value: "lstk -h"}}, + }) + return output.NewSilentError(fmt.Errorf("unknown command %q for lstk", name)) + } + return err + } + + emulators := resolveEmulators(ctx, cfg, logger) + configDir, err := config.ConfigDir() + if err != nil { + return fmt.Errorf("resolving config directory: %w", err) + } + + runCtx := extension.Context{ + ConfigDir: configDir, + AuthToken: cfg.AuthToken, + NonInteractive: !isInteractiveMode(cfg), + Emulators: emulators, + } + + logger.Info("extension: dispatching %q (bundled=%v) at %s", name, ext.Bundled, ext.Path) + return extension.Invoke(ctx, ext, extArgs, runCtx) +} + +// resolveEmulators best-effort discovers every running LocalStack emulator and +// returns them for the LSTK_EXT_CONTEXT `emulators` array. lstk can run several +// emulators at once (e.g. AWS + Snowflake + Azure), so all running ones are +// reported, not just the first. When no emulator is running (or the runtime is +// unavailable) it returns nil, which Environ renders as an empty array; the +// extension is still executed. +func resolveEmulators(ctx context.Context, cfg *env.Env, logger log.Logger) []extension.Emulator { + rt, err := runtime.NewDockerRuntime(cfg.DockerHost) + if err != nil { + logger.Info("extension: runtime unavailable, omitting emulator context: %v", err) + return nil + } + if err := rt.IsHealthy(ctx); err != nil { + logger.Info("extension: runtime not healthy, omitting emulator context: %v", err) + return nil + } + + var emulators []extension.Emulator + for _, c := range emulatorCandidates() { + name, err := container.ResolveRunningContainerName(ctx, rt, c) + if err != nil || name == "" { + continue + } + // Ask the runtime for the actual bound port rather than trusting config: + // the user may have changed the config port while the container still + // runs on the old one (mirrors `lstk status`). + hostPort := c.Port + if containerPort, err := c.ContainerPort(); err == nil { + if actual, err := rt.GetBoundPort(ctx, name, containerPort); err == nil { + hostPort = actual + } + } + host, _ := endpoint.ResolveHost(ctx, hostPort, cfg.LocalStackHost) + emulators = append(emulators, extension.Emulator{ + Type: string(c.Type), + Endpoint: "http://" + host, + Port: hostPort, + }) + } + return emulators +} + +// emulatorCandidates returns the containers to probe for a running emulator: the +// configured containers first, then a default container for every other +// selectable emulator type, so a running emulator is found even if the config +// names a different one. +func emulatorCandidates() []config.ContainerConfig { + var candidates []config.ContainerConfig + seen := map[config.EmulatorType]struct{}{} + + if appCfg, err := config.Get(); err == nil { + for _, c := range appCfg.Containers { + candidates = append(candidates, c) + seen[c.Type] = struct{}{} + } + } + for _, t := range config.SelectableEmulatorTypes { + if _, ok := seen[t]; ok { + continue + } + candidates = append(candidates, config.ContainerConfig{Type: t, Port: config.DefaultPort}) + } + return candidates +} + +// registerExtensionHelp wires an "extensions" template function that renders the +// Extensions section of `lstk --help`. It scans the bundled dir + PATH for +// `lstk-*` executables (de-duplicated, bundled wins) and attaches descriptions +// for bundled extensions from the hand-authored descriptions file; PATH and +// custom extensions, and bundled names missing from the file, are name-only. +// Rendering never executes an extension. A scan happens on each help render so +// freshly installed extensions appear without restarting. +func registerExtensionHelp(logger log.Logger) { + cobra.AddTemplateFunc("extensions", func(namePadding int) string { + resolver := extension.NewResolver(logger) + list := resolver.List() + if len(list) == 0 { + return "" + } + descriptions := extension.LoadDescriptions(resolver.BundledDir, logger) + return formatExtensionList(list, descriptions, namePadding) + }) +} + +// formatExtensionList renders the extension help lines so they align with the +// command sections above them. It mirrors Cobra's own scheme (see the usage +// template's "{{rpad .Name .NamePadding}} {{.Short}}"): each name is right-padded +// to namePadding, then a single space, then its description (bundled extensions +// only, from the descriptions file). namePadding is the root command's +// .NamePadding, so the description column matches the Commands/Tools sections; a +// name longer than namePadding widens its own row exactly as Cobra's per-row +// rpad does. Lines are sorted by name (List already sorts). +func formatExtensionList(list []extension.Extension, descriptions map[string]string, namePadding int) string { + width := namePadding + for _, ext := range list { + if len(ext.Name) > width { + width = len(ext.Name) + } + } + + var b strings.Builder + for _, ext := range list { + desc := "" + if ext.Bundled { + desc = descriptions[ext.Name] + } + if desc != "" { + fmt.Fprintf(&b, " %-*s %s\n", width, ext.Name, desc) + } else { + fmt.Fprintf(&b, " %s\n", ext.Name) + } + } + return strings.TrimRight(b.String(), "\n") +} diff --git a/cmd/help.go b/cmd/help.go index 7d04e2a2..53f8dfce 100644 --- a/cmd/help.go +++ b/cmd/help.go @@ -25,7 +25,10 @@ Commands:{{range $cmds}}{{if (or .IsAvailableCommand (eq .Name "help"))}} {{rpad .Name .NamePadding }} {{.Short}}{{end}}{{end}}{{end}}{{if not .AllChildCommandsHaveGroup}} Additional Commands:{{range $cmds}}{{if (and (eq .GroupID "") (or .IsAvailableCommand (eq .Name "help")))}} - {{rpad .Name .NamePadding }} {{.Short}}{{end}}{{end}}{{end}}{{end}}{{end}}{{if .HasAvailableLocalFlags}} + {{rpad .Name .NamePadding }} {{.Short}}{{end}}{{end}}{{end}}{{end}}{{end}}{{if not .HasParent}}{{if extensions .NamePadding}} + +Extensions: +{{extensions .NamePadding}}{{end}}{{end}}{{if .HasAvailableLocalFlags}} Options: {{.LocalFlags.FlagUsages | trimTrailingWhitespaces}}{{end}}{{if .HasAvailableInheritedFlags}} diff --git a/cmd/root.go b/cmd/root.go index 8054882c..a8cf95e1 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -48,7 +48,18 @@ func NewRootCmd(cfg *env.Env, tel *telemetry.Client, logger log.Logger) *cobra.C Short: "LocalStack CLI", Long: "lstk is the command-line interface for LocalStack.", PreRunE: initConfigDeferCreate(&firstRun), + // ArbitraryArgs stops Cobra from rejecting an unknown first arg with its + // own "unknown command" error before RunE runs, so an unknown `lstk + // ` falls through to extension dispatch. Built-in commands are still + // matched by Cobra's command resolution first, so they always win. + Args: cobra.ArbitraryArgs, RunE: func(cmd *cobra.Command, args []string) error { + // A non-empty arg here means the first positional was not a built-in + // command (Cobra would have routed those to their own command), so it + // is an extension name; everything after it is forwarded verbatim. + if len(args) > 0 { + return dispatchExtension(cmd.Context(), cfg, logger, args) + } rt, err := runtime.NewDockerRuntime(cfg.DockerHost) if err != nil { return err @@ -69,7 +80,18 @@ func NewRootCmd(cfg *env.Env, tel *telemetry.Client, logger log.Logger) *cobra.C root.PersistentFlags().BoolVar(&cfg.NonInteractive, "non-interactive", false, "Disable interactive mode") root.Flags().Bool("persist", false, "Persist emulator state across restarts") + // Parse lstk's global flags only when they precede the command name: with + // interspersing disabled, Cobra consumes leading flags and hands everything + // from the first positional (the command/extension name) onward to the + // dispatch path verbatim. This gives Git-style "globals only before the + // command" and lets an extension own its entire flag space — a flag after the + // name (even one named like an lstk global) is forwarded untouched. Only the + // root's own flag set is affected; built-in subcommands keep their own + // (interspersing) flag parsing. + root.Flags().SetInterspersed(false) + configureHelp(root) + registerExtensionHelp(logger) root.InitDefaultVersionFlag() root.Flags().Lookup("version").Shorthand = "v" diff --git a/internal/extension/context.go b/internal/extension/context.go new file mode 100644 index 00000000..9e7914ff --- /dev/null +++ b/internal/extension/context.go @@ -0,0 +1,85 @@ +package extension + +import ( + "encoding/json" + "fmt" + "strconv" + "strings" +) + +// The runtime-context contract is conveyed to an extension through exactly two +// environment variables: +// +// - EnvAPIVersion (LSTK_EXT_API_VERSION) — a flat integer kept outside the JSON +// payload so an extension can check contract compatibility before parsing. +// - EnvContext (LSTK_EXT_CONTEXT) — a single JSON object (see Context) carrying +// the resolved config directory, auth token, non-interactive state, and the +// list of running emulators. +// +// APIVersion is bumped ONLY on a breaking change (a field removed or +// repurposed); adding a field does not bump it. Extensions therefore detect +// additive fields by their presence in the JSON object — not via the version — +// and use the version only to refuse a contract generation they predate. So any +// field added after version 1 must be distinguishable when absent (omitempty / +// pointer / null) for that presence check to work. +const ( + EnvAPIVersion = "LSTK_EXT_API_VERSION" + EnvContext = "LSTK_EXT_CONTEXT" +) + +// envPrefix is the common prefix of every contract variable. Inherited values +// under this prefix are stripped from the environment before the resolved +// contract is applied, so lstk fully owns the LSTK_EXT_ namespace handed to the +// child: every LSTK_EXT_* an extension sees came from this lstk invocation. This +// is an ownership invariant, not a correctness fix for the two vars we always +// set — exec.Cmd deduplicates Env keeping the last entry, so those override an +// inherited value regardless; the strip's job is to also remove LSTK_EXT_* names +// lstk does not set. +const envPrefix = "LSTK_EXT_" + +// Emulator describes one running LocalStack emulator in the context payload. +type Emulator struct { + Type string `json:"type"` // emulator type, e.g. "aws", "snowflake", "azure" + Endpoint string `json:"endpoint"` // full URL, e.g. "http://localhost:4566" + Port string `json:"port"` // resolved host port, e.g. "4566" +} + +// Context is the resolved runtime context lstk conveys to an extension, rendered +// as the LSTK_EXT_CONTEXT JSON object. The command boundary populates it +// (resolving running emulators, config dir, auth token, and interactivity) and +// Environ renders it. An empty AuthToken is omitted from the JSON; Emulators is +// always present, marshalling to [] when no emulator is running so an extension +// always decodes a list. +type Context struct { + ConfigDir string `json:"configDir"` + AuthToken string `json:"authToken,omitempty"` + NonInteractive bool `json:"nonInteractive"` + Emulators []Emulator `json:"emulators"` +} + +// Environ layers the resolved contract on top of the inherited host environment +// base (typically os.Environ()), returning a new slice suitable for +// exec.Cmd.Env. The host environment is preserved so extensions inherit the +// user's PATH, locale, and tool configuration; only LSTK_EXT_API_VERSION and +// LSTK_EXT_CONTEXT are added. Any inherited LSTK_EXT_* is stripped first so a +// stray value cannot shadow lstk's resolved context. +func (c Context) Environ(base []string) ([]string, error) { + if c.Emulators == nil { + c.Emulators = []Emulator{} + } + payload, err := json.Marshal(c) + if err != nil { + return nil, fmt.Errorf("marshal extension context: %w", err) + } + + env := make([]string, 0, len(base)+2) + for _, entry := range base { + if strings.HasPrefix(entry, envPrefix) { + continue + } + env = append(env, entry) + } + env = append(env, EnvAPIVersion+"="+strconv.Itoa(APIVersion)) + env = append(env, EnvContext+"="+string(payload)) + return env, nil +} diff --git a/internal/extension/descriptions.go b/internal/extension/descriptions.go new file mode 100644 index 00000000..ec97e308 --- /dev/null +++ b/internal/extension/descriptions.go @@ -0,0 +1,44 @@ +package extension + +import ( + "os" + "path/filepath" + + "github.com/localstack/lstk/internal/log" + "github.com/pelletier/go-toml/v2" +) + +// DescriptionsFileName is the static, hand-authored file shipped alongside +// the bundled extensions that maps a bundled extension's command name to a +// one-line description for help rendering. It is a single LocalStack-controlled +// file (not a per-extension manifest), version-locked to the bundled binaries +// and validated against them at release time. +// Its TOML body is a flat table of name = "description" entries, e.g.: +// +// deploy = "Deploy your application to LocalStack" +const DescriptionsFileName = "lstk-extensions.toml" + +// LoadDescriptions reads the bundled descriptions file from dir and returns a +// map of extension command name to one-line description. A missing or unreadable +// file degrades to an empty map without error, so help rendering never fails on +// account of descriptions. dir is the bundled-extensions directory; an empty dir +// yields an empty map. +func LoadDescriptions(dir string, logger log.Logger) map[string]string { + if dir == "" { + return map[string]string{} + } + path := filepath.Join(dir, DescriptionsFileName) + data, err := os.ReadFile(path) + if err != nil { + if !os.IsNotExist(err) { + logger.Info("extension: could not read descriptions file %s: %v", path, err) + } + return map[string]string{} + } + descriptions := map[string]string{} + if err := toml.Unmarshal(data, &descriptions); err != nil { + logger.Info("extension: could not parse descriptions file %s: %v", path, err) + return map[string]string{} + } + return descriptions +} diff --git a/internal/extension/descriptions_test.go b/internal/extension/descriptions_test.go new file mode 100644 index 00000000..d31e701d --- /dev/null +++ b/internal/extension/descriptions_test.go @@ -0,0 +1,47 @@ +package extension + +import ( + "os" + "path/filepath" + "testing" + + "github.com/localstack/lstk/internal/log" +) + +func TestLoadDescriptionsReadsFile(t *testing.T) { + dir := t.TempDir() + body := "deploy = \"Deploy your application to LocalStack\"\nbackup = \"Back up emulator state\"\n" + if err := os.WriteFile(filepath.Join(dir, DescriptionsFileName), []byte(body), 0o644); err != nil { + t.Fatal(err) + } + got := LoadDescriptions(dir, log.Nop()) + if got["deploy"] != "Deploy your application to LocalStack" { + t.Errorf("deploy description = %q", got["deploy"]) + } + if got["backup"] != "Back up emulator state" { + t.Errorf("backup description = %q", got["backup"]) + } +} + +func TestLoadDescriptionsMissingFileDegrades(t *testing.T) { + got := LoadDescriptions(t.TempDir(), log.Nop()) + if len(got) != 0 { + t.Fatalf("expected empty map for missing file, got %+v", got) + } +} + +func TestLoadDescriptionsEmptyDir(t *testing.T) { + if got := LoadDescriptions("", log.Nop()); len(got) != 0 { + t.Fatalf("expected empty map for empty dir, got %+v", got) + } +} + +func TestLoadDescriptionsMalformedDegrades(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, DescriptionsFileName), []byte("this is not = valid = toml ="), 0o644); err != nil { + t.Fatal(err) + } + if got := LoadDescriptions(dir, log.Nop()); len(got) != 0 { + t.Fatalf("expected empty map for malformed file, got %+v", got) + } +} diff --git a/internal/extension/exec.go b/internal/extension/exec.go new file mode 100644 index 00000000..e2289ce9 --- /dev/null +++ b/internal/extension/exec.go @@ -0,0 +1,67 @@ +package extension + +import ( + "context" + "errors" + "os" + "os/exec" + + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/codes" + + "github.com/localstack/lstk/internal/output" +) + +// Invoke executes the resolved extension with args forwarded unmodified, +// connecting the child's stdin/stdout/stderr to lstk's own so the user's +// terminal is wired straight through. The runtime context is layered on the +// inherited host environment. +// +// The invocation is wrapped in an OpenTelemetry span recording the extension +// name, whether it was bundled, and the exit code, so extension usage is visible +// when telemetry is enabled (LSTK_OTEL); when telemetry is disabled the global +// no-op tracer makes this free and emits nothing. lstk does not inject trace +// context into the extension process, so an extension's own spans do not yet +// nest under lstk's trace (deferred). +// +// A non-zero exit from the extension is wrapped as a silent error carrying the +// *exec.ExitError, so the top-level handler propagates the child's exit code as +// lstk's own (via main.go's errors.As check) without printing an extra +// lstk-level error line over the extension's output. Modelled on the IaC +// proxies' exec path. +func Invoke(ctx context.Context, ext *Extension, args []string, runCtx Context) error { + ctx, span := otel.Tracer("github.com/localstack/lstk/internal/extension").Start(ctx, "extension") + defer span.End() + span.SetAttributes( + attribute.String("extension.name", ext.Name), + attribute.Bool("extension.bundled", ext.Bundled), + ) + + envv, err := runCtx.Environ(os.Environ()) + if err != nil { + span.RecordError(err) + span.SetStatus(codes.Error, err.Error()) + return err + } + + cmd := exec.CommandContext(ctx, ext.Path, args...) + cmd.Env = envv + cmd.Stdin = os.Stdin + cmd.Stdout = os.Stdout + cmd.Stderr = os.Stderr + + if err := cmd.Run(); err != nil { + var exitErr *exec.ExitError + if errors.As(err, &exitErr) { + span.SetAttributes(attribute.Int("extension.exit_code", exitErr.ExitCode())) + span.SetStatus(codes.Error, "extension exited non-zero") + return output.NewSilentError(err) + } + span.RecordError(err) + span.SetStatus(codes.Error, err.Error()) + return err + } + span.SetAttributes(attribute.Int("extension.exit_code", 0)) + return nil +} diff --git a/internal/extension/extension.go b/internal/extension/extension.go new file mode 100644 index 00000000..928b5cd1 --- /dev/null +++ b/internal/extension/extension.go @@ -0,0 +1,33 @@ +// Package extension implements lstk's Git-style extension mechanism: when a user +// runs `lstk ` and `` is not a built-in command, lstk resolves and +// executes an external `lstk-` executable, forwarding arguments, streams, +// and the exit code. This mirrors Git's `git-` model and lstk's own IaC +// proxies, and is the only model that cleanly supports closed-source and +// third-party extensions written in any language: an extension is an opaque +// binary that never touches the core repository. +package extension + +// APIVersion is the integer version of the LSTK_EXT_* runtime-context contract +// that this lstk implements. It is exposed to extensions as +// LSTK_EXT_API_VERSION. Bump it only when a variable is removed or repurposed; +// adding a new variable is an additive change that keeps the same version. +const APIVersion = 1 + +// NamePrefix is the executable-name prefix that identifies an extension: an +// executable named "lstk-" provides the "" extension. +const NamePrefix = "lstk-" + +// Extension is a resolved extension executable: its command name (the part after +// the "lstk-" prefix) and the absolute path to the executable that provides it. +// Bundled reports whether it was resolved from the bundled-extensions directory +// (which ships with lstk and takes precedence over PATH) rather than from PATH. +type Extension struct { + Name string + Path string + Bundled bool +} + +// NewExtension returns an Extension for the given command name and executable path. +func NewExtension(name, path string, bundled bool) *Extension { + return &Extension{Name: name, Path: path, Bundled: bundled} +} diff --git a/internal/extension/extension_test.go b/internal/extension/extension_test.go new file mode 100644 index 00000000..d4a2f2e0 --- /dev/null +++ b/internal/extension/extension_test.go @@ -0,0 +1,307 @@ +package extension + +import ( + "encoding/json" + "os" + "os/exec" + "path/filepath" + goruntime "runtime" + "strings" + "testing" + + "github.com/localstack/lstk/internal/log" +) + +// writeExe writes an executable file named base (with a platform extension on +// Windows) into dir and returns its path. +func writeExe(t *testing.T, dir, base string) string { + t.Helper() + name := base + if goruntime.GOOS == "windows" { + name += ".exe" + } + path := filepath.Join(dir, name) + if err := os.WriteFile(path, []byte("#!/bin/sh\n"), 0o755); err != nil { + t.Fatalf("write %s: %v", path, err) + } + return path +} + +func TestNew(t *testing.T) { + ext := NewExtension("hello", "/usr/bin/lstk-hello", false) + if ext.Name != "hello" || ext.Path != "/usr/bin/lstk-hello" || ext.Bundled { + t.Fatalf("unexpected extension: %+v", ext) + } +} + +func TestResolveBundledWinsOverPath(t *testing.T) { + bundled := t.TempDir() + pathDir := t.TempDir() + bundledPath := writeExe(t, bundled, "lstk-deploy") + writeExe(t, pathDir, "lstk-deploy") + t.Setenv("PATH", pathDir) + + r := &Resolver{BundledDir: bundled, logger: log.Nop()} + ext, err := r.Resolve("deploy") + if err != nil { + t.Fatalf("Resolve: %v", err) + } + if !ext.Bundled { + t.Fatalf("expected bundled extension to win, got %+v", ext) + } + if ext.Path != bundledPath { + t.Fatalf("expected path %s, got %s", bundledPath, ext.Path) + } +} + +func TestResolveFallsBackToPath(t *testing.T) { + bundled := t.TempDir() + pathDir := t.TempDir() + writeExe(t, pathDir, "lstk-hello") + t.Setenv("PATH", pathDir) + + r := &Resolver{BundledDir: bundled, logger: log.Nop()} + ext, err := r.Resolve("hello") + if err != nil { + t.Fatalf("Resolve: %v", err) + } + if ext.Bundled { + t.Fatalf("expected PATH extension, got bundled %+v", ext) + } +} + +func TestResolveNotFound(t *testing.T) { + t.Setenv("PATH", t.TempDir()) + r := &Resolver{BundledDir: t.TempDir(), logger: log.Nop()} + if _, err := r.Resolve("doesnotexist"); err != ErrNotFound { + t.Fatalf("expected ErrNotFound, got %v", err) + } +} + +func TestListDeduplicatesBundledWins(t *testing.T) { + bundled := t.TempDir() + pathDir := t.TempDir() + writeExe(t, bundled, "lstk-deploy") + writeExe(t, pathDir, "lstk-deploy") // shadowed by bundled + writeExe(t, pathDir, "lstk-hello") + // A non-extension file and a non-executable must be ignored. + if err := os.WriteFile(filepath.Join(pathDir, "unrelated"), []byte("x"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", pathDir) + + r := &Resolver{BundledDir: bundled, logger: log.Nop()} + list := r.List() + + if len(list) != 2 { + t.Fatalf("expected 2 extensions, got %d: %+v", len(list), list) + } + // Sorted by name: deploy, hello. + if list[0].Name != "deploy" || !list[0].Bundled { + t.Fatalf("expected bundled deploy first, got %+v", list[0]) + } + if list[1].Name != "hello" || list[1].Bundled { + t.Fatalf("expected PATH hello second, got %+v", list[1]) + } +} + +func TestListIgnoresNonExecutableOnUnix(t *testing.T) { + if goruntime.GOOS == "windows" { + t.Skip("execute-bit semantics are Unix-only") + } + pathDir := t.TempDir() + // lstk-noexec exists but is not executable, so it must not be listed. + if err := os.WriteFile(filepath.Join(pathDir, "lstk-noexec"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + t.Setenv("PATH", pathDir) + + r := &Resolver{logger: log.Nop()} + if list := r.List(); len(list) != 0 { + t.Fatalf("expected no extensions, got %+v", list) + } +} + +func TestBundledDirResolvesThroughSymlink(t *testing.T) { + if goruntime.GOOS == "windows" { + t.Skip("symlink test is Unix-only") + } + realDir := t.TempDir() + realExe := filepath.Join(realDir, "lstk") + if err := os.WriteFile(realExe, []byte("#!/bin/sh\n"), 0o755); err != nil { + t.Fatal(err) + } + linkDir := t.TempDir() + link := filepath.Join(linkDir, "lstk") + if err := os.Symlink(realExe, link); err != nil { + t.Fatal(err) + } + // EvalSymlinks(link) must resolve to realExe, whose dir is realDir. + resolved, err := filepath.EvalSymlinks(link) + if err != nil { + t.Fatal(err) + } + if got := filepath.Dir(resolved); got != realDir { + // On macOS, TempDir may itself be under a symlinked /var → /private/var. + realResolved, _ := filepath.EvalSymlinks(realDir) + if got != realResolved { + t.Fatalf("expected bundled dir %s (or %s), got %s", realDir, realResolved, got) + } + } +} + +// decodeContext extracts and JSON-decodes the LSTK_EXT_CONTEXT value from a +// rendered environment, failing the test if it is absent or malformed. +func decodeContext(t *testing.T, env map[string]string) Context { + t.Helper() + raw, ok := env[EnvContext] + if !ok { + t.Fatalf("%s not set in environment", EnvContext) + } + var c Context + if err := json.Unmarshal([]byte(raw), &c); err != nil { + t.Fatalf("decode %s: %v (raw: %s)", EnvContext, err, raw) + } + return c +} + +func mustEnviron(t *testing.T, c Context, base []string) map[string]string { + t.Helper() + entries, err := c.Environ(base) + if err != nil { + t.Fatalf("Environ: %v", err) + } + return envMap(entries) +} + +func TestEnvironRendersJSONContext(t *testing.T) { + env := mustEnviron(t, Context{ + ConfigDir: "/home/u/.config/lstk", + AuthToken: "tok-123", + NonInteractive: true, + Emulators: []Emulator{ + {Type: "aws", Endpoint: "http://localhost:4566", Port: "4566"}, + {Type: "snowflake", Endpoint: "http://localhost:4566", Port: "4566"}, + }, + }, nil) + + if env[EnvAPIVersion] != "1" { + t.Errorf("%s = %q, want \"1\"", EnvAPIVersion, env[EnvAPIVersion]) + } + c := decodeContext(t, env) + if c.ConfigDir != "/home/u/.config/lstk" || c.AuthToken != "tok-123" || !c.NonInteractive { + t.Errorf("decoded scalars wrong: %+v", c) + } + if len(c.Emulators) != 2 || c.Emulators[0].Type != "aws" || c.Emulators[1].Type != "snowflake" { + t.Errorf("emulators wrong: %+v", c.Emulators) + } + if c.Emulators[0].Endpoint != "http://localhost:4566" || c.Emulators[0].Port != "4566" { + t.Errorf("emulator[0] fields wrong: %+v", c.Emulators[0]) + } +} + +func TestEnvironOmitsAbsentValues(t *testing.T) { + env := mustEnviron(t, Context{ConfigDir: "/cfg"}, nil) // no emulator, no token, interactive + + if env[EnvAPIVersion] != "1" { + t.Error("version must always be set") + } + // authToken must be omitted from the JSON (not present as an empty string). + if strings.Contains(env[EnvContext], "authToken") { + t.Errorf("authToken must be omitted when unauthenticated, got: %s", env[EnvContext]) + } + c := decodeContext(t, env) + if c.ConfigDir != "/cfg" { + t.Errorf("configDir = %q, want /cfg", c.ConfigDir) + } + if c.AuthToken != "" || c.NonInteractive { + t.Errorf("token/non-interactive should be zero values: %+v", c) + } + // emulators is always present and an empty (non-nil) array. + if c.Emulators == nil || len(c.Emulators) != 0 { + t.Errorf("emulators must be an empty array when none running, got: %+v", c.Emulators) + } +} + +func TestEnvironSetsOnlyTheTwoContractVariables(t *testing.T) { + env := mustEnviron(t, Context{ConfigDir: "/cfg", AuthToken: "t"}, nil) + var prefixed []string + for k := range env { + if strings.HasPrefix(k, envPrefix) { + prefixed = append(prefixed, k) + } + } + if len(prefixed) != 2 { + t.Errorf("expected exactly LSTK_EXT_API_VERSION and LSTK_EXT_CONTEXT, got %v", prefixed) + } +} + +func TestEnvironInheritsHostEnvAndStripsStaleContract(t *testing.T) { + base := []string{"HTTP_PROXY=http://proxy:8080", "LSTK_EXT_CONTEXT=stale", "LSTK_EXT_API_VERSION=99"} + env := mustEnviron(t, Context{ConfigDir: "/cfg"}, base) + + if env["HTTP_PROXY"] != "http://proxy:8080" { + t.Error("host environment must be inherited") + } + // Stale contract values in the inherited env must be replaced, not duplicated. + if env[EnvAPIVersion] != "1" { + t.Errorf("stale %s from host env must be overridden, got %q", EnvAPIVersion, env[EnvAPIVersion]) + } + if env[EnvContext] == "stale" { + t.Error("stale LSTK_EXT_CONTEXT from host env must be stripped") + } +} + +func TestInvokeForwardsArgsAndPropagatesExitCode(t *testing.T) { + if goruntime.GOOS == "windows" { + t.Skip("shell-script reference extension is Unix-only") + } + dir := t.TempDir() + // Reference extension: echoes args and the JSON context var, exits with the + // code given by its first argument. + script := "#!/bin/sh\n" + + "echo \"args: $*\"\n" + + "echo \"context: $LSTK_EXT_CONTEXT\"\n" + + "exit \"$1\"\n" + path := filepath.Join(dir, "lstk-ref") + if err := os.WriteFile(path, []byte(script), 0o755); err != nil { + t.Fatal(err) + } + + ext := NewExtension("ref", path, false) + err := Invoke(t.Context(), ext, []string{"3", "--flag"}, Context{ConfigDir: "/cfg", AuthToken: "tok"}) + + var exitErr *exec.ExitError + if err == nil || !asExit(err, &exitErr) { + t.Fatalf("expected exit error, got %v", err) + } + if exitErr.ExitCode() != 3 { + t.Fatalf("expected exit code 3, got %d", exitErr.ExitCode()) + } +} + +func envMap(entries []string) map[string]string { + m := map[string]string{} + for _, e := range entries { + if k, v, ok := strings.Cut(e, "="); ok { + m[k] = v + } + } + return m +} + +// asExit unwraps err (including through output.SilentError) into an *exec.ExitError. +func asExit(err error, target **exec.ExitError) bool { + for err != nil { + if ee, ok := err.(*exec.ExitError); ok { + *target = ee + return true + } + u, ok := err.(interface{ Unwrap() error }) + if !ok { + return false + } + err = u.Unwrap() + } + return false +} diff --git a/internal/extension/resolve.go b/internal/extension/resolve.go new file mode 100644 index 00000000..98d34dc6 --- /dev/null +++ b/internal/extension/resolve.go @@ -0,0 +1,198 @@ +package extension + +import ( + "errors" + "os" + "os/exec" + "path/filepath" + goruntime "runtime" + "sort" + "strings" + + "github.com/localstack/lstk/internal/log" +) + +// ErrNotFound is returned by Resolve when no matching extension executable +// exists in the bundled directory or on PATH. +var ErrNotFound = errors.New("extension not found") + +// Resolver discovers and resolves extension executables. It searches the +// bundled-extensions directory (BundledDir) before PATH, so a bundled extension +// wins over a same-named executable on PATH. A zero BundledDir disables the +// bundled search (used in tests that exercise only the PATH path). +type Resolver struct { + BundledDir string + logger log.Logger +} + +// NewResolver returns a Resolver whose bundled-extensions directory is derived +// from the symlink-resolved location of the running lstk executable, so it is +// found even when lstk is invoked through a symlink or package shim. +func NewResolver(logger log.Logger) *Resolver { + return &Resolver{ + BundledDir: BundledDir(logger), + logger: logger, + } +} + +// BundledDir returns the directory in which lstk looks for bundled extensions: +// the directory containing the symlink-resolved lstk executable. Resolving +// symlinks is what makes this work through npm `.bin` links and Homebrew shims, +// where the invoked `lstk` is a link to the real binary living next to its +// bundled siblings. It returns "" when the executable path cannot be resolved. +func BundledDir(logger log.Logger) string { + exe, err := os.Executable() + if err != nil { + logger.Info("extension: cannot determine executable path: %v", err) + return "" + } + resolved, err := filepath.EvalSymlinks(exe) + if err != nil { + logger.Info("extension: cannot resolve executable symlinks: %v", err) + resolved = exe + } + return filepath.Dir(resolved) +} + +// Resolve returns the extension for the given command name, searching the +// bundled directory first and then PATH. It returns ErrNotFound when no +// matching executable exists anywhere. +func (r *Resolver) Resolve(name string) (*Extension, error) { + base := NamePrefix + name + + if r.BundledDir != "" { + if path := findExecutable(r.BundledDir, base); path != "" { + return NewExtension(name, path, true), nil + } + } + + if path, err := exec.LookPath(base); err == nil { + return NewExtension(name, path, false), nil + } + + return nil, ErrNotFound +} + +// List returns the extensions resolvable from the bundled directory and PATH, +// de-duplicated by command name with bundled-then-PATH precedence (so a bundled +// extension shadows a same-named PATH executable), sorted by command name. It +// never executes an extension. +func (r *Resolver) List() []Extension { + seen := map[string]struct{}{} + var found []Extension + + add := func(dir string, bundled bool) { + for _, name := range scanDir(dir) { + if _, ok := seen[name]; ok { + continue + } + seen[name] = struct{}{} + path := findExecutable(dir, NamePrefix+name) + found = append(found, Extension{Name: name, Path: path, Bundled: bundled}) + } + } + + if r.BundledDir != "" { + add(r.BundledDir, true) + } + for _, dir := range pathDirs() { + if dir == "" || dir == r.BundledDir { + continue + } + add(dir, false) + } + + sort.Slice(found, func(i, j int) bool { return found[i].Name < found[j].Name }) + return found +} + +// scanDir returns the extension command names (the part after the "lstk-" +// prefix, with any platform executable extension stripped) of the executable +// "lstk-*" files in dir. Non-executable files and subdirectories are ignored. +func scanDir(dir string) []string { + entries, err := os.ReadDir(dir) + if err != nil { + return nil + } + var names []string + for _, entry := range entries { + fileName := entry.Name() + if !strings.HasPrefix(fileName, NamePrefix) { + continue + } + if entry.IsDir() { + continue + } + if !isExecutableFile(filepath.Join(dir, fileName)) { + continue + } + name := strings.TrimPrefix(fileName, NamePrefix) + if goruntime.GOOS == "windows" { + name = strings.TrimSuffix(name, filepath.Ext(name)) + } + if name == "" { + continue + } + names = append(names, name) + } + return names +} + +// findExecutable returns the path to an executable named base in dir, honoring +// platform executable extensions on Windows (PATHEXT), or "" if none is found. +func findExecutable(dir, base string) string { + if goruntime.GOOS == "windows" { + for _, ext := range windowsExts() { + path := filepath.Join(dir, base+ext) + if isExecutableFile(path) { + return path + } + } + return "" + } + path := filepath.Join(dir, base) + if isExecutableFile(path) { + return path + } + return "" +} + +// isExecutableFile reports whether path is a regular file that is executable. On +// Windows, executability is determined by extension (handled by the caller), so +// any regular file here is considered executable; on Unix it must have an +// execute bit set. +func isExecutableFile(path string) bool { + info, err := os.Stat(path) + if err != nil || info.IsDir() || !info.Mode().IsRegular() { + return false + } + if goruntime.GOOS == "windows" { + return true + } + return info.Mode().Perm()&0111 != 0 +} + +// windowsExts returns the executable extensions to try on Windows, derived from +// PATHEXT with a conventional default, lower-cased and dot-prefixed. +func windowsExts() []string { + raw := os.Getenv("PATHEXT") + if raw == "" { + raw = ".COM;.EXE;.BAT;.CMD" + } + var exts []string + for _, ext := range strings.Split(raw, string(os.PathListSeparator)) { + ext = strings.ToLower(strings.TrimSpace(ext)) + if ext == "" { + continue + } + if !strings.HasPrefix(ext, ".") { + ext = "." + ext + } + exts = append(exts, ext) + } + return exts +} + +func pathDirs() []string { + return filepath.SplitList(os.Getenv("PATH")) +} diff --git a/test/integration/extension_test.go b/test/integration/extension_test.go new file mode 100644 index 00000000..b7d212d9 --- /dev/null +++ b/test/integration/extension_test.go @@ -0,0 +1,395 @@ +package integration_test + +import ( + "fmt" + "io" + "os" + "os/exec" + "path/filepath" + "runtime" + "strings" + "sync" + "testing" + + "github.com/localstack/lstk/test/integration/env" + "github.com/stretchr/testify/require" +) + +// The extension mechanism resolves and execs `lstk-` executables for +// unknown commands, conveying runtime context via LSTK_EXT_* env vars. These +// tests use the in-tree reference extension (test-samples/extensions/lstk-ref) built under +// various names and placed on PATH or in the binary's bundled directory. + +var ( + refExtOnce sync.Once + refExtPath string + refExtErr error +) + +// referenceExtensionBinary builds test-samples/extensions/lstk-ref once and returns the path to +// the compiled binary. The same binary backs every extension name in the tests +// (it just echoes its decoded LSTK_EXT_CONTEXT, forwards args, and can exit with a +// chosen code or perform a stubbed self-authorization). The reference extension +// lives inside this `test/integration` module (its own go.mod), so it is built +// from the module root, not the repo root. +func referenceExtensionBinary(t *testing.T) string { + t.Helper() + refExtOnce.Do(func() { + // The test runs with its working directory at the integration module root. + moduleRoot, err := filepath.Abs(".") + if err != nil { + refExtErr = err + return + } + dir, err := os.MkdirTemp("", "lstk-ref-build-*") + if err != nil { + refExtErr = err + return + } + out := filepath.Join(dir, "lstk-ref-bin") + cmd := exec.Command("go", "build", "-o", out, "./test-samples/extensions/lstk-ref") + cmd.Dir = moduleRoot + if b, err := cmd.CombinedOutput(); err != nil { + refExtErr = fmt.Errorf("build reference extension: %w: %s", err, b) + return + } + refExtPath = out + }) + require.NoError(t, refExtErr) + return refExtPath +} + +func execName(base string) string { + if runtime.GOOS == "windows" { + return base + ".exe" + } + return base +} + +func copyExecutable(t *testing.T, src, dst string) { + t.Helper() + in, err := os.Open(src) + require.NoError(t, err) + defer func() { _ = in.Close() }() + out, err := os.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o755) + require.NoError(t, err) + _, err = io.Copy(out, in) + require.NoError(t, err) + require.NoError(t, out.Close()) +} + +// installExtension places the reference extension under the name `lstk-` +// in dir and returns the directory. +func installExtension(t *testing.T, dir, name string) { + t.Helper() + copyExecutable(t, referenceExtensionBinary(t), filepath.Join(dir, execName("lstk-"+name))) +} + +// installLstkBundle copies the built lstk binary into dir so that dir becomes +// lstk's bundled-extensions directory (lstk derives it from its own +// symlink-resolved executable location). Returns the path to the copied binary. +func installLstkBundle(t *testing.T, dir string) string { + t.Helper() + binPath, err := filepath.Abs(binaryPath()) + require.NoError(t, err) + dst := filepath.Join(dir, execName("lstk")) + copyExecutable(t, binPath, dst) + return dst +} + +// envWithPath returns a test environment with extDir prepended to PATH and an +// isolated HOME, so extensions placed in extDir resolve and no real user state +// is touched. Extra DOCKER_HOST etc. can be appended by the caller. +func envWithPath(tmpHome, extDir string) []string { + e := testEnvWithHome(tmpHome, "") + e = append(e, "PATH="+extDir+string(os.PathListSeparator)+os.Getenv("PATH")) + return e +} + +// runBinary runs an arbitrary lstk binary (not necessarily ../../bin/lstk) with +// the given env, returning trimmed stdout, stderr, and the run error. +func runBinary(t *testing.T, dir string, environ []string, binPath string, args ...string) (string, string, error) { + t.Helper() + cmd := exec.CommandContext(testContext(t), binPath, args...) + cmd.Dir = dir + cmd.Env = environ + var stdout, stderr strings.Builder + cmd.Stdout = &stdout + cmd.Stderr = &stderr + err := cmd.Run() + return strings.TrimSpace(stdout.String()), strings.TrimSpace(stderr.String()), err +} + +func TestExtensionBuiltinTakesPrecedence(t *testing.T) { + t.Parallel() + extDir := t.TempDir() + // An lstk-config on PATH must NOT shadow the built-in `config` command. + installExtension(t, extDir, "config") + + tmpHome := t.TempDir() + stdout, stderr, err := runLstk(t, testContext(t), t.TempDir(), envWithPath(tmpHome, extDir), "config", "path") + require.NoError(t, err, stderr) + // The built-in prints a config path; the extension would have printed ARGS=. + require.Contains(t, stdout, "config.toml") + require.NotContains(t, stdout, "ARGS=") +} + +func TestExtensionUnknownCommandDispatches(t *testing.T) { + t.Parallel() + extDir := t.TempDir() + installExtension(t, extDir, "hello") + + tmpHome := t.TempDir() + stdout, stderr, err := runLstk(t, testContext(t), t.TempDir(), envWithPath(tmpHome, extDir), "hello", "world") + require.NoError(t, err, stderr) + require.Contains(t, stdout, "ARGS=[world]") + require.Contains(t, stdout, "API_VERSION=1") + require.Contains(t, stdout, "CONFIG_DIR=") +} + +func TestExtensionUnknownCommandNoExtensionErrors(t *testing.T) { + t.Parallel() + tmpHome := t.TempDir() + // Empty extension dir so `nope` resolves nowhere. The error event renders + // through the plain sink (stdout), consistent with other lstk commands. + stdout, _, err := runLstk(t, testContext(t), t.TempDir(), envWithPath(tmpHome, t.TempDir()), "nope") + requireExitCode(t, 1, err) + require.Contains(t, stdout, "unknown command") +} + +func TestExtensionExitCodePropagates(t *testing.T) { + t.Parallel() + extDir := t.TempDir() + installExtension(t, extDir, "hello") + + tmpHome := t.TempDir() + _, _, err := runLstk(t, testContext(t), t.TempDir(), envWithPath(tmpHome, extDir), "hello", "exit", "7") + requireExitCode(t, 7, err) +} + +func TestExtensionGlobalFlagConveyedNotForwarded(t *testing.T) { + t.Parallel() + extDir := t.TempDir() + installExtension(t, extDir, "hello") + + tmpHome := t.TempDir() + stdout, stderr, err := runLstk(t, testContext(t), t.TempDir(), envWithPath(tmpHome, extDir), "--non-interactive", "hello", "--foo") + require.NoError(t, err, stderr) + // --non-interactive is consumed by lstk and conveyed via env, not forwarded. + require.Contains(t, stdout, "ARGS=[--foo]") + require.Contains(t, stdout, "NON_INTERACTIVE=true") +} + +func TestExtensionAuthTokenConveyedWhenAuthed(t *testing.T) { + t.Parallel() + extDir := t.TempDir() + installExtension(t, extDir, "ref") + + tmpHome := t.TempDir() + environ := envWithPath(tmpHome, extDir) + environ = append(environ, string(env.AuthToken)+"=tok-abc-123") + stdout, stderr, err := runLstk(t, testContext(t), t.TempDir(), environ, "ref") + require.NoError(t, err, stderr) + require.Contains(t, stdout, "AUTH_TOKEN=tok-abc-123") +} + +func TestExtensionEndpointOmittedWhenNoRuntime(t *testing.T) { + t.Parallel() + extDir := t.TempDir() + installExtension(t, extDir, "ref") + + tmpHome := t.TempDir() + environ := envWithPath(tmpHome, extDir) + // Point DOCKER_HOST at a closed port so the runtime is unavailable and the + // emulator context is deterministically omitted. + environ = append(environ, "DOCKER_HOST=tcp://127.0.0.1:1") + stdout, stderr, err := runLstk(t, testContext(t), t.TempDir(), environ, "ref") + require.NoError(t, err, stderr) + require.Contains(t, stdout, "EMULATOR_COUNT=0") + require.NotContains(t, stdout, "EMULATOR=") + // The extension still runs and still receives the always-present variables. + require.Contains(t, stdout, "API_VERSION=1") +} + +func TestExtensionSelfAuthorizationRefusesWithoutToken(t *testing.T) { + t.Parallel() + extDir := t.TempDir() + installExtension(t, extDir, "deploy") + + tmpHome := t.TempDir() + environ := envWithPath(tmpHome, extDir) + environ = append(environ, "DOCKER_HOST=tcp://127.0.0.1:1") + // No auth token: the extension's stubbed self-authorization refuses (exit 13). + _, stderr, err := runLstk(t, testContext(t), t.TempDir(), environ, "deploy", "auth") + requireExitCode(t, 13, err) + require.Contains(t, stderr, "not authorized") +} + +func TestExtensionEndpointConveyedWhenEmulatorRunning(t *testing.T) { + requireDocker(t) + cleanup() + t.Cleanup(cleanup) + + ctx := testContext(t) + startTestContainer(t, ctx) + + extDir := t.TempDir() + installExtension(t, extDir, "ref") + + tmpHome := t.TempDir() + environ := envWithPath(tmpHome, extDir) + environ = append(environ, string(env.AuthToken)+"=tok-xyz") + stdout, stderr, err := runLstk(t, ctx, t.TempDir(), environ, "ref") + require.NoError(t, err, stderr) + require.Contains(t, stdout, "EMULATOR=aws http") + require.Contains(t, stdout, "AUTH_TOKEN=tok-xyz") +} + +func TestExtensionHelpListsBundledWithDescriptionAndPathNameOnly(t *testing.T) { + t.Parallel() + bundleDir := t.TempDir() + lstkBin := installLstkBundle(t, bundleDir) + installExtension(t, bundleDir, "deploy") // bundled + require.NoError(t, os.WriteFile( + filepath.Join(bundleDir, "lstk-extensions.toml"), + []byte("deploy = \"Deploy your application to LocalStack\"\n"), 0o644)) + + extDir := t.TempDir() + installExtension(t, extDir, "hello") // PATH-only + + tmpHome := t.TempDir() + stdout, stderr, err := runBinary(t, t.TempDir(), envWithPath(tmpHome, extDir), lstkBin, "--help") + require.NoError(t, err, stderr) + + require.Contains(t, stdout, "Extensions:") + require.Contains(t, stdout, "deploy") + require.Contains(t, stdout, "Deploy your application to LocalStack") + require.Contains(t, stdout, "hello") + // Help must not execute any extension. + require.NotContains(t, stdout, "ARGS=") +} + +// TestExtensionHelpDescriptionColumnAlignsWithCommands guards the bug where the +// Extensions section computed its own padding (local-max name width + 2 spaces) +// and rendered descriptions in a different column than the Commands/Tools +// sections (Cobra's NamePadding + 1 space). The fix has the Extensions section +// reuse the root command's NamePadding, so both columns line up. +func TestExtensionHelpDescriptionColumnAlignsWithCommands(t *testing.T) { + t.Parallel() + bundleDir := t.TempDir() + lstkBin := installLstkBundle(t, bundleDir) + installExtension(t, bundleDir, "deploy") + require.NoError(t, os.WriteFile( + filepath.Join(bundleDir, "lstk-extensions.toml"), + []byte("deploy = \"Deploy your application to LocalStack\"\n"), 0o644)) + + tmpHome := t.TempDir() + stdout, stderr, err := runBinary(t, t.TempDir(), envWithPath(tmpHome, t.TempDir()), lstkBin, "--help") + require.NoError(t, err, stderr) + + lines := strings.Split(stdout, "\n") + // The built-in `aws` command (in the Tools group) sets the reference column. + cmdCol := descriptionColumn(t, lines, "aws", "Run AWS CLI commands against LocalStack") + extCol := descriptionColumn(t, lines, "deploy", "Deploy your application to LocalStack") + require.Equal(t, cmdCol, extCol, + "extension description column (%d) must align with command description column (%d)", extCol, cmdCol) +} + +// descriptionColumn returns the byte index at which desc begins on the help line +// for the command/extension named name (a " ... " row). It fails +// the test if no such line is found. +func descriptionColumn(t *testing.T, lines []string, name, desc string) int { + t.Helper() + for _, ln := range lines { + if strings.HasPrefix(ln, " "+name+" ") && strings.Contains(ln, desc) { + return strings.Index(ln, desc) + } + } + t.Fatalf("no help line for %q with description %q in:\n%s", name, desc, strings.Join(lines, "\n")) + return -1 +} + +func TestExtensionHelpMissingDescriptionsFileDegrades(t *testing.T) { + t.Parallel() + bundleDir := t.TempDir() + lstkBin := installLstkBundle(t, bundleDir) + installExtension(t, bundleDir, "deploy") // bundled, but no descriptions file + + tmpHome := t.TempDir() + stdout, stderr, err := runBinary(t, t.TempDir(), envWithPath(tmpHome, t.TempDir()), lstkBin, "--help") + require.NoError(t, err, stderr) + require.Contains(t, stdout, "Extensions:") + require.Contains(t, stdout, "deploy") +} + +func TestExtensionResolvableViaSymlinkedLstk(t *testing.T) { + t.Parallel() + if runtime.GOOS == "windows" { + t.Skip("symlink shim test is Unix-only") + } + // Real install dir holds lstk + its bundled extension; lstk is invoked via a + // symlink elsewhere (mimicking an npm .bin link or Homebrew shim). lstk must + // resolve its real location to find the bundled sibling. + realDir := t.TempDir() + realLstk := installLstkBundle(t, realDir) + installExtension(t, realDir, "deploy") + + linkDir := t.TempDir() + link := filepath.Join(linkDir, "lstk") + require.NoError(t, os.Symlink(realLstk, link)) + + tmpHome := t.TempDir() + // Empty PATH extension dir: deploy can only come from the bundled location. + stdout, stderr, err := runBinary(t, t.TempDir(), envWithPath(tmpHome, t.TempDir()), link, "deploy", "ok") + require.NoError(t, err, stderr) + require.Contains(t, stdout, "ARGS=[ok]") +} + +func TestExtensionBundledPremiumSelfAuthorizes(t *testing.T) { + t.Parallel() + bundleDir := t.TempDir() + lstkBin := installLstkBundle(t, bundleDir) + installExtension(t, bundleDir, "deploy") // bundled "premium" extension + + tmpHome := t.TempDir() + noRuntime := append(envWithPath(tmpHome, t.TempDir()), "DOCKER_HOST=tcp://127.0.0.1:1") + + // Unentitled (no token): lstk still dispatches to the bundled extension, which + // performs its own authorization and refuses. + _, _, err := runBinary(t, t.TempDir(), noRuntime, lstkBin, "deploy", "auth") + requireExitCode(t, 13, err) + + // Authed: the bundled extension authorizes successfully. + authed := append(noRuntime, string(env.AuthToken)+"=tok-premium") + stdout, stderr, err := runBinary(t, t.TempDir(), authed, lstkBin, "deploy", "auth") + require.NoError(t, err, stderr) + require.Contains(t, stdout, "authorized") +} + +func TestExtensionBundledWinsOverPath(t *testing.T) { + t.Parallel() + bundleDir := t.TempDir() + lstkBin := installLstkBundle(t, bundleDir) + installExtension(t, bundleDir, "deploy") // bundled + + extDir := t.TempDir() + installExtension(t, extDir, "deploy") // same name on PATH + + tmpHome := t.TempDir() + // The reference extension echoes its own resolved executable path as SELF=, + // so we can confirm the *bundled* copy ran, not the same-named PATH copy. + stdout, stderr, err := runBinary(t, t.TempDir(), envWithPath(tmpHome, extDir), lstkBin, "deploy", "world") + require.NoError(t, err, stderr) + require.Contains(t, stdout, "ARGS=[world]") + // lstk derives its bundled dir from the symlink-resolved executable path, so + // compare SELF against the resolved bundle dir (TempDir may be a symlink, + // e.g. /var → /private/var on macOS). + resolvedBundle, err := filepath.EvalSymlinks(bundleDir) + require.NoError(t, err) + require.Contains(t, stdout, "SELF="+resolvedBundle, "expected the bundled extension to run, not the PATH one") + + // Help lists the de-duplicated name exactly once. + helpOut, _, err := runBinary(t, t.TempDir(), envWithPath(tmpHome, extDir), lstkBin, "--help") + require.NoError(t, err) + require.Equal(t, 1, strings.Count(helpOut, "\n deploy")) +} diff --git a/test/integration/test-samples/extensions/lstk-ref/main.go b/test/integration/test-samples/extensions/lstk-ref/main.go new file mode 100644 index 00000000..27671a2e --- /dev/null +++ b/test/integration/test-samples/extensions/lstk-ref/main.go @@ -0,0 +1,124 @@ +// Command lstk-ref is a reference lstk extension used by lstk's own integration +// tests. It exercises the manifest-free contract: lstk resolves it as the `ref` +// extension (`lstk ref ...`), forwards arguments verbatim, and conveys runtime +// context via the LSTK_EXT_API_VERSION + LSTK_EXT_CONTEXT (JSON) environment +// variables. The extension decodes that context, echoes it back so tests can +// assert on it, and shows the recommended self-authorization pattern. The prose +// author guide in docs/extensions-authoring.md is the canonical reference for +// extension authors; this binary exists for tests. +// +// Subcommands: +// +// (default) Echo the received args and decoded context, then exit 0. +// exit N Echo, then exit with status N (for exit-code propagation tests). +// auth Perform a stubbed self-authorization: succeed (exit 0) only when +// the conveyed context carries an auth token, otherwise refuse +// (exit 13). A real extension would verify the token server-side +// against the LocalStack platform — authorization must never rely on +// lstk, which is open source and rebuildable. +// +// The extension also self-enforces contract compatibility: it requires +// LSTK_EXT_API_VERSION >= minAPIVersion and refuses to run otherwise, rather +// than relying on lstk to gate it. +package main + +import ( + "encoding/json" + "fmt" + "os" + "strconv" +) + +// minAPIVersion is the lowest contract version this extension supports. lstk +// advertises its version via LSTK_EXT_API_VERSION; the extension checks it itself. +const minAPIVersion = 1 + +// exitNotAuthorized is the status the stubbed self-authorization returns when no +// auth token was conveyed. +const exitNotAuthorized = 13 + +// emulator mirrors one entry of the LSTK_EXT_CONTEXT `emulators` array. +type emulator struct { + Type string `json:"type"` + Endpoint string `json:"endpoint"` + Port string `json:"port"` +} + +// extContext mirrors the LSTK_EXT_CONTEXT JSON object lstk conveys. +type extContext struct { + ConfigDir string `json:"configDir"` + AuthToken string `json:"authToken"` + NonInteractive bool `json:"nonInteractive"` + Emulators []emulator `json:"emulators"` +} + +func main() { + os.Exit(run(os.Args[1:])) +} + +func run(args []string) int { + if v, err := strconv.Atoi(os.Getenv("LSTK_EXT_API_VERSION")); err == nil && v < minAPIVersion { + fmt.Fprintf(os.Stderr, "lstk-ref: requires LSTK_EXT_API_VERSION >= %d, got %d\n", minAPIVersion, v) + return 1 + } + + ctx := decodeContext() + echo(args, ctx) + + if len(args) == 0 { + return 0 + } + switch args[0] { + case "exit": + if len(args) < 2 { + fmt.Fprintln(os.Stderr, "lstk-ref: exit requires a status code") + return 1 + } + code, err := strconv.Atoi(args[1]) + if err != nil { + fmt.Fprintf(os.Stderr, "lstk-ref: invalid exit code %q\n", args[1]) + return 1 + } + return code + case "auth": + if ctx.AuthToken == "" { + fmt.Fprintln(os.Stderr, "lstk-ref: not authorized (no auth token conveyed)") + return exitNotAuthorized + } + fmt.Println("lstk-ref: authorized") + return 0 + default: + return 0 + } +} + +// decodeContext decodes LSTK_EXT_CONTEXT, reporting a malformed payload but still +// returning the zero context so tests can observe the absence of fields. +func decodeContext() extContext { + var c extContext + if raw := os.Getenv("LSTK_EXT_CONTEXT"); raw != "" { + if err := json.Unmarshal([]byte(raw), &c); err != nil { + fmt.Fprintf(os.Stderr, "lstk-ref: invalid LSTK_EXT_CONTEXT: %v\n", err) + } + } + return c +} + +// echo prints the received args and decoded context in a stable, line-oriented +// form so integration tests can assert on individual fields. +func echo(args []string, c extContext) { + fmt.Printf("ARGS=%v\n", args) + if self, err := os.Executable(); err == nil { + fmt.Printf("SELF=%s\n", self) + } + fmt.Printf("API_VERSION=%s\n", os.Getenv("LSTK_EXT_API_VERSION")) + fmt.Printf("CONFIG_DIR=%s\n", c.ConfigDir) + if c.AuthToken != "" { + fmt.Printf("AUTH_TOKEN=%s\n", c.AuthToken) + } + fmt.Printf("NON_INTERACTIVE=%t\n", c.NonInteractive) + fmt.Printf("EMULATOR_COUNT=%d\n", len(c.Emulators)) + for _, e := range c.Emulators { + fmt.Printf("EMULATOR=%s %s %s\n", e.Type, e.Endpoint, e.Port) + } +} From cf3131587b37317b0849b0ba3a6f3061f3c43d72 Mon Sep 17 00:00:00 2001 From: Peter Smith Date: Wed, 1 Jul 2026 10:02:36 +1200 Subject: [PATCH 5/6] fix(extensions): record extension usage in product telemetry; route errors to stderr MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Telemetry (incorporates PR feedback): extension invocations now reach the analytics pipeline / data warehouse, not just the OTel tracing exporter. dispatchExtension emits an `lstk_command` event named `ext:` (duration + exit code) for a resolved extension, so the warehouse tracks which extension ran. instrumentCommands skips its generic emit for the extension-dispatch path so the run is no longer mislabeled as `start`, and an unresolved command records nothing. extension.Invoke keeps its OTel span for tracing. CI fixes (failures after merging main #346): - The unknown-command error now goes to stderr (matching Cobra's own output), so `lstk ` puts "unknown command" on stderr and exits 1 — fixes TestInvalidUsageExitsNonZero/unknown_command and the in-tree TestExtensionUnknownCommandNoExtensionErrors. - The self-authorization integration tests strip the inherited LOCALSTACK_AUTH_TOKEN (CI sets one) so the "without token" path truly conveys no token and the extension refuses (exit 13). - Add TestExtensionInvocationRecordedInTelemetry asserting the `ext:` event (and that it is not labeled `start`). Updates design Decision 8, the extension-runtime-context telemetry requirement, and task 3.6 to distinguish product telemetry (warehouse) from OTel tracing. Co-Authored-By: Claude Opus 4.8 (1M context) --- cmd/extension.go | 30 +++++++++++++--- cmd/root.go | 10 +++++- .../changes/add-extension-mechanism/design.md | 12 ++++--- .../specs/extension-runtime-context/spec.md | 18 +++++++--- .../changes/add-extension-mechanism/tasks.md | 2 +- test/integration/extension_test.go | 36 +++++++++++++++---- 6 files changed, 85 insertions(+), 23 deletions(-) diff --git a/cmd/extension.go b/cmd/extension.go index 00ca3131..385e8b99 100644 --- a/cmd/extension.go +++ b/cmd/extension.go @@ -5,7 +5,9 @@ import ( "errors" "fmt" "os" + "os/exec" "strings" + "time" "github.com/localstack/lstk/internal/config" "github.com/localstack/lstk/internal/container" @@ -15,6 +17,7 @@ import ( "github.com/localstack/lstk/internal/log" "github.com/localstack/lstk/internal/output" "github.com/localstack/lstk/internal/runtime" + "github.com/localstack/lstk/internal/telemetry" "github.com/spf13/cobra" ) @@ -24,16 +27,20 @@ import ( // remaining args verbatim and conveying runtime context via LSTK_EXT_*. Built-in // commands never reach here because Cobra routes them to their own command, so // they always take precedence. When no extension resolves, lstk emits its -// standard unknown-command error and returns a silent, non-zero error. -func dispatchExtension(ctx context.Context, cfg *env.Env, logger log.Logger, args []string) error { +// standard unknown-command error (to stderr, matching Cobra's own) and returns a +// silent, non-zero error. A resolved extension's invocation is recorded as a +// product-telemetry command event named "ext:" so the analytics pipeline +// can track which extension ran; this is separate from the OTel span emitted +// inside extension.Invoke (see internal/extension/exec.go). +func dispatchExtension(ctx context.Context, cfg *env.Env, tel *telemetry.Client, logger log.Logger, args []string) error { name, extArgs := args[0], args[1:] - sink := output.NewPlainSink(os.Stdout) resolver := extension.NewResolver(logger) ext, err := resolver.Resolve(name) if err != nil { if errors.Is(err, extension.ErrNotFound) { - sink.Emit(output.ErrorEvent{ + // Errors go to stderr, like Cobra's own unknown-command output. + output.NewPlainSink(os.Stderr).Emit(output.ErrorEvent{ Title: fmt.Sprintf("unknown command %q for lstk", name), Actions: []output.ErrorAction{{Label: "See help:", Value: "lstk -h"}}, }) @@ -56,7 +63,20 @@ func dispatchExtension(ctx context.Context, cfg *env.Env, logger log.Logger, arg } logger.Info("extension: dispatching %q (bundled=%v) at %s", name, ext.Bundled, ext.Path) - return extension.Invoke(ctx, ext, extArgs, runCtx) + start := time.Now() + runErr := extension.Invoke(ctx, ext, extArgs, runCtx) + + exitCode, errorMsg := 0, "" + if runErr != nil { + exitCode, errorMsg = 1, runErr.Error() + var exitErr *exec.ExitError + if errors.As(runErr, &exitErr) { + exitCode = exitErr.ExitCode() + } + } + tel.EmitCommand(ctx, "ext:"+name, nil, time.Since(start).Milliseconds(), exitCode, errorMsg) + + return runErr } // resolveEmulators best-effort discovers every running LocalStack emulator and diff --git a/cmd/root.go b/cmd/root.go index bbe6b8af..0c156803 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -58,7 +58,7 @@ func NewRootCmd(cfg *env.Env, tel *telemetry.Client, logger log.Logger) *cobra.C // command (Cobra would have routed those to their own command), so it // is an extension name; everything after it is forwarded verbatim. if len(args) > 0 { - return dispatchExtension(cmd.Context(), cfg, logger, args) + return dispatchExtension(cmd.Context(), cfg, tel, logger, args) } rt, err := runtime.NewDockerRuntime(cfg.DockerHost) if err != nil { @@ -327,6 +327,14 @@ func instrumentCommands(cmd *cobra.Command, tel *telemetry.Client) { startTime := time.Now() runErr := original(c, args) + // Extension dispatch (the root command invoked with a positional arg) + // records its own command event in dispatchExtension, which knows the + // resolved extension name; skip the generic emit here so the invocation + // is not mislabeled as "start". + if c == c.Root() && len(args) > 0 { + return runErr + } + var flags []string c.Flags().Visit(func(f *pflag.Flag) { flags = append(flags, "--"+f.Name) diff --git a/openspec/changes/add-extension-mechanism/design.md b/openspec/changes/add-extension-mechanism/design.md index 9985fb93..1f910a7a 100644 --- a/openspec/changes/add-extension-mechanism/design.md +++ b/openspec/changes/add-extension-mechanism/design.md @@ -111,13 +111,17 @@ The file is **hand-authored and owned by LocalStack's private extensions reposit **Alternatives considered**: A Go validator in-repo (`extension.ValidateDescriptions` reusing the runtime `scanDir` + go-toml, run via `go run ./...`) — rejected: it is the only thing that *cannot* drift from how lstk resolves extensions at runtime, but it costs an extra build entrypoint, runs against the repo's "domain logic in Go, helpers in shell" grain for what is a 20-line set-difference, and the drift it guards against is negligible here (validation reads only bare-identifier keys against a single clean dir). Acceptable to lose that guarantee while the descriptions file stays a flat name→string table; revisit if the file gains structure (per-extension metadata, min API version) that should share types with `LoadDescriptions`. Generating the file from a manifest in the open-source repo (the original design; rejected: the descriptions belong with the binaries in the private repo, and generation duplicated a list the private repo already owns — validation gives the same version-lock guarantee with one source of truth). A describe protocol that execs `lstk- lstk:describe` (rejected: turns inert help into code execution, needs timeout/cache/trust-boundary machinery, and a mistyped command rendering Cobra usage could fork extensions). Per-extension manifests (rejected with Decision 4). Name-only for everything (viable and simplest, but the user wants descriptions for bundled extensions like `lstk-deploy`, which this delivers with no exec). -### Decision 8: lstk records extension invocations as telemetry; trace propagation is deferred +### Decision 8: extension usage flows through the product-telemetry command event; trace propagation is deferred -When telemetry is enabled (lstk's existing `LSTK_OTEL` path), lstk SHALL record each extension invocation — the command name, duration, and exit code — through the same OpenTelemetry export the rest of lstk uses. This is **lstk-side only**: lstk does **not** inject trace context (W3C `traceparent`/`tracestate`) into the extension process, so an extension's own spans do not yet nest under lstk's trace. +lstk has **two independent instrumentation systems**, and extension usage must go through the right one. The *product telemetry* (`telemetry.Client.EmitCommand` → the `lstk_command` event → the analytics endpoint → the data warehouse) is what answers "which extension was invoked"; the *OpenTelemetry tracing* (`LSTK_OTEL` → an OTLP backend) is dev/observability only and never reaches the warehouse. The decision is: -**Rationale** (PR review): visibility into which extensions run, how often, and whether they succeed is valuable and cheap — lstk already has an OTEL pipeline, so the invocation is just one more span/event at the dispatch boundary, with no contract surface and no dependency on the extension being instrumented. Full distributed tracing *through* the extension is a larger, optional concern: it only helps extensions that are themselves OTEL-instrumented, and it is purely additive later (inject `traceparent`, or add a `trace` field to `LSTK_EXT_CONTEXT`, under a version bump) — so it is deferred rather than designed now. Recording respects the same opt-in/opt-out as all lstk telemetry; nothing is emitted when telemetry is disabled. +- On a **resolved** extension invocation, `dispatchExtension` emits a product-telemetry command event named **`ext:`** carrying the duration and exit code — so the warehouse records which extension ran and whether it succeeded, alongside built-in `lstk_command` events. This respects the same opt-out as all lstk telemetry (nothing is sent when disabled / to the unreachable test endpoint). +- The generic root-command instrumentation (`instrumentCommands`) **skips its emit for the extension-dispatch path**, because the root command's hardcoded name would otherwise mislabel every extension run as `start`. An **unresolved** command (a typo) emits nothing — matching the pre-change behavior where Cobra never reached `RunE` for an unknown command. +- `extension.Invoke` additionally opens an **OTel span** (extension name, bundled, exit code) for tracing. lstk does **not** inject trace context (W3C `traceparent`/`tracestate`) into the extension, so an extension's own spans do not yet nest under lstk's trace. -**Alternatives considered**: Inject standard `TRACEPARENT`/`TRACESTATE` so any instrumented extension auto-continues the trace (deferred, not rejected — clean and standard, but only useful once we have instrumented extensions, and addable without breaking the contract). An `LSTK_EXT_`-prefixed trace variable (rejected for now: non-standard, and the standard names are the natural choice when we do add propagation). No telemetry at all (rejected: the reviewer specifically wants extension observability). +**Rationale** (PR review): the reviewer asked for "telemetry **and** tracing of extensions." Telemetry-to-warehouse is the part with product value (which extensions run, how often, success rate), and it lives in the `lstk_command` pipeline, not the OTel exporter — wiring it into the OTel span only would have left the warehouse blind. Tracing is the OTel span, kept lstk-side. Full distributed tracing *through* the extension only helps OTel-instrumented extensions and is purely additive later (inject `traceparent`, or add a `trace` field to `LSTK_EXT_CONTEXT`, under a version bump), so it is deferred. + +**Alternatives considered**: Emitting `ext:` from the generic `instrumentCommands` wrapper rather than `dispatchExtension` (rejected: the wrapper can't tell a resolved extension from a typo and would record `ext:` for unknown commands; emitting from `dispatchExtension` after a successful resolve avoids that). A dedicated event type instead of reusing `lstk_command` (rejected: a new warehouse schema for no added value; `ext:`-prefixed command names bucket cleanly and a follow-up can split bundled vs third-party if needed). Recording only the OTel span (rejected: that was the original mistake — it never reaches the warehouse). Injecting `TRACEPARENT`/`TRACESTATE` now (deferred — useful only once extensions are instrumented; additive later). ### Decision 9: No shared TUI/library between lstk and extensions diff --git a/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md b/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md index ccc74992..fce7cf7d 100644 --- a/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md +++ b/openspec/changes/add-extension-mechanism/specs/extension-runtime-context/spec.md @@ -108,14 +108,22 @@ lstk SHALL pass the user's existing environment through to the extension and onl - **THEN** the extension's environment still contains `HTTP_PROXY` - **AND** also contains the `LSTK_EXT_API_VERSION` and `LSTK_EXT_CONTEXT` variables -### Requirement: Extension invocations are recorded as telemetry +### Requirement: Extension invocations are recorded in product telemetry -When lstk telemetry is enabled (the existing `LSTK_OTEL` path), lstk SHALL record each extension invocation — at least the extension command name, the duration, and the exit code — through the same OpenTelemetry export used by the rest of lstk. Recording SHALL respect the same opt-in/opt-out as all lstk telemetry: when telemetry is disabled, lstk SHALL emit nothing for the invocation. lstk SHALL NOT inject trace context into the extension process in this change (an extension's own spans nesting under lstk's trace is deferred and would be an additive change). +lstk SHALL record each invocation of a **resolved** extension as a product-telemetry command event (the same `lstk_command` event stream that built-in commands use, which flows to lstk's analytics pipeline), identifying the extension — for example a command name of the form `ext:` — and carrying the invocation's duration and exit code. This is distinct from the OpenTelemetry tracing span lstk may also open for the invocation; the product-telemetry event is the one that reaches the data warehouse. Recording SHALL respect the same opt-out as all lstk telemetry: when telemetry is disabled, lstk SHALL emit nothing. -#### Scenario: Invocation recorded when telemetry enabled +lstk SHALL NOT mislabel an extension invocation as a different command (e.g. the bare-root `start` event), and SHALL NOT record an unresolved command (one with no matching extension) as an extension invocation. lstk SHALL NOT inject trace context into the extension process in this change (an extension's own spans nesting under lstk's trace is deferred and would be an additive change). -- **WHEN** telemetry is enabled and lstk dispatches to an extension that exits with a status code -- **THEN** lstk records the extension's command name, duration, and exit code via its telemetry export +#### Scenario: Resolved extension invocation is recorded with its identity + +- **WHEN** telemetry is enabled and lstk dispatches to a resolved extension `deploy` +- **THEN** lstk records a command event identifying the extension (e.g. `ext:deploy`) with the invocation's duration and exit code +- **AND** the event is not labeled as the `start` command + +#### Scenario: Unknown command is not recorded as an extension + +- **WHEN** lstk is invoked with a name that resolves to no extension +- **THEN** lstk records no extension command event for it #### Scenario: Nothing recorded when telemetry disabled diff --git a/openspec/changes/add-extension-mechanism/tasks.md b/openspec/changes/add-extension-mechanism/tasks.md index 51727197..a80e0aa4 100644 --- a/openspec/changes/add-extension-mechanism/tasks.md +++ b/openspec/changes/add-extension-mechanism/tasks.md @@ -20,7 +20,7 @@ - [x] 3.3 Include `authToken` in the object only when a token is resolved (omit the field otherwise); always include `configDir`; no entitlement/grant fields - [x] 3.4 Set the `nonInteractive` field from lstk's resolved interactivity (the `isInteractive` condition: `--non-interactive` given or stdout not a TTY); document that future global flags are conveyed as additive JSON fields - [x] 3.5 Unit tests asserting the JSON shape across scenarios: zero/one/multiple emulators, authed vs not (field present/absent), non-interactive flag vs non-TTY, host env inherited, stray `LSTK_EXT_*` stripped -- [x] 3.6 Record each extension invocation (command name, duration, exit code) via the existing OTEL/telemetry path when telemetry is enabled; emit nothing when disabled; do **not** inject trace context into the child (Decision 8); unit/integration coverage for enabled vs disabled +- [x] 3.6 Record each **resolved** extension invocation in product telemetry: `dispatchExtension` emits an `lstk_command` event named `ext:` (duration + exit code) via `telemetry.Client`, so the warehouse tracks which extension ran; `instrumentCommands` skips its generic emit for the extension-dispatch path so it is not mislabeled `start`, and an unresolved command records nothing. `extension.Invoke` additionally opens an OTel tracing span (name, bundled, exit code); no trace-context injection into the child (Decision 8) ## 4. Invocation (exec) path diff --git a/test/integration/extension_test.go b/test/integration/extension_test.go index b7d212d9..0dd5db84 100644 --- a/test/integration/extension_test.go +++ b/test/integration/extension_test.go @@ -150,11 +150,11 @@ func TestExtensionUnknownCommandDispatches(t *testing.T) { func TestExtensionUnknownCommandNoExtensionErrors(t *testing.T) { t.Parallel() tmpHome := t.TempDir() - // Empty extension dir so `nope` resolves nowhere. The error event renders - // through the plain sink (stdout), consistent with other lstk commands. - stdout, _, err := runLstk(t, testContext(t), t.TempDir(), envWithPath(tmpHome, t.TempDir()), "nope") + // Empty extension dir so `nope` resolves nowhere. The unknown-command error + // goes to stderr, matching Cobra's own unknown-command output. + _, stderr, err := runLstk(t, testContext(t), t.TempDir(), envWithPath(tmpHome, t.TempDir()), "nope") requireExitCode(t, 1, err) - require.Contains(t, stdout, "unknown command") + require.Contains(t, stderr, "unknown command") } func TestExtensionExitCodePropagates(t *testing.T) { @@ -217,14 +217,34 @@ func TestExtensionSelfAuthorizationRefusesWithoutToken(t *testing.T) { installExtension(t, extDir, "deploy") tmpHome := t.TempDir() - environ := envWithPath(tmpHome, extDir) + // Strip any inherited LOCALSTACK_AUTH_TOKEN (CI sets one) so no token is + // conveyed; the extension's stubbed self-authorization must then refuse. + environ := env.Environ(envWithPath(tmpHome, extDir)).Without(env.AuthToken) environ = append(environ, "DOCKER_HOST=tcp://127.0.0.1:1") - // No auth token: the extension's stubbed self-authorization refuses (exit 13). _, stderr, err := runLstk(t, testContext(t), t.TempDir(), environ, "deploy", "auth") requireExitCode(t, 13, err) require.Contains(t, stderr, "not authorized") } +func TestExtensionInvocationRecordedInTelemetry(t *testing.T) { + t.Parallel() + extDir := t.TempDir() + installExtension(t, extDir, "hello") + + analyticsSrv, events := mockAnalyticsServer(t) + + tmpHome := t.TempDir() + environ := env.Environ(envWithPath(tmpHome, extDir)). + With(env.AnalyticsEndpoint, analyticsSrv.URL) + + _, stderr, err := runLstk(t, testContext(t), t.TempDir(), environ, "hello", "world") + require.NoError(t, err, stderr) + + // The invocation is recorded in product telemetry as ext:, so the + // warehouse tracks which extension ran — and is NOT mislabeled as "start". + assertCommandTelemetry(t, events, "ext:hello", 0) +} + func TestExtensionEndpointConveyedWhenEmulatorRunning(t *testing.T) { requireDocker(t) cleanup() @@ -352,7 +372,9 @@ func TestExtensionBundledPremiumSelfAuthorizes(t *testing.T) { installExtension(t, bundleDir, "deploy") // bundled "premium" extension tmpHome := t.TempDir() - noRuntime := append(envWithPath(tmpHome, t.TempDir()), "DOCKER_HOST=tcp://127.0.0.1:1") + // Strip any inherited LOCALSTACK_AUTH_TOKEN (CI sets one) so the unentitled + // case truly has no token to convey. + noRuntime := append(env.Environ(envWithPath(tmpHome, t.TempDir())).Without(env.AuthToken), "DOCKER_HOST=tcp://127.0.0.1:1") // Unentitled (no token): lstk still dispatches to the bundled extension, which // performs its own authorization and refuses. From 6b6b26245e48d002cbd13b3bb8293540898802a0 Mon Sep 17 00:00:00 2001 From: Peter Date: Thu, 2 Jul 2026 04:45:31 +1200 Subject: [PATCH 6/6] docs(readme): add Extensions section Briefly introduce the Git-style extension mechanism and point readers to the extensions-authoring.md guide for details. --- README.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/README.md b/README.md index 711e22f0..95bac133 100644 --- a/README.md +++ b/README.md @@ -423,6 +423,18 @@ lstk start --snapshot pod:other-baseline # load a different snapshot this run lstk start --no-snapshot # skip auto-loading this run ``` +## Extensions + +lstk supports Git-style extensions: running `lstk `, for a name that isn't a built-in command, delegates to an external `lstk-` executable found on your `PATH`, forwarding all arguments and passing stdin/stdout/stderr through. + +```bash +lstk my-tool --flag # resolves and runs lstk-my-tool, if it exists +``` + +Extensions receive context about the current lstk setup (config dir, auth token, running emulators) via environment variables, so they can integrate without reimplementing discovery. + +See [docs/extensions-authoring.md](docs/extensions-authoring.md) for the extension contract and how to author your own. + ## Reporting bugs Feedback is welcome! Use the repository issue tracker for bug reports or feature requests.