Add lstk extension mechanism#340
Conversation
OpenSpec change proposing a Git-style extension mechanism for lstk: when `lstk <name>` is not a built-in, lstk resolves and execs an `lstk-<name>` binary, forwarding args/streams/exit code and passing runtime context via a versioned `LSTK_EXT_*` environment contract. Scope (4 capabilities): - extension-framework: bundled-dir + PATH resolution (bundled wins), built-in precedence, leading-only global-flag handling, side-effect-free help listing. - extension-runtime-context: emulator endpoint/type/port, config dir, auth token, and resolved global flags (e.g. LSTK_EXT_NON_INTERACTIVE). - extension-entitlement: lstk passes the token; the extension self-authorizes. The lstk-side signed-grant mechanism is explicitly deferred. - extension-bundling: ship LocalStack-built (closed-source) extensions like lstk-deploy by default, with a release-generated descriptions file for help text and atomic version-matched updates. Deferred: emulator attestation, signed entitlement grants, and user-facing `lstk extension` management commands.
|
This overall looks good to me! Here are a few things that are not addressed as far as I could see, which I think deserve decisions before starting with implementation:
|
I'd also like to see answers to questions 3 and 4 |
|
Thanks for the discussion :-)
Great idea, I'll add that.
I've gone with the no-sharing approach. I did think about creating a shared library, but that makes building an extension more complicated and more dependent on Golang APIs (which can change without much control). Instead, I went with a simple environment variable-based contract that's less likely to change (and we have a version number if we need to change it). If a third-party wants to build an extension, they'll need to incorporate their own libraries for doing whatever they want their extension to do (including TUI).
I haven't worried about this, because the only way to inject a malicious extension is to install it onto your local machine, which should be secured in other ways. If I was a hacker, I'd target commands like I did notice that some tools have an
Good idea... changing to a JSON format and using arrays/objects will solve that. We'd definitely want extensions to be aware that multiple emulators might be running, and to not require modification in future. |
…ed-extension distribution Incorporate PR #340 review feedback into the OpenSpec artifacts and phase the launch: - Runtime context becomes a versioned JSON object (LSTK_EXT_API_VERSION + LSTK_EXT_CONTEXT) carrying an emulators array, so multiple simultaneously running emulators are representable from day one. - Record extension invocations as lstk-side telemetry; defer trace-context propagation, a shared SDK, and an extension allow-list. - Scope the first release to running extensions + bundled-dir resolution (validated by manual placement); defer automated distribution and atomic co-update to the new add-bundled-extension-distribution change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nership Resolve two concerns in the JSON runtime-context contract: - Versioning vs feature detection: LSTK_EXT_API_VERSION is bumped only on a breaking change (a field removed or repurposed), never on additions. An extension therefore confirms a field by its presence in LSTK_EXT_CONTEXT, not by the version number, and any field added after v1 must be distinguishable when absent. The author guide now shows presence checks instead of a version floor check, and the version is used only to refuse a contract generation the extension predates. - Namespace ownership: lstk strips inherited LSTK_EXT_* before adding its own, reframed as an ownership invariant (Go's exec dedup already overrides the two vars lstk sets, so the strip exists to keep the LSTK_EXT_ namespace fully owned by this invocation). Updates design.md, the extension-runtime-context spec, and the author guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dispatch `lstk <name>` to an external `lstk-<name>` executable when `<name>` is not a built-in, resolved from the bundled directory next to the binary and then PATH (built-ins always win; bundled wins over PATH). Arguments after the name, stdin/stdout/stderr, and the exit code are passed through unchanged. Runtime context is conveyed in two environment variables: LSTK_EXT_API_VERSION (a flat integer) and LSTK_EXT_CONTEXT (a JSON object with the config dir, auth token, non-interactive state, and an array of every running emulator). Each invocation is wrapped in an OpenTelemetry span (name, bundled, exit code) so usage is recorded when LSTK_OTEL is enabled and costs nothing when it is not. Authorization stays with the extension: lstk conveys the token and makes no entitlement decision. Help lists resolvable extensions side-effect-free, with one-line descriptions for bundled ones read from a descriptions file when present, aligned with the built-in command columns. Scope: this first release runs extensions (including a bundled one placed manually); automated bundled-extension distribution and atomic co-update are deferred to the add-bundled-extension-distribution change. Domain logic in internal/extension/; dispatch, help, and context wiring in cmd/extension.go (hooked from cmd/root.go). Covered by unit tests and integration tests using an in-tree reference extension. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…rrors to stderr Telemetry (incorporates PR feedback): extension invocations now reach the analytics pipeline / data warehouse, not just the OTel tracing exporter. dispatchExtension emits an `lstk_command` event named `ext:<name>` (duration + exit code) for a resolved extension, so the warehouse tracks which extension ran. instrumentCommands skips its generic emit for the extension-dispatch path so the run is no longer mislabeled as `start`, and an unresolved command records nothing. extension.Invoke keeps its OTel span for tracing. CI fixes (failures after merging main #346): - The unknown-command error now goes to stderr (matching Cobra's own output), so `lstk <unknown>` puts "unknown command" on stderr and exits 1 — fixes TestInvalidUsageExitsNonZero/unknown_command and the in-tree TestExtensionUnknownCommandNoExtensionErrors. - The self-authorization integration tests strip the inherited LOCALSTACK_AUTH_TOKEN (CI sets one) so the "without token" path truly conveys no token and the extension refuses (exit 13). - Add TestExtensionInvocationRecordedInTelemetry asserting the `ext:<name>` event (and that it is not labeled `start`). Updates design Decision 8, the extension-runtime-context telemetry requirement, and task 3.6 to distinguish product telemetry (warehouse) from OTel tracing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
anisaoshafi
left a comment
There was a problem hiding this comment.
LGTM 👍🏼
Could you add a small section in the README explaining how to use extension feature?
What this is
lstk gains Git-style extensions: when you run
lstk <name>and<name>isn't a built-in command, lstk finds and runs an executable calledlstk-<name>, forwarding your arguments and the child's stdin/stdout/stderr and exit code. Same model asgit-<name>. Extensions can be written in any language, open or closed source, with no manifest and no registration step.This PR contains the implementation plus the proposal/specs (the
openspec/change artifacts) and the author-facing guide. Readdocs/extensions-authoring.mdfor user-facing details on how extensions work.How dispatch works
start,snapshot, etc.lstkbinary →PATH. A bundled extension wins over a same-named one onPATH.lstk --non-interactive deploy --fooruns the extension with just--foo; everything after<name>is the extension's and is forwarded verbatim.What lstk passes to an extension
So extensions don't re-implement endpoint discovery or read lstk's config, lstk hands them context through two environment variables:
LSTK_EXT_API_VERSION— an integer contract version.LSTK_EXT_CONTEXT— a JSON object: the config dir, the auth token (when logged in), whether the session is non-interactive, and an array of every running emulator (type,endpoint,port).Emulators is an array because lstk can run AWS, Snowflake, and Azure emulators at once. New fields may be added without bumping the version — an extension confirms a field by its presence in the JSON; the version bumps only on a breaking change.
Authorization
lstk passes the user's auth token and makes no entitlement decision. An extension that needs to gate access (e.g. a paid feature) authorizes the user itself, server-side, with that token. This is deliberate: lstk is open source and rebuildable, so any client-side check is a speed bump, not a control.
Telemetry
When telemetry is enabled (
LSTK_OTEL), lstk records each extension invocation (name, duration, exit code) as an OpenTelemetry span. Nothing is emitted when telemetry is off.Scope of the first release
The first release runs extensions and passes context to them — including a bundled extension placed manually next to
lstk(that is, in the same directory), so LocalStack's own bundled extensions can be validated by hand.This code does not download or auto-update bundled extensions. Automated packaging into the install artifacts (binary archive / Homebrew / npm) and atomic version-matched updates are deferred to a follow-up change,
add-bundled-extension-distribution(its artifacts are also in this PR'sopenspec/tree).What's in the PR
internal/extension/— domain logic: resolution (bundled dir + PATH), the JSON runtime context, exec/telemetry.cmd/extension.go(hooked fromcmd/root.go) — unknown-command dispatch, help listing, and context wiring;cmd/help.goaligns the Extensions help section with the built-in command columns.test/integration/— integration tests driving a reallstkbinary against an in-tree reference extension (test-samples/extensions/lstk-ref).openspec/changes/add-extension-mechanism/— proposal, design, specs (framework, runtime-context, entitlement, bundling), and tasks.openspec/changes/add-bundled-extension-distribution/— the deferred distribution + auto-update change.docs/extensions-authoring.md— the author-facing guide to the contract.Review feedback incorporated
lstk-<name>like Git; trust is the local machine's, and bundled extensions ship inside thelstkartifact rather than as separate downloads.