Skip to content

Add lstk extension mechanism#340

Open
peter-smith-phd wants to merge 6 commits into
mainfrom
dpx-517-lstk-extension-mechanism
Open

Add lstk extension mechanism#340
peter-smith-phd wants to merge 6 commits into
mainfrom
dpx-517-lstk-extension-mechanism

Conversation

@peter-smith-phd

@peter-smith-phd peter-smith-phd commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

What this is

lstk gains Git-style extensions: when you run lstk <name> and <name> isn't a built-in command, lstk finds and runs an executable called lstk-<name>, forwarding your arguments and the child's stdin/stdout/stderr and exit code. Same model as git-<name>. Extensions can be written in any language, open or closed source, with no manifest and no registration step.

This PR contains the implementation plus the proposal/specs (the openspec/ change artifacts) and the author-facing guide. Read docs/extensions-authoring.md for user-facing details on how extensions work.

How dispatch works

  • Built-ins always win. Dispatch happens only on Cobra's unknown-command path, so an extension can never shadow start, snapshot, etc.
  • Resolution order: built-ins → a bundled directory next to the lstk binary → PATH. A bundled extension wins over a same-named one on PATH.
  • Global flags are consumed before the name. lstk --non-interactive deploy --foo runs the extension with just --foo; everything after <name> is the extension's and is forwarded verbatim.

What lstk passes to an extension

So extensions don't re-implement endpoint discovery or read lstk's config, lstk hands them context through two environment variables:

  • LSTK_EXT_API_VERSION — an integer contract version.
  • LSTK_EXT_CONTEXT — a JSON object: the config dir, the auth token (when logged in), whether the session is non-interactive, and an array of every running emulator (type, endpoint, port).

Emulators is an array because lstk can run AWS, Snowflake, and Azure emulators at once. New fields may be added without bumping the version — an extension confirms a field by its presence in the JSON; the version bumps only on a breaking change.

Authorization

lstk passes the user's auth token and makes no entitlement decision. An extension that needs to gate access (e.g. a paid feature) authorizes the user itself, server-side, with that token. This is deliberate: lstk is open source and rebuildable, so any client-side check is a speed bump, not a control.

Telemetry

When telemetry is enabled (LSTK_OTEL), lstk records each extension invocation (name, duration, exit code) as an OpenTelemetry span. Nothing is emitted when telemetry is off.

Scope of the first release

The first release runs extensions and passes context to them — including a bundled extension placed manually next to lstk (that is, in the same directory), so LocalStack's own bundled extensions can be validated by hand.

This code does not download or auto-update bundled extensions. Automated packaging into the install artifacts (binary archive / Homebrew / npm) and atomic version-matched updates are deferred to a follow-up change, add-bundled-extension-distribution (its artifacts are also in this PR's openspec/ tree).

What's in the PR

  • internal/extension/ — domain logic: resolution (bundled dir + PATH), the JSON runtime context, exec/telemetry.
  • cmd/extension.go (hooked from cmd/root.go) — unknown-command dispatch, help listing, and context wiring; cmd/help.go aligns the Extensions help section with the built-in command columns.
  • test/integration/ — integration tests driving a real lstk binary against an in-tree reference extension (test-samples/extensions/lstk-ref).
  • openspec/changes/add-extension-mechanism/ — proposal, design, specs (framework, runtime-context, entitlement, bundling), and tasks.
  • openspec/changes/add-bundled-extension-distribution/ — the deferred distribution + auto-update change.
  • docs/extensions-authoring.md — the author-facing guide to the contract.

Review feedback incorporated

  • Multi-emulator support → the context carries a JSON array of emulators from day one.
  • Telemetry of extension invocations.
  • TUI: no shared library — extensions couple to lstk only through the env contract and bring their own libraries.
  • Malicious extensions: no allow-list — lstk runs any resolvable lstk-<name> like Git; trust is the local machine's, and bundled extensions ship inside the lstk artifact rather than as separate downloads.

OpenSpec change proposing a Git-style extension mechanism for lstk: when
`lstk <name>` is not a built-in, lstk resolves and execs an `lstk-<name>`
binary, forwarding args/streams/exit code and passing runtime context via a
versioned `LSTK_EXT_*` environment contract.

Scope (4 capabilities):
- extension-framework: bundled-dir + PATH resolution (bundled wins), built-in
  precedence, leading-only global-flag handling, side-effect-free help listing.
- extension-runtime-context: emulator endpoint/type/port, config dir, auth
  token, and resolved global flags (e.g. LSTK_EXT_NON_INTERACTIVE).
- extension-entitlement: lstk passes the token; the extension self-authorizes.
  The lstk-side signed-grant mechanism is explicitly deferred.
- extension-bundling: ship LocalStack-built (closed-source) extensions like
  lstk-deploy by default, with a release-generated descriptions file for help
  text and atomic version-matched updates.

Deferred: emulator attestation, signed entitlement grants, and user-facing
`lstk extension` management commands.
@peter-smith-phd peter-smith-phd added docs: needed Pull request requires documentation updates semver: minor labels Jun 25, 2026
@peter-smith-phd peter-smith-phd marked this pull request as ready for review June 28, 2026 17:45
@peter-smith-phd peter-smith-phd requested a review from a team as a code owner June 28, 2026 17:45
@carole-lavillonniere

Copy link
Copy Markdown
Collaborator

This overall looks good to me! Here are a few things that are not addressed as far as I could see, which I think deserve decisions before starting with implementation:

  • Telemetry and tracing of extensions
  • TUI: the proposal does not say if we share any components with main binary or re-implement?
  • Possibility of malicious extensions: do we want to implement something such as a whitelist of extensions to prevent that?
  • Multi-emulator support: passing the endpoint as a simple env var is conflicting with that. Can we go with the json format from day 1?

@mmaureenliu

mmaureenliu commented Jun 30, 2026

Copy link
Copy Markdown

This overall looks good to me! Here are a few things that are not addressed as far as I could see, which I think deserve decisions before starting with implementation:

  • Telemetry and tracing of extensions
  • TUI: the proposal does not say if we share any components with main binary or re-implement?
  • Possibility of malicious extensions: do we want to implement something such as a whitelist of extensions to prevent that?
  • Multi-emulator support: passing the endpoint as a simple env var is conflicting with that. Can we go with the json format from day 1?

I'd also like to see answers to questions 3 and 4

@peter-smith-phd

Copy link
Copy Markdown
Contributor Author

Thanks for the discussion :-)

  • Telemetry and tracing of extensions

Great idea, I'll add that.

  • TUI: the proposal does not say if we share any components with main binary or re-implement?

I've gone with the no-sharing approach. I did think about creating a shared library, but that makes building an extension more complicated and more dependent on Golang APIs (which can change without much control). Instead, I went with a simple environment variable-based contract that's less likely to change (and we have a version number if we need to change it). If a third-party wants to build an extension, they'll need to incorporate their own libraries for doing whatever they want their extension to do (including TUI).

  • Possibility of malicious extensions: do we want to implement something such as a whitelist of extensions to prevent that?

I haven't worried about this, because the only way to inject a malicious extension is to install it onto your local machine, which should be secured in other ways. If I was a hacker, I'd target commands like ls or cat, rather than lstk-xxx. I know that some extension mechanisms can download arbitrary extensions directly from the internet, but that seems overkill for us. Note that we will have "bundle extensions" (e.g. lstk-deploy and lstk-doctor), but those will be bundled with the main lstk binary, rather than separate downloads.

I did notice that some tools have an extension add command, but that seems overkill.

  • Multi-emulator support: passing the endpoint as a simple env var is conflicting with that. Can we go with the json format from day 1?

Good idea... changing to a JSON format and using arrays/objects will solve that. We'd definitely want extensions to be aware that multiple emulators might be running, and to not require modification in future.

peter-smith-phd and others added 2 commits July 1, 2026 07:11
…ed-extension distribution

Incorporate PR #340 review feedback into the OpenSpec artifacts and phase the
launch:

- Runtime context becomes a versioned JSON object (LSTK_EXT_API_VERSION +
  LSTK_EXT_CONTEXT) carrying an emulators array, so multiple simultaneously
  running emulators are representable from day one.
- Record extension invocations as lstk-side telemetry; defer trace-context
  propagation, a shared SDK, and an extension allow-list.
- Scope the first release to running extensions + bundled-dir resolution
  (validated by manual placement); defer automated distribution and atomic
  co-update to the new add-bundled-extension-distribution change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nership

Resolve two concerns in the JSON runtime-context contract:

- Versioning vs feature detection: LSTK_EXT_API_VERSION is bumped only on a
  breaking change (a field removed or repurposed), never on additions. An
  extension therefore confirms a field by its presence in LSTK_EXT_CONTEXT, not
  by the version number, and any field added after v1 must be distinguishable
  when absent. The author guide now shows presence checks instead of a version
  floor check, and the version is used only to refuse a contract generation the
  extension predates.
- Namespace ownership: lstk strips inherited LSTK_EXT_* before adding its own,
  reframed as an ownership invariant (Go's exec dedup already overrides the two
  vars lstk sets, so the strip exists to keep the LSTK_EXT_ namespace fully
  owned by this invocation).

Updates design.md, the extension-runtime-context spec, and the author guide.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@peter-smith-phd peter-smith-phd changed the title Add scoped extension-mechanism proposal Add lstk extension mechanism (proposal + specs) Jun 30, 2026
Dispatch `lstk <name>` to an external `lstk-<name>` executable when `<name>` is
not a built-in, resolved from the bundled directory next to the binary and then
PATH (built-ins always win; bundled wins over PATH). Arguments after the name,
stdin/stdout/stderr, and the exit code are passed through unchanged.

Runtime context is conveyed in two environment variables: LSTK_EXT_API_VERSION
(a flat integer) and LSTK_EXT_CONTEXT (a JSON object with the config dir, auth
token, non-interactive state, and an array of every running emulator). Each
invocation is wrapped in an OpenTelemetry span (name, bundled, exit code) so
usage is recorded when LSTK_OTEL is enabled and costs nothing when it is not.

Authorization stays with the extension: lstk conveys the token and makes no
entitlement decision. Help lists resolvable extensions side-effect-free, with
one-line descriptions for bundled ones read from a descriptions file when
present, aligned with the built-in command columns.

Scope: this first release runs extensions (including a bundled one placed
manually); automated bundled-extension distribution and atomic co-update are
deferred to the add-bundled-extension-distribution change.

Domain logic in internal/extension/; dispatch, help, and context wiring in
cmd/extension.go (hooked from cmd/root.go). Covered by unit tests and
integration tests using an in-tree reference extension.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@peter-smith-phd peter-smith-phd changed the title Add lstk extension mechanism (proposal + specs) Add lstk extension mechanism Jun 30, 2026
peter-smith-phd and others added 2 commits July 1, 2026 09:54
…rrors to stderr

Telemetry (incorporates PR feedback): extension invocations now reach the
analytics pipeline / data warehouse, not just the OTel tracing exporter.
dispatchExtension emits an `lstk_command` event named `ext:<name>` (duration +
exit code) for a resolved extension, so the warehouse tracks which extension
ran. instrumentCommands skips its generic emit for the extension-dispatch path
so the run is no longer mislabeled as `start`, and an unresolved command records
nothing. extension.Invoke keeps its OTel span for tracing.

CI fixes (failures after merging main #346):
- The unknown-command error now goes to stderr (matching Cobra's own output),
  so `lstk <unknown>` puts "unknown command" on stderr and exits 1 — fixes
  TestInvalidUsageExitsNonZero/unknown_command and the in-tree
  TestExtensionUnknownCommandNoExtensionErrors.
- The self-authorization integration tests strip the inherited
  LOCALSTACK_AUTH_TOKEN (CI sets one) so the "without token" path truly conveys
  no token and the extension refuses (exit 13).
- Add TestExtensionInvocationRecordedInTelemetry asserting the `ext:<name>`
  event (and that it is not labeled `start`).

Updates design Decision 8, the extension-runtime-context telemetry requirement,
and task 3.6 to distinguish product telemetry (warehouse) from OTel tracing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@anisaoshafi anisaoshafi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍🏼
Could you add a small section in the README explaining how to use extension feature?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs: needed Pull request requires documentation updates semver: minor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants