Skip to content

feat(registry): tag-based versioning with # ref syntax and CDN-bypass#151

Open
jrob5756 wants to merge 1 commit intomainfrom
feat/registry-tag-versioning
Open

feat(registry): tag-based versioning with # ref syntax and CDN-bypass#151
jrob5756 wants to merge 1 commit intomainfrom
feat/registry-tag-versioning

Conversation

@jrob5756
Copy link
Copy Markdown
Collaborator

@jrob5756 jrob5756 commented May 5, 2026

Summary

Reworks the workflow registry to behave more like a plugin system — versions are auto-discovered from git tags instead of being explicitly listed, refs can pin to any tag/branch/SHA, and stale CDN content is no longer a problem.

Closes the two pain points raised in discussion:

  1. `conductor registry update` doesn't reflect changes immediately — root cause was Fastly's CDN cache on `raw.githubusercontent.com`. Index fetches now resolve the ref to a commit SHA via the GitHub API and fetch from `…//index.yaml`. Unique URL per commit → CDN can't serve stale data. No `--force` flag needed.

  2. Versioning was awkward — required maintainers to enumerate every release in `index.yaml`. Now versions are auto-discovered from the registry repo's git tags, semver-sorted.

New ref grammar

```
[@][#]
```

Example Resolves to
`qa-bot` default registry, latest tag (or default branch HEAD if no tags)
`qa-bot#v1.2.3` default registry, ref `v1.2.3`
`qa-bot#main` default registry, branch `main`
`qa-bot@team` registry `team`, latest
`qa-bot@team#v1.2.3` registry `team`, tag `v1.2.3`
`qa-bot@#abc123` default registry, commit `abc123`

Hard errors: empty ref after `#`, multiple `#` or `@`, empty workflow name, path registry with a ref.

Behavior changes

  • `versions: [...]` field dropped from `index.yaml` schema. Existing indexes with this field still parse (Pydantic ignores it) — no migration required.
  • `latest` = newest semver-sorted tag (with leading `v` stripped for parsing). Falls back to default branch HEAD if no tags exist.
  • Cache key is commit SHA (`//<sha[:12]>/`). Mutable branch refs re-resolve fresh on every fetch; cache hit only when SHAs match.
  • Atomic cache writes: fetch into `tempfile.mkdtemp`, then `os.replace` into the final dir. No partial-state visibility for concurrent runs.
  • Index and workflow always come from the same commit (`load_index(entry, ref=sha)`) — fixes a subtle bug where `latest` could resolve to a tag that doesn't contain a workflow listed in the default-branch index.
  • `registry list` / `registry show` drop the per-workflow "Versions" column and add a "Latest tags:" footer (up to 5 newest, with `, ...` if more) for github registries.
  • `registry update` help text updated to explain the SHA-based CDN bypass.

Implementation

Module Change
`registry/github.py` Added `get_default_branch`, `resolve_ref_to_sha`; `list_tags` now paginates (cap 1000)
`registry/version_resolver.py` New module: `resolve_ref` (latest tag → default branch fallback) and `materialize_to_sha`
`registry/index.py` `load_index(entry, ref=None)` accepts an optional ref; github path always resolves to SHA before fetching
`registry/resolver.py` New `#`-based parser; `ResolvedRef.version` → `ResolvedRef.ref`
`registry/cache.py` Ref-aware, SHA-keyed, atomic fetch_workflow
`cli/registry.py` Drop Versions column, add Latest tags footer
`cli/app.py` Update 4 callers from `.version` → `.ref`

Tests

  • 153 registry tests pass (including 14 new tests in `test_version_resolver.py`).
  • Full suite: 2324 pass, 9 skipped, 0 regressions.
  • New coverage: tag pagination, semver sort with prereleases, mixed parseable/non-parseable tags, default-branch fallback, atomic-write failure cleanup, branch ref re-resolution, cache-hit skip, race-on-rename handling, path-registry-with-ref error.

Docs updated

  • `docs/design/registry.md` — full rewrite of Reference syntax, Versioning, and Caching sections; updated Implementation table.
  • `docs/cli-reference.md` — new syntax examples; `registry update`/`show`/`list` descriptions.
  • `README.md` — quick-start example uses `#ref` form.

Notes for reviewers

  • The `#` separator may need quoting in shells if surrounded by spaces (e.g. `conductor run 'qa-bot @ team # v1'`). Mid-token use (the typical case) is fine unquoted.
  • Branch refs are re-resolved to SHAs on every `fetch_workflow` call, which means an extra GitHub API roundtrip when running with a branch ref. Acceptable trade-off for correctness — explicit tag refs hit the cache directly.

- New ref grammar: <workflow>[@<registry>][#<ref>] (was @@Version)
- Auto-discover versions from git tags; semver-sort with v-prefix strip
- Fall back to default branch HEAD when no tags exist
- Any tag/branch/SHA can be pinned via #ref
- Drop versions: field from index.yaml (silently ignored for back-compat)
- Cache by commit SHA (sha[:12]); mutable refs re-resolve fresh
- Atomic cache writes (tempfile.mkdtemp + os.replace)
- Index loads always resolve ref->SHA, fetching at /<sha>/index.yaml
  to bypass raw.githubusercontent.com Fastly CDN
- registry list/show display 'Latest tags:' footer
- Path registries hard-error on non-empty refs
- Empty # and multiple @/# are hard errors

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant