-
Notifications
You must be signed in to change notification settings - Fork 8k
feat: automate community catalog submissions with validation and PR generation #2401
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
mnriem
wants to merge
13
commits into
github:main
Choose a base branch
from
mnriem:feat/catalog-submission-automation
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
0e86c76
feat: automate community catalog submissions with validation and PR g…
mnriem 02872e7
fix: use URL hostname allowlist instead of substring match for token …
mnriem 4f1b42a
fix: address Copilot review findings from PR #2401
mnriem a8b7025
fix: address Copilot review round 3 findings
mnriem 606eb9a
fix: address Copilot review round 4 findings
mnriem f6b5341
fix: address Copilot review round 5 findings
mnriem c9ddb13
fix: address Copilot review round 6 findings
mnriem 8278474
fix: fix SSRF comment and remove redundant validate re-runs
mnriem 1c10491
fix: address Copilot review round 8 findings
mnriem b65d14a
fix: address Copilot review round 9 findings
mnriem d627a64
fix: address Copilot review round 10 findings
mnriem 3740d4a
fix: address Copilot review round 11 findings
mnriem 199f6bf
fix: address Copilot review round 12 findings
mnriem File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,8 @@ | ||
| # Global code owner | ||
| * @mnriem | ||
|
|
||
| # Community catalog files — require maintainer approval even for bot PRs | ||
| extensions/catalog.community.json @mnriem | ||
| integrations/catalog.community.json @mnriem | ||
| presets/catalog.community.json @mnriem | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,208 @@ | ||
| #!/usr/bin/env python3 | ||
| """Generate a markdown table from a community catalog JSON file. | ||
|
|
||
| Reads a catalog.community.json and replaces content between marker comments | ||
| in a target markdown file. When ``--target`` is provided and markers are | ||
| missing, the script exits with an error. Without ``--target`` the table is | ||
| printed to stdout. | ||
|
|
||
| Markers expected in the markdown file: | ||
| <!-- catalog-table-start --> | ||
| ... (old table content replaced) ... | ||
| <!-- catalog-table-end --> | ||
|
|
||
| Usage: | ||
| python .github/scripts/catalog-generate-table.py \ | ||
| --catalog presets/catalog.community.json \ | ||
| --type preset \ | ||
| --target docs/community/presets.md | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import json | ||
| import re | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| START_MARKER = "<!-- catalog-table-start -->" | ||
| END_MARKER = "<!-- catalog-table-end -->" | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Table builders — one per catalog type | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
| def _repo_display_name(url: str) -> str: | ||
| """Extract the repository name from a GitHub URL.""" | ||
| # https://github.com/user/spec-kit-foo → spec-kit-foo | ||
| return url.rstrip("/").rsplit("/", 1)[-1] | ||
|
|
||
|
|
||
| def _provides_str_preset(provides: dict) -> str: | ||
| parts: list[str] = [] | ||
| t = provides.get("templates", 0) | ||
| c = provides.get("commands", 0) | ||
| s = provides.get("scripts", 0) | ||
| if t: | ||
| parts.append(f"{t} template{'s' if t != 1 else ''}") | ||
| if c: | ||
| parts.append(f"{c} command{'s' if c != 1 else ''}") | ||
| if s: | ||
| parts.append(f"{s} script{'s' if s != 1 else ''}") | ||
| return ", ".join(parts) or "—" | ||
|
|
||
|
|
||
| def _requires_str_preset(requires: dict) -> str: | ||
| exts = requires.get("extensions", []) | ||
| if exts: | ||
| return ", ".join(f"{e} extension" for e in exts) | ||
| return "—" | ||
|
|
||
|
|
||
| def _escape_cell(value: str) -> str: | ||
| """Escape pipe characters and collapse whitespace for markdown table cells.""" | ||
| return value.replace("|", "\\|").replace("\n", " ").replace("\r", "").strip() | ||
|
|
||
|
|
||
| def build_preset_table(catalog: dict) -> str: | ||
| """Build a markdown table for presets.""" | ||
| entries = catalog.get("presets", {}) | ||
| lines: list[str] = [] | ||
| lines.append("| Preset | Purpose | Provides | Requires | URL |") | ||
| lines.append("|--------|---------|----------|----------|-----|") | ||
|
|
||
| for _id in sorted(entries): | ||
| e = entries[_id] | ||
| name = _escape_cell(e.get("name", _id)) | ||
| desc = _escape_cell(e.get("description", "")) | ||
| provides = _provides_str_preset(e.get("provides", {})) | ||
| requires = _requires_str_preset(e.get("requires", {})) | ||
| repo_url = e.get("repository", "") | ||
| repo_name = _repo_display_name(repo_url) | ||
| lines.append( | ||
| f"| {name} | {desc} | {provides} | {requires} " | ||
| f"| [{repo_name}]({repo_url}) |" | ||
| ) | ||
|
mnriem marked this conversation as resolved.
|
||
|
|
||
| return "\n".join(lines) | ||
|
|
||
|
|
||
| def _provides_str_extension(provides: dict) -> str: | ||
| parts: list[str] = [] | ||
| c = provides.get("commands", 0) | ||
| h = provides.get("hooks", 0) | ||
| if c: | ||
| parts.append(f"{c} command{'s' if c != 1 else ''}") | ||
| if h: | ||
| parts.append(f"{h} hook{'s' if h != 1 else ''}") | ||
| return ", ".join(parts) or "—" | ||
|
|
||
|
|
||
| def build_extension_table(catalog: dict) -> str: | ||
| """Build a markdown table for extensions. | ||
|
|
||
| Note: Category and Effect columns will be empty unless the catalog | ||
| entries include ``category`` and ``effect`` fields (not yet part of | ||
| the standard catalog schema). | ||
| """ | ||
| entries = catalog.get("extensions", {}) | ||
| lines: list[str] = [] | ||
| lines.append("| Extension | Purpose | Category | Effect | URL |") | ||
| lines.append("|-----------|---------|----------|--------|-----|") | ||
|
|
||
| for _id in sorted(entries): | ||
| e = entries[_id] | ||
| name = _escape_cell(e.get("name", _id)) | ||
| desc = _escape_cell(e.get("description", "")) | ||
| category = _escape_cell(e.get("category", "")) | ||
| if category: | ||
| category = f"`{category}`" | ||
| effect = _escape_cell(e.get("effect", "")) | ||
| repo_url = e.get("repository", "") | ||
| repo_name = _repo_display_name(repo_url) | ||
| lines.append( | ||
| f"| {name} | {desc} | {category} | {effect} " | ||
| f"| [{repo_name}]({repo_url}) |" | ||
| ) | ||
|
mnriem marked this conversation as resolved.
|
||
|
|
||
|
mnriem marked this conversation as resolved.
|
||
| return "\n".join(lines) | ||
|
|
||
|
|
||
| BUILDERS = { | ||
| "preset": build_preset_table, | ||
| "extension": build_extension_table, | ||
| } | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # File updater | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
| def update_file(path: Path, table: str) -> bool: | ||
| """Replace content between markers in *path*. Returns True if updated.""" | ||
| content = path.read_text() | ||
|
|
||
| pattern = re.compile( | ||
| rf"({re.escape(START_MARKER)})\n.*?\n({re.escape(END_MARKER)})", | ||
| re.DOTALL, | ||
| ) | ||
|
|
||
| if not pattern.search(content): | ||
| return False | ||
|
|
||
| new_content = pattern.sub(lambda m: f"{m.group(1)}\n{table}\n{m.group(2)}", content) | ||
|
|
||
| if new_content != content: | ||
| path.write_text(new_content) | ||
| return True | ||
|
mnriem marked this conversation as resolved.
|
||
| return False | ||
|
|
||
|
|
||
| # --------------------------------------------------------------------------- | ||
| # Main | ||
| # --------------------------------------------------------------------------- | ||
|
|
||
| def main() -> None: | ||
| parser = argparse.ArgumentParser(description=__doc__) | ||
| parser.add_argument( | ||
| "--catalog", required=True, | ||
| help="Path to catalog.community.json", | ||
| ) | ||
| parser.add_argument( | ||
| "--type", required=True, choices=list(BUILDERS), | ||
| help="Catalog type", | ||
| ) | ||
| parser.add_argument( | ||
| "--target", | ||
| help="Markdown file to update (must contain marker comments)", | ||
| ) | ||
| args = parser.parse_args() | ||
|
|
||
| with open(args.catalog) as f: | ||
| catalog = json.load(f) | ||
|
|
||
| builder = BUILDERS[args.type] | ||
| table = builder(catalog) | ||
|
|
||
| if args.target: | ||
| target = Path(args.target) | ||
| if not target.exists(): | ||
| print(f"Error: target file not found: {target}", file=sys.stderr) | ||
| sys.exit(1) | ||
| if update_file(target, table): | ||
| print(f"Updated {target}") | ||
| else: | ||
| print( | ||
| f"Error: markers {START_MARKER} / {END_MARKER} not found " | ||
| f"in {target}.", | ||
| file=sys.stderr, | ||
| ) | ||
|
mnriem marked this conversation as resolved.
|
||
| sys.exit(1) | ||
| else: | ||
| print(table) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,138 @@ | ||
| #!/usr/bin/env python3 | ||
| """Update a community catalog JSON file with a validated entry. | ||
|
|
||
| Reads the validation result and entry produced by catalog-validate.py, | ||
| inserts or replaces the entry in the catalog, sorts entries alphabetically, | ||
| and optionally regenerates a docs table. | ||
|
|
||
| Usage (typically called from GitHub Actions): | ||
| python .github/scripts/catalog-pr.py \ | ||
| --catalog extensions/catalog.community.json \ | ||
| --type extension | ||
|
|
||
| python .github/scripts/catalog-pr.py \ | ||
| --catalog presets/catalog.community.json \ | ||
| --type preset \ | ||
| --table-target docs/community/presets.md | ||
|
|
||
| Environment variables: | ||
| GITHUB_OUTPUT — Path to GitHub Actions output file (optional) | ||
|
|
||
| Inputs (files produced by catalog-validate.py): | ||
| /tmp/validation-result.json — metadata including item_id, valid, is_update | ||
| /tmp/catalog-entry.json — the entry to insert/replace | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import json | ||
| import os | ||
| import subprocess | ||
| import sys | ||
| from datetime import datetime, timezone | ||
| from pathlib import Path | ||
|
|
||
| CATALOG_KEY = { | ||
| "extension": "extensions", | ||
| "preset": "presets", | ||
| } | ||
|
|
||
|
|
||
| def main() -> None: | ||
| parser = argparse.ArgumentParser(description=__doc__) | ||
| parser.add_argument( | ||
| "--catalog", required=True, | ||
| help="Path to the catalog JSON file", | ||
| ) | ||
| parser.add_argument( | ||
| "--type", required=True, choices=list(CATALOG_KEY), | ||
| help="Catalog type", | ||
| ) | ||
| parser.add_argument( | ||
| "--result", default="/tmp/validation-result.json", | ||
| help="Path to validation result JSON", | ||
| ) | ||
| parser.add_argument( | ||
| "--entry", default="/tmp/catalog-entry.json", | ||
| help="Path to catalog entry JSON", | ||
| ) | ||
| parser.add_argument( | ||
| "--table-target", | ||
| help="Markdown file to regenerate table in (must contain marker comments)", | ||
| ) | ||
| args = parser.parse_args() | ||
|
|
||
| # Load validation result | ||
| result_path = Path(args.result) | ||
| if not result_path.exists(): | ||
| print(f"Error: result file not found: {result_path}", file=sys.stderr) | ||
| sys.exit(2) | ||
| result = json.loads(result_path.read_text()) | ||
|
|
||
| if not result.get("valid"): | ||
| print("Submission is not valid — skipping catalog update.") | ||
| _set_output("skipped", "true") | ||
| sys.exit(0) | ||
|
|
||
| # Load entry | ||
| entry_path = Path(args.entry) | ||
| if not entry_path.exists(): | ||
| print(f"Error: entry file not found: {entry_path}", file=sys.stderr) | ||
| sys.exit(2) | ||
| new_entry = json.loads(entry_path.read_text()) | ||
|
|
||
| item_id = result["item_id"] | ||
| is_update = result.get("is_update", False) | ||
| cat_key = CATALOG_KEY[args.type] | ||
|
|
||
| # Update catalog | ||
| catalog_path = Path(args.catalog) | ||
| with open(catalog_path) as f: | ||
| catalog = json.load(f) | ||
|
|
||
| catalog[cat_key][item_id] = new_entry | ||
| catalog["updated_at"] = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") | ||
| catalog[cat_key] = dict(sorted(catalog[cat_key].items())) | ||
|
|
||
| with open(catalog_path, "w") as f: | ||
| json.dump(catalog, f, indent=2) | ||
| f.write("\n") | ||
|
|
||
| print(f"Updated {catalog_path}: {'replaced' if is_update else 'added'} {item_id}") | ||
|
|
||
| # Regenerate docs table if requested | ||
| if args.table_target: | ||
| table_script = Path(__file__).parent / "catalog-generate-table.py" | ||
| subprocess.run( | ||
| [ | ||
| sys.executable, str(table_script), | ||
| "--catalog", args.catalog, | ||
| "--type", args.type, | ||
| "--target", args.table_target, | ||
| ], | ||
| check=True, | ||
| ) | ||
|
|
||
|
Comment on lines
+89
to
+116
|
||
| # Set outputs for the workflow | ||
| action = "update" if is_update else "add" | ||
| _set_output("skipped", "false") | ||
| _set_output("item_id", item_id) | ||
| _set_output("is_update", str(is_update).lower()) | ||
| _set_output("action", action.title()) | ||
| _set_output("action_verb", f"{action.title()}s") | ||
| _set_output("branch", f"community/{action}-{args.type}-{item_id}") | ||
|
|
||
|
|
||
| def _set_output(name: str, value: str) -> None: | ||
| """Write a GitHub Actions output variable.""" | ||
| gh_output = os.environ.get("GITHUB_OUTPUT") | ||
| if gh_output: | ||
| with open(gh_output, "a") as f: | ||
| f.write(f"{name}={value}\n") | ||
| # Also print for local debugging | ||
| print(f" output: {name}={value}") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.