From a21fe17d8ff2b405332fc02befd03ae93a284d71 Mon Sep 17 00:00:00 2001 From: DevCats Date: Mon, 22 Jun 2026 14:50:28 -0500 Subject: [PATCH 1/5] feat(config.yaml): add scanners.skillspector.llm block Document the new llm.provider config knob (default nv_build) and the workflow contract: empty flags + workflow appends --no-llm dynamically when the matching credential secret is unset. Removes --no-llm from the static flags list now that the workflow drives it. This commit was prepared with help from Coder Agents. --- config.yaml | 37 +++++++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/config.yaml b/config.yaml index b20462a..31be704 100644 --- a/config.yaml +++ b/config.yaml @@ -39,8 +39,41 @@ scanners: # so a bumper bot lives outside the loop until the upstream # publishes to PyPI and the pin can move into pyproject.toml. pin: "skillspector @ git+https://github.com/NVIDIA/SkillSpector.git@2eb844780ab163f01468ecf142c40a2ec0fcaec0" - flags: - - "--no-llm" + # Extra CLI flags passed to every SkillSpector invocation. Empty by + # default; the scan workflow appends --no-llm dynamically when the + # LLM credential secret is not set (see llm: block below). CI runs + # do not invoke SkillSpector live. + flags: [] + # SkillSpector ships a two-stage analyser: fast static rules followed + # by an optional LLM semantic pass. The LLM pass lifts precision + # from roughly 70% to roughly 87% per upstream docs by filtering + # context-aware false positives, classifying intent on prompt + # injection patterns, and producing human-readable explanations. + # + # The scheduled scan reads the credential matching the provider + # below from a repository secret. When the secret is configured, + # LLM mode is on. When the secret is missing, the workflow falls + # back to --no-llm automatically so a fresh fork is never broken + # by an unset secret. + # + # Provider options and the env var SkillSpector consumes: + # + # provider env var(s) + # nv_build NVIDIA_INFERENCE_KEY (free; build.nvidia.com) + # openai OPENAI_API_KEY (+ OPENAI_BASE_URL for AI gateways) + # anthropic ANTHROPIC_API_KEY + # anthropic_proxy ANTHROPIC_PROXY_API_KEY + ANTHROPIC_PROXY_ENDPOINT_URL + # + # Changing provider also requires updating the env block in + # .github/workflows/scan.yaml so the matching secret is wired in, + # and adding the secret under Settings > Secrets and variables > + # Actions. + llm: + provider: nv_build + # Empty model uses the provider's bundled default. Override here + # to pin a specific revision (e.g. "claude-opus-4-6" for + # provider=anthropic). + model: "" # Per-skill verdict policy. v1 has one input (SkillSpector risk_score). # When more scanners join the pipeline we add new threshold fields here From c3f42dbd21be10b86869a9249af54229e6fc2968 Mon Sep 17 00:00:00 2001 From: DevCats Date: Mon, 22 Jun 2026 14:53:56 -0500 Subject: [PATCH 2/5] docs(CALIBRATION.md): add LLM semantic pass section Document what flipping LLM mode on does (and does not do) to the verdict math, the precision delta we expect, and what to expect for the five in-tree skills. Adds "LLM provider changes" to the "When to revisit" list. This commit was prepared with help from Coder Agents. --- docs/CALIBRATION.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/CALIBRATION.md b/docs/CALIBRATION.md index e4cf5a0..5091163 100644 --- a/docs/CALIBRATION.md +++ b/docs/CALIBRATION.md @@ -99,6 +99,42 @@ verdict: This avoids broadcasting the ~half-of-catalogue base rate that ClawHub measured. +## LLM semantic pass + +SkillSpector ships a two-stage analyser: fast static rules (the 64 +patterns SkillSpector documents) followed by an optional LLM semantic +pass. Upstream's published precision numbers are: + +- `--no-llm` (static only): high recall, moderate precision (~70%). + False positives on context-sensitive patterns are common; for + example, EA2 ("autonomous decision making") fires on prose that + documents safeguards as well as prose that bypasses them. +- Default (LLM on): ~87% precision. The LLM pass reads each finding's + surrounding context, classifies intent, filters context-aware false + positives, and writes a human-readable explanation that ships in the + per-finding output. + +The scheduled scan runs LLM mode when the workflow's chosen credential +secret (`NVIDIA_INFERENCE_KEY` for the default `nv_build` provider) is +configured. The fallback to `--no-llm` is automatic when the secret is +missing, so an unset secret on a fresh fork degrades the scan rather +than breaking it. + +The LLM pass does not affect the threshold math: SkillSpector's +`risk_score` is still a 0-100 weighted sum of rule hits, and the +51/81 cutoffs above still map directly to `HIGH` and `CRITICAL` bands. +It does affect which findings reach the verdict: false positives that +the LLM filters out no longer contribute to the score. Expect verdicts +to move down (or stay the same) when LLM mode flips on, not up. + +For the five existing in-tree skills, the static-only scan placed +`coder/setup` at 100 / `malicious`. With LLM mode on we expect the +findings list to shrink (the EA2 prose hits and the asset-path MP2 +hits should be filtered) but the score will still be high. Reducing +`coder/setup`'s verdict below `suspicious` requires the upcoming +permissions-manifest layer (Phase 3 of the v3 plan), not the LLM pass +alone. + ## What we did not change (and why) - We did not raise `suspicious_risk_score` above `51`. SkillSpector @@ -127,6 +163,9 @@ Re-run this analysis when any of: that shifts where its bands sit. The pinned commit in `config.yaml` protects us from drifting silently; a deliberate bump should walk through this doc. +- The LLM provider changes (e.g., moving from `nv_build` to + `anthropic`). Different models filter differently; spot-check the + five in-tree skills before merging the provider swap. - We observe a real-world skill that lands in an obviously wrong bucket (false positive or false negative). Open a tracking issue, link it from this doc, and adjust with evidence in the next PR. From c10bf521be005930f60a53acc870f347f9b12d6b Mon Sep 17 00:00:00 2001 From: DevCats Date: Mon, 22 Jun 2026 14:54:39 -0500 Subject: [PATCH 3/5] docs(README.md): document LLM mode and the one-time secret setup Update step 3 of the architecture summary to reflect that the scheduled scan now runs SkillSpector with the LLM semantic pass on by default. Add a new "One-time setup on the repo" section that lists the three repo-level configurations needed for a useful scan, including the new LLM credential secret. Mirror the LLM secret note into "Forking for your own catalogue". This commit was prepared with help from Coder Agents. --- README.md | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index f662a82..0aacfe4 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,10 @@ Every 6 hours, the scheduled workflow in this repo: 1. Enumerates every skill in `coder/registry` (both the in-tree `.agents/skills/` format and the future external-sources format). 2. Shallow-clones each source repo. -3. Runs [NVIDIA SkillSpector](https://github.com/NVIDIA/SkillSpector) in - `--no-llm` static mode over the upstream content. +3. Runs [NVIDIA SkillSpector](https://github.com/NVIDIA/SkillSpector) over + the upstream content. The scheduled scan uses LLM semantic analysis + when the credential secret is configured, and falls back to + `--no-llm` static-only mode otherwise. 4. Builds a per-skill verdict (`clean`, `suspicious`, `malicious`, `unknown`) from `risk_score` plus the thresholds in `config.yaml`. 5. Builds the React SPA in `site/` and ships it together with @@ -60,6 +62,26 @@ Vite's dev proxy (see `site/vite.config.ts`) forwards `latest.json`, app sees real scanner output without CORS shenanigans. SPA routes such as `/skills/coder/setup` stay client-side. +## One-time setup on the repo + +Three things have to be configured once on the GitHub repo before the +scheduled scan publishes a useful result: + +1. **Settings > Pages**: set source to "GitHub Actions". The + `publish-pages` job in `scan.yaml` will fail until this is set. +2. **Settings > Actions**: workflow permissions "Read and write" so + `publish-release` can create the rolling `latest` release. +3. **Settings > Secrets and variables > Actions**: add the LLM + credential matching the provider in `config.yaml`'s + `scanners.skillspector.llm.provider`. For the default `nv_build` + provider this is `NVIDIA_INFERENCE_KEY` (sign up free at + [build.nvidia.com](https://build.nvidia.com)). Without the secret + the scan still runs, but SkillSpector falls back to + `--no-llm` static-only mode and precision drops from roughly 87% + to roughly 70%. See `docs/CALIBRATION.md` for the precision + discussion. The optional `SLACK_WEBHOOK_URL` secret enables the + `notify-slack-on-failure` job; without it that job is a no-op. + ## Repo layout ```text @@ -97,7 +119,10 @@ This scanner is data-driven. To run it against a different registry: "GitHub Actions"). 4. Set Actions workflow permissions to "Read and write" so the publish-release job can create releases. -5. Enable Actions. +5. Add the LLM credential secret matching your chosen provider + (see "One-time setup on the repo" above). Optional; static-only + mode works without it. +6. Enable Actions. No source changes required for catalogue changes. @@ -115,7 +140,8 @@ SkillSpector's `risk_score` (0-100) is the only input. The thresholds are aligned to SkillSpector's own `HIGH` and `CRITICAL` bands; [`docs/CALIBRATION.md`](./docs/CALIBRATION.md) walks through the evidence (SkillSpector source, the ClawHub paper, our in-tree -catalogue) behind the chosen numbers. +catalogue) behind the chosen numbers, and the LLM-on-vs-off precision +discussion behind running the semantic pass on every scheduled scan. The architecture keeps room for additional scanners (gitleaks, Semgrep, VirusTotal Premium, etc.); adding one is a new module under `scanner/`, From 8c57f9a637bec637cd63766f5402dedafba1bb01 Mon Sep 17 00:00:00 2001 From: DevCats Date: Mon, 22 Jun 2026 17:23:24 -0500 Subject: [PATCH 4/5] feat(config.yaml): switch LLM provider to Anthropic Sonnet 4.6 Swap the default LLM provider from nv_build (free NVIDIA Build) to anthropic with model pinned to claude-sonnet-4-6. Rationale: - Removes the second-vendor signup. The Coder org already has an Anthropic billing relationship, so the credential is one secret away from working. - Sonnet 4.6 is roughly 5x cheaper than the anthropic default (Opus 4.6) and is well matched to SkillSpector's LLM pass, which is finding-by-finding intent classification rather than long-form reasoning. Cost ballpark for 5 skills x 4 scans/day is small. - The other provider options (anthropic_proxy via Vertex, openai via any OpenAI-compatible gateway, nv_build) stay documented in the config comments and are still a one-line swap. This commit was prepared with help from Coder Agents. --- config.yaml | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/config.yaml b/config.yaml index 31be704..9cc2328 100644 --- a/config.yaml +++ b/config.yaml @@ -59,21 +59,23 @@ scanners: # Provider options and the env var SkillSpector consumes: # # provider env var(s) - # nv_build NVIDIA_INFERENCE_KEY (free; build.nvidia.com) - # openai OPENAI_API_KEY (+ OPENAI_BASE_URL for AI gateways) - # anthropic ANTHROPIC_API_KEY + # anthropic ANTHROPIC_API_KEY (api.anthropic.com) # anthropic_proxy ANTHROPIC_PROXY_API_KEY + ANTHROPIC_PROXY_ENDPOINT_URL + # openai OPENAI_API_KEY (+ OPENAI_BASE_URL for AI gateways) + # nv_build NVIDIA_INFERENCE_KEY (free; build.nvidia.com) # # Changing provider also requires updating the env block in # .github/workflows/scan.yaml so the matching secret is wired in, # and adding the secret under Settings > Secrets and variables > # Actions. llm: - provider: nv_build - # Empty model uses the provider's bundled default. Override here - # to pin a specific revision (e.g. "claude-opus-4-6" for - # provider=anthropic). - model: "" + provider: anthropic + # SkillSpector's bundled default for the anthropic provider is + # claude-opus-4-6. Sonnet 4.6 is roughly 5x cheaper than Opus and + # is well-suited for the finding-classification task the LLM pass + # actually does, so it is the better cost/quality choice for + # periodic scanning. Override here to pin a different revision. + model: "claude-sonnet-4-6" # Per-skill verdict policy. v1 has one input (SkillSpector risk_score). # When more scanners join the pipeline we add new threshold fields here From 3333b8117369b441b8697350984b8cd036a915a9 Mon Sep 17 00:00:00 2001 From: DevCats Date: Mon, 22 Jun 2026 17:24:12 -0500 Subject: [PATCH 5/5] docs(README.md): point setup at Anthropic provider, not nv_build Follow-up to the provider swap. The one-time-setup section now points at console.anthropic.com and ANTHROPIC_API_KEY instead of build.nvidia.com / NVIDIA_INFERENCE_KEY. This commit was prepared with help from Coder Agents. --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 0aacfe4..f486179 100644 --- a/README.md +++ b/README.md @@ -73,10 +73,10 @@ scheduled scan publishes a useful result: `publish-release` can create the rolling `latest` release. 3. **Settings > Secrets and variables > Actions**: add the LLM credential matching the provider in `config.yaml`'s - `scanners.skillspector.llm.provider`. For the default `nv_build` - provider this is `NVIDIA_INFERENCE_KEY` (sign up free at - [build.nvidia.com](https://build.nvidia.com)). Without the secret - the scan still runs, but SkillSpector falls back to + `scanners.skillspector.llm.provider`. For the default `anthropic` + provider this is `ANTHROPIC_API_KEY` (from + [console.anthropic.com](https://console.anthropic.com)). Without + the secret the scan still runs, but SkillSpector falls back to `--no-llm` static-only mode and precision drops from roughly 87% to roughly 70%. See `docs/CALIBRATION.md` for the precision discussion. The optional `SLACK_WEBHOOK_URL` secret enables the