From e6b5cfa4210bde8360957c02a5027c0454494ce1 Mon Sep 17 00:00:00 2001 From: Dima Birenbaum Date: Fri, 24 Apr 2026 09:34:35 +0300 Subject: [PATCH 1/5] docs: add spec for agentic-workflows noop fix --- ...04-24-agentic-workflows-noop-fix-design.md | 155 ++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100644 docs/superpowers/specs/2026-04-24-agentic-workflows-noop-fix-design.md diff --git a/docs/superpowers/specs/2026-04-24-agentic-workflows-noop-fix-design.md b/docs/superpowers/specs/2026-04-24-agentic-workflows-noop-fix-design.md new file mode 100644 index 00000000..753e0ca3 --- /dev/null +++ b/docs/superpowers/specs/2026-04-24-agentic-workflows-noop-fix-design.md @@ -0,0 +1,155 @@ +# Agentic Workflows — `noop` Fix Design + +**Date:** 2026-04-24 +**Tracking issue:** [#247 — [aw] MSDO Issue Triage Assistant failed](https://github.com/microsoft/security-devops-action/issues/247) + +## Problem + +Three agentic workflows in `.github/workflows/` each set `safe-outputs.noop: false` +while their prompts instruct the agent to call `noop` or stay silent under +various conditions: + +| Workflow | File | Silent-path trigger | +|---|---|---| +| MSDO Issue Triage Assistant | `msdo-issue-assistant.md` | "Don't respond if" rules (off-topic issue, closed, non-author, `status:team-review`, already-responded, etc.) | +| CI Doctor | `ci-doctor.md` | "If the workflow succeeded, do nothing (noop)"; duplicate-issue check | +| MSDO Toolchain Breach Monitor | `msdo-breach-monitor.md` | "Call `noop` with a one-line summary" when no new CVEs | + +With `noop` disabled, the agent has no way to signal "intentional no-op." gh-aw +reads no `agent_output.json`, treats the run as failure, and files an issue +titled `[aw] failed`. + +In this repository, **every new GitHub issue opens a CRI IcM ticket that pages +on-call**. Each false-positive failure issue is therefore a human page. For +`msdo-issue-assistant`, the failure issue itself (#247) carries the +`agentic-workflows` label and re-triggers the bot on every comment, producing +a self-sustaining spam loop. + +Evidence from run `24783399971`: + +``` +Agent conclusion: success +Error reading agent output file: ENOENT: no such file or directory, open '/tmp/gh-aw/agent_output.json' +Agent succeeded but produced no safe outputs +Found existing issue #247: https://github.com/microsoft/security-devops-action/issues/247 +Added comment to existing issue #247 +``` + +## Goals + +1. Stop paging IcM on-call for false-positive agent failures. +2. Let each bot explicitly signal "nothing to do" when its prompt says to. +3. Preserve normal behaviour on genuine user questions and genuine incidents. + +## Non-goals + +- Closing issue #247 (user closes manually after merge). +- Adding non-paging alerting for real agent failures (future work). +- Fixing the unrelated broken `ContainerMapping` tests on main. +- Changes to non-agentic workflows. + +## Approach + +Hybrid fix — enable `noop` as the root-cause fix, add `report-failure-as-issue: false` +as a safety net so any edge case that still produces no output never pages IcM. + +### Changes to `safe-outputs` (all three `.md` files) + +```yaml +safe-outputs: + noop: true # was: false + report-failure-as-issue: false # new + # ... existing keys (add-comment, add-labels, create-issue, etc.) unchanged +``` + +Semantics: +- `noop: true` registers the `noop` safe-output handler so the agent can call it + with a reason. gh-aw records a successful no-op run and does not treat it as + failure. +- `report-failure-as-issue: false` prevents gh-aw from filing an issue when the + run ends in failure or with no outputs. Genuine failures remain visible in the + Actions tab. + +### Additional edits in `msdo-issue-assistant.md` only + +The prompt currently uses `## Important Rules → Don't respond if` and +`## Do NOT Respond Examples`. Update both to direct the agent to call `noop` +explicitly. + +**Rule replacement** (replace the existing rule 4 "Don't respond if" block): + +```markdown +4. **Call `noop` instead of staying silent** when any of these apply. Pass a + one-line reason so the decision is auditable: + - The issue is not related to MSDO or security-devops-action + - The issue title starts with `[aw]` or is labeled `agentic-workflows` + (auto-generated failure reports, not user issues) + - The issue is closed + - The commenter is not the issue author (unless it's a new issue) + - You have already responded twice and there is no new technical + information in the latest user message + - The issue has a `status:team-review` label +``` + +**New entry in "Do NOT Respond Examples"** (append): + +```markdown +**Workflow failure issue (auto-generated):** Title starts with `[aw]` +(e.g. "[aw] MSDO Issue Triage Assistant failed") or labeled +`agentic-workflows`. +→ Call `noop` with reason "auto-generated failure report, not a user issue". +``` + +No prompt edits in `ci-doctor.md` or `msdo-breach-monitor.md` — their prompts +already say "call noop" / "do nothing (noop)" and will work correctly once +`noop: true` is set. + +### Lock-file regeneration + +After `.md` edits, run `gh aw compile` locally (gh-aw CLI v0.61.0, +matching the version recorded in the existing lock-file header) to +regenerate the three `.lock.yml` files. Both `.md` and `.lock.yml` go in +the same PR so reviewers can diff intent against generated output. + +## Validation + +Existing unit tests on main are broken (ContainerMapping) and do not cover +agentic-workflow behaviour. Validation is behavioural, via `workflow_dispatch` +runs on the PR branch: + +1. **Compile check:** `gh aw compile` succeeds without error; lock-file diff + contains the expected `noop` handler wiring and no other unintended changes. +2. **`msdo-issue-assistant` negative path:** on the PR branch, post a comment + on an existing off-topic issue or on issue #247 itself (this fires the + `issue_comment: created` trigger against the PR-branch workflow via the + normal gh-aw activation flow). Expect the run to succeed, no new comment + posted, no new `[aw] ... failed` issue filed. +3. **`msdo-issue-assistant` positive path:** open a test issue with a real MSDO + question (for example "how do I pass `--download-external-modules` to + checkov?"). Expect the bot to reply normally, citing the wiki, applying the + `area:msdo-cli` label. +4. **`ci-doctor` negative path:** trigger a CI run that succeeds on `main` or a + `release/**` branch (the workflow auto-fires on `workflow_run: CI completed`), + or dispatch manually and point it at a successful run. Expect noop, no + issue filed. +5. **`msdo-breach-monitor` negative path:** `workflow_dispatch` manually when + no new CVEs are in the advisory window. Expect noop, no issue filed. + +If any dry run still files a `[aw] ... failed` issue, the safety net +(`report-failure-as-issue: false`) has not taken effect — investigate before +merging. + +## Rollout + +- One PR on branch `fix/agentic-workflows-noop`, base `main`. +- PR title: `fix(ci): enable noop on agentic workflows to stop IcM page spam`. +- Merge once dry-run validation passes. +- User closes #247 manually after merge. + +## Risks + +| Risk | Mitigation | +|---|---| +| `report-failure-as-issue: false` hides a real agent failure | Accepted trade-off — false positives page IcM; real failures remain in Actions tab and can be wired to non-paging alerts later | +| gh-aw v0.61.0 interprets `noop: true` differently than expected | Lock-file diff is reviewed before merge; fall back to `report-failure-as-issue: false` only (Approach 2) if the generated handler looks wrong | +| Prompt edits cause `msdo-issue-assistant` to noop on cases users want a reply on | Conditions are identical to existing "Don't respond" rules — behaviour unchanged, only the exit mechanism becomes explicit. Positive-path dry run catches regressions | From b9bfeb7679d7d1d3a266753e669a3da973d42990 Mon Sep 17 00:00:00 2001 From: Dima Birenbaum Date: Fri, 24 Apr 2026 09:40:24 +0300 Subject: [PATCH 2/5] docs: add implementation plan for agentic-workflows noop fix --- .../2026-04-24-agentic-workflows-noop-fix.md | 593 ++++++++++++++++++ 1 file changed, 593 insertions(+) create mode 100644 docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md diff --git a/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md b/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md new file mode 100644 index 00000000..b07f3d92 --- /dev/null +++ b/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md @@ -0,0 +1,593 @@ +# Agentic Workflows noop Fix — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Stop the three gh-aw agentic workflows from filing false-positive `[aw] ... failed` issues that page the on-call IcM rotation. + +**Architecture:** For all three agentic workflows, set `safe-outputs.noop: true` (lets the agent explicitly signal "nothing to do") and `safe-outputs.report-failure-as-issue: false` (blocks the auto-filed failure issue even when no output is produced). Update the `msdo-issue-assistant` prompt so its existing "Don't respond if" rules now direct the agent to call the `noop` tool explicitly. Regenerate the three `.lock.yml` files with `gh aw compile` and ship everything in one PR. + +**Tech Stack:** GitHub Actions, gh-aw CLI v0.61.0, YAML, Markdown prompts. + +**Spec:** [docs/superpowers/specs/2026-04-24-agentic-workflows-noop-fix-design.md](../specs/2026-04-24-agentic-workflows-noop-fix-design.md) + +**Branch:** `fix/agentic-workflows-noop` (already created; the spec is committed there as `e6b5cfa`) + +--- + +## Task 1: Verify gh-aw CLI is installed at the right version + +**Files:** none (local tooling check) + +- [ ] **Step 1: Check the gh-aw version** + +Run: +```bash +gh aw version +``` + +Expected output contains `v0.61.0` (this matches the version recorded in the existing lock-file headers at [.github/workflows/msdo-issue-assistant.lock.yml:15](../../.github/workflows/msdo-issue-assistant.lock.yml#L15)). + +If `gh aw` is not installed, install it first: +```bash +gh extension install github/gh-aw +``` + +If a different version is installed, upgrade: +```bash +gh extension upgrade gh-aw +``` + +- [ ] **Step 2: Confirm we are on the right branch** + +Run: +```bash +git branch --show-current +``` + +Expected: `fix/agentic-workflows-noop` + +If not on that branch: +```bash +git checkout fix/agentic-workflows-noop +``` + +--- + +## Task 2: Edit `ci-doctor.md` safe-outputs + +**Files:** +- Modify: [.github/workflows/ci-doctor.md](../../.github/workflows/ci-doctor.md) (lines 32-40) + +- [ ] **Step 1: Replace the `safe-outputs` block** + +In [.github/workflows/ci-doctor.md](../../.github/workflows/ci-doctor.md), replace this exact block: + +```yaml +safe-outputs: + noop: false + create-issue: + max: 1 + add-labels: + allowed: [ci-failure, flaky-test, build-failure, dependency-issue, needs-maintainer] + add-comment: null + create-pull-request: null +``` + +With: + +```yaml +safe-outputs: + noop: true + report-failure-as-issue: false + create-issue: + max: 1 + add-labels: + allowed: [ci-failure, flaky-test, build-failure, dependency-issue, needs-maintainer] + add-comment: null + create-pull-request: null +``` + +The only changes are `noop: false` → `noop: true` and inserting `report-failure-as-issue: false` as the second key. + +- [ ] **Step 2: Verify the edit** + +Run: +```bash +grep -nE "noop|report-failure-as-issue" .github/workflows/ci-doctor.md +``` + +Expected output: +``` +33: noop: true +34: report-failure-as-issue: false +``` + +--- + +## Task 3: Edit `msdo-breach-monitor.md` safe-outputs + +**Files:** +- Modify: [.github/workflows/msdo-breach-monitor.md](../../.github/workflows/msdo-breach-monitor.md) (lines 41-47) + +- [ ] **Step 1: Replace the `safe-outputs` block** + +In [.github/workflows/msdo-breach-monitor.md](../../.github/workflows/msdo-breach-monitor.md), replace this exact block: + +```yaml +safe-outputs: + noop: false + create-issue: + max: 1 + add-labels: + allowed: [security-breach, supply-chain, toolchain-alert, critical, high, medium] +``` + +With: + +```yaml +safe-outputs: + noop: true + report-failure-as-issue: false + create-issue: + max: 1 + add-labels: + allowed: [security-breach, supply-chain, toolchain-alert, critical, high, medium] +``` + +Only changes are `noop: false` → `noop: true` and inserting `report-failure-as-issue: false`. + +- [ ] **Step 2: Verify the edit** + +Run: +```bash +grep -nE "noop|report-failure-as-issue" .github/workflows/msdo-breach-monitor.md +``` + +Expected output: +``` +42: noop: true +43: report-failure-as-issue: false +``` + +--- + +## Task 4: Edit `msdo-issue-assistant.md` safe-outputs + +**Files:** +- Modify: [.github/workflows/msdo-issue-assistant.md](../../.github/workflows/msdo-issue-assistant.md) (lines 32-38) + +- [ ] **Step 1: Replace the `safe-outputs` block** + +In [.github/workflows/msdo-issue-assistant.md](../../.github/workflows/msdo-issue-assistant.md), replace this exact block: + +```yaml +safe-outputs: + noop: false + add-comment: + max: 4 + add-labels: + allowed: ["type:bug", "type:feature", "type:docs", "type:question", "type:security", "type:maintenance", "status:triage", "status:waiting-on-author", "status:repro-needed", "status:team-review", "area:action", "area:msdo-cli", "area:ci", "area:container-mapping"] +``` + +With: + +```yaml +safe-outputs: + noop: true + report-failure-as-issue: false + add-comment: + max: 4 + add-labels: + allowed: ["type:bug", "type:feature", "type:docs", "type:question", "type:security", "type:maintenance", "status:triage", "status:waiting-on-author", "status:repro-needed", "status:team-review", "area:action", "area:msdo-cli", "area:ci", "area:container-mapping"] +``` + +Only changes are `noop: false` → `noop: true` and inserting `report-failure-as-issue: false`. + +- [ ] **Step 2: Verify the edit** + +Run: +```bash +grep -nE "noop|report-failure-as-issue" .github/workflows/msdo-issue-assistant.md +``` + +Expected output: +``` +33: noop: true +34: report-failure-as-issue: false +``` + +--- + +## Task 5: Update `msdo-issue-assistant.md` rule 4 to call noop explicitly + +**Files:** +- Modify: [.github/workflows/msdo-issue-assistant.md](../../.github/workflows/msdo-issue-assistant.md) (around lines 182-188, inside the `## Important Rules` section) + +- [ ] **Step 1: Replace the rule-4 block** + +Replace this exact block: + +```markdown +4. **Don't respond** if: + - The issue is not related to MSDO or security-devops-action + - The issue is closed + - The commenter is not the issue author (unless it's a new issue) + - You've already responded twice and there is no new technical information in the latest user message + - The issue has a `status:team-review` label +``` + +With: + +```markdown +4. **Call `noop` instead of staying silent** when any of these apply. Pass a one-line reason so the decision is auditable: + - The issue is not related to MSDO or security-devops-action + - The issue title starts with `[aw]` or is labeled `agentic-workflows` (auto-generated failure reports, not user issues) + - The issue is closed + - The commenter is not the issue author (unless it's a new issue) + - You have already responded twice and there is no new technical information in the latest user message + - The issue has a `status:team-review` label +``` + +Changes: title reworded from "Don't respond" to "Call `noop` instead of staying silent"; new bullet added for `[aw]`-title / `agentic-workflows`-label issues. + +- [ ] **Step 2: Verify the edit** + +Run: +```bash +grep -n "Call \`noop\` instead" .github/workflows/msdo-issue-assistant.md +``` + +Expected: one match pointing to the rule-4 line. + +--- + +## Task 6: Add `[aw]` example to `msdo-issue-assistant.md` "Do NOT Respond Examples" + +**Files:** +- Modify: [.github/workflows/msdo-issue-assistant.md](../../.github/workflows/msdo-issue-assistant.md) (at the end of the `## Do NOT Respond Examples` section, currently ending around line 213) + +- [ ] **Step 1: Append a new example at the end of the section** + +Find this existing last entry in the `## Do NOT Respond Examples` section: + +```markdown +**Non-author comment on existing issue:** A third party comments "I have the same problem." +→ Do not respond. The commenter is not the issue author. +``` + +Append **after** that block (preserve a blank line before the new entry): + +```markdown + +**Workflow failure issue (auto-generated):** Title starts with `[aw]` (e.g. "[aw] MSDO Issue Triage Assistant failed") or labeled `agentic-workflows`. +→ Call `noop` with reason "auto-generated failure report, not a user issue". +``` + +- [ ] **Step 2: Verify the edit** + +Run: +```bash +grep -n "Workflow failure issue" .github/workflows/msdo-issue-assistant.md +``` + +Expected: one match, appearing after the `Non-author comment on existing issue` example. + +Also run: +```bash +tail -5 .github/workflows/msdo-issue-assistant.md +``` + +Expected: the tail shows the new example as the last content in the file. + +--- + +## Task 7: Regenerate all three lock files + +**Files:** +- Modify (via compile): [.github/workflows/ci-doctor.lock.yml](../../.github/workflows/ci-doctor.lock.yml) +- Modify (via compile): [.github/workflows/msdo-breach-monitor.lock.yml](../../.github/workflows/msdo-breach-monitor.lock.yml) +- Modify (via compile): [.github/workflows/msdo-issue-assistant.lock.yml](../../.github/workflows/msdo-issue-assistant.lock.yml) + +- [ ] **Step 1: Run `gh aw compile`** + +Run from repo root: +```bash +gh aw compile +``` + +Expected: the command exits 0 and reports recompiling the three workflows. Any non-zero exit or schema error indicates the YAML edits are malformed — fix the `.md` files and retry. + +- [ ] **Step 2: Inspect the lock-file diff** + +Run: +```bash +git diff -- .github/workflows/*.lock.yml | head -120 +``` + +Expected: three lock files touched. In each diff, the `frontmatter_hash` near the top of the lock file changes (because the `.md` frontmatter changed). Look for new handler wiring for the noop safe output, and the absence of a `handle_missing_safe_outputs` or similar failure-issue step (because `report-failure-as-issue: false` disables it). + +If the diff shows only the `frontmatter_hash` change and no handler wiring change, the schema interpretation of `noop`/`report-failure-as-issue` may differ from expectation — pause and escalate before committing. + +--- + +## Task 8: Commit the changes + +**Files:** all six touched files in this commit. + +- [ ] **Step 1: Stage all changes** + +Run: +```bash +git add .github/workflows/ci-doctor.md \ + .github/workflows/ci-doctor.lock.yml \ + .github/workflows/msdo-breach-monitor.md \ + .github/workflows/msdo-breach-monitor.lock.yml \ + .github/workflows/msdo-issue-assistant.md \ + .github/workflows/msdo-issue-assistant.lock.yml +``` + +- [ ] **Step 2: Verify the staged diff** + +Run: +```bash +git diff --cached --stat +``` + +Expected: six files listed, three `.md` and three `.lock.yml`. + +- [ ] **Step 3: Commit with the project's oneliner style** + +Run: +```bash +git commit -m "fix(ci): enable noop on agentic workflows to stop IcM page spam" +``` + +No Co-Authored-By line; no multi-line body. + +- [ ] **Step 4: Verify the commit landed** + +Run: +```bash +git log --oneline -2 +``` + +Expected top commit: `fix(ci): enable noop on agentic workflows to stop IcM page spam`. +Second commit from top should be the earlier spec commit (`docs: add spec for agentic-workflows noop fix`). + +--- + +## Task 9: Push the branch and open the PR + +**Files:** none (GitHub operations). + +- [ ] **Step 1: Push the branch** + +Run: +```bash +git push -u origin fix/agentic-workflows-noop +``` + +Expected: branch published to `origin` with tracking configured. + +- [ ] **Step 2: Create the PR** + +Run (use `DimaBir` as the author account per the user's PR-account preference — if the git remote is already using that identity, a plain `gh pr create` is fine; otherwise the user handles account selection manually before this step): + +```bash +gh pr create \ + --repo microsoft/security-devops-action \ + --base main \ + --head fix/agentic-workflows-noop \ + --title "fix(ci): enable noop on agentic workflows to stop IcM page spam" \ + --body "$(cat <<'EOF' +## Summary +- Sets `safe-outputs.noop: true` on all three agentic workflows so the agent can explicitly signal "nothing to do" instead of exiting silent. +- Sets `safe-outputs.report-failure-as-issue: false` so edge-case silent exits no longer file `[aw] ... failed` issues that page the IcM on-call rotation. +- Updates the `msdo-issue-assistant` prompt to call `noop` in its existing "don't respond" conditions and to recognise auto-generated `[aw]` failure issues. + +Fixes the false-positive failure loop documented in #247 and in [docs/superpowers/specs/2026-04-24-agentic-workflows-noop-fix-design.md](docs/superpowers/specs/2026-04-24-agentic-workflows-noop-fix-design.md). + +## Test plan +- [ ] `gh aw compile` recompiles all three workflows cleanly +- [ ] `msdo-issue-assistant` negative path: post a comment on an off-topic or `[aw]`-titled issue on the PR branch — no new `[aw] ... failed` issue filed, no comment posted +- [ ] `msdo-issue-assistant` positive path: open a test issue asking a real MSDO question — bot replies normally with wiki citations and `area:msdo-cli` label +- [ ] `ci-doctor` negative path: dispatch against a successful CI run — noop, no issue filed +- [ ] `msdo-breach-monitor` negative path: `workflow_dispatch` with no new CVEs — noop, no issue filed +EOF +)" +``` + +No `🤖 Generated with Claude Code` footer (per user preference). + +- [ ] **Step 3: Capture the PR URL** + +`gh pr create` prints the PR URL on success. Record it for the validation tasks below. + +--- + +## Task 10: Negative-path validation — `msdo-issue-assistant` + +**Files:** none (exercises the PR-branch workflow). + +- [ ] **Step 1: Trigger the bot against a known don't-respond case** + +Option A — post a comment on issue #247 (`[aw]`-titled, will exercise the new rule): + +```bash +gh issue comment 247 --repo microsoft/security-devops-action --body "test: verifying fix/agentic-workflows-noop — expect noop" +``` + +Option B — open a new test issue with clearly off-topic content, e.g. title "How do I deploy to AWS?" body "not MSDO-related, just testing". Close it after the run completes. + +Note: the workflow runs off whatever is merged on the default branch for new issues, **unless** gh-aw activation is configured to pick up the PR head. If the run still uses the current `main` version, either (a) merge first and validate post-merge, or (b) on a fork/test repo, push the branch and re-open the same test issue. For this repo, merging first is the likely path — log this as a deliberate choice in the PR review. + +- [ ] **Step 2: Observe the workflow run** + +Run: +```bash +gh run list --repo microsoft/security-devops-action --workflow "MSDO Issue Triage Assistant" --limit 5 +``` + +Expected: newest run's conclusion is `success`. Then inspect that specific run: + +```bash +gh run view --repo microsoft/security-devops-action --log | grep -E "noop|safe output|agent_output|failure" +``` + +Expected markers: +- Evidence of the `noop` handler firing (log line referencing `noop` or `handle_noop`). +- No `"Agent succeeded but produced no safe outputs"` line. +- No step that creates or comments on a `[aw] ... failed` issue. + +- [ ] **Step 3: Confirm no new `[aw]` issue was filed** + +Run: +```bash +gh issue list --repo microsoft/security-devops-action --search "[aw] MSDO Issue Triage Assistant failed" --state open --limit 5 +``` + +Expected: only the pre-existing #247 listed (or none, if it was closed). No newer `[aw]` issues. + +--- + +## Task 11: Positive-path validation — `msdo-issue-assistant` + +**Files:** none (exercises the workflow). + +- [ ] **Step 1: Open a test issue with a real MSDO question** + +Run: +```bash +gh issue create --repo microsoft/security-devops-action \ + --title "How do I pass --download-external-modules to checkov?" \ + --body "I want checkov (run via MSDO) to fetch external Terraform modules. How do I enable this?" +``` + +- [ ] **Step 2: Wait for the bot to respond (up to ~3 minutes), then inspect** + +Run: +```bash +gh issue view --repo microsoft/security-devops-action --comments +``` + +Expected: +- One new comment from the bot citing the wiki, mentioning `GDN_CHECKOV_DOWNLOADEXTERNALMODULES` or linking the Tool Configuration wiki page. +- The issue has the `area:msdo-cli` label applied. +- No `[aw] ... failed` issue created for this run. + +- [ ] **Step 3: Close the test issue** + +Run: +```bash +gh issue close --repo microsoft/security-devops-action --comment "test issue — closing" +``` + +--- + +## Task 12: Negative-path validation — `ci-doctor` + +**Files:** none. + +- [ ] **Step 1: Find a successful CI run on main** + +Run: +```bash +gh run list --repo microsoft/security-devops-action --workflow CI --branch main --status success --limit 3 +``` + +Expected: at least one green CI run. Record its run ID. + +- [ ] **Step 2: Manually dispatch `ci-doctor` against it (pre-merge, from the fix branch)** + +Run: +```bash +gh workflow run "CI Doctor" --repo microsoft/security-devops-action --ref fix/agentic-workflows-noop +``` + +Using `--ref fix/agentic-workflows-noop` makes GitHub pick up the updated `.lock.yml` on the PR branch, so this exercises the fix pre-merge. + +Wait ~1-2 minutes. Then: + +```bash +gh run list --repo microsoft/security-devops-action --workflow "CI Doctor" --limit 3 +``` + +Expected: newest run's conclusion is `success`. + +- [ ] **Step 3: Confirm no new `[aw]` or CI Doctor diagnostic issue was filed** + +Run: +```bash +gh issue list --repo microsoft/security-devops-action \ + --search "[aw] CI Doctor failed OR [CI Doctor]" \ + --state open --limit 5 +``` + +Expected: no newer entries than the pre-existing baseline. If CI Doctor found nothing new to diagnose (green run), it must have noop'd cleanly. + +--- + +## Task 13: Negative-path validation — `msdo-breach-monitor` + +**Files:** none. + +- [ ] **Step 1: Dispatch the monitor (pre-merge, from the fix branch)** + +Run: +```bash +gh workflow run "MSDO Toolchain Breach Monitor" --repo microsoft/security-devops-action --ref fix/agentic-workflows-noop +``` + +Using `--ref fix/agentic-workflows-noop` makes GitHub pick up the updated `.lock.yml` on the PR branch. + +- [ ] **Step 2: Observe the workflow run** + +Run: +```bash +gh run list --repo microsoft/security-devops-action --workflow "MSDO Toolchain Breach Monitor" --limit 3 +``` + +Expected: newest run's conclusion is `success`. + +Inspect the log for the noop call: + +```bash +gh run view --repo microsoft/security-devops-action --log | grep -E "noop|no new incidents|toolchain-alert" +``` + +Expected: evidence of a noop call (unless a genuine CVE in the window would produce a `toolchain-alert` issue — which is a positive-path outcome, not a failure). + +- [ ] **Step 3: Confirm no new `[aw] MSDO Toolchain Breach Monitor failed` issue was filed** + +Run: +```bash +gh issue list --repo microsoft/security-devops-action \ + --search "[aw] MSDO Toolchain Breach Monitor failed" \ + --state open --limit 5 +``` + +Expected: no newer entries. + +--- + +## Task 14: Complete the PR + +**Files:** none (GitHub). + +- [ ] **Step 1: Tick the PR's Test plan checkboxes** + +In the PR description, tick each checkbox that the validation tasks confirmed. + +Run: +```bash +gh pr view --repo microsoft/security-devops-action +``` + +Edit description via: +```bash +gh pr edit --repo microsoft/security-devops-action --body "" +``` + +- [ ] **Step 2: Hand off for human review and merge** + +The PR is now complete. Post a short summary comment and leave the merge to the repository maintainer per normal review process. The user will close #247 manually after merge. From 9bc950c9c170206bd5d627bbd479c21ccbdf14d0 Mon Sep 17 00:00:00 2001 From: Dima Birenbaum Date: Fri, 24 Apr 2026 10:05:33 +0300 Subject: [PATCH 3/5] fix(ci): enable noop on agentic workflows to stop IcM page spam --- .github/workflows/ci-doctor.lock.yml | 36 ++++++++++++++++--- .github/workflows/ci-doctor.md | 4 ++- .../workflows/msdo-breach-monitor.lock.yml | 36 ++++++++++++++++--- .github/workflows/msdo-breach-monitor.md | 4 ++- .../workflows/msdo-issue-assistant.lock.yml | 36 ++++++++++++++++--- .github/workflows/msdo-issue-assistant.md | 12 +++++-- 6 files changed, 111 insertions(+), 17 deletions(-) diff --git a/.github/workflows/ci-doctor.lock.yml b/.github/workflows/ci-doctor.lock.yml index 5ae5c95e..3082894f 100644 --- a/.github/workflows/ci-doctor.lock.yml +++ b/.github/workflows/ci-doctor.lock.yml @@ -21,7 +21,7 @@ # For more information: https://github.github.com/gh-aw/introduction/overview/ # # -# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"0de0b4ed23dc52687ceb1b6a9959941b552fe02d240da7798c789c86c45691f5","compiler_version":"v0.61.0","strict":true} +# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"fe5e4a384d919733e6f15f7f5c94214a34a50028234b069e4f38e860c2a37977","compiler_version":"v0.61.0","strict":true} name: "CI Doctor" "on": @@ -137,7 +137,7 @@ jobs: cat "/opt/gh-aw/prompts/safe_outputs_prompt.md" cat << 'GH_AW_PROMPT_EOF' - Tools: add_comment, create_issue, create_pull_request, add_labels, missing_tool, missing_data + Tools: add_comment, create_issue, create_pull_request, add_labels, missing_tool, missing_data, noop GH_AW_PROMPT_EOF cat "/opt/gh-aw/prompts/safe_outputs_create_pull_request.md" cat << 'GH_AW_PROMPT_EOF' @@ -335,7 +335,7 @@ jobs: mkdir -p /tmp/gh-aw/safeoutputs mkdir -p /tmp/gh-aw/mcp-logs/safeoutputs cat > /opt/gh-aw/safeoutputs/config.json << 'GH_AW_SAFE_OUTPUTS_CONFIG_EOF' - {"add_comment":{"max":1},"add_labels":{"allowed":["ci-failure","flaky-test","build-failure","dependency-issue","needs-maintainer"],"max":3},"create_issue":{"max":1},"create_pull_request":{"max":1},"missing_data":{},"missing_tool":{}} + {"add_comment":{"max":1},"add_labels":{"allowed":["ci-failure","flaky-test","build-failure","dependency-issue","needs-maintainer"],"max":3},"create_issue":{"max":1},"create_pull_request":{"max":1},"missing_data":{},"missing_tool":{},"noop":{"max":1}} GH_AW_SAFE_OUTPUTS_CONFIG_EOF - name: Write Safe Outputs Tools run: | @@ -504,6 +504,17 @@ jobs: "maxLength": 128 } } + }, + "noop": { + "defaultMax": 1, + "fields": { + "message": { + "required": true, + "type": "string", + "sanitize": true, + "maxLength": 65000 + } + } } } GH_AW_SAFE_OUTPUTS_VALIDATION_EOF @@ -931,6 +942,7 @@ jobs: group: "gh-aw-conclusion-ci-doctor" cancel-in-progress: false outputs: + noop_message: ${{ steps.noop.outputs.noop_message }} tools_reported: ${{ steps.missing_tool.outputs.tools_reported }} total_count: ${{ steps.missing_tool.outputs.total_count }} steps: @@ -951,6 +963,20 @@ jobs: mkdir -p /tmp/gh-aw/ find "/tmp/gh-aw/" -type f -print echo "GH_AW_AGENT_OUTPUT=/tmp/gh-aw/agent_output.json" >> "$GITHUB_ENV" + - name: Process No-Op Messages + id: noop + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 + env: + GH_AW_AGENT_OUTPUT: ${{ env.GH_AW_AGENT_OUTPUT }} + GH_AW_NOOP_MAX: "1" + GH_AW_WORKFLOW_NAME: "CI Doctor" + with: + github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} + script: | + const { setupGlobals } = require('/opt/gh-aw/actions/setup_globals.cjs'); + setupGlobals(core, github, context, exec, io); + const { main } = require('/opt/gh-aw/actions/noop.cjs'); + await main(); - name: Record Missing Tool id: missing_tool uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 @@ -979,7 +1005,7 @@ jobs: GH_AW_CODE_PUSH_FAILURE_ERRORS: ${{ needs.safe_outputs.outputs.code_push_failure_errors }} GH_AW_CODE_PUSH_FAILURE_COUNT: ${{ needs.safe_outputs.outputs.code_push_failure_count }} GH_AW_GROUP_REPORTS: "false" - GH_AW_FAILURE_REPORT_AS_ISSUE: "true" + GH_AW_FAILURE_REPORT_AS_ISSUE: "false" GH_AW_TIMEOUT_MINUTES: "20" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} @@ -996,6 +1022,8 @@ jobs: GH_AW_WORKFLOW_NAME: "CI Doctor" GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} GH_AW_AGENT_CONCLUSION: ${{ needs.agent.result }} + GH_AW_NOOP_MESSAGE: ${{ steps.noop.outputs.noop_message }} + GH_AW_NOOP_REPORT_AS_ISSUE: "false" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} script: | diff --git a/.github/workflows/ci-doctor.md b/.github/workflows/ci-doctor.md index 855cf14c..787512d2 100644 --- a/.github/workflows/ci-doctor.md +++ b/.github/workflows/ci-doctor.md @@ -30,7 +30,9 @@ tools: allowed: [] safe-outputs: - noop: false + noop: + report-as-issue: false + report-failure-as-issue: false create-issue: max: 1 add-labels: diff --git a/.github/workflows/msdo-breach-monitor.lock.yml b/.github/workflows/msdo-breach-monitor.lock.yml index d1208426..0a442ddc 100644 --- a/.github/workflows/msdo-breach-monitor.lock.yml +++ b/.github/workflows/msdo-breach-monitor.lock.yml @@ -21,7 +21,7 @@ # For more information: https://github.github.com/gh-aw/introduction/overview/ # # -# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"8aff8c918da79899626a7f1870cfdc2c94bba2f747ff53f3abfd9892ab61aaf7","compiler_version":"v0.61.0","strict":true} +# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"ccb5fde04f4c7256d8b743110bed1df58ef26d32a932c77575344a90eab7943a","compiler_version":"v0.61.0","strict":true} name: "MSDO Toolchain Breach Monitor" "on": @@ -123,7 +123,7 @@ jobs: cat "/opt/gh-aw/prompts/safe_outputs_prompt.md" cat << 'GH_AW_PROMPT_EOF' - Tools: create_issue, add_labels, missing_tool, missing_data + Tools: create_issue, add_labels, missing_tool, missing_data, noop The following GitHub context information is available for this workflow: @@ -313,7 +313,7 @@ jobs: mkdir -p /tmp/gh-aw/safeoutputs mkdir -p /tmp/gh-aw/mcp-logs/safeoutputs cat > /opt/gh-aw/safeoutputs/config.json << 'GH_AW_SAFE_OUTPUTS_CONFIG_EOF' - {"add_labels":{"allowed":["security-breach","supply-chain","toolchain-alert","critical","high","medium"],"max":3},"create_issue":{"max":1},"missing_data":{},"missing_tool":{}} + {"add_labels":{"allowed":["security-breach","supply-chain","toolchain-alert","critical","high","medium"],"max":3},"create_issue":{"max":1},"missing_data":{},"missing_tool":{},"noop":{"max":1}} GH_AW_SAFE_OUTPUTS_CONFIG_EOF - name: Write Safe Outputs Tools run: | @@ -426,6 +426,17 @@ jobs: "maxLength": 128 } } + }, + "noop": { + "defaultMax": 1, + "fields": { + "message": { + "required": true, + "type": "string", + "sanitize": true, + "maxLength": 65000 + } + } } } GH_AW_SAFE_OUTPUTS_VALIDATION_EOF @@ -851,6 +862,7 @@ jobs: group: "gh-aw-conclusion-msdo-breach-monitor" cancel-in-progress: false outputs: + noop_message: ${{ steps.noop.outputs.noop_message }} tools_reported: ${{ steps.missing_tool.outputs.tools_reported }} total_count: ${{ steps.missing_tool.outputs.total_count }} steps: @@ -871,6 +883,20 @@ jobs: mkdir -p /tmp/gh-aw/ find "/tmp/gh-aw/" -type f -print echo "GH_AW_AGENT_OUTPUT=/tmp/gh-aw/agent_output.json" >> "$GITHUB_ENV" + - name: Process No-Op Messages + id: noop + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 + env: + GH_AW_AGENT_OUTPUT: ${{ env.GH_AW_AGENT_OUTPUT }} + GH_AW_NOOP_MAX: "1" + GH_AW_WORKFLOW_NAME: "MSDO Toolchain Breach Monitor" + with: + github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} + script: | + const { setupGlobals } = require('/opt/gh-aw/actions/setup_globals.cjs'); + setupGlobals(core, github, context, exec, io); + const { main } = require('/opt/gh-aw/actions/noop.cjs'); + await main(); - name: Record Missing Tool id: missing_tool uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 @@ -897,7 +923,7 @@ jobs: GH_AW_CHECKOUT_PR_SUCCESS: ${{ needs.agent.outputs.checkout_pr_success }} GH_AW_INFERENCE_ACCESS_ERROR: ${{ needs.agent.outputs.inference_access_error }} GH_AW_GROUP_REPORTS: "false" - GH_AW_FAILURE_REPORT_AS_ISSUE: "true" + GH_AW_FAILURE_REPORT_AS_ISSUE: "false" GH_AW_TIMEOUT_MINUTES: "20" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} @@ -914,6 +940,8 @@ jobs: GH_AW_WORKFLOW_NAME: "MSDO Toolchain Breach Monitor" GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} GH_AW_AGENT_CONCLUSION: ${{ needs.agent.result }} + GH_AW_NOOP_MESSAGE: ${{ steps.noop.outputs.noop_message }} + GH_AW_NOOP_REPORT_AS_ISSUE: "false" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} script: | diff --git a/.github/workflows/msdo-breach-monitor.md b/.github/workflows/msdo-breach-monitor.md index 61c89a05..6b2b4a97 100644 --- a/.github/workflows/msdo-breach-monitor.md +++ b/.github/workflows/msdo-breach-monitor.md @@ -39,7 +39,9 @@ tools: - registry.npmjs.org safe-outputs: - noop: false + noop: + report-as-issue: false + report-failure-as-issue: false create-issue: max: 1 add-labels: diff --git a/.github/workflows/msdo-issue-assistant.lock.yml b/.github/workflows/msdo-issue-assistant.lock.yml index 34c14261..54e329ab 100644 --- a/.github/workflows/msdo-issue-assistant.lock.yml +++ b/.github/workflows/msdo-issue-assistant.lock.yml @@ -21,7 +21,7 @@ # For more information: https://github.github.com/gh-aw/introduction/overview/ # # -# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"b9853605bc6fd41a4d81ec4728106d1ffdc01e2dbcf460d6aaea1620c94a3367","compiler_version":"v0.61.0","strict":true} +# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"1dced2a7773143b01044e0ace3fa6b13dc03efadb64e0bad57014801b6e3fa94","compiler_version":"v0.61.0","strict":true} name: "MSDO Issue Triage Assistant" "on": @@ -141,7 +141,7 @@ jobs: cat "/opt/gh-aw/prompts/safe_outputs_prompt.md" cat << 'GH_AW_PROMPT_EOF' - Tools: add_comment, add_labels, missing_tool, missing_data + Tools: add_comment, add_labels, missing_tool, missing_data, noop The following GitHub context information is available for this workflow: @@ -336,7 +336,7 @@ jobs: mkdir -p /tmp/gh-aw/safeoutputs mkdir -p /tmp/gh-aw/mcp-logs/safeoutputs cat > /opt/gh-aw/safeoutputs/config.json << 'GH_AW_SAFE_OUTPUTS_CONFIG_EOF' - {"add_comment":{"max":4},"add_labels":{"allowed":["type:bug","type:feature","type:docs","type:question","type:security","type:maintenance","status:triage","status:waiting-on-author","status:repro-needed","status:team-review","area:action","area:msdo-cli","area:ci","area:container-mapping"],"max":3},"missing_data":{},"missing_tool":{}} + {"add_comment":{"max":4},"add_labels":{"allowed":["type:bug","type:feature","type:docs","type:question","type:security","type:maintenance","status:triage","status:waiting-on-author","status:repro-needed","status:team-review","area:action","area:msdo-cli","area:ci","area:container-mapping"],"max":3},"missing_data":{},"missing_tool":{},"noop":{"max":1}} GH_AW_SAFE_OUTPUTS_CONFIG_EOF - name: Write Safe Outputs Tools run: | @@ -434,6 +434,17 @@ jobs: "maxLength": 128 } } + }, + "noop": { + "defaultMax": 1, + "fields": { + "message": { + "required": true, + "type": "string", + "sanitize": true, + "maxLength": 65000 + } + } } } GH_AW_SAFE_OUTPUTS_VALIDATION_EOF @@ -860,6 +871,7 @@ jobs: group: "gh-aw-conclusion-msdo-issue-assistant" cancel-in-progress: false outputs: + noop_message: ${{ steps.noop.outputs.noop_message }} tools_reported: ${{ steps.missing_tool.outputs.tools_reported }} total_count: ${{ steps.missing_tool.outputs.total_count }} steps: @@ -880,6 +892,20 @@ jobs: mkdir -p /tmp/gh-aw/ find "/tmp/gh-aw/" -type f -print echo "GH_AW_AGENT_OUTPUT=/tmp/gh-aw/agent_output.json" >> "$GITHUB_ENV" + - name: Process No-Op Messages + id: noop + uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 + env: + GH_AW_AGENT_OUTPUT: ${{ env.GH_AW_AGENT_OUTPUT }} + GH_AW_NOOP_MAX: "1" + GH_AW_WORKFLOW_NAME: "MSDO Issue Triage Assistant" + with: + github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} + script: | + const { setupGlobals } = require('/opt/gh-aw/actions/setup_globals.cjs'); + setupGlobals(core, github, context, exec, io); + const { main } = require('/opt/gh-aw/actions/noop.cjs'); + await main(); - name: Record Missing Tool id: missing_tool uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 @@ -906,7 +932,7 @@ jobs: GH_AW_CHECKOUT_PR_SUCCESS: ${{ needs.agent.outputs.checkout_pr_success }} GH_AW_INFERENCE_ACCESS_ERROR: ${{ needs.agent.outputs.inference_access_error }} GH_AW_GROUP_REPORTS: "false" - GH_AW_FAILURE_REPORT_AS_ISSUE: "true" + GH_AW_FAILURE_REPORT_AS_ISSUE: "false" GH_AW_TIMEOUT_MINUTES: "20" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} @@ -923,6 +949,8 @@ jobs: GH_AW_WORKFLOW_NAME: "MSDO Issue Triage Assistant" GH_AW_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} GH_AW_AGENT_CONCLUSION: ${{ needs.agent.result }} + GH_AW_NOOP_MESSAGE: ${{ steps.noop.outputs.noop_message }} + GH_AW_NOOP_REPORT_AS_ISSUE: "false" with: github-token: ${{ secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN }} script: | diff --git a/.github/workflows/msdo-issue-assistant.md b/.github/workflows/msdo-issue-assistant.md index 18cd4ca5..b8f331f2 100644 --- a/.github/workflows/msdo-issue-assistant.md +++ b/.github/workflows/msdo-issue-assistant.md @@ -30,7 +30,9 @@ tools: - raw.githubusercontent.com safe-outputs: - noop: false + noop: + report-as-issue: false + report-failure-as-issue: false add-comment: max: 4 add-labels: @@ -179,11 +181,12 @@ Keep responses: - docs.microsoft.com - aka.ms 3. **Stay on topic** - Only respond to issues related to MSDO, security-devops-action, or the supported security tools. If an issue is unrelated (e.g. general GitHub Actions questions, unrelated security tools, off-topic discussions), do not respond. -4. **Don't respond** if: +4. **Call `noop` instead of staying silent** when any of these apply. Pass a one-line reason so the decision is auditable: - The issue is not related to MSDO or security-devops-action + - The issue title starts with `[aw]` or is labeled `agentic-workflows` (auto-generated failure reports, not user issues) - The issue is closed - The commenter is not the issue author (unless it's a new issue) - - You've already responded twice and there is no new technical information in the latest user message + - You have already responded twice and there is no new technical information in the latest user message - The issue has a `status:team-review` label (a maintainer is handling it) 5. **Be honest** - if you don't know something, say so and suggest checking the wiki or waiting for a maintainer @@ -211,3 +214,6 @@ Keep responses: **Non-author comment on existing issue:** A third party comments "I have the same problem." → Do not respond. The commenter is not the issue author. + +**Workflow failure issue (auto-generated):** Title starts with `[aw]` (e.g. "[aw] MSDO Issue Triage Assistant failed") or labeled `agentic-workflows`. +→ Call `noop` with reason "auto-generated failure report, not a user issue". From 58e0e1947023457c7a0d0480eb9cba5dc033b484 Mon Sep 17 00:00:00 2001 From: Dima Birenbaum Date: Fri, 24 Apr 2026 10:39:57 +0300 Subject: [PATCH 4/5] fix(ci): make msdo-issue-assistant prompt consistently call noop --- .github/workflows/msdo-issue-assistant.md | 10 ++--- .../2026-04-24-agentic-workflows-noop-fix.md | 44 +++++++++++-------- 2 files changed, 31 insertions(+), 23 deletions(-) diff --git a/.github/workflows/msdo-issue-assistant.md b/.github/workflows/msdo-issue-assistant.md index b8f331f2..c8a6d8bb 100644 --- a/.github/workflows/msdo-issue-assistant.md +++ b/.github/workflows/msdo-issue-assistant.md @@ -180,7 +180,7 @@ Keep responses: - learn.microsoft.com - docs.microsoft.com - aka.ms -3. **Stay on topic** - Only respond to issues related to MSDO, security-devops-action, or the supported security tools. If an issue is unrelated (e.g. general GitHub Actions questions, unrelated security tools, off-topic discussions), do not respond. +3. **Stay on topic** - Only respond to issues related to MSDO, security-devops-action, or the supported security tools. If an issue is unrelated (e.g. general GitHub Actions questions, unrelated security tools, off-topic discussions), call `noop` with a reason — see rule 4. 4. **Call `noop` instead of staying silent** when any of these apply. Pass a one-line reason so the decision is auditable: - The issue is not related to MSDO or security-devops-action - The issue title starts with `[aw]` or is labeled `agentic-workflows` (auto-generated failure reports, not user issues) @@ -204,16 +204,16 @@ Keep responses: ## Do NOT Respond Examples **Off-topic issue:** "How do I set up GitHub Actions for deploying to AWS?" -→ Do not respond. This is unrelated to MSDO. +→ Call `noop` with reason "off-topic — unrelated to MSDO". **Issue labeled `status:team-review`:** Any issue with this label. -→ Do not respond. A maintainer is already handling it. +→ Call `noop` with reason "status:team-review — maintainer is handling it". **Repeated comments with no new info:** User says "Any update?" or "bump" after you already responded. -→ Do not respond. No new technical information to act on. +→ Call `noop` with reason "no new technical information since prior response". **Non-author comment on existing issue:** A third party comments "I have the same problem." -→ Do not respond. The commenter is not the issue author. +→ Call `noop` with reason "commenter is not the issue author". **Workflow failure issue (auto-generated):** Title starts with `[aw]` (e.g. "[aw] MSDO Issue Triage Assistant failed") or labeled `agentic-workflows`. → Call `noop` with reason "auto-generated failure report, not a user issue". diff --git a/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md b/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md index b07f3d92..089421cc 100644 --- a/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md +++ b/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md @@ -4,7 +4,9 @@ **Goal:** Stop the three gh-aw agentic workflows from filing false-positive `[aw] ... failed` issues that page the on-call IcM rotation. -**Architecture:** For all three agentic workflows, set `safe-outputs.noop: true` (lets the agent explicitly signal "nothing to do") and `safe-outputs.report-failure-as-issue: false` (blocks the auto-filed failure issue even when no output is produced). Update the `msdo-issue-assistant` prompt so its existing "Don't respond if" rules now direct the agent to call the `noop` tool explicitly. Regenerate the three `.lock.yml` files with `gh aw compile` and ship everything in one PR. +**Architecture:** For all three agentic workflows, enable the `noop` safe output with `report-as-issue: false` (lets the agent explicitly signal "nothing to do" without itself filing an issue) and set `safe-outputs.report-failure-as-issue: false` (blocks the auto-filed failure issue even when no output is produced). Update the `msdo-issue-assistant` prompt so its "don't respond" rules now direct the agent to call the `noop` tool explicitly. Regenerate the three `.lock.yml` files with `gh aw compile` and ship everything in one PR. + +> **Note on syntax:** gh-aw v0.61.0 rejects `noop: true` as a boolean. The correct YAML shape is an object: `noop:\n report-as-issue: false`. All YAML blocks below use that shape. If you see `noop: true` anywhere, the compile will fail with "value must be false. Expected format: {...}". **Tech Stack:** GitHub Actions, gh-aw CLI v0.61.0, YAML, Markdown prompts. @@ -77,7 +79,8 @@ With: ```yaml safe-outputs: - noop: true + noop: + report-as-issue: false report-failure-as-issue: false create-issue: max: 1 @@ -87,19 +90,20 @@ safe-outputs: create-pull-request: null ``` -The only changes are `noop: false` → `noop: true` and inserting `report-failure-as-issue: false` as the second key. +Changes: `noop: false` replaced with the `noop:\n report-as-issue: false` object form (enables the noop tool without having it file its own issue), plus `report-failure-as-issue: false` inserted as the next key. - [ ] **Step 2: Verify the edit** Run: ```bash -grep -nE "noop|report-failure-as-issue" .github/workflows/ci-doctor.md +grep -nE "noop|report-failure-as-issue|report-as-issue" .github/workflows/ci-doctor.md ``` Expected output: ``` -33: noop: true -34: report-failure-as-issue: false +33: noop: +34: report-as-issue: false +35: report-failure-as-issue: false ``` --- @@ -126,7 +130,8 @@ With: ```yaml safe-outputs: - noop: true + noop: + report-as-issue: false report-failure-as-issue: false create-issue: max: 1 @@ -134,19 +139,20 @@ safe-outputs: allowed: [security-breach, supply-chain, toolchain-alert, critical, high, medium] ``` -Only changes are `noop: false` → `noop: true` and inserting `report-failure-as-issue: false`. +Changes: `noop: false` replaced with the `noop:\n report-as-issue: false` object form, plus `report-failure-as-issue: false` inserted as the next key. - [ ] **Step 2: Verify the edit** Run: ```bash -grep -nE "noop|report-failure-as-issue" .github/workflows/msdo-breach-monitor.md +grep -nE "noop|report-failure-as-issue|report-as-issue" .github/workflows/msdo-breach-monitor.md ``` Expected output: ``` -42: noop: true -43: report-failure-as-issue: false +42: noop: +43: report-as-issue: false +44: report-failure-as-issue: false ``` --- @@ -173,7 +179,8 @@ With: ```yaml safe-outputs: - noop: true + noop: + report-as-issue: false report-failure-as-issue: false add-comment: max: 4 @@ -181,19 +188,20 @@ safe-outputs: allowed: ["type:bug", "type:feature", "type:docs", "type:question", "type:security", "type:maintenance", "status:triage", "status:waiting-on-author", "status:repro-needed", "status:team-review", "area:action", "area:msdo-cli", "area:ci", "area:container-mapping"] ``` -Only changes are `noop: false` → `noop: true` and inserting `report-failure-as-issue: false`. +Changes: `noop: false` replaced with the `noop:\n report-as-issue: false` object form, plus `report-failure-as-issue: false` inserted as the next key. - [ ] **Step 2: Verify the edit** Run: ```bash -grep -nE "noop|report-failure-as-issue" .github/workflows/msdo-issue-assistant.md +grep -nE "noop|report-failure-as-issue|report-as-issue" .github/workflows/msdo-issue-assistant.md ``` -Expected output: +Expected output (the first three lines — additional matches will appear later in the file inside the prompt text): ``` -33: noop: true -34: report-failure-as-issue: false +33: noop: +34: report-as-issue: false +35: report-failure-as-issue: false ``` --- @@ -381,7 +389,7 @@ gh pr create \ --title "fix(ci): enable noop on agentic workflows to stop IcM page spam" \ --body "$(cat <<'EOF' ## Summary -- Sets `safe-outputs.noop: true` on all three agentic workflows so the agent can explicitly signal "nothing to do" instead of exiting silent. +- Enables `safe-outputs.noop` (with `report-as-issue: false`) on all three agentic workflows so the agent can explicitly signal "nothing to do" instead of exiting silent. - Sets `safe-outputs.report-failure-as-issue: false` so edge-case silent exits no longer file `[aw] ... failed` issues that page the IcM on-call rotation. - Updates the `msdo-issue-assistant` prompt to call `noop` in its existing "don't respond" conditions and to recognise auto-generated `[aw]` failure issues. From 7c2112b65197d69394401b7d2f6af061aac3b1fc Mon Sep 17 00:00:00 2001 From: Dima Birenbaum Date: Fri, 24 Apr 2026 10:46:07 +0300 Subject: [PATCH 5/5] docs(ci): document v9.0.0 SHA restoration and rename noop examples heading --- .github/workflows/ci-doctor.lock.yml | 2 +- .github/workflows/ci-doctor.md | 6 ++++++ .github/workflows/msdo-breach-monitor.lock.yml | 2 +- .github/workflows/msdo-breach-monitor.md | 6 ++++++ .github/workflows/msdo-issue-assistant.lock.yml | 2 +- .github/workflows/msdo-issue-assistant.md | 8 +++++++- .../plans/2026-04-24-agentic-workflows-noop-fix.md | 4 ++++ 7 files changed, 26 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci-doctor.lock.yml b/.github/workflows/ci-doctor.lock.yml index 3082894f..1f0fab42 100644 --- a/.github/workflows/ci-doctor.lock.yml +++ b/.github/workflows/ci-doctor.lock.yml @@ -21,7 +21,7 @@ # For more information: https://github.github.com/gh-aw/introduction/overview/ # # -# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"fe5e4a384d919733e6f15f7f5c94214a34a50028234b069e4f38e860c2a37977","compiler_version":"v0.61.0","strict":true} +# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"e0a10012ec11f9360eb65d497093ec0ba53c0a1f14cfbb5e21200dcc08055474","compiler_version":"v0.61.0","strict":true} name: "CI Doctor" "on": diff --git a/.github/workflows/ci-doctor.md b/.github/workflows/ci-doctor.md index 787512d2..e4772f77 100644 --- a/.github/workflows/ci-doctor.md +++ b/.github/workflows/ci-doctor.md @@ -1,6 +1,12 @@ --- # CI Doctor - GitHub Agentic Workflow # Investigates failed CI workflows and opens diagnostic issues +# +# MAINTENANCE NOTE: after running `gh aw compile` with gh-aw v0.61.0, verify +# that the `actions/github-script` SHA in the generated .lock.yml stays pinned +# to v9.0.0 (`3a2844b7e9c422d3c10d287c895573f7108da1b3`). v0.61.0's bundled +# scaffolding emits the older v8 SHA and would silently revert PR #244. See +# PR #252 for context. on: workflow_run: diff --git a/.github/workflows/msdo-breach-monitor.lock.yml b/.github/workflows/msdo-breach-monitor.lock.yml index 0a442ddc..948c0df6 100644 --- a/.github/workflows/msdo-breach-monitor.lock.yml +++ b/.github/workflows/msdo-breach-monitor.lock.yml @@ -21,7 +21,7 @@ # For more information: https://github.github.com/gh-aw/introduction/overview/ # # -# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"ccb5fde04f4c7256d8b743110bed1df58ef26d32a932c77575344a90eab7943a","compiler_version":"v0.61.0","strict":true} +# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"73ddd2b5a2fc15ff120245519bd10f342dd3d1a0925df30be6453378664b4c29","compiler_version":"v0.61.0","strict":true} name: "MSDO Toolchain Breach Monitor" "on": diff --git a/.github/workflows/msdo-breach-monitor.md b/.github/workflows/msdo-breach-monitor.md index 6b2b4a97..2607ea3c 100644 --- a/.github/workflows/msdo-breach-monitor.md +++ b/.github/workflows/msdo-breach-monitor.md @@ -1,6 +1,12 @@ --- # MSDO Toolchain Breach Monitor - GitHub Agentic Workflow # Nightly supply chain breach monitor for MSDO toolchain dependencies +# +# MAINTENANCE NOTE: after running `gh aw compile` with gh-aw v0.61.0, verify +# that the `actions/github-script` SHA in the generated .lock.yml stays pinned +# to v9.0.0 (`3a2844b7e9c422d3c10d287c895573f7108da1b3`). v0.61.0's bundled +# scaffolding emits the older v8 SHA and would silently revert PR #244. See +# PR #252 for context. on: workflow_dispatch: diff --git a/.github/workflows/msdo-issue-assistant.lock.yml b/.github/workflows/msdo-issue-assistant.lock.yml index 54e329ab..e412326d 100644 --- a/.github/workflows/msdo-issue-assistant.lock.yml +++ b/.github/workflows/msdo-issue-assistant.lock.yml @@ -21,7 +21,7 @@ # For more information: https://github.github.com/gh-aw/introduction/overview/ # # -# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"1dced2a7773143b01044e0ace3fa6b13dc03efadb64e0bad57014801b6e3fa94","compiler_version":"v0.61.0","strict":true} +# gh-aw-metadata: {"schema_version":"v2","frontmatter_hash":"ad862ac3404b6a5b9235e75266770d3df954d43cd43d766661be878e414e622b","compiler_version":"v0.61.0","strict":true} name: "MSDO Issue Triage Assistant" "on": diff --git a/.github/workflows/msdo-issue-assistant.md b/.github/workflows/msdo-issue-assistant.md index c8a6d8bb..8f5ba757 100644 --- a/.github/workflows/msdo-issue-assistant.md +++ b/.github/workflows/msdo-issue-assistant.md @@ -1,6 +1,12 @@ --- # MSDO Issue Assistant - GitHub Agentic Workflow # Automatically triage and respond to issues using wiki knowledge +# +# MAINTENANCE NOTE: after running `gh aw compile` with gh-aw v0.61.0, verify +# that the `actions/github-script` SHA in the generated .lock.yml stays pinned +# to v9.0.0 (`3a2844b7e9c422d3c10d287c895573f7108da1b3`). v0.61.0's bundled +# scaffolding emits the older v8 SHA and would silently revert PR #244. See +# PR #252 for context. on: issues: @@ -201,7 +207,7 @@ Keep responses: **User reports:** "Trivy is failing with container image not found" **Response:** This error typically occurs when Docker isn't available. Trivy requires Docker for container scanning. Please ensure you have `docker/setup-buildx-action@v3` in your workflow before the MSDO action. Can you share your workflow YAML so I can help verify the configuration? -## Do NOT Respond Examples +## Noop Examples **Off-topic issue:** "How do I set up GitHub Actions for deploying to AWS?" → Call `noop` with reason "off-topic — unrelated to MSDO". diff --git a/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md b/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md index 089421cc..b01b180f 100644 --- a/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md +++ b/docs/superpowers/plans/2026-04-24-agentic-workflows-noop-fix.md @@ -8,6 +8,10 @@ > **Note on syntax:** gh-aw v0.61.0 rejects `noop: true` as a boolean. The correct YAML shape is an object: `noop:\n report-as-issue: false`. All YAML blocks below use that shape. If you see `noop: true` anywhere, the compile will fail with "value must be false. Expected format: {...}". +> **Post-implementation addenda (for traceability):** +> - The `gh aw compile` step with v0.61.0 silently downgrades `actions/github-script` from v9.0.0 (per PR #244) back to v8. The v9.0.0 SHA (`3a2844b7e9c422d3c10d287c895573f7108da1b3`) was restored via sed after compile. A maintenance note to this effect is embedded as a YAML comment at the top of each `.md` source file. +> - A second commit (after the initial review) extended the `msdo-issue-assistant` prompt edits beyond what Tasks 5-6 specified: rule 3 was updated to redirect to rule 4, and the four pre-existing "Do NOT Respond Examples" arrows were changed from "→ Do not respond" to "→ Call `noop` with reason ...". The `## Do NOT Respond Examples` heading was also renamed to `## Noop Examples`. These changes eliminated an internal contradiction between rule 3 and rule 4 and made the examples match the new noop-centric behaviour. They are not reflected in the task descriptions below. + **Tech Stack:** GitHub Actions, gh-aw CLI v0.61.0, YAML, Markdown prompts. **Spec:** [docs/superpowers/specs/2026-04-24-agentic-workflows-noop-fix-design.md](../specs/2026-04-24-agentic-workflows-noop-fix-design.md)