From cad946576000054a41b3cc75e670101ca45a81e0 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Thu, 16 Jul 2026 10:25:11 -0500 Subject: [PATCH 1/4] fix: select latest reused sweep artifacts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：修复复用扫描产物的选择逻辑，只保留每个逻辑基准测试最新上传的产物。 --- .github/workflows/run-sweep.yml | 42 +++++++++++++++++++-------------- 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/.github/workflows/run-sweep.yml b/.github/workflows/run-sweep.yml index 53a1739ad..50cc89158 100644 --- a/.github/workflows/run-sweep.yml +++ b/.github/workflows/run-sweep.yml @@ -741,25 +741,31 @@ jobs: GH_TOKEN: ${{ secrets.REPO_PAT || github.token }} SOURCE_RUN_ID: ${{ needs.setup.outputs.reuse-source-run-id }} run: | - gh run download "$SOURCE_RUN_ID" \ - --repo "${{ github.repository }}" \ - -D source-artifacts + mkdir -p source-artifacts - # Keep only artifacts consumed by the official ingest path. - # The merge run uploads its own changelog metadata; reusable - # benchmark/eval rows are attributed to the source PR sweep. - rm -rf source-artifacts/changelog-metadata - for artifact_dir in source-artifacts/*; do - [ -e "$artifact_dir" ] || continue - name=$(basename "$artifact_dir") - case "$name" in - results_bmk|eval_results_all|run-stats|bmk_*|eval_*|agentic_*|server_logs_*|multinode_server_logs_*) - ;; - *) - rm -rf "$artifact_dir" - ;; - esac - done + # Reruns share one run ID. Keep the newest upload for each + # runner-independent artifact name, matching normal ingest. + while IFS=$'\t' read -r name url; do + mkdir -p "source-artifacts/$name" + gh api "$url" > artifact.zip + unzip -q artifact.zip -d "source-artifacts/$name" + done < <( + gh api \ + "repos/${{ github.repository }}/actions/runs/$SOURCE_RUN_ID/artifacts" \ + --paginate | + jq -rs ' + [ + .[].artifacts[] + | select(.expired | not) + | select(.name | test("^(results_bmk$|eval_results_all$|run-stats$|bmk_|eval_|agentic_|server_logs_|multinode_server_logs_)")) + ] + | group_by(.name | sub("_[A-Za-z][A-Za-z0-9.-]*_[0-9]+$"; "")) + | map(max_by(.created_at))[] + | [.name, .archive_download_url] + | @tsv + ' + ) + rm -f artifact.zip mkdir -p source-artifacts/reused-ingest-metadata cat > source-artifacts/reused-ingest-metadata/reuse_source_run.json <<'JSON' From 2381e8283155f8f8e5e740716e2297a33f4b6406 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Thu, 16 Jul 2026 11:37:45 -0500 Subject: [PATCH 2/4] refactor: dispatch reusable sweep sources directly MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove the reusable artifact relay and send source and merge run IDs directly to InferenceX-app. Simplify manual recovery to the same two-ID dispatch and add structural workflow tests. 移除可复用工件中转流程，直接将源运行 ID 和合并运行 ID 发送到 InferenceX-app。手动恢复流程也简化为相同的双 ID 分发，并新增工作流结构测试。 --- .github/workflows/README.md | 13 +- .github/workflows/recover-reused-ingest.yml | 150 +----------------- .github/workflows/run-sweep.yml | 92 +---------- .../test_run_sweep_gating.py | 30 ++++ 4 files changed, 47 insertions(+), 238 deletions(-) diff --git a/.github/workflows/README.md b/.github/workflows/README.md index fdcdcad27..efd6e7c0d 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -186,12 +186,13 @@ Remove and re-add the sweep label to force one. It merges `main`, preserves changelog bytes, fixes an appended `XXX` PR link, pushes a synchronization commit, waits for checks, then merges. -The main run verifies the source, validates and uploads its ingest artifacts, -then ingests them with merge-run changelog metadata. Source coverage is -authoritative, so later matrix/eval policy changes do not invalidate reuse. -Validation rejects duplicate fixed rows, missing run stats, inconsistent -agentic artifacts, malformed eval metadata, and raw/aggregate eval mismatches. -Batched evals use only `completed_eval_concs`. +The main run passes the selected source run ID and its own merge run ID directly +to InferenceX-app. The app downloads source artifacts, keeps the newest valid +artifact family for each logical benchmark point, validates the selected set, +and ingests it with changelog metadata from the merge run. Benchmark rows and +public links retain source-run provenance. Source coverage is authoritative, so +later matrix/eval policy changes do not invalidate reuse. Batched evals use only +`completed_eval_concs`. Reuse fails closed when authorized but ineligible or invalid; without authorization, `main` runs the normal full sweep. diff --git a/.github/workflows/recover-reused-ingest.yml b/.github/workflows/recover-reused-ingest.yml index 73c924d37..2bcdba781 100644 --- a/.github/workflows/recover-reused-ingest.yml +++ b/.github/workflows/recover-reused-ingest.yml @@ -4,158 +4,16 @@ on: workflow_dispatch: inputs: source-run-id: - description: Source PR sweep run ID - required: true - type: string - source-run-attempt: - description: Source PR sweep attempt - required: true - default: "1" - type: string - source-pr-number: - description: Source PR number - required: true - type: string - source-head-sha: - description: Source sweep commit SHA + description: Source PR sweep run ID containing benchmark artifacts required: true type: string merge-run-id: - description: Original failed merge run ID + description: Main publication run ID containing changelog metadata required: true type: string - merge-run-attempt: - description: Original failed merge run attempt - required: true - default: "1" - type: string jobs: - package-reused-artifacts: - runs-on: blacksmith-4vcpu-ubuntu-2404 - permissions: - contents: read - steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - - - name: Download source artifacts - env: - GH_TOKEN: ${{ secrets.REPO_PAT || github.token }} - SOURCE_RUN_ID: ${{ inputs.source-run-id }} - ARTIFACTS_PATH: ${{ github.workspace }}/source-artifacts - run: | - mkdir -p "$ARTIFACTS_PATH" - - download_artifact() { - local name="$1" - local url="$2" - local zip_path="$RUNNER_TEMP/artifact.zip" - - echo "Downloading artifact: $name" - for attempt in 1 2 3; do - if gh api "$url" > "$zip_path"; then - mkdir -p "$ARTIFACTS_PATH/$name" - if unzip -oq "$zip_path" -d "$ARTIFACTS_PATH/$name"; then - rm -f "$zip_path" - return 0 - fi - rm -rf "${ARTIFACTS_PATH:?}/$name" - fi - rm -f "$zip_path" - echo " Attempt $attempt/3 failed, retrying in ${attempt}s..." - sleep "$attempt" - done - return 1 - } - - gh api "repos/${{ github.repository }}/actions/runs/${SOURCE_RUN_ID}/artifacts?per_page=100" --paginate --slurp \ - | jq -r ' - [.[].artifacts[] - | select(.name != "changelog-metadata") - | . + {logical_name: (.name | sub("_[A-Za-z][A-Za-z0-9.-]*_[0-9]+$"; ""))} - ] - | group_by(.logical_name) - | map(sort_by(.created_at) | last)[] - | "\(.name)\t\(.archive_download_url)"' \ - | while IFS=$'\t' read -r name url; do - if ! download_artifact "$name" "$url"; then - echo "::error::Failed to download source artifact: $name" - exit 1 - fi - done - - if [ -z "$(find "$ARTIFACTS_PATH" -mindepth 1 -maxdepth 1 -print -quit)" ]; then - echo "::error::No source artifacts downloaded from run $SOURCE_RUN_ID" - exit 1 - fi - - - name: Record reuse provenance - env: - ARTIFACTS_PATH: ${{ github.workspace }}/source-artifacts - SOURCE_RUN_ID: ${{ inputs.source-run-id }} - SOURCE_RUN_ATTEMPT: ${{ inputs.source-run-attempt }} - SOURCE_PR_NUMBER: ${{ inputs.source-pr-number }} - SOURCE_HEAD_SHA: ${{ inputs.source-head-sha }} - MERGE_RUN_ID: ${{ inputs.merge-run-id }} - MERGE_RUN_ATTEMPT: ${{ inputs.merge-run-attempt }} - run: | - mkdir -p "$ARTIFACTS_PATH/reused-ingest-metadata" - jq -n \ - --arg source_run_id "$SOURCE_RUN_ID" \ - --arg source_run_attempt "$SOURCE_RUN_ATTEMPT" \ - --arg source_run_url "${{ github.server_url }}/${{ github.repository }}/actions/runs/$SOURCE_RUN_ID" \ - --arg source_pr_number "$SOURCE_PR_NUMBER" \ - --arg source_head_sha "$SOURCE_HEAD_SHA" \ - --arg ingest_run_id "$MERGE_RUN_ID" \ - --arg ingest_run_attempt "$MERGE_RUN_ATTEMPT" \ - --arg ingest_run_url "${{ github.server_url }}/${{ github.repository }}/actions/runs/$MERGE_RUN_ID" \ - '{ - source_run_id: $source_run_id, - source_run_attempt: $source_run_attempt, - source_run_url: $source_run_url, - source_pr_number: $source_pr_number, - source_head_sha: $source_head_sha, - ingest_run_id: $ingest_run_id, - ingest_run_attempt: $ingest_run_attempt, - ingest_run_url: $ingest_run_url - }' > "$ARTIFACTS_PATH/reused-ingest-metadata/reuse_source_run.json" - - - name: Validate reusable artifacts - run: | - python3 utils/validate_reusable_sweep_artifacts.py \ - --artifacts-dir "${{ github.workspace }}/source-artifacts" - - - name: Download original merge changelog - env: - GH_TOKEN: ${{ secrets.REPO_PAT || github.token }} - MERGE_RUN_ID: ${{ inputs.merge-run-id }} - CHANGELOG_PATH: ${{ github.workspace }}/merge-changelog - run: | - url=$(gh api "repos/${{ github.repository }}/actions/runs/${MERGE_RUN_ID}/artifacts?per_page=100" --paginate --slurp \ - | jq -r '[.[].artifacts[] | select(.name == "changelog-metadata")] | sort_by(.created_at) | last | .archive_download_url // empty') - if [ -z "$url" ]; then - echo "::error::No changelog-metadata artifact found on run $MERGE_RUN_ID" - exit 1 - fi - mkdir -p "$CHANGELOG_PATH" - gh api "$url" > "$RUNNER_TEMP/changelog.zip" - unzip -oq "$RUNNER_TEMP/changelog.zip" -d "$CHANGELOG_PATH" - rm -f "$RUNNER_TEMP/changelog.zip" - - - name: Upload reusable ingest artifacts - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 - with: - name: reused-ingest-artifacts - path: source-artifacts/* - - - name: Upload merge changelog - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 - with: - name: changelog-metadata - path: merge-changelog/* - trigger-agentic-ingest: - needs: package-reused-artifacts runs-on: ubuntu-latest steps: - name: Trigger agentic database ingest @@ -167,8 +25,8 @@ jobs: -d '{ "event_type": "ingest-agentic-results", "client_payload": { - "run-id": "${{ github.run_id }}", - "run-attempt": "${{ github.run_attempt }}", + "source-run-id": "${{ inputs.source-run-id }}", + "merge-run-id": "${{ inputs.merge-run-id }}", "database-target": "production" } }' diff --git a/.github/workflows/run-sweep.yml b/.github/workflows/run-sweep.yml index 50cc89158..011353c68 100644 --- a/.github/workflows/run-sweep.yml +++ b/.github/workflows/run-sweep.yml @@ -719,84 +719,6 @@ jobs: uses: ./.github/workflows/collect-evals.yml secrets: inherit - reuse-ingest-artifacts: - needs: setup - # `setup` runs via always() and depends on `check-changelog` and - # `reuse-sweep-gate`, which are skipped on every push-to-main. Without - # always() here, those skipped jobs poison this job's implicit - # success() check and it is skipped even when setup succeeded with - # reuse-enabled=true — silently breaking the merge-time ingest. Guard - # explicitly on setup success instead (same pattern as - # comment-unofficial-run-visualizer). - if: >- - always() && - needs.setup.result == 'success' && - needs.setup.outputs.reuse-enabled == 'true' - runs-on: blacksmith-4vcpu-ubuntu-2404 - steps: - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - - - name: Download reusable source artifacts - env: - GH_TOKEN: ${{ secrets.REPO_PAT || github.token }} - SOURCE_RUN_ID: ${{ needs.setup.outputs.reuse-source-run-id }} - run: | - mkdir -p source-artifacts - - # Reruns share one run ID. Keep the newest upload for each - # runner-independent artifact name, matching normal ingest. - while IFS=$'\t' read -r name url; do - mkdir -p "source-artifacts/$name" - gh api "$url" > artifact.zip - unzip -q artifact.zip -d "source-artifacts/$name" - done < <( - gh api \ - "repos/${{ github.repository }}/actions/runs/$SOURCE_RUN_ID/artifacts" \ - --paginate | - jq -rs ' - [ - .[].artifacts[] - | select(.expired | not) - | select(.name | test("^(results_bmk$|eval_results_all$|run-stats$|bmk_|eval_|agentic_|server_logs_|multinode_server_logs_)")) - ] - | group_by(.name | sub("_[A-Za-z][A-Za-z0-9.-]*_[0-9]+$"; "")) - | map(max_by(.created_at))[] - | [.name, .archive_download_url] - | @tsv - ' - ) - rm -f artifact.zip - - mkdir -p source-artifacts/reused-ingest-metadata - cat > source-artifacts/reused-ingest-metadata/reuse_source_run.json <<'JSON' - { - "source_run_id": "${{ needs.setup.outputs.reuse-source-run-id }}", - "source_run_attempt": "${{ needs.setup.outputs.reuse-source-run-attempt }}", - "source_run_url": "${{ needs.setup.outputs.reuse-source-run-url }}", - "source_pr_number": "${{ needs.setup.outputs.reuse-source-pr-number }}", - "source_head_sha": "${{ needs.setup.outputs.reuse-source-head-sha }}", - "ingest_run_id": "${{ github.run_id }}", - "ingest_run_attempt": "${{ github.run_attempt }}", - "ingest_run_url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" - } - JSON - - echo "Reusing artifacts from $SOURCE_RUN_ID:" - find source-artifacts -maxdepth 1 -mindepth 1 -type d -printf ' %f\n' | sort - - - name: Validate reusable artifacts - # Collapses reran (flaky) eval duplicates to the latest result - # per identity, then validates consistency for ingest. - run: | - python3 utils/validate_reusable_sweep_artifacts.py \ - --artifacts-dir source-artifacts - - - name: Upload reusable ingest artifacts - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 - with: - name: reused-ingest-artifacts - path: source-artifacts/* - upload-changelog-metadata: needs: [setup, collect-results] if: ${{ always() && needs.setup.result == 'success' }} @@ -887,7 +809,6 @@ jobs: calc-success-rate, setup, upload-changelog-metadata, - reuse-ingest-artifacts, ] if: >- always() && @@ -907,7 +828,7 @@ jobs: ( needs.collect-results.result != 'skipped' || needs.collect-evals.result != 'skipped' || - needs.reuse-ingest-artifacts.result == 'success' + needs.setup.outputs.reuse-enabled == 'true' ) runs-on: ubuntu-latest steps: @@ -920,8 +841,8 @@ jobs: -d '{ "event_type": "ingest-results", "client_payload": { - "run-id": "${{ github.run_id }}", - "run-attempt": "${{ github.run_attempt }}" + "source-run-id": "${{ needs.setup.outputs.reuse-enabled == 'true' && needs.setup.outputs.reuse-source-run-id || github.run_id }}", + "merge-run-id": "${{ github.run_id }}" } }' @@ -932,7 +853,6 @@ jobs: sweep-agentic, sweep-multi-node-agentic, upload-changelog-metadata, - reuse-ingest-artifacts, ] if: >- always() && @@ -942,7 +862,7 @@ jobs: needs.upload-changelog-metadata.result == 'success' && ( ( - needs.reuse-ingest-artifacts.result == 'success' && + needs.setup.outputs.reuse-enabled == 'true' && ( ( toJson(fromJson(needs.setup.outputs.search-space-config).single_node['agentic']) != 'null' && @@ -980,8 +900,8 @@ jobs: -d '{ "event_type": "ingest-agentic-results", "client_payload": { - "run-id": "${{ github.run_id }}", - "run-attempt": "${{ github.run_attempt }}", + "source-run-id": "${{ needs.setup.outputs.reuse-enabled == 'true' && needs.setup.outputs.reuse-source-run-id || github.run_id }}", + "merge-run-id": "${{ github.run_id }}", "database-target": "production" } }' diff --git a/utils/changelog_gate_tests/test_run_sweep_gating.py b/utils/changelog_gate_tests/test_run_sweep_gating.py index a36f1eb72..a1bbc5e9f 100644 --- a/utils/changelog_gate_tests/test_run_sweep_gating.py +++ b/utils/changelog_gate_tests/test_run_sweep_gating.py @@ -346,6 +346,36 @@ def test_trigger_types_enable_gated_events() -> None: assert {"opened", "reopened"}.isdisjoint(PR_TYPES) +def test_reuse_dispatches_source_directly_without_artifact_relay() -> None: + jobs = _WF["jobs"] + assert "reuse-ingest-artifacts" not in jobs + + for job_name in ("trigger-ingest", "trigger-agentic-ingest"): + job = jobs[job_name] + assert "reuse-ingest-artifacts" not in job["needs"] + dispatch = job["steps"][0]["run"] + assert '"source-run-id"' in dispatch + assert '"merge-run-id"' in dispatch + assert '"source-run-attempt"' not in dispatch + assert '"merge-run-attempt"' not in dispatch + + +def test_reuse_recovery_dispatches_only_run_ids() -> None: + recovery = yaml.safe_load( + (REPO_ROOT / ".github/workflows/recover-reused-ingest.yml").read_text() + ) + inputs = recovery[True]["workflow_dispatch"]["inputs"] + assert set(inputs) == {"source-run-id", "merge-run-id"} + + jobs = recovery["jobs"] + assert set(jobs) == {"trigger-agentic-ingest"} + dispatch = jobs["trigger-agentic-ingest"]["steps"][0]["run"] + assert '"source-run-id"' in dispatch + assert '"merge-run-id"' in dispatch + assert '"source-run-attempt"' not in dispatch + assert '"merge-run-attempt"' not in dispatch + + # -------------------------------------------------------------------------- # Independent reference spec of the INTENDED gating, plus an exhaustive # cross-product cross-check: every combination of the input axes is fed to From 1159c4886eb581d6b3361ee380c94d796605e319 Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Thu, 16 Jul 2026 11:46:29 -0500 Subject: [PATCH 3/4] docs: update reusable ingest recovery flow MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace stale relay-job references with the direct source/merge dispatch and app-side preparation and validation steps. 将过时的中转作业引用替换为直接的源运行/合并运行分发，以及应用侧的工件准备和验证步骤。 --- .claude/commands/recover-failed-ingest.md | 24 ++++++++++++++--------- KLAUD_DEBUG.md | 10 ++++++---- 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/.claude/commands/recover-failed-ingest.md b/.claude/commands/recover-failed-ingest.md index ebbfb54f5..dd147ec61 100644 --- a/.claude/commands/recover-failed-ingest.md +++ b/.claude/commands/recover-failed-ingest.md @@ -111,10 +111,10 @@ TARGET_RUN_ID= Require event `push`, workflow path `.github/workflows/run-sweep.yml`, and branch `main`; confirm the target is no longer running before recovering. The disqualifying state is broader than `failure`/`skipped`: when `/reuse-sweep-run` -was forgotten before merge, `reuse-ingest-artifacts` is skipped, the GPU jobs run -(often `cancelled` to save cost), and because `collect-results`/`collect-evals` -are not skipped, `trigger-ingest` still fires `always()` and lands a *bogus* -ingest under the target's own `run_id`. So a target showing +was forgotten before merge, setup leaves reuse disabled, the GPU jobs run (often +`cancelled` to save cost), and because `collect-results`/`collect-evals` are not +skipped, `trigger-ingest` still fires `always()` and lands a *bogus* ingest under +the target's own `run_id`. So a target showing `trigger-ingest=success` (and concluding `success` or `cancelled`) can still hold no valid benchmark data — recovery is required. That bogus row is keyed on the target `run_id` and is superseded by the recovery ingest under a new `run_id`; @@ -324,6 +324,9 @@ python3 utils/validate_reusable_sweep_artifacts.py \ --artifacts-dir /tmp/source-artifacts ``` +This local preflight uses the same validator that InferenceX-app runs after it +selects and downloads the reusable artifact set. + The validator first collapses reran (flaky) eval duplicates in place — keeping the latest result per eval identity when a retried eval left duplicate raw dirs / aggregate rows — so a legitimate rerun does not fail validation. It only @@ -449,15 +452,15 @@ The push-to-main `Run Sweep` must: - run `setup` even if the merge message contains `[skip-sweep]`; - resolve the recovery PR and pinned source run; - set `reuse-enabled=true`; -- pass `reuse-ingest-artifacts` consistency validation; - upload recovery changelog metadata; -- run `trigger-ingest`. +- dispatch `source-run-id` and `merge-run-id` from `trigger-ingest`. Then locate the resulting `repository_dispatch` run in `SemiAnalysisAI/InferenceX-app`. In the forgotten-`/reuse` case the target's bogus ingest is also a recent successful `ingest-results` run, so do not pick by -recency — pick the run whose `Download artifacts from InferenceX run` step logs -`RUN_ID: `: +recency — pick the run whose `Prepare artifacts from InferenceX` step logs both +the expected source run and recovery merge run. That app workflow must prepare +the artifact family, pass reusable-artifact validation, and complete ingestion: ```bash gh run list --repo SemiAnalysisAI/InferenceX-app \ @@ -467,7 +470,10 @@ gh run list --repo SemiAnalysisAI/InferenceX-app \ INGEST_RUN_ID= gh run view "$INGEST_RUN_ID" --repo SemiAnalysisAI/InferenceX-app --log \ - | grep -m1 "RUN_ID: $RECOVERY_RUN_ID" # must match before you trust this run + | grep -m1 "Source run: $SOURCE_RUN_ID" # must match before you trust this run + +gh run view "$INGEST_RUN_ID" --repo SemiAnalysisAI/InferenceX-app --log \ + | grep -m1 "Merge run: $RECOVERY_RUN_ID" # must also match gh run watch "$INGEST_RUN_ID" \ --repo SemiAnalysisAI/InferenceX-app --exit-status diff --git a/KLAUD_DEBUG.md b/KLAUD_DEBUG.md index a3b0cb659..7ffa8bcfa 100644 --- a/KLAUD_DEBUG.md +++ b/KLAUD_DEBUG.md @@ -206,10 +206,12 @@ Or check whether any other recipe on main uses the proposed tag — if zero uses ### 7.1 Reuse after matrix-generation policy changes Reusable source artifacts are authoritative. The merge-time -`reuse-ingest-artifacts` job validates that downloaded artifacts are readable, -non-duplicated, and internally consistent, but it does not require them to -match a matrix regenerated from the merge commit. A generator-policy change -between the PR sweep and merge therefore does not require another GPU sweep. +dispatch passes the source and merge run IDs directly to InferenceX-app. The app +selects the newest valid artifact family for each logical point and validates +that the prepared artifacts are readable, non-duplicated, and internally +consistent, but it does not require them to match a matrix regenerated from the +merge commit. A generator-policy change between the PR sweep and merge therefore +does not require another GPU sweep. Raw and aggregate eval identities must still match, as must agentic point/raw artifacts and summaries. Batched eval identities come from From 3e5d83f92b2dc9c6a0209bbbf650b25316dcd6dc Mon Sep 17 00:00:00 2001 From: Cam Quilici Date: Thu, 16 Jul 2026 12:13:41 -0500 Subject: [PATCH 4/4] docs: simplify reusable ingest guidance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Document that InferenceX-app downloads normal source artifacts, substitutes the merge changelog, and relies on existing ingestion behavior for failed rows. Remove obsolete reusable-artifact validation instructions.\n\n中文：说明 InferenceX-app 直接下载常规源运行产物、替换合并运行变更日志，并沿用现有摄取逻辑跳过失败结果。删除过时的复用产物验证说明。 --- .claude/commands/recover-failed-ingest.md | 83 ++++------------------- .github/workflows/README.md | 17 +++-- KLAUD_DEBUG.md | 15 ++-- 3 files changed, 24 insertions(+), 91 deletions(-) diff --git a/.claude/commands/recover-failed-ingest.md b/.claude/commands/recover-failed-ingest.md index dd147ec61..cf8503948 100644 --- a/.claude/commands/recover-failed-ingest.md +++ b/.claude/commands/recover-failed-ingest.md @@ -4,8 +4,8 @@ argument-hint: [source-run-id] --- Recover the official database ingest for a failed or skipped InferenceX -push-to-main `Run Sweep` workflow by creating a recovery PR that reuses validated -artifacts from an earlier PR sweep. Do not add a one-off recovery workflow. +push-to-main `Run Sweep` workflow by creating a recovery PR that reuses artifacts +from an earlier PR sweep. Do not add a one-off recovery workflow. Inputs from `$ARGUMENTS`: @@ -29,9 +29,8 @@ Run from a clean InferenceX checkout with authenticated `gh`, `git`, `jq`, and `.github/workflows/run-sweep.yml` on `main` whose official ingest did not complete. - Reuse only a completed `pull_request` run of `run-sweep.yml`. Unpinned reuse - requires success. A specifically pinned failed run is allowed only when - artifact validation proves its available result set is internally consistent; - only completed points are recovered. + requires success. A specifically pinned failed run is allowed; normal + ingestion skips failed benchmark rows, so only completed points are recovered. - The source run must belong to the original PR being recovered. - Stop if that PR changed the recovered configuration's execution semantics after the source SHA: image, model, recipe, runner, launcher, benchmark @@ -132,7 +131,7 @@ git diff "$ORIGINAL_BASE_SHA" "$ORIGINAL_MERGE_SHA" -- \ perf-changelog.yaml ``` -## 2. Select and validate the source run +## 2. Select the source run Find candidates from the original PR branch when no run ID was supplied: @@ -238,7 +237,7 @@ Keep exactly one of `full-sweep-enabled`, `non-canary-full-sweep-enabled`, `full-sweep-fail-fast`, or `full-sweep-fail-fast-no-canary`. -## 5. Append the recovery changelog and validate source artifacts +## 5. Append and validate the recovery changelog Append recovery entries to the end of `perf-changelog.yaml`. Preserve the original entries' `config-keys`, `description`, `evals-only`, and @@ -281,64 +280,6 @@ python3 utils/process_changelog.py \ Confirm the generated config contains only the intended recovery scope. Its row counts may differ from the source run. -Download only the result artifacts needed for local validation. This avoids the -large server-log artifacts retained in the official ingest bundle. Raw per-config -`bmk__*` artifacts are intentionally not selected — they fall through the -`case` below; the aggregate `results_bmk` is what the validator reads: - -```bash -rm -rf /tmp/source-artifacts -ARTIFACT_ARGS=() -while IFS= read -r name; do - case "$name" in - results_bmk|eval_results_all|run-stats|bmk_agentic_*|agentic_*) - ARTIFACT_ARGS+=(-n "$name") - ;; - eval_server_logs_*|eval_gpu_metrics_*) - ;; - eval_*) - ARTIFACT_ARGS+=(-n "$name") - ;; - esac -done < <( - gh api \ - "repos/SemiAnalysisAI/InferenceX/actions/runs/$SOURCE_RUN_ID/artifacts?per_page=100" \ - --paginate --jq '.artifacts[] | select(.expired == false) | .name' | - sort -u -) - -((${#ARTIFACT_ARGS[@]})) || { - echo "No unexpired result artifacts found" >&2 - exit 1 -} -gh run download "$SOURCE_RUN_ID" \ - --repo SemiAnalysisAI/InferenceX \ - -D /tmp/source-artifacts \ - "${ARTIFACT_ARGS[@]}" -``` - -Validate the source artifacts: - -```bash -python3 utils/validate_reusable_sweep_artifacts.py \ - --artifacts-dir /tmp/source-artifacts -``` - -This local preflight uses the same validator that InferenceX-app runs after it -selects and downloads the reusable artifact set. - -The validator first collapses reran (flaky) eval duplicates in place — keeping -the latest result per eval identity when a retried eval left duplicate raw dirs -/ aggregate rows — so a legitimate rerun does not fail validation. It only -collapses identities with a clear latest result; genuinely ambiguous duplicates -are still rejected. - -The validator does not compare source coverage with -`/tmp/recovery-full-config.json`. It rejects duplicate fixed rows, missing run -stats, inconsistent agentic artifacts, malformed eval metadata, raw/aggregate -eval mismatches, or an empty result set. For a pinned failed batched eval run, -only `completed_eval_concs` are recovered. - ## 6. Attach the source SHA without changing the tree Make the ancestry carrier the final branch commit. `git commit-tree` guarantees @@ -459,8 +400,8 @@ Then locate the resulting `repository_dispatch` run in `SemiAnalysisAI/InferenceX-app`. In the forgotten-`/reuse` case the target's bogus ingest is also a recent successful `ingest-results` run, so do not pick by recency — pick the run whose `Prepare artifacts from InferenceX` step logs both -the expected source run and recovery merge run. That app workflow must prepare -the artifact family, pass reusable-artifact validation, and complete ingestion: +the expected source run and recovery merge run. That app workflow must download +the source artifacts, substitute the merge changelog, and complete ingestion: ```bash gh run list --repo SemiAnalysisAI/InferenceX-app \ @@ -479,13 +420,13 @@ gh run watch "$INGEST_RUN_ID" \ --repo SemiAnalysisAI/InferenceX-app --exit-status ``` -The ingest's first step is a `sleep 300` "wait for source run to finish", so the -run idles ~5 minutes before doing work — that is normal, not a hang. +Reused ingests skip the normal five-minute wait because the source run is already +complete. Verify its logs identify `RECOVERY_RUN_ID` as the trigger and `SOURCE_RUN_ID` plus `SOURCE_RUN_ATTEMPT` as the reused source. Require successful artifact -download, flattening, database ingest, run overrides, database verification, -cache invalidation, and unmapped-entity checks. +download, database ingest, run overrides, database verification, cache +invalidation, and unmapped-entity checks. Post a final recovery PR comment with the original failed or skipped run/job, source run/attempt/SHA, recovery merge run, downstream ingest run, recovered diff --git a/.github/workflows/README.md b/.github/workflows/README.md index efd6e7c0d..e330a9484 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -177,9 +177,9 @@ in the PR. A run ID can pin an eligible successful or failed run: /reuse-sweep-run ``` -Failed-run artifacts must still validate. The latest matching comment by an -`OWNER`, `MEMBER`, or `COLLABORATOR` wins. Comments do not trigger or cancel -sweeps; later commits skip a new sweep after changelog/matrix validation. +The latest matching comment by an `OWNER`, `MEMBER`, or `COLLABORATOR` wins. +Comments do not trigger or cancel sweeps; later commits skip a new sweep after +changelog/matrix validation. Remove and re-add the sweep label to force one. `utils/merge_with_reuse.sh ` is the supported merge path for reuse. @@ -187,12 +187,11 @@ It merges `main`, preserves changelog bytes, fixes an appended `XXX` PR link, pushes a synchronization commit, waits for checks, then merges. The main run passes the selected source run ID and its own merge run ID directly -to InferenceX-app. The app downloads source artifacts, keeps the newest valid -artifact family for each logical benchmark point, validates the selected set, -and ingests it with changelog metadata from the merge run. Benchmark rows and -public links retain source-run provenance. Source coverage is authoritative, so -later matrix/eval policy changes do not invalidate reuse. Batched evals use only -`completed_eval_concs`. +to InferenceX-app. The app downloads source artifacts, keeps the newest upload +for each exact artifact name, and ingests them with changelog metadata from the +merge run. The normal ingestion code skips failed benchmark rows. Benchmark +rows and public links retain source-run provenance. Source coverage is +authoritative, so later matrix/eval policy changes do not invalidate reuse. Reuse fails closed when authorized but ineligible or invalid; without authorization, `main` runs the normal full sweep. diff --git a/KLAUD_DEBUG.md b/KLAUD_DEBUG.md index 7ffa8bcfa..6760b37f8 100644 --- a/KLAUD_DEBUG.md +++ b/KLAUD_DEBUG.md @@ -207,17 +207,10 @@ Or check whether any other recipe on main uses the proposed tag — if zero uses Reusable source artifacts are authoritative. The merge-time dispatch passes the source and merge run IDs directly to InferenceX-app. The app -selects the newest valid artifact family for each logical point and validates -that the prepared artifacts are readable, non-duplicated, and internally -consistent, but it does not require them to match a matrix regenerated from the -merge commit. A generator-policy change between the PR sweep and merge therefore -does not require another GPU sweep. - -Raw and aggregate eval identities must still match, as must agentic point/raw -artifacts and summaries. Batched eval identities come from -`completed_eval_concs`, so an explicitly pinned failed run may reuse only the -points it completed. Missing or invalid metadata, duplicate identities, and -raw/aggregate disagreement still fail reuse. +keeps the newest upload for each exact artifact name and sends the resulting +directory to the normal ingestion code. The only reuse-specific substitution is +that changelog metadata comes from the merge run. A generator-policy change +between the PR sweep and merge therefore does not require another GPU sweep. ---