Skip to content

Simplify reusable sweep ingestion with direct source dispatch / 通过直接分发源运行简化复用扫描摄取 #2244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cquil11 merged 4 commits into from
Jul 16, 2026
Merged

Simplify reusable sweep ingestion with direct source dispatch / 通过直接分发源运行简化复用扫描摄取 #2244

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cad9465
fix: select latest reused sweep artifacts
cquil11 Jul 16, 2026
2381e82
refactor: dispatch reusable sweep sources directly
cquil11 Jul 16, 2026
1159c48
docs: update reusable ingest recovery flow
cquil11 Jul 16, 2026
3e5d83f
docs: simplify reusable ingest guidance
cquil11 Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 22 additions & 75 deletions .claude/commands/recover-failed-ingest.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ argument-hint: <failed-run-or-job-url | pr-number> [source-run-id]
---

Recover the official database ingest for a failed or skipped InferenceX
push-to-main `Run Sweep` workflow by creating a recovery PR that reuses validated
artifacts from an earlier PR sweep. Do not add a one-off recovery workflow.
push-to-main `Run Sweep` workflow by creating a recovery PR that reuses artifacts
from an earlier PR sweep. Do not add a one-off recovery workflow.

Inputs from `$ARGUMENTS`:

Expand All @@ -29,9 +29,8 @@ Run from a clean InferenceX checkout with authenticated `gh`, `git`, `jq`, and
`.github/workflows/run-sweep.yml` on `main` whose official ingest did not
complete.
- Reuse only a completed `pull_request` run of `run-sweep.yml`. Unpinned reuse
requires success. A specifically pinned failed run is allowed only when
artifact validation proves its available result set is internally consistent;
only completed points are recovered.
requires success. A specifically pinned failed run is allowed; normal
ingestion skips failed benchmark rows, so only completed points are recovered.
- The source run must belong to the original PR being recovered.
- Stop if that PR changed the recovered configuration's execution semantics
after the source SHA: image, model, recipe, runner, launcher, benchmark
Expand Down Expand Up @@ -111,10 +110,10 @@ TARGET_RUN_ID=<matching-run-id>
Require event `push`, workflow path `.github/workflows/run-sweep.yml`, and branch
`main`; confirm the target is no longer running before recovering. The
disqualifying state is broader than `failure`/`skipped`: when `/reuse-sweep-run`
was forgotten before merge, `reuse-ingest-artifacts` is skipped, the GPU jobs run
(often `cancelled` to save cost), and because `collect-results`/`collect-evals`
are not skipped, `trigger-ingest` still fires `always()` and lands a *bogus*
ingest under the target's own `run_id`. So a target showing
was forgotten before merge, setup leaves reuse disabled, the GPU jobs run (often
`cancelled` to save cost), and because `collect-results`/`collect-evals` are not
skipped, `trigger-ingest` still fires `always()` and lands a *bogus* ingest under
the target's own `run_id`. So a target showing
`trigger-ingest=success` (and concluding `success` or `cancelled`) can still hold
no valid benchmark data — recovery is required. That bogus row is keyed on the
target `run_id` and is superseded by the recovery ingest under a new `run_id`;
Expand All @@ -132,7 +131,7 @@ git diff "$ORIGINAL_BASE_SHA" "$ORIGINAL_MERGE_SHA" -- \
perf-changelog.yaml
```

## 2. Select and validate the source run
## 2. Select the source run

Find candidates from the original PR branch when no run ID was supplied:

Expand Down Expand Up @@ -238,7 +237,7 @@ Keep exactly one of `full-sweep-enabled`,
`non-canary-full-sweep-enabled`, `full-sweep-fail-fast`, or
`full-sweep-fail-fast-no-canary`.

## 5. Append the recovery changelog and validate source artifacts
## 5. Append and validate the recovery changelog

Append recovery entries to the end of `perf-changelog.yaml`. Preserve the
original entries' `config-keys`, `description`, `evals-only`, and
Expand Down Expand Up @@ -281,61 +280,6 @@ python3 utils/process_changelog.py \
Confirm the generated config contains only the intended recovery scope. Its row
counts may differ from the source run.

Download only the result artifacts needed for local validation. This avoids the
large server-log artifacts retained in the official ingest bundle. Raw per-config
`bmk_<model>_*` artifacts are intentionally not selected — they fall through the
`case` below; the aggregate `results_bmk` is what the validator reads:

```bash
rm -rf /tmp/source-artifacts
ARTIFACT_ARGS=()
while IFS= read -r name; do
case "$name" in
results_bmk|eval_results_all|run-stats|bmk_agentic_*|agentic_*)
ARTIFACT_ARGS+=(-n "$name")
;;
eval_server_logs_*|eval_gpu_metrics_*)
;;
eval_*)
ARTIFACT_ARGS+=(-n "$name")
;;
esac
done < <(
gh api \
"repos/SemiAnalysisAI/InferenceX/actions/runs/$SOURCE_RUN_ID/artifacts?per_page=100" \
--paginate --jq '.artifacts[] | select(.expired == false) | .name' |
sort -u
)

((${#ARTIFACT_ARGS[@]})) || {
echo "No unexpired result artifacts found" >&2
exit 1
}
gh run download "$SOURCE_RUN_ID" \
--repo SemiAnalysisAI/InferenceX \
-D /tmp/source-artifacts \
"${ARTIFACT_ARGS[@]}"
```

Validate the source artifacts:

```bash
python3 utils/validate_reusable_sweep_artifacts.py \
--artifacts-dir /tmp/source-artifacts
```

The validator first collapses reran (flaky) eval duplicates in place — keeping
the latest result per eval identity when a retried eval left duplicate raw dirs
/ aggregate rows — so a legitimate rerun does not fail validation. It only
collapses identities with a clear latest result; genuinely ambiguous duplicates
are still rejected.

The validator does not compare source coverage with
`/tmp/recovery-full-config.json`. It rejects duplicate fixed rows, missing run
stats, inconsistent agentic artifacts, malformed eval metadata, raw/aggregate
eval mismatches, or an empty result set. For a pinned failed batched eval run,
only `completed_eval_concs` are recovered.

## 6. Attach the source SHA without changing the tree

Make the ancestry carrier the final branch commit. `git commit-tree` guarantees
Expand Down Expand Up @@ -449,15 +393,15 @@ The push-to-main `Run Sweep` must:
- run `setup` even if the merge message contains `[skip-sweep]`;
- resolve the recovery PR and pinned source run;
- set `reuse-enabled=true`;
- pass `reuse-ingest-artifacts` consistency validation;
- upload recovery changelog metadata;
- run `trigger-ingest`.
- dispatch `source-run-id` and `merge-run-id` from `trigger-ingest`.

Then locate the resulting `repository_dispatch` run in
`SemiAnalysisAI/InferenceX-app`. In the forgotten-`/reuse` case the target's
bogus ingest is also a recent successful `ingest-results` run, so do not pick by
recency — pick the run whose `Download artifacts from InferenceX run` step logs
`RUN_ID: <RECOVERY_RUN_ID>`:
recency — pick the run whose `Prepare artifacts from InferenceX` step logs both
the expected source run and recovery merge run. That app workflow must download
the source artifacts, substitute the merge changelog, and complete ingestion:

```bash
gh run list --repo SemiAnalysisAI/InferenceX-app \
Expand All @@ -467,19 +411,22 @@ gh run list --repo SemiAnalysisAI/InferenceX-app \

INGEST_RUN_ID=<candidate-run-id>
gh run view "$INGEST_RUN_ID" --repo SemiAnalysisAI/InferenceX-app --log \
| grep -m1 "RUN_ID: $RECOVERY_RUN_ID" # must match before you trust this run
| grep -m1 "Source run: $SOURCE_RUN_ID" # must match before you trust this run

gh run view "$INGEST_RUN_ID" --repo SemiAnalysisAI/InferenceX-app --log \
| grep -m1 "Merge run: $RECOVERY_RUN_ID" # must also match

gh run watch "$INGEST_RUN_ID" \
--repo SemiAnalysisAI/InferenceX-app --exit-status
```

The ingest's first step is a `sleep 300` "wait for source run to finish", so the
run idles ~5 minutes before doing work — that is normal, not a hang.
Reused ingests skip the normal five-minute wait because the source run is already
complete.

Verify its logs identify `RECOVERY_RUN_ID` as the trigger and `SOURCE_RUN_ID`
plus `SOURCE_RUN_ATTEMPT` as the reused source. Require successful artifact
download, flattening, database ingest, run overrides, database verification,
cache invalidation, and unmapped-entity checks.
download, database ingest, run overrides, database verification, cache
invalidation, and unmapped-entity checks.

Post a final recovery PR comment with the original failed or skipped run/job,
source run/attempt/SHA, recovery merge run, downstream ingest run, recovered
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,21 +177,21 @@ in the PR. A run ID can pin an eligible successful or failed run:
/reuse-sweep-run <run_id>
```

Failed-run artifacts must still validate. The latest matching comment by an
`OWNER`, `MEMBER`, or `COLLABORATOR` wins. Comments do not trigger or cancel
sweeps; later commits skip a new sweep after changelog/matrix validation.
The latest matching comment by an `OWNER`, `MEMBER`, or `COLLABORATOR` wins.
Comments do not trigger or cancel sweeps; later commits skip a new sweep after
changelog/matrix validation.
Remove and re-add the sweep label to force one.

`utils/merge_with_reuse.sh <pr-number>` is the supported merge path for reuse.
It merges `main`, preserves changelog bytes, fixes an appended `XXX` PR link,
pushes a synchronization commit, waits for checks, then merges.

The main run verifies the source, validates and uploads its ingest artifacts,
then ingests them with merge-run changelog metadata. Source coverage is
The main run passes the selected source run ID and its own merge run ID directly
to InferenceX-app. The app downloads source artifacts, keeps the newest upload
for each exact artifact name, and ingests them with changelog metadata from the
merge run. The normal ingestion code skips failed benchmark rows. Benchmark
rows and public links retain source-run provenance. Source coverage is
authoritative, so later matrix/eval policy changes do not invalidate reuse.
Validation rejects duplicate fixed rows, missing run stats, inconsistent
agentic artifacts, malformed eval metadata, and raw/aggregate eval mismatches.
Batched evals use only `completed_eval_concs`.

Reuse fails closed when authorized but ineligible or invalid; without
authorization, `main` runs the normal full sweep.
Expand Down
150 changes: 4 additions & 146 deletions .github/workflows/recover-reused-ingest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,158 +4,16 @@ on:
workflow_dispatch:
inputs:
source-run-id:
description: Source PR sweep run ID
required: true
type: string
source-run-attempt:
description: Source PR sweep attempt
required: true
default: "1"
type: string
source-pr-number:
description: Source PR number
required: true
type: string
source-head-sha:
description: Source sweep commit SHA
description: Source PR sweep run ID containing benchmark artifacts
required: true
type: string
merge-run-id:
description: Original failed merge run ID
description: Main publication run ID containing changelog metadata
required: true
type: string
merge-run-attempt:
description: Original failed merge run attempt
required: true
default: "1"
type: string

jobs:
package-reused-artifacts:
runs-on: blacksmith-4vcpu-ubuntu-2404
permissions:
contents: read
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0

- name: Download source artifacts
env:
GH_TOKEN: ${{ secrets.REPO_PAT || github.token }}
SOURCE_RUN_ID: ${{ inputs.source-run-id }}
ARTIFACTS_PATH: ${{ github.workspace }}/source-artifacts
run: |
mkdir -p "$ARTIFACTS_PATH"

download_artifact() {
local name="$1"
local url="$2"
local zip_path="$RUNNER_TEMP/artifact.zip"

echo "Downloading artifact: $name"
for attempt in 1 2 3; do
if gh api "$url" > "$zip_path"; then
mkdir -p "$ARTIFACTS_PATH/$name"
if unzip -oq "$zip_path" -d "$ARTIFACTS_PATH/$name"; then
rm -f "$zip_path"
return 0
fi
rm -rf "${ARTIFACTS_PATH:?}/$name"
fi
rm -f "$zip_path"
echo " Attempt $attempt/3 failed, retrying in ${attempt}s..."
sleep "$attempt"
done
return 1
}

gh api "repos/${{ github.repository }}/actions/runs/${SOURCE_RUN_ID}/artifacts?per_page=100" --paginate --slurp \
| jq -r '
[.[].artifacts[]
| select(.name != "changelog-metadata")
| . + {logical_name: (.name | sub("_[A-Za-z][A-Za-z0-9.-]*_[0-9]+$"; ""))}
]
| group_by(.logical_name)
| map(sort_by(.created_at) | last)[]
| "\(.name)\t\(.archive_download_url)"' \
| while IFS=$'\t' read -r name url; do
if ! download_artifact "$name" "$url"; then
echo "::error::Failed to download source artifact: $name"
exit 1
fi
done

if [ -z "$(find "$ARTIFACTS_PATH" -mindepth 1 -maxdepth 1 -print -quit)" ]; then
echo "::error::No source artifacts downloaded from run $SOURCE_RUN_ID"
exit 1
fi

- name: Record reuse provenance
env:
ARTIFACTS_PATH: ${{ github.workspace }}/source-artifacts
SOURCE_RUN_ID: ${{ inputs.source-run-id }}
SOURCE_RUN_ATTEMPT: ${{ inputs.source-run-attempt }}
SOURCE_PR_NUMBER: ${{ inputs.source-pr-number }}
SOURCE_HEAD_SHA: ${{ inputs.source-head-sha }}
MERGE_RUN_ID: ${{ inputs.merge-run-id }}
MERGE_RUN_ATTEMPT: ${{ inputs.merge-run-attempt }}
run: |
mkdir -p "$ARTIFACTS_PATH/reused-ingest-metadata"
jq -n \
--arg source_run_id "$SOURCE_RUN_ID" \
--arg source_run_attempt "$SOURCE_RUN_ATTEMPT" \
--arg source_run_url "${{ github.server_url }}/${{ github.repository }}/actions/runs/$SOURCE_RUN_ID" \
--arg source_pr_number "$SOURCE_PR_NUMBER" \
--arg source_head_sha "$SOURCE_HEAD_SHA" \
--arg ingest_run_id "$MERGE_RUN_ID" \
--arg ingest_run_attempt "$MERGE_RUN_ATTEMPT" \
--arg ingest_run_url "${{ github.server_url }}/${{ github.repository }}/actions/runs/$MERGE_RUN_ID" \
'{
source_run_id: $source_run_id,
source_run_attempt: $source_run_attempt,
source_run_url: $source_run_url,
source_pr_number: $source_pr_number,
source_head_sha: $source_head_sha,
ingest_run_id: $ingest_run_id,
ingest_run_attempt: $ingest_run_attempt,
ingest_run_url: $ingest_run_url
}' > "$ARTIFACTS_PATH/reused-ingest-metadata/reuse_source_run.json"

- name: Validate reusable artifacts
run: |
python3 utils/validate_reusable_sweep_artifacts.py \
--artifacts-dir "${{ github.workspace }}/source-artifacts"

- name: Download original merge changelog
env:
GH_TOKEN: ${{ secrets.REPO_PAT || github.token }}
MERGE_RUN_ID: ${{ inputs.merge-run-id }}
CHANGELOG_PATH: ${{ github.workspace }}/merge-changelog
run: |
url=$(gh api "repos/${{ github.repository }}/actions/runs/${MERGE_RUN_ID}/artifacts?per_page=100" --paginate --slurp \
| jq -r '[.[].artifacts[] | select(.name == "changelog-metadata")] | sort_by(.created_at) | last | .archive_download_url // empty')
if [ -z "$url" ]; then
echo "::error::No changelog-metadata artifact found on run $MERGE_RUN_ID"
exit 1
fi
mkdir -p "$CHANGELOG_PATH"
gh api "$url" > "$RUNNER_TEMP/changelog.zip"
unzip -oq "$RUNNER_TEMP/changelog.zip" -d "$CHANGELOG_PATH"
rm -f "$RUNNER_TEMP/changelog.zip"

- name: Upload reusable ingest artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: reused-ingest-artifacts
path: source-artifacts/*

- name: Upload merge changelog
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: changelog-metadata
path: merge-changelog/*

trigger-agentic-ingest:
needs: package-reused-artifacts
runs-on: ubuntu-latest
steps:
- name: Trigger agentic database ingest
Expand All @@ -167,8 +25,8 @@ jobs:
-d '{
"event_type": "ingest-agentic-results",
"client_payload": {
"run-id": "${{ github.run_id }}",
"run-attempt": "${{ github.run_attempt }}",
"source-run-id": "${{ inputs.source-run-id }}",
"merge-run-id": "${{ inputs.merge-run-id }}",
"database-target": "production"
}
}'
Loading