[WIP] Add CI job priority system

[adibarra]

Priority-aware CI scheduling

GitHub Actions does not natively prioritize self-hosted jobs. This PR adds an opt-in controller that decides which queued GPU job each compatible idle runner may accept.

Key mechanism: GitHub uses AND matching for runs-on labels. A runner must have every label requested by a job.

How dispatch works

With priority scheduling enabled, a job requests labels similar to:

• self-hosted
• b200
• ci-job-6.000-a81f23...

Runners permanently carry only their normal hardware and identity labels. They do not permanently carry ci-job-* labels.

The controller:

1. Reads queued jobs and online idle runners.
2. Parses the score and queue hash from each job's ci-job-<score>-<hash> label.
3. Adds queue aging and sorts higher scores first.
4. Finds a compatible runner using existing hardware labels.
5. Temporarily adds the selected job's combined ci-job-* label.
6. Lets GitHub dispatch the now-exact label match.
7. Removes stale scheduling labels after the runner becomes idle.

The combined label transports both the human-readable score and a unique 128-bit job hash through GitHub's queued-job API. Because every label must match, the hash prevents GitHub from selecting another job with the same hardware and priority.

Running jobs are never preempted or relabeled.

Single-node and multinode awareness

Existing runner labels remain compatibility constraints:

• Single-node example: b200
• Multinode example: b200-multinode

A single-node runner cannot accept a multinode job unless it already satisfies the multinode label. The controller schedules the coordinating GitHub runner; existing Slurm launchers continue provisioning worker nodes.

Automatic scoring

All ordinary priorities are numeric, nonnegative, automatic, and uncapped. Higher scores run first.

For PR sweeps, Claude Fable inspects the diff and returns only verified priority facts, such as topology, workload type, precision, speculative decoding, framework, model family, checklist completion, and patchwork.

Fable never receives weights and never calculates a score. ci_priority.py:

1. Rejects unknown facts.
2. Intersects the facts with each matrix job's actual configuration.
3. Reads weights from configs/ci-priority.yaml.
4. Performs deterministic arithmetic.
5. Emits the combined ci-job-<score>-<hash> label metadata.

This intersection keeps mixed matrices correct: a PR may contain both vLLM and SGLang jobs, but each job receives only its matching criteria. Main-branch pushes retain deterministic matrix-field scoring without a Fable call.

Malformed or unavailable Fable output conservatively classifies the PR as patchwork. Arbitrary ci-priority:pN overrides are not supported.

Ties are resolved by queue timestamp and job ID. Runner selection preserves scarce exact-match runners when a less-constrained runner can accept the higher-priority job.

Job hashes

Generated job hashes are the first 128 bits of a SHA-256 digest over:

• Workflow run namespace
• Matrix path/index
• Canonical job configuration

Manual matrix workflows derive the same 32-character hash from their run and matrix-cell identity. Hashes are routing identifiers, not credentials or secrets.

A score alone could release any compatible job with that priority. ci-job-<score>-<hash> combines ordering and one-job identity in a single exact-match label.

Protected skip_queue

skip_queue is the only absolute bypass mechanism. The workflow keeps the job's ordinary numeric priority and adds a PR-scoped skip request label.

The trusted controller then:

1. Reads the PR timeline.
2. Finds the active label's latest labeling actor.
3. Verifies active SemiAnalysisAI/Core membership.
4. Treats the job's effective priority as absolute only when authorized.

Unauthorized or unverifiable requests retain their numeric priority. The scheduler credential remains inside the trusted controller and is never passed to PR-controlled workflow code.

Required controller-token permissions:

• Actions: Read
• Administration: Write
• Issues: Read
• Organization Members: Read

Safety and failure behavior

• Busy and offline runners are never relabeled.
• Malformed or duplicate scheduler labels fail loudly.
• Transient GitHub API failures retry in watch mode.
• Partial label updates converge on the next controller cycle.
• Controller downtime leaves priority-enabled jobs queued.
• Run exactly one active controller instance.
• Existing routing remains unchanged while disabled.

Activation requires deploying the controller and setting:

PRIORITY_SCHEDULER_ENABLED=true

Verification

• 28 focused priority tests cover scoring, rounding, aging, authorization, malformed labels, compatibility, exact-runner preservation, and dispatch.
• 27 sweep-gating tests cover named cases and the exhaustive event/label cross-product.
• The complete changelog and sweep-gating CI suite passes: 138 tests.
• Changed workflows pass actionlint.
• Simulations verified automatic numeric scoring and unique queue tokens.
• Core-authorized skip_queue outranked numeric priority 100.000 while retaining its numeric job label.