feat: add experimental BinEval evaluation support

[Copilot]

Adds native BinEval-style evaluations to gh-aw — small, binary questions declared in workflow frontmatter, executed post-run via an LLM harness, with results aggregated and reported as CI artifacts.

Schema (evals frontmatter)

• New optional evals array with id + question fields; validated for unique IDs and non-empty questions
• Emits experimental warning at compile time

evals:
  - id: builds
    question: Does the generated code compile?
  - id: focused
    question: Is the implementation limited to the requested change?

Evaluation model

• EvalDefinition, EvalResult, EvalSummary types in frontmatter_types.go
• WorkflowData.Evals []EvalDefinition for downstream consumers

Eval job

• New eval job injected after agent + detection jobs in the compiled workflow
• JS harness (eval_harness.cjs) calls GitHub Models API (gpt-4o-mini) per question independently — no MCPs, no checkout
• Prompt generation produces per-question binary prompts with rationale; no holistic scoring
• Results aggregated (total/passed/failed/pass-rate) and uploaded as a eval artifact with a markdown step summary

Not included

Phase 8 (persisting results to a git branch, à la experiments) is deferred.

[pelikhan]

@copilot run inference in AWF and use /reflect to find a suitable inference endpoint.

[github-actions]

Hey @Copilot 👋 — great work on the BinEval evaluation support! The end-to-end shape is clean: the evals frontmatter schema, the Go compiler module, the JS harness, and the wiring into the orchestrator all land as a cohesive unit.

A few things that stand out positively:

• Well-scoped: every changed file is in service of the same feature — nothing unrelated was mixed in.
• Tests at every layer: compiler_evals_test.go (Go unit tests), eval_harness.test.cjs (JS unit tests), and evals_experimental_warning_test.go (integration guard) give solid coverage across the stack.
• Clear description: the PR body explains the schema, the evaluation model, the harness design, and explicitly calls out what was deferred (Phase 8 persistence) — exactly the context reviewers need.
• Experimental gating: emitting a compile-time warning via emitExperimentalFeatureWarnings is the right pattern for a feature that isn't production-ready yet.

This looks ready for review. 🚀

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• patchdiff.githubusercontent.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "patchdiff.githubusercontent.com"

See Network Configuration for more information.

Generated by ✅ Contribution Check · 293.1 AIC · ⌖ 20.4 AIC · ⊞ 6K · ◷

[pelikhan]

@copilot keep working

[github-actions]

🤖 PR Triage — §28332039983

Field	Value
Category	feature
Risk	high
Priority	medium
Score	38 / 100 (impact 25 · urgency 5 · quality 8)
Action	defer
Batch	—

Rationale: Large experimental draft (+2 181 lines). Adds a BinEval-style LLM evaluation harness in workflow frontmatter — significant new capability but high scope/risk. No CI results yet (merge state: UNSTABLE). Defer until draft is promoted to ready and CI passes.

Labels applied: pr-type:feature · pr-risk:high · pr-priority:medium · pr-action:defer · pr-agent:copilot-swe-agent

Generated by 🔧 PR Triage Agent · 65.9 AIC · ⌖ 11.5 AIC · ⊞ 5.4K · ◷

[github-actions]

🤖 PR Triage — §28342769269

Field	Value
Category	feature
Risk	high
Score	38/100
Priority	low
Action	defer
Status	Draft — no CI, no reviews

Score breakdown: Impact 25 · Urgency 6 · Quality 7

Rationale: Experimental BinEval LLM evaluation harness (+2181 lines). Large addition, still draft with no CI checks passing. ~10h old with no new activity. Defer until promoted to ready and CI established.

i️ Carried over. Defer until draft promoted to ready.

Generated by 🔧 PR Triage Agent · 107.8 AIC · ⌖ 10.9 AIC · ⊞ 5.4K · ◷

[github-actions]

🤖 PR Triage — §28357644191

Field	Value
Category	feature
Risk	🔴 High
Score	30/100 (Impact 20 · Urgency 5 · Quality 5)
Action	defer

Carried over — 17.3h old. Experimental BinEval LLM evaluation harness (+2181 lines). Large addition, no CI yet, draft. Defer until promoted to ready and CI validates.

Generated by 🔧 PR Triage Agent · 89.9 AIC · ⌖ 12 AIC · ⊞ 5.4K · ◷

[github-actions]

🤖 PR Triage — §28376613466

Field	Value
Category	feature (experimental)
Risk	🔴 High
Priority	🟢 Low
Score	30 / 100
Action	⏸️ defer
Age	23h

Score breakdown: Impact 20 + Urgency 5 + Quality 5

Rationale: Adds native BinEval-style evaluations — large experimental addition (2181+/0−, 19 files), draft, no CI. High risk, no reviewer engagement yet. Defer until promoted from draft, CI passes, and feature scope is scoped down or approved.

i️ pr-priority:medium label is stale — score is 30 (low boundary). Carried over from §28357644191.

Generated by 🔧 PR Triage Agent · 93.2 AIC · ⌖ 14.6 AIC · ⊞ 5.4K · ◷

[github-actions]

🔍 PR Triage — §28395315609

Field	Value
Category	feature
Risk	high
Score	26 / 100
Action	defer
Batch	—

Score breakdown: impact 15 + urgency 3 + quality 8

Carried over (28h). Experimental BinEval evaluation support. Large draft (19 files, +2181/-0), no CI, Phase 8 deferred by author. Priority corrected: pr-priority:medium → pr-priority:low (score 26). Defer until out of draft with CI.

Generated by 🔧 PR Triage Agent · 99.1 AIC · ⌖ 11.6 AIC · ⊞ 5.4K · ◷