fix(v1): resume must match saved rollouts by load position, not data.idx

[dumko2001]

Description

The runner selects and dispatches tasks by load position (enumerate(tasks)), but resume.load matched saved traces.jsonl rows by task.data.idx. In LeanTaskset, data.idx is the raw dataset enumeration index and empty statements are filtered out, so data.idx and the load position diverge. On resume, finished rollouts were dropped and the wrong tasks reran.

This change:

1. Records the load position in trace.info["task_idx"] in both the local and server eval paths.
2. Makes resume.load prefer info["task_idx"] and fall back to task.data.idx mapped to the current load position via a new data_idx_to_pos argument.
3. Leaves LeanTaskset.data.idx as the stable raw dataset index; the runner builds a data.idx -> load-position map from the selected task slice and passes it to resume.load, so legacy traces without info.task_idx still resume correctly.

#1962 rewrote resume.load to reload finished traces, but it did not fix this load-position mismatch. This PR is a follow-up to that work.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Test improvement

Testing

• All existing tests pass when running uv run pytest locally.
• New tests have been added to cover the changes

uv run pytest tests/v1/test_taskset.py tests/test_eval_utils.py tests/test_eval_display.py -q and the full uv run pytest tests/v1 -m 'not e2e' pass.

Checklist

• My code follows the style guidelines of this project as outlined in AGENTS.md
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published

Additional Notes

• Rebased onto the current PrimeIntellect-ai/verifiers:main (commit 9b01a8172).
• No documentation, recipe, config, or hardware-support changes were required: docs/v1/evaluation.md and skills/evaluate-environments/references/REFERENCE.md already describe resume as keeping good rollouts and rerunning missing/errored ones, including group-scored tasks being resumed as a whole group. This fix aligns the implementation with that documented behavior.
• Per AGENTS.md, no unit tests were added.

Note

Fix resume to match saved rollouts by positional index rather than data.idx

• Resumed eval runs previously matched saved traces to tasks using task.data.idx, which breaks when the current selection differs from the original run's ordering.
• resume.py now reads info.task_idx from saved traces when present, falling back to task.data.idx remapped through a caller-supplied data_idx_to_pos dict, or raw data.idx as a last resort.
• runner.py builds a task_positions map (data.idx → position in selected slice) and passes it to resume.load; owed rollouts and post-resume filtering are now driven by these positional indices.
• All newly written traces now include info.task_idx recording the task's positional index, enabling correct matching in future resumes.
• Behavioral Change: traces without a resolvable index are ignored rather than matched by raw data.idx.

^{Macroscope summarized 627ff7b.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a3fd46612e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve legacy data.idx matches when resuming

When resuming output dirs written before info.task_idx existed, those rows fall back to task.data.idx, but this call now passes selected positions instead of the saved data indices. For tasksets whose saved ids are sparse or raw dataset rows (the Lean case this change targets), completed legacy rows no longer match selected_idxs, so resume.load drops them, rewrites traces.jsonl without them, and reruns work that had already finished.

Useful? React with 👍 / 👎.