fix(v1): strip the harness's bootstrap venv from run_bash's PATH#2011
Open
eligotts wants to merge 1 commit into
Open
fix(v1): strip the harness's bootstrap venv from run_bash's PATH#2011eligotts wants to merge 1 commit into
eligotts wants to merge 1 commit into
Conversation
The default harness bootstraps itself as a uv script, running inside a uv-managed venv (VIRTUAL_ENV set, its bin/ prepended to PATH). run_bash's subprocess inherited that environment unchanged, so an agent invoking bare `python`/`python3` resolved to the harness's own venv — which only has the harness's deps (openai, mcp, httpx), not the task container's provisioned interpreter or packages. This produced misleading failures that looked like sandbox/network issues: an agent running `python -m pip install X` would see "No module named pip" (the harness venv has no pip) and conclude the sandbox had no internet access, when a working interpreter was one directory away the whole time. Observed distorting real eval scores across multiple runs/models before the root cause was traced. Strip VIRTUAL_ENV and its bin/ entry from run_bash's env before exec, so the agent's shell sees the runtime's actual default environment. This extends the existing run_bash precedent of not leaking harness-private state into the agent's subprocess (secrets already arrive via argv, not env, for the same reason) to the bootstrap venv. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ApprovabilityVerdict: Approved Straightforward bug fix that prevents the harness's virtual environment from leaking into agent bash commands. The change is self-contained, well-documented, and the fix is obvious—stripping VIRTUAL_ENV and its bin directory from PATH before running subprocess. You can customize Macroscope's approvability policy. Learn more. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The default harness bootstraps itself as a uv script, running inside a uv-managed venv
(
VIRTUAL_ENVset, itsbin/prepended toPATH).run_bash's subprocess inherited thatenvironment unchanged, so an agent invoking bare
python/python3resolved to the harness'sown venv — which only has the harness's deps (openai, mcp, httpx) — instead of the task
container's provisioned interpreter or packages.
This produces misleading failures that look like sandbox/network issues: an agent running
python -m pip install XseesNo module named pip(the harness venv has no pip) andconcludes the sandbox has no internet access, when a working interpreter is one directory
away the whole time (
/usr/local/bin/python, for example). We traced this after several evalrollouts appeared to fail on "no network" while other rollouts in the same sandbox, same run
succeeded at outbound network calls once they stumbled onto a working interpreter — the
harness venv leak was silently distorting model scores.
Fix
Strip
VIRTUAL_ENVand itsbin/entry fromrun_bash's env before exec, so the agent'sshell sees the runtime's actual default environment. This extends the existing
run_bashprecedent of not leaking harness-private state into the agent's subprocess (secrets already
arrive via argv, not env, for the same reason) to the bootstrap venv.
Test plan
ruff check/ruff formatpassbefore the fix,
python -m pipinsiderun_bashresolved to the harness venv andfailed; after, it resolves to the container's provisioned interpreter.
run_bashvs. scrubbingat the bootstrap/launch layer is the preferred fix location — happy to move it if so.
🤖 Generated with Claude Code
Note
Low Risk
Small, localized change to the default harness bash tool env; improves eval fidelity with no auth or data-path impact.
Overview
Agent
bashtool commands no longer inherit the harness’s uv bootstrap virtualenv, so barepython/python3resolve to the task container’s interpreter instead of the harness-only venv (openai/mcp/httpx, often without pip).run_bashnow builds a copy of the process environment, removesVIRTUAL_ENV, and drops that venv’sbinfromPATHbeforesubprocess.run, matching the existing pattern of not leaking harness-private state into agent shells (same rationale as passing secrets via argv rather than env).Reviewed by Cursor Bugbot for commit 259e729. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Strip harness bootstrap venv from
run_bashPATH in v1 default harnessThe harness's own virtualenv was leaking into commands spawned by
run_bash, causing incorrect interpreter resolution. program.py now builds a filtered environment before callingsubprocess.run: it removesVIRTUAL_ENVand strips the venv'sbindirectory fromPATH. Behavioral Change: commands run viarun_bashnow resolve binaries against the system PATH instead of the harness venv.📊 Macroscope summarized 259e729. 1 file reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted
🗂️ Filtered Issues
No issues evaluated.