Add native OpenEnv V1 taskset support#1997
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review Unable to check for correctness in 29ea522. This PR introduces a new OpenEnv V1 taskset feature with ~170 lines of new integration code, new components (OpenEnvTaskset, OpenEnvUser), and modifications to core runtime behavior in launch.py and server.py. New feature with external integrations warrants human review. You can customize Macroscope's approvability policy. Learn more. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: aed0c1aa11
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e0d114956
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f8ad2c423e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f6c4116031
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad39bead64
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
AI Code Review (PR Review Agent)
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c7d3b606cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
91231ec to
0a98a78
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0a98a783eb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
d15e217 to
c173e7f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c173e7f8e9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 83f968b. Configure here.
2 participants
Overview
Adds a reusable
openenv-v1adapter package that lets OpenEnv own environment startup and interaction while Verifiers maps reset observations, actions, rewards, and termination into the standard V1 task, user, and trace model.The public surface stays small: import
OpenEnvConfigandOpenEnvTasksetfromopenenv_v1for any OpenEnv environment id or existing server, or subclass them to provide environment-specific defaults.openenv_wordle_v1demonstrates the thin-wrapper pattern by supplying only Wordle's non-default ASGI app.Execution model
OpenEnv runs as the task's V1 user simulator, independently of the model harness:
harness.runtime.*places the model harness.taskset.task.user.runtime.*places the OpenEnv user for UV andbase_urlruns.subprocessruntime.taskset.resetsbecomes one finite task and is passed directly to OpenEnv'sreset.The reset observation opens the conversation, so the adapter does not add a generic OpenEnv prompt. Each user turn contains the current observation and live action schema. Step rewards are accumulated on the V1 trace, and OpenEnv's
donesignal stops the rollout.Environment startup modes
UV provider (default)
env="openenv/<name>"callsGenericEnvClient.from_env(..., use_docker=False). OpenEnv resolves the environment and launches its ASGI application with UV. Remote ids may require Git, network access, and environment-specific system packages in the selected user runtime;project_pathcan point at source already present in that runtime.OpenEnv-native launch options live under
task.user.provider_kwargs, includingapp,env_vars,project_path,port,workers,host,reload, and startup timeout settings.UV works with host subprocess, Verifiers Docker, Prime, Modal, or a colocated harness runtime when that runtime provides the source and system requirements. It is the default because it does not require an inner Docker daemon.
OpenEnv Docker provider
use_docker=trueasks OpenEnv to pull and run the environment's published image. Verifiers does not build an environment image.Docker-backed OpenEnv always receives its own Prime VM user runtime (
colocated=false,vm=true) so the Docker daemon runs inside a VM rather than inside another container. Existing Prime settings such as image, resources, networking, region, and labels are retained; onlyvmis forced on. A non-Prime user runtime selection is replaced with the Prime VM configuration.Existing OpenEnv server
base_urlconnects to an already-running OpenEnv server. It performs no clone, dependency resolution, image pull, or server launch. The endpoint must be reachable from the configured user runtime. An inheritedenvdefault or unused provider settings do not affect this connection mode.Configuration and extension
env: OpenEnv environment id.base_url: existing OpenEnv server; required only whenenvis absent.use_docker: use OpenEnv's Docker provider instead of UV.resets: finite list of dictionaries passed toreset; defaults to one empty reset.task.user.provider_kwargs: options passed toGenericEnvClient.from_env.task.user.runtime: placement for UV orbase_url; Docker always selects a Prime VM.task.user.colocated: reuse the harness runtime for UV orbase_url; Docker forces this off.On setup the adapter connects the client, reads the environment's live schema, discovers concrete tools for generic MCP-style environments, and resets the environment. It accepts structured JSON actions plus raw values for single-field schemas, sums per-step rewards, and maps
doneto V1 stopping.The included Wordle wrapper can be run directly:
Other OpenEnv packages can depend on
openenv-v1, subclassOpenEnvConfigto setenvand any required provider defaults, then inheritOpenEnvTaskset.Supporting V1 server changes
The standalone adapter declares its own OpenEnv dependency, keeping it out of the main Verifiers package extras while letting sandbox launch install it with the adapter. Server setup, serving, and cleanup now share one async event loop, so OpenEnv's native context manager closes UV processes, Docker containers, and remote sessions on graceful shutdown. Sandboxed server processes retain UV's user-bin path, and subprocess provider temp files live in the runtime workdir so normal runtime cleanup removes them.
Compared with the legacy integration
OpenEnvEnvbase_urlenvironments/<name>/proj/treeopenenv.yaml, server package/Dockerfile, thenvf-buildLegacy
vf-buildpushed a Prime image and generated the metadata used to start each rollout. V1 removes that Verifiers-specific build and lifecycle layer:base_urldoes neither.The legacy API and
vf-buildremain separate; V1 OpenEnv tasksets do not use them.Note
Medium Risk
New external OpenEnv integration and Docker-forced Prime VM placement affect rollout runtime behavior; lockfile adds a large dependency surface but core verifiers paths are mostly additive.
Overview
Introduces a reusable
openenv-v1package that wires OpenEnv into Verifiers V1:OpenEnvUserconnects viaGenericEnvClient(UV, Docker, orbase_url), loads the live action schema (including MCP tool discovery), maps agent messages tostep/reset, and scores rollouts from accumulated rewards and OpenEnv’sdoneflag.openenv-wordle-v1is a thin subclass that pinsopenenv/wordleand defaults Wordle’s non-standard ASGI app inprovider_kwargs.Both packages are added to the repo
examplesgroup anduv.sources;uv.lockgains theopenenvdependency tree.Supporting MCP changes: sandbox launches set
TMPDIRon subprocess runtimes and wrap server startup in one async loop withAsyncExitStackso providers like OpenEnv can shut down cleanly; non-subprocess launches preservePATHforuv.Reviewed by Cursor Bugbot for commit 29ea522. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add native OpenEnv V1 taskset support for Wordle and generic environments
openenv_v1package withOpenEnvTaskset,OpenEnvUser, and supporting config/data models to run OpenEnv-compatible environments as verifier tasksets.OpenEnvUser.setup_taskconnects to or launches an OpenEnv instance, introspects the action schema over HTTP (including MCP tool listing), and performs an initial reset before interaction begins.OpenEnvUser.respondsteps the environment on each turn, accumulates reward, and returns the observation plus action schema; termination and scoring are driven by the environment'sdoneflag and cumulative reward.openenv_wordle_v1package as a concrete specialization, defaulting the provider app totextarena_env.server.app:app.OpenEnvConfigvalidation enforces that eitherenvorbase_urlis set, and automatically switches the user runtime to VM mode whenuse_dockeris true.serve_in_runtimein launch.py to setTMPDIRto the runtime workdir for subprocess providers and to launch sandboxed providers through a shell that exports$HOME/.local/binonPATH.Macroscope summarized 29ea522.