Skip to content

Make cache:prepare self-contained in fresh worktrees#4292

Open
habdelra wants to merge 9 commits intomainfrom
worktree-cs-10560-cache-prepare
Open

Make cache:prepare self-contained in fresh worktrees#4292
habdelra wants to merge 9 commits intomainfrom
worktree-cs-10560-cache-prepare

Conversation

@habdelra
Copy link
Copy Markdown
Contributor

@habdelra habdelra commented Apr 1, 2026

Summary

Resolves CS-10560: cache:prepare now works in a completely fresh worktree with zero running services.

  • Auto-provision build artifacts: Host dist, boxel-ui dist, and boxel-icons dist are automatically symlinked from the root repo (fast) or built from source (fallback) when missing in a worktree. Uses fs.symlinkSync for cross-platform compatibility.

  • Managed icon server lifecycle: Always starts our own icon server process instead of hoping an externally-detected server stays alive. Falls back to the existing dev server if port 4206 is already taken (with EADDRINUSE retry for slow-starting external servers).

  • Indexing progress bar: Pre-allocates the worker-manager port and polls its /_indexing-status JSON endpoint every 2s. Shows an in-place single-line progress bar that updates as each realm is indexed. File counts come from the invalidation graph so they're precise. The reporter tracks seen realms to show how many remain:

      indexing: waiting for status 3s
      indexing base: discovering files... 8s (2 realms remaining)
      indexing base [===============>              ] 105/153 files (69%) 50s (2 realms remaining)
         ↓ (line switches in-place when realm finishes)
      indexing software-factory [============>         ] 113/204 files (55%) 155s (1 realm remaining)
         ↓
      indexing test [=======================>      ] 10/12 files (83%) 230s
      indexing complete: 12/12 files in 235.0s
    
  • Graceful pg unavailability: databaseExists() returns false instead of throwing when postgres isn't running yet.

  • Stale context validation: Validates both hostURL and matrixURL in cached support.json, discarding stale contexts.

Test plan

  • cache:prepare succeeds in a fresh worktree with no services running (pg, synapse, host, icons all auto-started)
  • cache:prepare succeeds alongside a running mise run dev-all without disrupting dev services
  • Host app at localhost:4200 remains fully functional during and after cache:prepare
  • Progress bar updates live during indexing (verified by polling /_indexing-status endpoint directly: base 0→150/153, software-factory 4→198/204, test 0→10/12)
  • totalFiles from the invalidation graph is precise
  • Remaining realm count decrements as realms complete (2 → 1 → 0)
  • Playwright tests pass with the updated harness

🤖 Generated with Claude Code

cache:prepare now works in a completely fresh worktree with no running
services. Previously it would fail if host dist, boxel-ui dist,
boxel-icons dist, or postgres weren't already available.

Changes:

- Auto-build host dist: when no pre-built host dist exists in the
  worktree or root repo, build it automatically instead of erroring.
  In worktrees, symlinks from the root repo's built dist when available.

- Auto-provision boxel-ui and boxel-icons dist: symlink from root repo
  in worktrees (fast path), or build from source as fallback.

- Managed icon server lifecycle: instead of detecting an external icon
  server and hoping it stays alive, always start our own managed
  process. Falls back to the existing dev server if port 4206 is
  already taken.

- Indexing progress bar: polls the boxel_index table every 2s during
  template builds and renders an in-place progress bar on stderr:

    indexing [===============               ] 199/373 files (53%) 80s
    indexing [==============================] 603/373 files (100%) 212s
    indexing complete: 603 files indexed in 229.5s

- Graceful pg unavailability: databaseExists() now returns false
  instead of throwing when postgres isn't running yet, allowing
  startFactorySupportServices() to start it.

- Stale context validation: cache-realm.ts now validates both hostURL
  and matrixURL in cached support.json, discarding stale contexts
  where either service is unreachable.

Tested: cache:prepare succeeds in a fresh worktree with zero services
running, and also succeeds alongside a running dev-all without
disrupting it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@chatgpt-codex-connector
Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the software-factory cache:prepare flow to be self-contained in fresh git worktrees by auto-provisioning required build artifacts, taking direct ownership of the icon server lifecycle, and adding indexing progress reporting during template builds.

Changes:

  • Auto-build or (attempt to) symlink missing host/boxel-ui/boxel-icons dist artifacts to avoid manual setup in new worktrees.
  • Always start a managed icon server process (with intended fallback to an already-running dev server) and ensure icon dist exists before serving.
  • Add stderr progress bar for indexing by polling boxel_index, plus make databaseExists() tolerant of Postgres being down and validate cached support context URLs.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File Description
packages/software-factory/src/harness/support-services.ts Adds dist auto-provisioning and managed icon server startup/stop logic.
packages/software-factory/src/harness/database.ts Makes DB existence checks resilient and adds indexing progress reporter + file counting.
packages/software-factory/src/cli/cache-realm.ts Validates cached support.json context by probing hostURL and matrixURL.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Replace spawnSync('ln', ...) with Node fs.symlinkSync for
  cross-platform symlink creation in ensureBoxelUIDist and
  ensureBoxelIconsDist
- Re-validate boxel-icons dist after build to catch partial output
- Add child.on('error', ...) handler to icon server process to catch
  spawn failures
- Use fallback child.kill() when process group kill fails in icon
  server stop
- Keep polling probe URL when icon server exits with EADDRINUSE
  instead of throwing immediately (allows external server startup time)
- Guard against overlapping DB polls in progress reporter with
  in-flight flag

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@habdelra habdelra force-pushed the worktree-cs-10560-cache-prepare branch from 2cc315a to 7a91f1c Compare April 1, 2026 03:42
Pre-allocate the worker-manager port via findAvailablePort() and pass
it to both startIsolatedRealmStack and the progress reporter. This
lets the reporter poll the /_indexing-status JSON endpoint which has
the exact invalidation graph computed by the index runner.

Progress output shows the current realm, file counts, and queued realms:

    indexing: waiting for status 3s
    indexing base: discovering files... 8s (2 realms remaining)
    indexing base [=====>                        ] 50/153 files (33%) 30s (2 realms remaining)
    indexing software-factory [========>             ] 22/204 files (11%) 94s
    indexing test [==============================] 13/13 files (100%) 210s
    indexing complete: 13/13 files in 229.5s

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@habdelra habdelra force-pushed the worktree-cs-10560-cache-prepare branch from 7a91f1c to ff78820 Compare April 1, 2026 06:55
@habdelra habdelra requested a review from a team April 1, 2026 06:55
habdelra and others added 2 commits April 1, 2026 09:27
Each harness instance now gets a fresh port via findAvailablePort()
instead of falling back to the static DEFAULT_WORKER_MANAGER_PORT
env var, which caused EADDRINUSE collisions in parallel test runs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each harness instance now gets a fresh port via findAvailablePort()
instead of using a pre-allocated port from the test fixture, which
caused EADDRINUSE collisions due to TOCTOU races.

- Remove DEFAULT_WORKER_MANAGER_PORT constant and
  SOFTWARE_FACTORY_WORKER_MANAGER_PORT env var entirely
- Remove workerManagerPort from TestWorkerPortSet (no longer
  pre-allocated per worker)
- startIsolatedRealmStack always uses findAvailablePort() when no
  explicit port is provided
- cache:prepare path unchanged: still pre-allocates port for progress
  monitoring via /_indexing-status

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
habdelra and others added 4 commits April 1, 2026 14:52
The comment and sizing rationale now accurately reflects that only
compat proxy, realm-server, and prerender ports are pre-allocated.
The worker-manager port is dynamically assigned per harness instance.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…uild

- startIconServerProcess: track spawn errors via spawnFailed() flag so
  ensureIconsReady fails immediately instead of polling for 30s
- ensureBoxelUIDist/ensureBoxelIconsDist: remove leftover symlinks
  before falling back to pnpm build, preventing writes through the
  symlink into the root repo's dist directory

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… repo

Check icons.js and styles/global.css in addition to components.js and
helpers.js when deciding whether the root repo's boxel-ui dist is
complete enough to symlink. Prevents symlinking a partially built dist
that would fail the usability check and fall through to the build
fallback.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tions

The report() callback runs on a setInterval — any thrown error would
surface as an unhandled promise rejection. Add a catch block so
progress reporting failures are silently swallowed instead of crashing
the process.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants