Drop the boxel_index HTML columns and the dual read/write paths#5476
Open
habdelra wants to merge 5 commits into
Open
Drop the boxel_index HTML columns and the dual read/write paths#5476habdelra wants to merge 5 commits into
habdelra wants to merge 5 commits into
Conversation
prerendered_html is the sole home of rendered output. The migration drops isolated_html, head_html, embedded_html, fitted_html, atom_html, and markdown from boxel_index / boxel_index_working along with their markdown FTS GIN indexes (icon_html stays — the icon renders in the index visit). Readers lose the ph-or-boxel_index CASE fallbacks and select the HTML channel columns directly; the FTS matches predicate reads only ph.markdown. The fused (SQLite) indexing path writes each entry's HTML half straight to prerendered_html_working in updateEntry and mirrors tombstones there, replacing the boxel_index_working projection; the realm copy fills the channel solely from the source's prerendered_html rows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A setupIndex fixture row with no rendering fields now gets no prerendered_html row — it reads as indexed-but-unrendered, matching how a row awaiting its render job reads in production. Also removes a leftover reference to the dropped boxel_index HTML-columns helper in the split integration test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b7a39690fa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
Preview deploymentsHost Test Results 1 files ±0 1 suites ±0 3h 2m 41s ⏱️ + 5m 50s Results for commit 18a22e1. ± Comparison against earlier commit 28858e1. Realm Server Test Results 1 files ±0 1 suites ±0 11m 33s ⏱️ -5s Results for commit 18a22e1. ± Comparison against earlier commit 28858e1. |
…TML half first The matches-filter Postgres integration suite seeds markdown directly onto prerendered_html (the sole markdown column) and asserts the planner against prerendered_html_markdown_fts_idx. In the fused updateEntry the prerendered_html_working write lands before the boxel_index_working row: that row is the resume marker, so a crash between the two writes re-visits the URL instead of resuming a row whose rendering never landed. Drops an unused eslint-disable directive in the migration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR completes the transition to prerendered_html as the sole storage for rendered output by removing the legacy HTML/markdown columns from boxel_index and deleting the remaining dual read/write behavior across runtime code, realm-server readers, and test harnesses.
Changes:
- Drops
boxel_index{,_working}HTML/markdown columns (and related FTS/GiN indexes) via a Postgres migration and regenerates the host SQLite schema accordingly. - Updates runtime query + read paths to source rendered output and full-text
matchesexclusively from theprerendered_htmljoin (ph.*), while keepingicon_htmlonboxel_index. - Updates writer + copy/tombstone logic and reshapes tests/fixtures so rendered output is seeded and asserted via
prerendered_htmlrather thanboxel_index.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/runtime-common/realm-index-query-engine.ts | Updates generation comment to reflect single-source prerendered HTML. |
| packages/runtime-common/index-writer.ts | Removes boxel_index HTML writes; writes/tombstones prerendered_html directly on fused path; keeps split-job behavior; updates copy semantics. |
| packages/runtime-common/index-structure.ts | Removes HTML/markdown fields from BoxelIndexTable; documents icon_html remaining. |
| packages/runtime-common/index-runner/visit-file.ts | Removes dual-read explanation; clarifies split-mode writes only search-doc half. |
| packages/runtime-common/index-query-engine.ts | Collapses dual-read CASE logic to direct ph.* selects; matches reads only ph.markdown; refactors select helpers/types accordingly. |
| packages/realm-test-harness/src/database.ts | Stops rewriting/rehydrating removed boxel_index HTML columns; keeps prerendered_html rewrite path. |
| packages/realm-server/tests/server-endpoints/index-responses-test.ts | Updates tests to mutate only prerendered_html for head HTML behaviors. |
| packages/realm-server/tests/search-entries-engine-test.ts | Updates mixed-index tests to clear renderings only on prerendered_html. |
| packages/realm-server/tests/realm-endpoints/search-test.ts | Updates mixed-index fallback test to clear renderings only on prerendered_html. |
| packages/realm-server/tests/prerendering-test.ts | Updates isolated-html injection test setup to write only prerendered_html. |
| packages/realm-server/tests/prerender-html-split-integration-test.ts | Removes assertions about boxel_index HTML columns; asserts icon stays on boxel_index. |
| packages/realm-server/tests/matches-filter-integration-test.ts | Seeds markdown in prerendered_html; asserts planner uses prerendered_html_markdown_fts_idx. |
| packages/realm-server/tests/card-source-endpoints-test.ts | Updates comment to reflect rendered output living on prerendered_html. |
| packages/realm-server/lib/retrieve-scoped-css.ts | Switches scoped CSS deps sourcing to prerendered_html-only (inner join). |
| packages/realm-server/lib/index-url-utils.ts | Updates comment wording for join context. |
| packages/realm-server/lib/index-html-injection.ts | Switches head/isolated HTML retrieval to prerendered_html-only (inner join). |
| packages/realm-server/handlers/handle-publish-realm.ts | Updates comment describing stale-source risk to refer to prerendered_html. |
| packages/postgres/migrations/1783724976754_drop-boxel-index-html-columns.js | Drops legacy HTML/markdown columns + markdown FTS indexes from boxel_index tables; restores on down. |
| packages/host/tests/unit/prerendered-html-read-test.ts | Replaces dual-read suite with prerendered-html-only read-path assertions (including FTS semantics). |
| packages/host/tests/unit/index-writer-test.ts | Updates expectations to assert markdown/render output in prerendered_html; icon remains in boxel_index; updates copy semantics. |
| packages/host/tests/helpers/indexer.ts | Changes fixture seeding to split index-vs-render halves into boxel_index and prerendered_html inserts. |
| packages/host/config/schema/1783724976754_schema.sql | Regenerates SQLite schema without boxel_index HTML/markdown columns. |
| .claude/skills/indexing-diagnostics/SKILL.md | Updates diagnostic note to reflect inline prerendered_html write (not projection). |
| .claude/skills/index-query-engine/SKILL.md | Updates documentation to describe prerendered_html as sole rendered-output channel. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Matches the form the live prerendered_html writes use, so alias-keyed consumers see one alias per URL whether the row is live or tombstoned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ents Boxel tombstones write the same .json-stripped file_alias as updateEntry so alias-keyed lookups see one alias per URL, matching the prerendered_html tombstone form. The down migration drops the markdown FTS index before recreating it, so an INVALID leftover from a failed CONCURRENTLY build can't be skipped by IF NOT EXISTS. Two comments that described the removed boxel_index projection/fallback now describe the current behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jurgenwerk
approved these changes
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Indexing writes two kinds of output for every card: the search doc (queryable field data) and the prerendered HTML for each format. These used to live side by side in
boxel_index, and when the render was split off the indexing hot path the HTML gained its own home — theprerendered_html/prerendered_html_workingtables — with theboxel_indexcopies kept around as a dual-read fallback while the new path soaked. Every reader has been serving HTML and markdown fromprerendered_html(falling back only for rows with noprerendered_htmlrow), and every writer has been landing renderings on that channel.This PR is the final cleanup of that transition:
prerendered_htmlis now the sole home of rendered output, and theboxel_indexcopies are gone.What changed
Migration. Drops
isolated_html,head_html,embedded_html,fitted_html,atom_html, andmarkdownfromboxel_indexandboxel_index_working, along with their markdown FTS GIN indexes (the fitted/embedded GIN indexes fall with their columns).icon_htmlstays — the icon renders in the index visit and lives onboxel_index. The host's generated SQLite schema is regenerated to match.Readers. The query engine's per-column
CASE WHEN ph.url IS NULL THEN i.… ELSE ph.… ENDfallbacks collapse to plainph.…selects, and the full-textmatchespredicate reads onlyph.markdown— a row with noprerendered_htmlrow has no rendering and no full-text membership until its render lands. Theprerendered_htmlLEFT JOIN itself stays: it is how reads attach the HTML channel, the rendering generation, and the render-error side of the effective error state. The realm server's head-HTML / isolated-HTML injection and scoped-CSS retrieval likewise read the channel directly.BoxelIndexTableloses the HTML/markdown fields; the single-entry read paths type their joined rows asboxel_indexrow +prerendered_htmlrendering columns.Writers.
updateEntryno longer writes HTML intoboxel_index_working. On the fused path (the in-browser SQLite realm, which has no separate render job) it lands each entry's HTML half directly onprerendered_html_working— sharing the row writer with theprerender_htmljob's batch — andtombstoneEntriesmirrors deletions onto the channel. The batch-commit projection that copied HTML fromboxel_index_workingontoprerendered_html_workingis gone. The realm copy operation fills the destination's channel solely from the source'sprerendered_htmlrows; a source URL with no rendering stays unrendered at the destination and reads as such (the catch-up sweep enqueues its render).Tests. The direct-construction harnesses (
setupIndex, the realm-test-harness clone/rebuild SQL) seed each fixture row's HTML half ontoprerendered_htmldirectly instead of projecting it fromboxel_index. The dual-read unit suite is reshaped into aprerendered_htmlread-path suite asserting the channel is the sole source: every read path serves from it, an absent row reads as unrendered, and FTS membership followsph.markdown.Test plan
pnpm migrate up) and the SQLite schema regenerated (pnpm make-schema).ember test --filter "Unit |") — covers the index writer, query engine, and the new prerendered-html read-path suite.realm indexingmodule — the fused end-to-end path._searchendpoint, index responses, and the matches-filter Postgres integration suite (seeds markdown ontoprerendered_html; planner asserted againstprerendered_html_markdown_fts_idx).