feat(docs): serve per-page markdown variants for LLMs#22712
Draft
critesjosh wants to merge 3 commits into
Draft
Conversation
Generate clean .md siblings for every versioned developer/operate page at build time and serve them either via `.md` URL suffix or `Accept: text/markdown` content negotiation (per acceptmarkdown.com / RFC 9110), so LLMs and CLI tools can consume docs without scraping rendered HTML.
- Strip trailing /index from slugs so /developers/docs/aztec-js.md resolves to the section landing page instead of being missing. - Honour CONTEXT env for the operate instance: mainnet routes to /operate/ in production and /operate/alpha/ elsewhere, matching docusaurus.config.js. - Rewrite Markdown images that use the @site/static alias so the generated .md doesn't leak a Docusaurus-only path. - Make snippet replacement context-aware: inline inside list items, padded with blank lines only when the tag stands alone. - Fix snippet HTML -> markdown so <p>...<ul>...</ul></p> doesn't collapse into "text:- item". - Skip snippet exports that throw during render instead of failing the build. - Tidy dead code (KNOWN_BENIGN_TAGS, unused args in outputPathFor) and align the Accept tie-breaker comment with the RFC-compatible behaviour.
react-dom/server emits `'` for apostrophes, which the prior literal entity table missed. Handle any decimal or hex numeric reference so snippets don't leak `protocol's` into generated markdown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
.mdsiblings for every versioned developer/operate docs page during build..mdto the URL, or sendAccept: text/markdownon the canonical URL.What's included
docs/scripts/generate_markdown_variants.js— walksdeveloper_versioned_docs/andnetwork_versioned_docs/, strips frontmatter/imports, resolves partial imports, flattens<Tabs>/<TabItem>to##headings, rewrites<Image img={require(...)} />to markdown image syntax, renders<General.*>/<Fees.*>/top-level snippets via esbuild + react-dom/server, and drops<DocCardList/>. A drift check fails the build if any unhandled<CapitalizedTag>remains, so new MDX components surface immediately.docs/netlify/edge-functions/accept-markdown.ts— RFC 9110 content negotiation. Passes through by default; rewrites to the.mdsibling only whentext/markdownis strictly preferred overtext/html; returns 406 when the client explicitly rejects both.docs/netlify.toml— setsContent-Type: text/markdown; charset=utf-8,Vary: Accept, andX-Robots-Tag: noindexfor/*.md(avoids duplicate-content penalty while keeping CDN caching correct).docs/package.json— runs the generator afterdocusaurus build, beforeappend_api_docs_to_llms.js.docs/docs-developers/ai_tooling.md— documents both access paths for LLM tools.docs/docs-words.txt— addsacceptmarkdownfor cspell.Test plan
yarn buildcompletes without drift-check failuresbuild/developers/<slug>.mdexists for every mainnet developer doc; same forbuild/operate/build/developers/testnet/andbuild/operate/testnet/curl https://<preview>/developers/<slug>.mdreturns markdown withContent-Type: text/markdowncurl -H 'Accept: text/markdown' https://<preview>/developers/<slug>returns the same markdowncurl https://<preview>/developers/<slug>still returns HTML (default browser request)curl -H 'Accept: image/png' https://<preview>/developers/<slug>returns 406