Open
Conversation
Adds an explicit Content Signals declaration (contentsignals.org,
2025) to the existing minimal wildcard file:
Content-Signal: search=yes, ai-input=yes, ai-train=yes
The Content Signals Policy — rolled out by Cloudflare to 3.8M+
domains since Sept 2025 — is the emerging post-user-agent-allowlist
mechanism for expressing AI policy in robots.txt. Three signals:
search — appear in traditional search indexes
ai-input — used in live retrieval / AI answer citations
(ChatGPT, Claude, Perplexity, etc.)
ai-train — used to train or fine-tune AI models
Flow developer docs are open source and we want maximum
discoverability for humans and AI agents, so all three are set to
yes. Rest of the file unchanged: still a wildcard allow-all (matches
the pattern used by Mintlify, Supabase, Vercel, Anthropic, OpenAI,
Stripe, Python docs, and developers.google.com).
Also documents the companion /llms.txt and /llms-full.txt AI-indexed
documentation paths plus the cadence-lang.org companion index.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an explicit Content Signals Policy declaration to
static/robots.txt:search=yes— appear in traditional search indexesai-input=yes— used in live AI retrieval / citation (ChatGPT, Claude, Perplexity, etc.)ai-train=yes— used to train or fine-tune AI modelsThe Content Signals Policy — rolled out by Cloudflare to 3.8M+ domains since Sept 2025 — is the emerging post-user-agent-allowlist mechanism for expressing AI policy in
robots.txt. Flow developer docs are open source and we want maximum discoverability for humans and AI agents, so all three signals are set toyes.What this is NOT
anthropic-aiin favor ofClaude-User/Claude-SearchBot).User-agent: */Disallow:. This PR just adds the explicit AI policy declaration.Test plan
curl https://<preview>/robots.txtreturns the new contentCompanion work
A parallel PR on onflow/cadence-lang.org (#292) applies the same Content Signals policy plus adds llms.txt generation. These two sites should stay in lockstep on AI policy.