robots.txt review

[ghostdevv]

I can remove the redundant rules. I would like someone on the core or maintainer teams to request/sign off because I want changes to be as focused as possible, and my main goal is to fix the malformed robots.txt that is giving incorrect access to bots

I had Composer 2.5 LLM run a report for me on redundant fields, and this is what it came up with, and all items seem reasonable to me :

editing to remove direct quote of AI response for human-written summary. Thanks to a suggestion from @trueberryless

1. Remove the disallow rules after Disallow: / covering lines 19–44 because Disallow: /. covers everything that is not explicitly allowed. Thus, delete - I think this is a good idea, so we do not reveal extra paths that we do not want bots/people to see.
2. `/opensearch.xml' is both allowed and disallowed (line 15, 41). I think this should be allowed. There feels like there is a story around this that I would like to understand before taking any action.
3. Specific AI classifier blocks are duplicates and can be simplified to just list out the crawlers and blocks once. I am neutral on this and can see value in having it both ways.

If I should implement any of these items, please let me know, and I can do it.

Originally posted by @MaxwellCohen in #2892 (comment)

[MaxwellCohen]

I can have a pr doing these changes in a few hours

[ghostdevv]

Remove the disallow rules after Disallow: / covering lines 19–44 because Disallow: /. covers everything that is not explicitly allowed. Thus, delete - I think this is a good idea, so we do not reveal extra paths that we do not want bots/people to see.

I'm half tempted to suggest we keep it for clarity, but not sure - what do other people think? the paths aren't a secret we just want to avoid automated traffic/search engines I guess

`/opensearch.xml' is both allowed and disallowed (line 15, 41). I think this should be allowed. There feels like there is a story around this that I would like to understand before taking any action.

I assume this should be disallowed as it's something that the browser looks at, not people - but @danielroe might know why

Specific AI classifier blocks are duplicates and can be simplified to just list out the crawlers and blocks once. I am neutral on this and can see value in having it both ways.

wdym?

[ghostdevv]

I also think we should check if there are any routes that should be accessible by bots that currently aren't - it's probable we've added new pages and not updated the robots.txt

[MaxwellCohen]

Based on what @ghostdevv suggests, it seems like we should not remove extra Disallow lines. But will update `opensearch.xml' to be either fully disabled or enabled based on more info from @danielroe.

---

Doing a quick manual check of new pages that are not specifically called out, here are my thoughts:

Some pages that are currently blocked, but I think should not be:

• leaderboard pages such as https://npmx.dev/leaderboard/likes -> thus add Allow: /leaderboard

Pages that are blocked that I think should be, but could be easily convinced otherwise, and if we go with calling out path disallow:

• profile pages: https://npmx.dev/profile/danielroe.dev
• package timeline: https://npmx.dev/package-timeline/vue/v/3.5.36
• diff pages: https://npmx.dev/diff/vue/v/3.6.0-alpha.5...3.5.35

---

Specific AI classifier blocks are duplicates and can be simplified to just list out the crawlers and blocks once. I am neutral on this and can see value in having it both ways.
wdym?

I am saying that we could combine user agents like this:

# Block AI training crawlers from scraping package content
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: CCBot
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: Google-Extended
User-agent: Bytespider
User-agent: cohere-ai
Disallow: /

I am not seeing a clear advantage or disadvantage in doing it or not doing it. When I look at major sites, they are divided between the approaches.

• OpenAI and Amazon are doing what we are doing today, having sections for each bot
• Anthropic and Google does the combined approach

So I vote for keeping what we have today.

---

In doing more research, some other items we can do:

• Content-Signal: search=yes, ai-input=no, ai-train=no line in like what Vercel has
• Another common thing in robots.txt that we are missing is sitemaps, which would require creating the sitemaps and splitting them into chunks to stay under search engine limits.