The Missing Line of Code
Anthropic's 'Share with link' pages for Claude conversations were never marked with a noindex meta tag or HTTP header - the standard directive that tells search engines not to list a page [1]. Instead, Anthropic relied on a basic robots.txt exclusion for the share directory, and robots.txt is a request, not an enforceable rule: crawlers routinely disregard it once an external site links directly to a specific URL, which is exactly what happens when a shared chat gets posted to a forum, a tweet, or a Reddit thread [2].
The distinction sounds like a technicality, but it is the entire story. noindex lives on the page itself and search engines must obey it. robots.txt lives at the domain root and only politely asks crawlers to stay out of a folder - it does nothing once a page is individually linked from somewhere else on the open web. Anthropic built a feature whose entire premise was 'share with a link, not with the world,' then protected it with the one mechanism that cannot actually keep a linked page off Google.


