Disallow Is Not Noindex: The One-Line SEO Bug Anthropic Missed
A Reddit user's routine Google search first surfaced the exposure, turning up hundreds of readable Claude conversations days before journalists identified the cause. The technical explanation for why hundreds of private-feeling Claude conversations turned into Google search results is almost mundane: a well-known but widely misunderstood distinction in how search engines crawl the web. Shared Claude pages relied on two signals meant to keep them out of search results - a robots.txt rule disallowing the /share/* path, and an X-Robots-Tag: none HTTP header. Neither one actually stops indexing on its own. As search engine analysts explained after the leak, a noindex tag only works if the crawler is allowed to access and read the page; if a page is blocked by robots.txt instead, it can still get indexed - without Google ever reading its content - as long as other pages around the web link to that URL [1]. That's exactly what happened: outside links to shared Claude and Artifacts pages gave Google and Bing enough of a signal to index the URLs anyway, even with the crawler nominally blocked from reading them. Reporting on the incident noted the exposed pages simply lacked the noindex tag that both Bing and Google say they take into account [2]. Once Anthropic added a real, crawlable noindex tag around July 26, the fix worked almost immediately: Google pulled the indexed conversations, and by Monday afternoon the site:claude.ai/share search trick that had surfaced hundreds of chats stopped returning results [3].


