News
OpenAI said it will honor a shared robots.txt signal so website operators can express AI training preferences to participating providers.
OpenAI said it will honor a shared robots.txt signal, giving website operators a way to communicate their AI training preferences to participating providers, according to a company announcement published on Fri, 30 Oct 2026.
Evidence and context
The announcement frames the change as a coordination mechanism rather than a security control. robots.txt is a public file at a site’s /robots.txt path, and its User-agent, Allow and Disallow directives communicate crawler-access preferences. Under RFC 9309, those rules are not access authorization and do not replace security controls; a scraper that does not cooperate can ignore them.
OpenAI’s crawler documentation also distinguishes between training and search behavior. The background material says that a directive such as User-agent: GPTBot with Disallow: / indicates content should not be crawled for training, while allowing OAI-SearchBot can preserve eligibility for ChatGPT search. OpenAI says search systems can take approximately 24 hours to adjust to robots.txt changes.
The broader context is that other providers have introduced similar content-preference tools. Cloudflare announced a tool for creators to control use of their content on 2025-09-24, and earlier, on 2025-07-01, it announced managed robots.txt and blocking options for monetized content. The provided background notes that such signals are purpose-specific preferences and should not be treated as universally enforced technical restrictions.
The source material also cautions that updating robots.txt does not show that previously collected material has been removed from datasets or existing models. No additional implementation details, dates for rollout beyond the announcement date, or evidence of enforcement across all providers were supplied.
Credit: Source link
