Google, Microsoft, OpenAI and Anthropic back one robots.txt line to opt out of AI training

Until now, opting out meant writing separate rules for each AI company's crawler or control. OpenAI says operators had to "identify individual crawlers and maintain separate rules for each and every provider"[1], and Google says site owners will no longer be managing separate instructions for every AI developer[2].

The draft adds no new syntax. A site writes an ordinary robots.txt group whose User-Agent line names a policy token, __usage_ai-training, and uses Disallow and Allow to state a preference against or in favour of training on each path[5]. The draft's own example opts a whole site out except /press-kit/[5]. Two details matter: a User-agent: * group does not count as a training preference, and the token states how content may be used, not whether a crawler may fetch it[5].

Microsoft is the only company with a firm date. It will honour the new preferences as it crawls from 1 February 2027, adds a page-level training override and a separate page-level grounding preference, and is retiring NoArchive and NoCache as its AI controls[3]. A new DisallowExtended grounding value keeps content in Microsoft's own AI experiences such as Copilot while excluding grounding services Microsoft sells to third parties[3]. Microsoft also says plainly that neither choice guarantees inclusion, traffic or payment[3].

Google aims to support the control in 2027 and will treat a shared-token opt-out as if the site had disallowed Google-Extended, unless Google-Extended is set explicitly[2]. That has a knock-on effect: Google says sites that allow Google-Extended are eligible to appear via grounding in Gemini Apps, so a blanket opt-out at Google also reaches Gemini answers, not just training[2][5].

OpenAI says it aims to implement the signal "in the coming months", that its existing crawler controls stay in place, and that it encourages other providers to honour it[1].

The IETF Datatracker lists the document as an individual submission that is not endorsed by the IETF and has no formal standing in its standards process[4]. Version 00 expires on 3 May 2027 unless revised[5]. Meta, Amazon and Apple are not among the companies that committed to read it[5].

For publishers this turns a growing, vendor-by-vendor robots.txt list into one line, and for the first time the biggest consumers of web text have committed to read the same one. The catch is that "training" and "being cited in AI answers" are handled differently at each company: Microsoft splits them, while Google ties a shared opt-out to Gemini grounding. Anyone who wants to stay visible in AI answers but out of training sets should read each provider's rules before adding the token.