Official Robots.txt Entry to Block AI from Training on Website Content

Previously, I wrote about how to block all AI scrapers by using robots.txt, then decided to allow them back, and then changed my mind again to welcome the AI bots limited to searching, not training, using Cloudflare’s Content Signal.

It was an early day and nothing was official across AI companies yet. But it’s all changed today.

You see, recently there was news that Microsoft Bing, Google, and even OpenAI announced that they will implement and honor the new AI training control in the robots.txt file as proposed by IETF.

This is good news for website owners. We finally have something official that hopefully will be respected by all the AI bots.

Robots.txt Entry To Block or Allow AI Training

If you allow your content to be used to train generative AI models, you don’t have to do anything. It’s an opt-in by default, similar to search indexing. Optionally, you can also add an entry to explicitly allow the bots to train their AI using your content.

# Allow is default. No action needed to allow ai training. To specify explicitly use
User-Agent: __usage_ai-training
Allow: /

But if you don’t want your content to be used to train generative AI models, you need to set a Disallow entry for user agent __usage_ai-training.

Here’s an example to stop the bots from using the content of the entire website as AI training data.

# Disallow ai training for whole site
User-Agent: __usage_ai-training
Disallow: /

You can also block AI training for certain paths of your website. Here’s an example of a robots.txt file that allows AI training of the entire website except for /wp-admin/.

# Disallow ai training at a path-level
User-Agent: __usage_ai-training
Disallow: /wp-admin/

All the companies that adopt this format will (hopefully and sincerely) respect this entry. So you don’t have to write the user agent of each AI company one by one. All you need to do is set the entry for the __usage_ai-training user agent!