ai.robots.txt
This list contains AI-related crawlers of all types, regardless of purpose. We encourage you to contribute to and implement this list on your own site. See information about the listed crawlers and the FAQ.
A number of these crawlers have been sourced from Known Agents and we appreciate the ongoing effort they put in to track these crawlers.
If you’d like to add an AI-related crawler to the list, please see “Contributing” below.
Usage
This repository provides the following files in the top-level directory[^root]:
[^root]: Other files with similar names appear in the directory code/test_files. These files are for test purposes only and you must not use them on your own site.
robots.txt implements the Robots Exclusion Protocol (RFC 9309).
.htaccess may be used to configure web servers such as Apache httpd to return an error page when one of the listed AI crawlers sends a request to the web server.
Note that, as stated in the httpd documentation, more performant methods than an .htaccess file exist.
nginx-block-ai-bots.conf implements a Nginx configuration snippet that can be included in any virtual host server {} block via the include directive.
Caddyfile includes a Header Regex matcher group you can copy or import into your Caddyfile, the rejection can then be handled as followed abort @aibots
haproxy-block-ai-bots.txt may be used to configure HAProxy to block AI bots. To implement it:
-
Add the file to the config directory of HAProxy
-
Add the following lines in the frontend section:
acl ai_robot hdr_sub(user-agent) -i -f /etc/haproxy/haproxy-block-ai-bots.txt
http-request deny if ai_robot
(Note that the path of the haproxy-block-ai-bots.txt may be different in your environment.)
lighttpd-block-ai-bots.conf can be included with include "fragments/lighttpd-block-ai-bots.conf" in your lighttpd configuration either globally or in any conditional section.
Bing uses the data it crawls for AI and training, you may opt out by adding a meta tag to the head of your site.
-
Robots.txt Traefik plugin:
middleware plugin for Traefik to automatically add rules of robots.txt
file on-the-fly.
-
Alternatively you can manually configure Traefik to centrally serve a static robots.txt.
-
Bot Ledger: free, static directory of verified AI crawlers with a one-click robots.txt and llms.txt generator. No signup required.
-
KI-Zugangsindex: open dataset on how widely this kind of blocking is actually deployed in the German (.de) web, measured on a fixed panel of 600 domains so the same sites can be re-checked over time.
-
AI Crawler Census: open dataset measuring which of these crawlers the Tranco top 5,000 sites allow or block, with per-domain results published for each run so the same sites can be compared over time. Raw JSON, CC BY 4.0.
-
AI Discovery Radar: monthly measurement of how many websites actually publish the files that tell AI systems what they may read or use (robots.txt, llms.txt, ai.txt, tdmrep.json and similar), and whether those files can be fetched at all. Open data, CC BY 4.0.
-
AI Visibility Checker: checks whether various AI crawlers can reach a site, testing robots.txt, the CDN and the server, and shows exactly what blocks each one and how to fix it.
Contributing
Please note that AI-generated contributions are not permitted.
A note about contributing: updates should be added/made to robots.json. A GitHub action will then generate the updated robots.txt, table-of-bot-metrics.md, .htaccess and nginx-block-ai-bots.conf.
You can run the tests by installing Python 3, installing the dependencies:
pip install -r requirements.txt
and then issuing:
code/tests.py
The .editorconfig file provides standard editor options for this project. See EditorConfig for more information.
Releasing
Admins may ship a new release v1.n (where n increments the minor version of the current release) as follows:
- Navigate to the new release page on GitHub.
- Click
Select tag, choose Create new tag, enter v1.n in the pop-up, and click Create.
- Enter a suitable release title (e.g.
v1.n: adds user-agent1, user-agent2).
- Click
Generate release notes.
- Click
Publish release.
A GitHub action will then add the asset robots.txt to the release. That’s it.
Subscribe to updates
You can subscribe to list updates via RSS/Atom with the releases feed:
https://github.com/ai-robots-txt/ai.robots.txt/releases.atom.
You can subscribe with Feedly, Inoreader, The Old Reader, Feedbin, or any other reader app.
Alternatively, you can also subscribe to new releases with your GitHub account by clicking the ⬇️ on “Watch” button at the top of this page, clicking “Custom” and selecting “Releases”.
License content with RSL
It is also possible to license your content to AI companies in robots.txt using
the Really Simple Licensing standard, with an option of
collective bargaining. A plugin currently
implements RSL as well as payment processing for WordPress sites.
Report abusive crawlers
If you use Cloudflare’s hard block alongside this list, you can report abusive crawlers that don’t respect robots.txt here.
But even if you don’t use Cloudflare’s hard block, their list of verified bots may come in handy.
Additional resources
评论