A plain text file at the root of a website has become one of the most contested documents in AI. The robots.txt file began in the early web as a polite way to tell search crawlers where not to go. Three decades later it is the main tool publishers have to say whether their pages may be used to train AI models, and it is at the center of this month’s dispute over Meta’s business terms. This timeline traces how it got there, and where its limits show.
1994: A convention among crawler authors
In the early days of the web, automated programs called crawlers fetched pages to build search indexes, and some overloaded small servers. In 1994, Martijn Koster, who was working on an early search engine, proposed the Robots Exclusion Protocol: a file named robots.txt placed at a site’s root that lists which paths particular crawlers should avoid. It was not a law or a technical barrier. It was an agreement that well-behaved crawler operators chose to follow, and that is still its nature. Our glossary entry on robots.txt gives the short definition.
1996 to 2007: Search engines make it standard
As search engines became commercial businesses, honoring robots.txt became a norm for major crawlers, because ignoring it risked complaints and blocks. The file stayed informal for decades. Different crawlers interpreted edge cases differently, and there was no official specification.
2007 to 2008: Open web archives appear
Common Crawl, a nonprofit, was founded to publish open crawls of the web for anyone to use. Its data later became a core ingredient of many language model training sets, which is one reason AI companies and publishers both pay attention to its crawler’s rules. Common Crawl’s crawler obeys robots.txt, according to the organization’s documentation.
2020 to 2022: Language models scale up
Large language models trained on web text grew quickly in the early 2020s. Few site owners had thought of robots.txt as an AI policy tool, because the crawlers feeding datasets were often run by third parties, and the data had already been collected by the time questions arose.
2022: A standard at last
In September 2022 the Internet Engineering Task Force published RFC 9309, formally specifying the Robots Exclusion Protocol, nearly three decades after the original proposal. The document clarifies syntax and behavior but keeps the core idea: it is a request, not an access control. The RFC itself states that the rules are not a form of access authorization.
2023: Opt-outs for AI crawlers
In August 2023, OpenAI published documentation for its GPTBot crawler and told site owners how to block it through robots.txt. Weeks later, Google introduced a separate token, Google-Extended, that lets publishers keep their content out of use for certain Google AI products while remaining in search, according to Google’s crawler documentation. These moves treated robots.txt as an AI consent mechanism for the first time at scale.
The approach has limits. A site must name each crawler it wants to block, new crawlers appear often, and the rules apply to future crawling, not to data already collected. Blocking a search crawler to avoid AI use can also remove a site from search results, which is why separate tokens mattered.
2023 to 2025: Lawsuits and licensing
Disputes moved from technical files to courts and contracts. Several news organizations sued AI developers over training on their articles, and many AI companies signed licensing deals with publishers. We explain both paths in AI and copyright: the lawsuits explained and AI content licensing deals explained. Infrastructure companies also entered the picture: in mid-2025, Cloudflare began blocking known AI crawlers by default for new sites on its network and introduced a pay-per-crawl idea for publishers, according to its announcements.
October 2026: Terms versus signals
This month’s Meta dispute shows a new layer. Meta planned business terms, set to take effect October 30, that publishers said could let it crawl sites linked to business accounts and use the content for AI. Meta says robots.txt still applies. After pushback it postponed the terms to November 20, The Verge reported. Our news report has the details. The episode raises a question robots.txt cannot answer alone: what happens when a contract between a platform and a publisher says one thing and the technical signal another?
What the timeline shows
Three layers now govern training-data access: the law (what is permitted), technical signals such as robots.txt (what a site requests) and contracts (what parties agree to). The 1994 convention worked when crawlers were few and mostly interested in indexing. It has strained as the number of AI crawlers grew, as value moved from links to content itself and as platforms began to combine terms with crawling.
What to watch next
- The final text of Meta’s terms and any change to how its crawlers handle robots.txt.
- Court rulings on whether training on web content is fair use.
- Adoption of newer standards or proposals for expressing AI-specific preferences.
- Whether pay-per-crawl or licensing marketplaces gain traction.
- Regulatory guidance in the EU and UK on opt-out mechanisms.
Key takeaways
- robots.txt began in 1994 as a voluntary convention among crawler operators and became an IETF standard (RFC 9309) in 2022.
- It is a request, not a security control; it cannot remove data already collected.
- AI-specific crawler names, such as GPTBot and Google-Extended, appeared in 2023.
- Disputes now span three layers: law, technical signals and contract terms.
Frequently asked questions
Does robots.txt stop AI companies from using my content?
Only if the crawler honors it and only for future crawling. It does not remove content that was collected before you added the rules.
Is robots.txt legally binding?
The standard describes it as a set of requests, not access authorization. Legal weight depends on jurisdiction and context.
Can I block AI crawlers without leaving search?
Often yes, where operators provide separate crawler names or tokens, but you must check each operator’s documentation because the rules differ.
