# robots.txt — Robots Exclusion Protocol

The original machine-readable web standard: a plain-text file at a site's root telling crawlers which paths they may visit. Advisory, not enforced.

# /robots.txt — Robots Exclusion Protocol

Source: https://www.robotstxt.org/ (returned 403 to programmatic fetch
2026-05-17; content reconstructed from RFC 9309 and Wikipedia)
RFC:    https://www.rfc-editor.org/rfc/rfc9309.html
Fetched: 2026-05-17

## Origin and history

- Proposed by **Martijn Koster** in **February 1994** while at Nexor.
- Provoked by Charles Stross, who wrote "a badly behaved web crawler that
  inadvertently caused a denial-of-service attack on Koster's server."
- De facto standard by June 1994; honored by WebCrawler, Lycos, AltaVista.
- Formally standardized as **RFC 9309** in September 2022 by Koster,
  Gary Illyes, Henner Zeller, and Lizzi Sassman — nearly 30 years after the
  original informal spec.

## File location

A plain text file at the top-level path of an origin:

```
scheme:[//authority]/robots.txt
```

E.g. `https://www.example.com/robots.txt`. Each subdomain needs its own.

## Syntax

Core directives:

- `User-agent:` — identifies the crawler the rules apply to (product tokens:
  letters, underscores, hyphens).
- `Disallow:` — URI path patterns the crawler must not access.
- `Allow:` — URI path patterns explicitly permitted.
- `Sitemap:` — URL of an XML sitemap (extension, widely supported).
- `Crawl-delay:` — requested interval between visits (non-standard).

Special characters:
- `#` — comment
- `$` — end-of-pattern anchor
- `*` — wildcard

Example:

```
User-agent: *
Disallow: /private/
Allow: /private/public-faq.html
Sitemap: https://example.com/sitemap.xml
```

## Advisory nature — load-bearing

"These rules are not a form of access authorization" (RFC 9309). The protocol
is "purely advisory and relies on the compliance of the web robot; it cannot
enforce any of what is stated in the file." For real access control, use
HTTP authentication or other security measures.

This is the central design property: robots.txt expresses a *request* to
well-behaved crawlers. Malicious crawlers ignore it.

## AI-crawler era (2020s–)

- Sites began listing AI-training agents (e.g. `GPTBot`, `Google-Extended`,
  `ClaudeBot`) as `Disallow`-targeted user-agents.
- **2025:** Really Simple Licensing (RSL) launched, letting publishers
  encode AI-bot permissions directly in robots.txt. Early adopters:
  Medium, Reddit, Yahoo.
- Adjacent (non-canonical) proposals: `ai.txt`, `/llms.txt`.

## Governance

- Original 1994 spec: informal, Martijn Koster.
- Current canonical reference: IETF RFC 9309 (September 2022).

---

## Backlinks

Pages that link here:

- [Agent-Readable Web Standards](/wiki/agent-readable-web-standards.md) — A family of plain-text conventions at well-known paths that let machine readers understand how a site or repository wants to be read — the protocol surface of the Independent Internet.
- [Glossary](/wiki/glossary.md) — Formal definitions of the key terms behind the Independent Internet and Web 4.0, as used across this wiki, with links to source material.
- [The Governance of Agent-Readable Standards](/wiki/governance-of-agent-readable-standards.md) — The five agent-readable web standards differ sharply in who governs them — and that trajectory is the real story.
- [IETF](/wiki/ietf.md) — The Internet Engineering Task Force — the open, consensus-driven body that standardizes the internet's core protocols through the RFC series.
- [Jeremy Howard](/wiki/jeremy-howard.md) — Data scientist and educator who proposed llms.txt, extending the agent-readable web standards from crawling to inference.
- [llms.txt](/wiki/llms-txt.md) — A proposed standard for a curated, LLM-shaped manifest at a site's root — telling language models what to read at inference time.
- [robots.txt — Robots Exclusion Protocol](/wiki/robots-txt.md) — The original machine-readable web standard: a plain-text file at a site's root telling crawlers which paths they may visit. Advisory, not enforced.
- [Sitemaps Protocol (sitemap.xml)](/wiki/sitemap-xml.md) — The positive complement to robots.txt: an XML file declaring what URLs exist on a site and what the operator wants crawled first.
- [Independent Internet (Web 4.0)](/wiki/web4.md) — The web forgot who it was for. Web 4.0 — the Independent Internet — is a resilient, self-hosted internet people own and control. Explore the research.