# Agent-Readable Web Standards

A family of plain-text conventions at well-known paths that let machine readers understand how a site or repository wants to be read — the protocol surface of the Independent Internet.

**Agent-readable web standards** are a family of plain-text and Markdown conventions placed at well-known paths that let machine readers — crawlers, indexers, LLMs, coding agents — understand how a site or repository wants to be read. Each is advisory, lives at a fixed path, and emerged as a response to a specific scaling failure of the existing web.

## The family

| Standard | Scope | Audience | Year | Governance |
|----------|-------|----------|------|------------|
| [robots.txt](/docs/robots-txt.md) | site | crawlers | 1994 | IETF (RFC 9309) |
| [sitemap.xml](/docs/sitemap-xml.md) | site | crawlers | 2005–06 | Google / Yahoo / Microsoft |
| [llms.txt](/docs/llms-txt.md) | site | LLMs (inference) | 2024 | Answer.AI |
| [AGENTS.md](/docs/agents-md.md) | repository | coding agents | 2024–25 | Linux Foundation |
| [design.md](/docs/design-md.md) | design system | coding agents | 2025–26 | Google Labs (alpha) |

## What they share

They are plain-text-class formats at a well-known location; advisory rather than enforced; operator-authored with no central registry; and additive — each slots beside the others to form a stack. A site that fully participates ships `robots.txt`, `sitemap.xml`, and `llms.txt`; a repository adds `AGENTS.md` and `design.md`.

## Why this is the substrate of Web 4.0

The Independent Internet frames Web 4.0 as *Read-Write-Own-Control*. Control means the operator declares the rules of engagement and machines honor them — and these standards are the first concrete artifacts of that control. `robots.txt` turned crawler behavior into operator policy in 1994; the rest extend the same pattern to inference, implementation, and design. Because compliance is a property of well-behaved readers rather than the network, the model scales by convention — the same [decentralization](/wiki/decentralization.md) logic the Web was born with. They also carry the [Semantic Web](/wiki/semantic-web.md)'s machine-readable ambition forward into the age of agents. The complementary, workspace-level pattern is [Interpretable Context Methodology](/wiki/interpretable-context-methodology.md).

---

## Backlinks

Pages that link here:

- [AGENTS.md](/wiki/agents-md.md) — An open Markdown format at a repository's root carrying the build, test, style, and security context a coding agent needs.
- [Disrupting the First Reported AI-Orchestrated Cyber Espionage Campaign](/wiki/ai-espionage-2025.md) — Anthropic's disclosure of a state-sponsored espionage campaign in which an AI agent executed an estimated 80–90% of the operation with only sporadic human direction.
- [design.md](/wiki/design-md.md) — An early Markdown convention for declaring a project's design system to design-aware coding agents.
- [Glossary](/wiki/glossary.md) — Formal definitions of the key terms behind the Independent Internet and Web 4.0, as used across this wiki, with links to source material.
- [Interpretable Context Methodology: Folder Structure as Agent Architecture](/wiki/icm-paper.md) — Van Clief and McDermott's paper arguing that a well-organized folder hierarchy can replace a multi-agent framework: the folder structure is the orchestration.
- [The Governance of Agent-Readable Standards](/wiki/governance-of-agent-readable-standards.md) — The five agent-readable web standards differ sharply in who governs them — and that trajectory is the real story.
- [Interpretable Context Methodology (ICM)](/wiki/interpretable-context-methodology.md) — Using folder structure — plain files in a well-organized hierarchy — as the orchestration mechanism for a multi-step AI workflow, instead of a code-level framework.
- [Jake Van Clief](/wiki/jake-van-clief.md) — AI-systems engineer, content architect, and Marine Corps veteran; lead author of the ICM paper and originator of Interpretable Context Methodology.
- [Jeremy Howard](/wiki/jeremy-howard.md) — Data scientist and educator who proposed llms.txt, extending the agent-readable web standards from crawling to inference.
- [LLM-Maintained Wiki](/wiki/llm-maintained-wiki.md) — A persistent, interlinked knowledge base that a language model builds and maintains — integrating each source once, rather than re-deriving answers from raw documents on every query.
- [LLM Wiki: A Pattern for Building Personal Knowledge Bases Using LLMs](/wiki/llm-wiki.md) — Karpathy's pattern for an LLM-maintained wiki: instead of re-deriving answers from raw documents on every query (RAG), the model integrates each source into a persistent, compounding knowledge base.
- [llms.txt](/wiki/llms-txt.md) — A proposed standard for a curated, LLM-shaped manifest at a site's root — telling language models what to read at inference time.
- [robots.txt — Robots Exclusion Protocol](/wiki/robots-txt.md) — The original machine-readable web standard: a plain-text file at a site's root telling crawlers which paths they may visit. Advisory, not enforced.
- [Sitemaps Protocol (sitemap.xml)](/wiki/sitemap-xml.md) — The positive complement to robots.txt: an XML file declaring what URLs exist on a site and what the operator wants crawled first.
- [Independent Internet (Web 4.0)](/wiki/web4.md) — The web forgot who it was for. Web 4.0 — the Independent Internet — is a resilient, self-hosted internet people own and control. Explore the research.