Agent-Readable Web Standards

Agent-readable web standards are a family of plain-text and Markdown conventions placed at well-known paths that let machine readers — crawlers, indexers, LLMs, coding agents — understand how a site or repository wants to be read. Each is advisory, lives at a fixed path, and emerged as a response to a specific scaling failure of the existing web.

The family

What they share

They are plain-text-class formats at a well-known location; advisory rather than enforced; operator-authored with no central registry; and additive — each slots beside the others to form a stack. A site that fully participates ships robots.txt, sitemap.xml, and llms.txt; a repository adds AGENTS.md and design.md.

Why this is the substrate of Web 4.0

The Independent Internet frames Web 4.0 as Read-Write-Own-Control. Control means the operator declares the rules of engagement and machines honor them — and these standards are the first concrete artifacts of that control. robots.txt turned crawler behavior into operator policy in 1994; the rest extend the same pattern to inference, implementation, and design. Because compliance is a property of well-behaved readers rather than the network, the model scales by convention — the same decentralization logic the Web was born with. They also carry the Semantic Web’s machine-readable ambition forward into the age of agents. The complementary, workspace-level pattern is Interpretable Context Methodology.