The agent-readable web standards look alike — plain-text files at well-known paths, advisory, operator-authored — but their governance diverges sharply, and that is the more interesting axis.
Five standards, five paths
- robots.txt — informal for nearly thirty years, then formalized by a neutral body (the IETF, RFC 9309). The mature case.
- sitemap.xml — born inside one company, adopted jointly by three, and then frozen under those corporate sponsors with no neutral steward. The cautionary case.
- AGENTS.md — moved early to a neutral home (the Linux Foundation). The encouraging case.
- llms.txt — still a single lab’s proposal (Jeremy Howard / Answer.AI).
- design.md — a single corporation’s alpha.
The argument
Critical web infrastructure tends to work best when it ends under neutral stewardship — the way HTML and CSS landed at the W3C. sitemap.xml shows the failure mode: a standard that worked in practice for two decades yet never acquired a neutral body, leaving its three sponsors as the de facto governance. For the Independent Internet, where operator control is the defining principle, who governs the rules of engagement is not a footnote — it is the point. The newest members of the family (llms.txt, design.md) are exactly where sitemap.xml was in 2006.