Cloudflare to Score Verified Bots on Whether They Respect robots.txt

Cloudflare is building a system that continuously checks whether Verified bots keep honoring website preferences after they are verified, the company said in a blog post published on October 16, 2026. The score will be shown as an “explainable trust band” rather than a permanent verdict on a bot or its operator.

The move targets a gap Cloudflare itself describes: verification proves who runs a bot, not how it will behave later. A known bot can still ignore site preferences, overwhelm origins or evade controls — sometimes through a misconfiguration or a bug rather than malice, the company notes.

What the score looks at

Cloudflare says its system measures how often a crawler respects site preferences, starting with two signals: whether the bot respects instructions, and whether it responded to direct feedback. It treats these as “common sense” rules and a courtesy to the operators who supply the information bots are after.

The primary input is robots.txt, the file sites use to tell bots which parts of a site they may crawl. Cloudflare says it looks for repeated patterns of behavior that disregard those published preferences, not isolated violations or one-offs, and expects responsible bots to follow the preferences in line with RFC 9309.

The company also stresses that robots.txt sets a minimum: anything not stated there is not automatically allowed. A site can signal that continued access is unwelcome through other means, and a bot is still expected to stop.

Trust over time, not a one-time check

Cloudflare frames the distinction as risk versus trust. Risk is the likelihood that a single request or point-in-time action may be harmful and can shift moment to moment; trust is built through repeated, consistent behavior. To be a Cloudflare Verified Bot, an operator broadly has to identify itself accurately and securely, do only what it says it will do, and avoid abusive behavior. Bots sign requests using Web Bot Auth, which BotBase catalogues in an open format.

As an example of de-listing, Cloudflare points to Perplexity’s bot, which it removed from its Verified bot list in 2025 after the bot sought access to resources on sites that had opted out of scraping and then attempted to circumvent controls when those requests were blocked.

Cloudflare says it will share these scores publicly on Radar in the future and expand how it measures behavior. For now, the company describes the effort as a way for responsible bot operators to show that trust is being upheld over time: verify, but continuously re-verify.

Related reading

Sources

• Continuous accountability for Verified bots — Cloudflare Blog