Cloudflare and OpenAI pilot indexes hundreds of millions of new pages for ChatGPT

Cloudflare and OpenAI say their joint research pilot has brought hundreds of millions of previously unknown pages into OpenAI’s search index, giving ChatGPT more sources to draw on. The two companies shared the early results on October 15, roughly three months after launching the pilot in July.

The test ran on a participating cohort of about 1% of Cloudflare’s total network. According to Cloudflare, signals from its network help OpenAI’s search crawler discover more useful pages across the web, which OpenAI then independently evaluates and ranks based on how relevant and helpful they are.

What the pilot found

Cloudflare frames the problem in practical terms: someone fixing a washing machine needs the manual for their exact model, someone applying for a passport needs current requirements and fees, and someone picking a restaurant needs the latest menu and opening hours. Those pages can be valuable even when few other sites link to them.

The company also described what it learned about low-quality signals. Bots repeatedly request files that do not exist, some sites redirect almost any address to their homepage, and other addresses lead to copies of the same content. Cloudflare says recognizing these patterns lets it cut wasted work before a crawler fetches a page.

Cloudflare says ChatGPT Search links back to relevant sources, giving people a way to reach the original content and the publishers behind it. The company ties the work to OpenAI’s GPT-6 and Intelligent UI push, describing the pilot as complementary to those efforts.

What is not yet public

The post does not name the participating sites, give a breakdown of which content types benefited, or set out a timeline for a wider rollout. The results are described as early, and the pilot remains a research effort between the two companies.

For site owners, the practical takeaway is that discovery is being treated as a network-level problem rather than a purely on-page one. Cloudflare has not announced any action publishers need to take to be included.

Related reading

Sources

• Cloudflare and OpenAI: making AI discovery better for everyone — Cloudflare Blog