Published by NewsPR Today | July 2025
A new battle line has been drawn in the ever-evolving landscape of the internet. Cloudflare, a company that provides critical infrastructure for a vast portion of the web, has just handed website owners a powerful, simple tool: a one-click button to block Google from using their content to train its AI and generate its “AI Overviews.”
This move signals a growing tension between content creators and the search giant, turning a technical setting into a declaration of independence for many online publishers.
The “Why”: A Publisher’s Worst Nightmare
Google’s new AI Overviews are designed to give users a direct, summarized answer at the very top of the search results page. While convenient for the user, this is a potential catastrophe for the websites that create the original content.
The business model of the open web is built on a simple exchange:
- A publisher creates valuable content (an article, a recipe, a how-to guide).
- This content is indexed by Google and displayed to users in search results.
- To access the website, the user clicks on the link.
- Through advertisements, affiliate links, or product sales, the website makes money from that visit.
This cycle could be broken by Google’s AI Overviews. The user has no reason to click through to the source website if the AI answers all of their questions directly. Devastating traffic loss results from this, which also means a loss of customers, ad revenue, and brand awareness. For many publishers, this isn’t just a problem; it’s an existential threat.
The “How”: A Simple, Elegant Solution
Cloudflare’s solution is powerful because of its simplicity. Here’s how it works:
Every website can have a file called robots.txt. This is a plain text file that acts as a set of rules for web crawlers, telling them which parts of a site they are allowed or forbidden to scan.
Google recently designated a specific crawler for its AI data collection, named Google-Extended. To opt out, a website owner simply needs to add the following rule to their robots.txt file:
User–agent: Google–Extended
Disallow: /
Related Article:
- How to Boost Your AdSense Earnings with Auto Ads and Intent-Driven Formats
- Google’s AI Overview Algorithm: Simple SEO Strategy Guide for Concrete Services
This command tells the Google-Extended crawler, “You are not allowed to access any part of this site.”
While anyone can do this manually, Cloudflare—which sits between a website and its visitors—has automated the process. With their new “Crawler Hints” feature, a website owner using Cloudflare can flip a single switch to automatically apply this rule, blocking Google’s AI without ever having to touch a line of code.
In short, Cloudflare has democratized the ability for websites to opt out of feeding Google’s AI.
This move positions Cloudflare as an ally to content creators in the fight for the future of web traffic, and it forces a critical conversation about the symbiotic—and now strained—relationship between search engines and the publishers who populate them with information.
If I could ask a couple follow ups (wrapping my head around this):
1) so does that still mean Gemini training is currently *not* blocked by default?
2) what would Cloudflare ideally want Google to do?
— Nate Hake (@natejhake) July 3, 2025
Building Your Own “Block Google AI” Script
Inspired by Cloudflare’s action, you might want to implement this block on your server without relying on a third-party service. The script below will safely check your robots.txt file and add the necessary block if it doesn’t already exist.
This is a simple shell script that can be run on most web servers (like those running Linux).
⚠️ Important: Always back up your robots.txt file before running any script that modifies it.
The Script: block-google-ai.sh
How to Use the Script
- Save the Code: Open a text editor on your server and paste the code above. Save the file as block-google-ai.sh.
- Make it Executable: Open your terminal, navigate to where you saved the file, and run this command to give it permission to execute:
./block-google-ai.sh
- Provide the Path: The script will ask you for the location of your robots.txt file. This is typically in your website’s root directory (e.g., /var/www/yourdomain.com/public_html/robots.txt or /home/user/public_html/robots.txt). Enter the full path and press Enter.
Disclaimer
The script provided in this guide is for educational and informational purposes only. It is designed to help website owners add a specific rule to their robots.txt file to block Google’s AI crawler (Google-Extended) from accessing their site content.
While the script checks for existing rules and aims to make safe changes, you are responsible for backing up your robots.txtfile and verifying that the changes made suit your website’s specific needs.
NewsPR Today and its authors are not liable for any unintended results, disruptions to your site’s indexing, or visibility issues that may occur as a result of using this script or modifying your site settings.
Use at your own discretion. If you’re unsure, consult with your web developer or hosting provider before making changes to your production environment.
