AI Bots on Your Website: When Should You Let Them In, and When Is It Better to Block Them?

In mid-September 2026, the owner of an online sports apparel store notices that his traffic from Google has dropped by a quarter over the course of a week. Yet nothing had been changed on the website—the content is the same, the prices are the same, and there are no penalties shown in Search Console. After an hour of investigating, he finds the culprit: the Cloudflare settings through which the website runs had started blocking AI bots—and, along with them, accidentally blocked the one that indexes the site for Google.

Among the affected bots was one that performs a dual role for Google: it indexes the web for traditional search while also training Google’s AI models. Cloudflare classifies such bots into a stricter category, so when it blocks AI training across the board, it can inadvertently shut out traditional search as well.

This didn’t have to happen just to you. Today, website owners, e-shop owners, and marketers all deal with AI robot settings—even if they have no background in programming—because a single checkbox in the admin panel can cost them actual revenue. Let’s take a look at who’s actually visiting your website, what changed on September 15, 2026, and how to decide—based on your website’s type—who to let in and who to block.

Should you block AI bots on your website, or let them in?

The answer depends mainly on how you make your living, with one important exception. If you sell goods or services and want people to find you even through AI search engines like ChatGPT, Perplexity, or Google’s AI mode, you should probably let the bots in. If, on the other hand, you make a living from your own exclusive content—such as news, premium research, or paid articles, blocking training bots makes sense, because otherwise the AI model would take your work for free and wouldn’t generate any traffic for you in return.

The exception applies to search bots. You should never block them, regardless of your line of business, because without them, you’ll disappear from standard Google search results as well. The problem is that some tools today combine both roles into a single bot, and it’s precisely this combination that turns a simple decision into a trap—one that even the e-shop mentioned at the beginning fell into.

Who Actually Visits Your Website: Three Types of AI Bots

You can find out who visits your website based on how a robot—technically known as a crawler (a program that automatically browses web pages and downloads their content)—identifies itself. Each one has a name in the request header—the so-called user-agent—and based on that, you can precisely determine what it’s up to. The more bots crawl your site, the more you should pay attention to how much they manage to crawl per day; this limit is called crawl budget And for larger online stores with thousands of products, it can determine whether the bots will even be able to go through the entire selection.

You can think of training bots as readers who borrow your book from the library, read it, but never return it and never tell anyone about you. Search and citation bots, on the other hand, are like a reader who recommends your book to others and sends you new customers as a result. A combined bot does both at the same time, which is why you can’t simply say, „I’ll block this, and nothing will happen.“.

Robots.txt is just a polite request, not a lock on the door

Robots.txt is a text file located in the root directory of a website, in which the website owner instructs web crawlers He'll write down where he's allowed to go and where he isn't. Most people confuse it with a lock, but in reality, it's just a sign on the gate. A well-behaved robot respects it, while a misbehaved one reads it and goes through anyway, because compliance with robots.txt is voluntary.

A new directive called "Content-Signal" has been added to robots.txt for AI bots; it goes one step further and tells the bot exactly what it is allowed to use your content for:

Content-Signal: ai-train=no, search=yes, ai-input=no

This entry says: Don’t train your models on my content, but feel free to index me for search and don’t quote me in live chatbot responses. However, this is still just a preference, not a technical measure against content scraping. If you really want to stop a bot, you have to go one level lower—to the server or firewall—which will immediately reject the bot’s request with a 403 Forbidden response before it even reaches your content. This is exactly what the “Block AI bots” feature in Cloudflare does, for example.

If, on the other hand, you want to allow robots access and make their job easier, there is a handy add-on to robots.txt. The file llms.txt It functions as a simplified sitemap designed specifically for language models, allowing the bot to find the most important pages more quickly, rather than having to search for them laboriously among hundreds of links.

What Changed on September 15, 2026, and Why It Might Affect You, Too

What changed on September 15, 2026, is simple: Cloudflare, which hosts a large portion of Czech websites, began enabling the blocking of AI training bots and AI agents as the default setting for all newly connected domains. According to existing customers, Cloudflare's official announcement They were given the option to opt out of the new default settings, and the company notified users of the change, so this may affect you if you overlooked the notification or never opened the settings.

Fortunately, the block does not target the entire website at once. It mainly targets pages that Cloudflare automatically identifies as containing ads, as well as bots designed to train AI models and so-called AI agents. It allows search engine bots to pass through. However, there is an important exception here: If a single bot combines the roles of a search bot and a training bot, a stricter rule applies, and Cloudflare classifies parts of Google’s and Bing’s infrastructure as such combined bots.

For a company website or online store, this leads to one practical recommendation: If you want to appear in AI search results, don't enable training blocking blindly and it’s best to check the settings directly in the Cloudflare dashboard, in the Security section under „AI Crawl Control.“ It’s difficult to determine this from the outside, as the settings are often “hidden” even from the website owner until they look there themselves or ask the person who manages the site.

When Is It Worth Letting AI Robots In?

It's worth letting AI robots in, especially if you want AI models to know about you the moment someone asks them a question about your field. This principle is called GEO, Generative Engine Optimization, that is, optimization for generative search engines. If you block the bot, the model won't know about you, and therefore won't be able to recommend you.

But it's not just about visibility for the sake of it. According to available data, visitors who arrive at the website via a click-through from an AI search convert at a rate of up to 4.4 times better than traditional organic traffic from Google, because they arrive already informed and ready to buy. A blocked bot means you’re competing for this group of customers with your hands tied behind your back.

Our practice: It’s not enough to just let the bots run—you also have to feed them. If you’re selling a product or service, add a comparison table to your website with real specifications and prices, along with an FAQ section written as actual customer questions, not marketing headlines. That’s because the model primarily cites content that it can quickly and reliably identify as a fact—not just well-written text without specific numbers.

The way the content is technically presented can also help. The cleaner and more machine-readable the page structure is, the easier it is for the model to extract facts from it; we discuss a similar principle in the article on Cloudflare Markdown for Agents, which provides robots with content directly in the format they work with best.

When Does Blocking Make Sense?

Paywalls make sense when the content itself is the product, not just a means to sell something else. Typically, this applies to news sites, specialized publications, creators of paid premium content, or companies that generate revenue through their expertise in the form of unique analyses and studies.

• Do you protect copyrighted content: You don't want an AI company to train its models for free using texts that your readers pay for through their subscriptions.
• Do you have sensitive or confidential data: internal documentation, price lists for specific clients, or content that is not intended for public distribution.
• The website is funded by advertising, not by sales: An AI response directly in the chat could reduce the traffic on which your ad revenue depends.

Even in these cases, however, you should only block training bots, not search or citation bots. Losing rankings in standard Google search usually hurts more than the benefits you gain from blocking AI model training.

Our Approach: How to Choose Settings Based on the Type of Website

Abstract rules are best understood through specific situations. Here are three that we most often encounter with clients.

1. A specialized online store with a limited product selection. Situation: An online store selling running gear is competing for attention alongside major retail players. Recommendation: Let absolutely all bots in—both training and search bots. Make sure your product listings have flawless structured data—including price, availability, and reviews—so that when an AI model is asked, „What are the best running shoes for beginners?“ it can use your product details directly in its response.

2. A B2B service provider with its own expertise. Situation: An agency or consultant relies on case studies and expert articles to generate leads. Recommendation: Keep public content, case studies, and your blog open to bots, because that’s exactly the kind of material you want to use to sound like an authority in AI-generated responses. Conversely, keep internal documentation, customized proposals, or price lists for specific clients behind a login screen, where no bot can access them anyway.

3. Editorial staff or creator of premium content. Situation: A small news website or news writer supports their business through subscriptions and ads on their own site. Recommendation: Block training bots using Content-Signal and actual server-level blocking, but let search and citation bots through, because without them you’d disappear even from regular Google searches, which still drive most of your traffic.

How to Check and, if Necessary, Change the Settings

If the website isn't hosted through Cloudflare or a similar service, the settings are configured directly in the robots.txt file. An example that blocks training bots while allowing search bots to pass through might look like this:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Googlebot
Allow: /

Content-Signal: ai-train=no, search=yes, ai-input=no

If you have a developer, this is exactly what you need to send them to work on. If you don't have a developer, you can manage this through your hosting account or CMS; most content management systems today allow you to edit the contents of the robots.txt file directly in the settings.

If your website runs through Cloudflare, you can find the settings in the admin panel under the "Security" section, in the "AI Crawl Control" section, where there are separate toggles for "Content Signals" (request) and "Block AI bots" (actual blocking). For both, pay attention to the scope—that is, whether they apply to the entire website or only to selected pages—and, if necessary, check to make sure you haven’t accidentally included Google and Bing search bots among the blocked bots.

How to Tell If an Open Door Is Actually Helping You

You don’t need to rely on estimates to see that opening your doors to AI bots actually helps you. The standard Google Search Console won’t tell you much about AI searches, because the model often generates the answer directly in the chat window, and a click-through to your website may not happen at all. This can help New "AI Performance" tab in Bing Webmaster Tools, which is the first to show how many times your website was actually cited as a source in an AI response, even if the reader didn't click through to the page at all.

Frequently Asked Questions About AI Robots on the Web

AI bots on your website are neither a phenomenon you should ignore nor a risk that requires you to shut down your site entirely. It’s a setting that deserves a conscious decision based on the nature of your business—not a default value chosen for you by your hosting or CDN provider without asking.

Not sure which settings are currently enabled on your website, or whether it might be blocking Googlebot? Contact us and we'll go over it together on your specific website.