Controlling AI Crawlers and Search Bots

The Crawlers tab controls which automated visitors are allowed to index your site, grouped into search engines, AI answer engines, SEO tools, and social preview bots. This guide explains what each group does, what Kapsule allows by default, and what blocking actually costs you.

Where Crawler Settings Live in KPanel

Sign in to KPanel.
Click Websites in the left sidebar, then click the site.
In the site's tab strip, open Performance, then Crawlers.

The direct address is /websites/<site-id>/crawlers.

Crawler controls for a site in KPanel

What This Page Actually Changes

Toggling a group rewrites your site's robots.txt file. That file is a published request to well-behaved automated visitors, and the major crawlers honour it.

Two consequences follow, and both matter.

It is a request, not a wall. A crawler that ignores robots.txt is unaffected by anything on this page. If your problem is scraping by something that does not identify itself honestly, this is the wrong tool: use the IP block list or country filtering on the site's Security page instead. See Site Security and Country Blocking for a Site.

It controls crawling, not what is already indexed. Blocking a crawler stops future fetches. Content already in an index usually falls out over time, but not instantly.

Changes save the moment you toggle a switch. The page shows Saving, then Saved, robots.txt updated.

The Four Groups

Search engines. The traditional crawlers behind ordinary search results: Googlebot, Bingbot, DuckDuckBot, Applebot, and YandexBot.

AI search and answer engines. Crawlers for systems that include your content in AI-generated answers and search overviews: GPTBot and OAI-SearchBot from OpenAI, ClaudeBot and anthropic-ai from Anthropic, PerplexityBot, Google-Extended, Cohere, Meta, and Diffbot.

SEO and analytics tools. Crawlers from platforms used to audit sites: AhrefsBot, SemrushBot, MJ12bot, DotBot, and Baiduspider.

Social media and preview. Crawlers that fetch a preview card when someone shares a link: Twitterbot, the Facebook preview fetcher, LinkedInBot, Slackbot, and Discordbot.

Each group card lists every crawler in it, with its owner, so you can see exactly what a toggle covers before you flip it.

Kapsule's Default: AI Crawlers Are Allowed

Every group is allowed by default, including AI crawlers, and that is a deliberate position rather than an oversight.

A number of hosts quietly block AI crawlers on their customers' behalf. We do not, because it is your traffic and your decision, and because the default costs you visibility. AI search results are a real and growing source of referrals. A site that AI systems cannot read does not appear in those answers, which means it does not get that traffic.

If you want to block them, the control is right here. We simply will not make that choice for you without telling you.

Deciding for Each Group

Search engines. Leave allowed unless the site is genuinely private, in which case password protection is the right tool anyway. See Password-Protecting a Site.

AI search and answer engines. This is the real decision on the page.

Allow them if you want traffic and citations from AI answers, which is the case for almost every commercial site, publisher, and service business.

Consider blocking them if your content is your product and being summarised removes the reason to visit, if you have contractual or licensing restrictions on how content may be reused, or if you have made a deliberate editorial choice about AI training.

Be honest about the trade. Blocking means your site disappears from AI-generated search overviews and answers, and that is measurable lost traffic, not a theoretical risk.

SEO and analytics tools. The most defensible group to block. They provide you with nothing directly: they let other people audit your site, and they can add real crawl load on a large site. Blocking them means your site is less visible in competitors' research tools, which some people consider a feature. The cost is that your own SEO tooling may also lose data about your site.

Social media and preview. Leave allowed unless you never want link previews. Blocking these means links to your site shared on social platforms and chat apps appear as bare URLs with no title, description, or image, which measurably reduces click-through.

Blocking All AI Crawlers at Once

At the bottom of the page there is a danger-zone card, Block all AI crawlers. It adds disallow rules for every crawler in the AI group in one action.

Clicking it asks for confirmation first, and the confirmation states the consequence plainly: your site will disappear from AI search overviews, and this costs real traffic.

This is a traffic decision, not a security one. Blocking AI crawlers does not protect your content from being copied: anyone can still read your pages, and a bad actor was never going to honour robots.txt in the first place. What it reliably does is remove you from AI-generated answers. Make sure that is the trade you want.

The action is reversible. Toggle the AI group back on and the disallow rules are removed on the next save. Getting back into an index afterwards takes time.

Who Can Change This

Changing crawler settings needs the site write permission. With a read-only role the switches are disabled, and hovering one explains why. Ask an account owner to make the change or adjust your role.

Troubleshooting

I blocked a group but the crawler is still visiting. Well-behaved crawlers re-read robots.txt periodically rather than before every request, so give it time. If it never stops, the crawler is not honouring the file and you need an access control instead. See Site Security.

My pages are still in search results after blocking. Blocking stops future crawling. Existing index entries expire over time. To remove content quickly, use the search engine's own removal tool.

Link previews stopped working. The social group is blocked. Turn it back on.

My SEO tool has no data for my site. The SEO group is blocked, which affects your tools as well as everyone else's.

Traffic dropped after I changed something here. Check which groups are blocked. If search or AI is off, that is the cause, and turning it back on starts the recovery.

A crawler I care about is not listed. The groups cover the major named crawlers. Anything not listed falls under the general allow rule at the top of the file and is not blocked by these toggles.

My site is not being indexed at all and everything is allowed here. Something else is blocking it. Check whether whole-site password protection is on, whether country filtering is blocking the crawler's region, or whether the site is on a shared address rather than your own domain.

Site Traffic Analytics to see what crawler traffic actually costs you in requests and bandwidth.
Site Security for IP blocking and other real access controls.
Country Blocking for a Site for geography-based filtering.
Password-Protecting a Site to hide a site from everything, crawlers included.