How is this different from Google Analytics?

Google Analytics shows you traffic. Shadow shows you traffic, AI bot activity, what AI platforms say about your brand, AND tells you what to do about all of it. It's analytics + AI intelligence + action steps in one tool.

Do I need to install anything?

For basic monitoring (bot detection, AI perception, readiness score) — nope, just enter your URL. For full visitor analytics (clicks, behavior, sessions), add one script tag. One-click integrations for Vercel, Shopify, WordPress, and more.

Will it slow down my site?

No. The script is under 5KB and loads async. Zero impact on page speed or Core Web Vitals. External monitoring has literally no impact — it watches from the outside.

What AI bots does Shadow detect?

All of them. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Amazonbot, and dozens more. The Shadow Network means new bots get identified across all users instantly.

What do you mean by "actionable steps"?

Shadow doesn't just show you graphs. It says things like: "ChatGPT has your pricing wrong — add structured data to /pricing to fix it" or "Your bounce rate on /features is 68% — here's why and what to change." Specific, do-it-today recommendations.

Can Shadow block bots?

Shadow is a telescope, not a shield. It shows you who's visiting and what AI says about you. It generates block rules and robots.txt configs you can apply — but it doesn't intercept traffic.

Yes. Shadow never collects PII. IP addresses are hashed after classification. No cookies on your visitors. All Shadow Network data is anonymized. GDPR compliant by design.

How do I add a noai meta tag to Webflow?

In the Webflow Designer, click the 'W' (Webflow logo) or gear icon to open Site Settings → Custom Code tab → paste in the 'Head Code' field. Click Save and publish your site. This adds the tag to every page. No paid plan required for custom code injection.

Does Webflow's free plan allow blocking AI bots?

Partially. Free/Starter plan users can add noai meta tags via Site Settings → Custom Code (head). This is the most important protection and works on all plans. Custom robots.txt editing requires a paid Webflow hosting plan. Cloudflare WAF works on any plan if you proxy your custom domain through Cloudflare.

Will blocking AI bots hurt my Webflow SEO?

No. Blocking GPTBot, ClaudeBot, CCBot, Google-Extended, and Bytespider has no effect on Googlebot or Bingbot. Your Webflow site will continue to rank normally in Google and Bing. The SEO Panel and Webflow's built-in SEO tools are completely unaffected. The only tradeoff is AI search visibility — blocking OAI-SearchBot or PerplexityBot removes you from ChatGPT Search or Perplexity results.

I publish to webflow.io — can I still block AI bots?

Yes. The noai meta tag via Site Settings → Custom Code works regardless of whether you use a custom domain or the webflow.io subdomain. robots.txt editing and Cloudflare proxying require a custom domain, but the meta tag protection is available on all plans and subdomains.

How do I add per-page noai tags in Webflow?

In the Webflow Designer, click on a page in the Pages panel → open Page Settings (gear icon) → scroll to the Custom Code section → paste the noai meta tag in the 'Before tag' field. This applies only to that specific page. Use Site Settings → Custom Code for sitewide protection instead.

WebflowNew9 min read

How to Block AI Bots on Webflow

Webflow gives you two clean methods to block AI crawlers — custom head code for noai meta tags (works on all plans) and a built-in robots.txt editor (paid plans). Here's the complete setup in under 10 minutes.

The fastest fix: Custom Head Code works on every Webflow plan

Unlike some platforms, Webflow lets you inject custom code into your site's <head> without a paid plan. This means you can add the noai meta tag — the most important AI training protection — right now, even on a free Webflow site. robots.txt editing is paid-only, but the meta tag covers the bots that respect it. Start there.

Quick block — paste into Webflow Site Settings → Custom Code → Head Code

This tag tells AI bots not to train on your content, even if they visit the page.

<meta name="robots" content="noai, noimageai">

For full bot blocking, also add the robots.txt rules below (Method 2).

robots.txt block — paste into Project Settings → SEO → Custom Robots.txt

Requires a paid Webflow hosting plan. Paste after any existing rules.

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: xAI-Bot
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: MistralBot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: AI2Bot
Disallow: /

User-agent: Ai2Bot-Dolma
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: omgili
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: webzio-extended
Disallow: /

User-agent: gemini-deep-research
Disallow: /

Googlebot, Bingbot, and traditional search crawlers are not affected.

Method 1: noai Meta Tag via Site Settings (All Plans)

The fastest and most broadly compatible method. Adding a noai meta tag to your site's <head> tells AI training bots not to use your content — and it works on any Webflow plan, including the free tier.

1
Open your project in the Webflow Designer.
2
Click the gear icon (⚙) in the top-left of the Designer to open Site Settings.
Alternative: In the Webflow Dashboard, open your project → Project Settings → Custom Code tab.
3
Go to the Custom Code tab.

In the Head Code field, paste:

<meta name="robots" content="noai, noimageai">

5
Click Save Changes.
6
Publish your site — go back to the Designer and click the Publish button. Custom code changes require a new publish to take effect.

Verify it worked: After publishing, right-click any page on your live site → View Page Source → search for noai. You should see the meta tag inside the <head> section.

Per-page noai tags: To protect only specific pages, open that page in the Designer → click the page name in the Pages panel → open Page Settings (⚙) → scroll to Custom Code → paste the meta tag in the "Before </head> tag" field. Per-page settings override site-wide settings for that page.

Method 2: Edit robots.txt in Project Settings (Paid Plans)

Webflow hosts your robots.txt for you, and paid plan users can fully customise it. This is the strongest defense against bots that crawl aggressively — combined with the noai meta tag, it covers both cooperative and scraper-type bots.

Webflow paid plan required

1Go to your Webflow Dashboard → click your project → Project Settings.
2Open the SEO tab (sometimes labelled "Hosting" or "Publishing" depending on your dashboard version).
3Find the Custom Robots.txt section and click to edit.
4Paste the AI bot rules from the block above. Preserve any existing rules (like User-agent: * / Allow: /).
5Click Save and republish your site. Verify at yourdomain.com/robots.txt.

Don't see a robots.txt editor? Webflow added custom robots.txt support for paid plans. If you're on a Basic or higher plan and still don't see it, try the Designer path: Webflow logo (top-left) → Site Settings → SEO tab. If your plan doesn't include it, use the Custom Head Code method (Method 1) and Cloudflare WAF (Method 3) instead.

Method 3: Cloudflare WAF (Strongest — Works on All Plans)

Proxying your Webflow site through Cloudflare and adding WAF rules blocks AI bots at the network edge — before they ever reach Webflow's servers. This is the only method that reliably stops Bytespider (known to ignore robots.txt) and other aggressive crawlers. Works with any Webflow plan, including free.

Setting up Cloudflare with Webflow

1.Add your custom domain to Cloudflare (free plan). Point your domain's nameservers to Cloudflare at your registrar (not inside Webflow).
2.In Cloudflare DNS, add a CNAME record: www → your Webflow subdomain (e.g. your-site.webflow.io). Enable the orange cloud (proxied).
3.For the apex domain (@), add an A record or CNAME pointing to Webflow's IP, with proxy enabled. Or set up a redirect rule from @ to www.
4.Set Cloudflare SSL/TLS mode to Full. Webflow handles its own SSL — don't use Flexible (it can cause redirect loops).
5.In your Webflow Project Settings → Hosting → Custom Domain, verify the domain is still connected and add your Cloudflare-proxied domain if needed.

Add the WAF blocking rule

1Cloudflare → your domain → Security → WAF → Custom Rules → Create rule.
2Click Edit expression and paste:

(http.user_agent contains "GPTBot") or
(http.user_agent contains "ClaudeBot") or
(http.user_agent contains "anthropic-ai") or
(http.user_agent contains "Google-Extended") or
(http.user_agent contains "Bytespider") or
(http.user_agent contains "CCBot") or
(http.user_agent contains "PerplexityBot") or
(http.user_agent contains "meta-externalagent") or
(http.user_agent contains "DeepSeekBot") or
(http.user_agent contains "MistralBot") or
(http.user_agent contains "xAI-Bot") or
(http.user_agent contains "Diffbot") or
(http.user_agent contains "cohere-ai") or
(http.user_agent contains "AI2Bot") or
(http.user_agent contains "DuckAssistBot") or
(http.user_agent contains "omgilibot") or
(http.user_agent contains "webzio-extended") or
(http.user_agent contains "gemini-deep-research")

Set action to Block. AI bots receive 403 Forbidden — Webflow never processes the request.

All 25 AI Bots to Block

User agents for the robots.txt rules — one per block:

GPTBot

ChatGPT-User

OAI-SearchBot

ClaudeBot

anthropic-ai

Google-Extended

Bytespider

CCBot

PerplexityBot

meta-externalagent

Amazonbot

Applebot-Extended

xAI-Bot

DeepSeekBot

MistralBot

Diffbot

cohere-ai

AI2Bot

Ai2Bot-Dolma

YouBot

DuckAssistBot

omgili

omgilibot

webzio-extended

gemini-deep-research

Will This Affect My Webflow SEO?

Safe to block

✓ Google Search rankings unaffected
✓ Bing rankings unaffected
✓ Webflow SEO Panel unaffected
✓ Google Image Search unaffected
✓ Webflow's sitemap generation unaffected

Consider before blocking

⚠ OAI-SearchBot → removes from ChatGPT Search
⚠ PerplexityBot → removes from Perplexity
⚠ DuckAssistBot → removes from Duck.ai
For most Webflow portfolio and business sites, AI search drives minimal direct traffic.

Webflow SEO Panel — no changes needed

Webflow's built-in SEO Panel generates your sitemap, handles canonical URLs, and manages noindex settings per page. None of these are affected by adding AI bot blocks to your robots.txt or adding a noai meta tag. The two systems are completely independent. Your Webflow SEO workflows continue exactly as before.

Verify Your Block Is Working

1. Check the noai meta tag is live

Visit your published site → right-click → View Page Source → search for noai. Confirm it appears inside the <head> section. Remember: changes require republishing in Webflow Designer.

2. Check robots.txt is live (paid plans)

Visit https://yourdomain.com/robots.txt. You should see your AI bot Disallow rules. If you see Webflow's default robots.txt without your changes, republish the site and wait a few minutes for CDN propagation.

3. Check Webflow hosting analytics

Webflow hosting analytics shows visitor traffic by source. Bot traffic appears in unusual spikes or as direct/unknown source sessions. After blocking, you may notice a reduction over 2–4 weeks as crawl frequency drops.

4. Scan with Open Shadow

Scan your Webflow site — get your AI Readiness Score, verify bot blocking status, check noai tag presence, and see which crawlers can still access your site.

What Works on Each Webflow Plan

Method	Free	Basic+	Custom Domain
noai meta tag (Site Settings)	✓	✓	—
robots.txt editor	✗	✓	—
Cloudflare WAF	Custom domain	Custom domain	✓

Frequently Asked Questions

Can you edit robots.txt in Webflow?↓

Yes, on paid Webflow hosting plans. Go to your Webflow Dashboard → Project Settings → SEO tab → Custom Robots.txt. Paste your AI bot rules, save, and republish. Free plan users can use the noai meta tag (Method 1) or Cloudflare WAF instead.

I'm on the Webflow free plan — what can I do?↓

Add a noai meta tag via Site Settings → Custom Code → Head Code. This is the most important protection and works on all plans including free. If you have a custom domain, route it through Cloudflare for WAF blocking too. robots.txt editing requires a paid Webflow plan.

How do I add a noai meta tag to every page in Webflow?↓

Add it to Site Settings → Custom Code → Head Code in the Webflow Designer. Code in this field gets injected into the <head> of every page on your site. Click Save and republish for the changes to go live.

I published my changes but they're not showing up. What's wrong?↓

Webflow caches aggressively. Try a hard refresh (Ctrl+Shift+R / Cmd+Shift+R) on the live site. For robots.txt, the change can take up to 24 hours to propagate through Webflow's CDN. Also double-check that you saved AND republished — custom code changes require a new publish to take effect.

Will blocking AI bots affect Webflow's sitemap generation?↓

No. Webflow auto-generates your sitemap.xml and submits it to Google Search Console. robots.txt Disallow rules for AI bots don't affect sitemap generation, Google's sitemap crawling, or any Webflow SEO Panel functionality.

Does Cloudflare affect Webflow's publishing or design previews?↓

Cloudflare proxying affects your live domain only — not the Webflow Designer, staging previews, or the webflow.io subdomain. Your design workflow is completely unaffected. Publishing from the Designer still pushes to Webflow's CDN; Cloudflare just sits in front of your custom domain.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.