How to Block AI Bots on BigCommerce
BigCommerce is fully hosted SaaS — no server access, no nginx config. But you can block AI crawlers through the admin panel's built-in robots.txt editor, Stencil theme customization, noai meta tags in base.html, and Cloudflare WAF for hard blocking at the network edge.
⚠️ Why e-commerce stores are high-value AI scraping targets
Diffbot and similar structured data crawlers specialize in extracting product names, prices, inventory signals, and pricing history from online stores. AI companies use this data to build price intelligence tools and training datasets. Your competitor could be querying an AI trained on your catalogue. Block Diffbot first — then the full list below.
Quick fix — Admin Panel robots.txt
Go to Store Setup → Store Settings → Search Engines. Paste these rules into the custom robots.txt field. No theme access needed.
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Diffbot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: /
All Methods
Admin Panel — robots.txt Editor (Recommended)
EasyNo theme access needed
Store Setup → Store Settings → Search Engines
BigCommerce has a built-in robots.txt text editor in the admin panel. Paste AI bot directives into the custom rules field — no Stencil CLI, no theme editing required.
BigCommerce appends your rules below its default directives. Works on all plans.
Stencil Theme — templates/pages/robots.txt.html
EasyStencil CLI required
templates/pages/robots.txt.html
The Stencil theme has a Handlebars template that renders the full robots.txt. Editing this gives you complete control and overrides the admin panel setting.
Requires Stencil CLI (stencil push). Overrides the admin panel robots.txt editor entirely.
noai meta tag via base.html
EasyStencil CLI required
templates/layout/base.html
Add <meta name="robots" content="noai, noimageai"> to the master layout file. Applies to every storefront page — product pages, category pages, homepage, blog.
Most reliable method for per-page AI training opt-out. Requires Stencil theme access.
Script Manager — noai tag injection
EasyNo theme access needed
Storefront → Script Manager
Inject a small JavaScript snippet that adds the noai meta tag to document.head on every page. Available without Stencil CLI — works via the admin panel only.
Less reliable than base.html — the tag is added after initial HTML parse. Use base.html when possible.
Cloudflare WAF — hard blocking
IntermediateRequires Cloudflare proxy
Cloudflare Dashboard → Security → WAF
Block AI bots at the network edge before requests reach BigCommerce. The only way to enforce hard blocking on SaaS-hosted stores. Bots receive a 403 and never touch your store.
Requires routing traffic through Cloudflare (change DNS nameservers or set up a Cloudflare-proxied CNAME). Free plan supports WAF custom rules.
Method 1: Admin Panel robots.txt Editor
BigCommerce provides a built-in robots.txt editor accessible from the admin panel — no developer access required. Navigate to Store Setup → Store Settings and scroll to the Search Engines section. You'll see a text field labelled "Robots.txt".
BigCommerce pre-generates a default robots.txt (blocking checkout, cart, search pages, etc.) and appends your custom rules below it. Paste the full AI bot block list:
User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Google-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: xAI-Bot Disallow: / User-agent: DeepSeekBot Disallow: / User-agent: MistralBot Disallow: / User-agent: Diffbot Disallow: / User-agent: cohere-ai Disallow: / User-agent: AI2Bot Disallow: / User-agent: Ai2Bot-Dolma Disallow: / User-agent: YouBot Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: omgili Disallow: / User-agent: omgilibot Disallow: / User-agent: webzio-extended Disallow: / User-agent: gemini-deep-research Disallow: /
Stencil theme override
If your Stencil theme contains a templates/pages/robots.txt.html file, it overrides the admin panel editor entirely. If you've customised your theme, check this file and update it directly (see Method 2).
Method 2: Stencil Theme — robots.txt.html
The Stencil theme (BigCommerce's native theme system) includes a Handlebars template at templates/pages/robots.txt.html that renders the complete robots.txt response. Editing this file via the Stencil CLI gives you full control and overrides the admin panel setting.
The default file typically contains something like:
{{!-- templates/pages/robots.txt.html --}}
User-agent: *
{{#each settings.robots_disallow}}
Disallow: {{this}}
{{/each}}
{{settings.robots_txt_custom}}Replace the entire file with your own static content — or append the AI bot rules after the existing Handlebars logic:
{{!-- templates/pages/robots.txt.html --}}
User-agent: *
{{#each settings.robots_disallow}}
Disallow: {{this}}
{{/each}}
{{!-- AI training bot block --}}
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
{{!-- ... rest of bot list ... --}}After editing, push the theme with Stencil CLI:
# Requires Stencil CLI and BigCommerce API credentials stencil push --activate
Method 3: noai Meta Tag via base.html
The templates/layout/base.html file is the master HTML template for every storefront page. Adding the noai meta tag here applies it globally — homepage, product pages, category pages, blog posts, and all other storefront pages.
Open templates/layout/base.html in your Stencil theme and add the meta tag just before the closing </head> tag:
{{!-- templates/layout/base.html (excerpt) --}}
{{{head.scripts}}}
{{!-- Block AI training crawlers --}}
<meta name="robots" content="noai, noimageai" />
</head>
<body class="{{#if pageType}}{{pageType}}{{else}}default{{/if}}">Push with stencil push to deploy. Verify by viewing source on any product page — the meta tag should appear in the <head> section.
Method 4: Script Manager (No Theme Access)
If you don't have Stencil CLI access, BigCommerce's Script Manager lets you inject JavaScript snippets on storefront pages. Navigate to Storefront → Script Manager and create a new script with placement Head (some plans) or Footer:
<!-- Script Manager — inject noai meta tag -->
<script>
(function() {
var meta = document.createElement('meta');
meta.name = 'robots';
meta.content = 'noai, noimageai';
document.head.appendChild(meta);
})();
</script>Less reliable than base.html
The Script Manager injects tags after the page HTML is parsed. Many AI crawlers parse the raw HTML response without executing JavaScript — they'll see the page before the meta tag is injected. Use base.html whenever possible. Script Manager is a fallback for merchants without theme file access.
Method 5: Cloudflare WAF (Hard Blocking)
Since BigCommerce is fully hosted, the only way to hard-block AI bots (return a 403 before they receive any page content) is at the network edge. Cloudflare is the standard approach — route your store's traffic through Cloudflare and create a WAF custom rule.
If your store already uses Cloudflare (DNS proxied through Cloudflare), go to Security → WAF → Custom rules and create:
# Cloudflare WAF — Custom Rule (Expression Editor) (http.user_agent contains "GPTBot") or (http.user_agent contains "ClaudeBot") or (http.user_agent contains "anthropic-ai") or (http.user_agent contains "CCBot") or (http.user_agent contains "Bytespider") or (http.user_agent contains "Google-Extended") or (http.user_agent contains "PerplexityBot") or (http.user_agent contains "Diffbot") or (http.user_agent contains "DeepSeekBot") or (http.user_agent contains "MistralBot") or (http.user_agent contains "cohere-ai") or (http.user_agent contains "meta-externalagent") or (http.user_agent contains "Amazonbot") or (http.user_agent contains "xAI-Bot") or (http.user_agent contains "AI2Bot") or (http.user_agent contains "OAI-SearchBot") or (http.user_agent contains "ChatGPT-User") or (http.user_agent contains "gemini-deep-research") → Action: Block
BigCommerce provides its own CDN powered by Fastly. For Enterprise plans, Fastly VCL rules can also block AI bots — contact BigCommerce support to request custom VCL if you need deeper integration without Cloudflare. For most stores, Cloudflare WAF is simpler and sufficient.
Setting up Cloudflare for BigCommerce
- 1. Add your domain to Cloudflare (free plan works)
- 2. Update your domain registrar's nameservers to Cloudflare's nameservers
- 3. In Cloudflare DNS, create a CNAME pointing to your BigCommerce store URL (e.g.
store-xxxx.mybigcommerce.com), with Proxy enabled (orange cloud) - 4. In BigCommerce admin, go to Store Setup → Store Profile → URL to confirm your domain
- 5. Create the WAF custom rule above in Cloudflare Security → WAF
Why Product Catalogues Are High-Risk
E-commerce stores are among the most valuable AI scraping targets. Structured product data — names, descriptions, pricing, inventory, customer reviews — is exactly what AI companies need to train product recommendation engines, price intelligence tools, and shopping assistants. BigCommerce stores expose this data in clean, crawlable HTML.
Diffbot
Extracts structured product/price data. Used by price intelligence platforms and competitors.
GPTBot
Trains OpenAI models on your product descriptions, content, and copy.
Bytespider
ByteDance crawler — aggressive. Scrapes content for TikTok Shop and AI training.
Google-Extended
Trains Google AI (Gemini, AI Overviews) separately from Google Search indexing.
AI Bots to Block
25 user agents covering AI training crawlers and AI search bots. The robots.txt block above includes all of them.
Frequently Asked Questions
How do I edit robots.txt on BigCommerce?
Two ways: (1) Admin panel — go to Store Setup → Store Settings → Search Engines tab. There is a 'Robots.txt' text area where you can append custom directives. BigCommerce generates a default robots.txt and appends your custom rules below it. This is the quickest method with no theme editing required. (2) Stencil theme — if you use the Stencil CLI, edit templates/pages/robots.txt.html in your theme. This Handlebars template renders the full robots.txt content and overrides the admin panel version.
Where do I add the noai meta tag in BigCommerce?
Edit your Stencil theme's master layout file: templates/layout/base.html. Find the closing </head> tag and add <meta name="robots" content="noai, noimageai" /> just before it. This injects the tag on every storefront page. If you don't have Stencil CLI access, use BigCommerce's Script Manager (Storefront → Script Manager) to inject a script that dynamically adds the meta tag to the document head — less reliable but available without theme file access.
Can I block AI bots at the server level on BigCommerce?
Not directly. BigCommerce is a fully hosted SaaS platform — you don't have access to the underlying web server, so you cannot configure nginx, Apache, or add server-side middleware. The only way to block AI bots at the request level is through a network proxy: Cloudflare is the most common option. Put your BigCommerce store behind Cloudflare, then create a WAF custom rule to block AI bot user agents. This intercepts requests before they reach BigCommerce's servers.
Does BigCommerce have a built-in robots.txt editor?
Yes. Go to Store Setup → Store Settings → scroll to the Search Engines section. There is a text field where you can enter custom robots.txt directives. BigCommerce pre-populates a default file (blocking /checkout/, /cart.php, etc.) and appends your custom rules. This is the easiest method for stores without Stencil CLI access. Note: if your Stencil theme has a custom robots.txt.html template, it overrides this admin panel setting.
Will blocking AI bots affect BigCommerce SEO or Google Shopping?
No. Blocking GPTBot, ClaudeBot, CCBot, and other AI training crawlers does not affect Googlebot, Bingbot, or any shopping feed crawlers. Your Google Shopping product listings, Google Merchant Center feeds, and organic search rankings are completely unaffected. The robots.txt directives and noai meta tags specifically target AI training and AI search aggregation bots — not traditional search engine bots.
What's the most important AI bot to block on an e-commerce store?
Diffbot is arguably the most dangerous for e-commerce. It specialises in structured data extraction — product names, prices, descriptions, inventory signals, and pricing history. AI companies and competitors use Diffbot to build price intelligence databases from your store. GPTBot (OpenAI), Google-Extended (Google AI training), and Bytespider (ByteDance/TikTok) also actively scrape product catalogues for training data. Block all of them with robots.txt and Cloudflare WAF.
Is your site protected from AI bots?
Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.