How to Block Amazonbot

Amazon's Three Crawlers

Unlike most AI companies which run a single crawler, Amazon operates three separate bots with distinct purposes. Getting these wrong — especially blocking Amzn-SearchBot when you're an e-commerce brand — can hurt visibility in Amazon's AI experiences.

Which Amazon Bot Should You Block?

Option 1: Block via robots.txt

# Block Amazon AI training crawler only
# Allows Amzn-SearchBot (Rufus + Alexa search) and Amzn-User (live queries)
User-agent: Amazonbot
Disallow: /

# Block all Amazon crawlers
User-agent: Amazonbot
Disallow: /

User-agent: Amzn-SearchBot
Disallow: /

User-agent: Amzn-User
Disallow: /

# Block Amazon AI training
User-agent: Amazonbot
Disallow: /

# Block other major AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: MistralBot
Disallow: /

User-agent: xAI-Bot
Disallow: /

User-agent: DeepSeekBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Search engines — unaffected
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Amzn-SearchBot
Allow: /

Option 2: Next.js App Router

import { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      // Block Amazon AI training only
      { userAgent: 'Amazonbot', disallow: ['/'] },

      // Allow Amazon search/Alexa experiences (Rufus, etc.)
      { userAgent: 'Amzn-SearchBot', allow: ['/'] },
      { userAgent: 'Amzn-User', allow: ['/'] },

      // Block other AI training crawlers
      { userAgent: 'GPTBot', disallow: ['/'] },
      { userAgent: 'ClaudeBot', disallow: ['/'] },
      { userAgent: 'anthropic-ai', disallow: ['/'] },
      { userAgent: 'Google-Extended', disallow: ['/'] },
      { userAgent: 'PerplexityBot', disallow: ['/'] },
      { userAgent: 'CCBot', disallow: ['/'] },

      // Allow search engines
      { userAgent: 'Googlebot', allow: ['/'] },
      { userAgent: '*', allow: ['/'] },
    ],
    sitemap: 'https://yoursite.com/sitemap.xml',
  };
}

Option 3: nginx — Hard Block

For server-level enforcement that doesn't depend on Amazon honoring robots.txt:

# Block Amazonbot (AI training) — hard 403
if ($http_user_agent ~* "Amazonbot") {
    return 403;
}

# Optionally block Amzn-SearchBot too
# if ($http_user_agent ~* "Amzn-SearchBot") {
#     return 403;
# }

Returns HTTP 403 before the request reaches your application. Combine with robots.txt for defense in depth.

Option 4: Cloudflare WAF Rule

(http.user_agent contains "Amazonbot")

Set the action to Block. To also block Amzn-SearchBot:

(http.user_agent contains "Amazonbot") or (http.user_agent contains "Amzn-SearchBot")

Cloudflare Dashboard → Security → WAF → Custom Rules → Create rule

What Is Rufus — And Why It Matters for Publishers

Rufus is Amazon's AI shopping assistant, launched in 2024 and now available to hundreds of millions of Amazon customers. When a customer asks Rufus a question — "What's the best laptop for video editing?" or "Compare these two coffee makers" — Rufus draws on content indexed by Amzn-SearchBot, including product reviews, buying guides, and editorial content from the open web.

For most content publishers, the right call is: block Amazonbot, allow Amzn-SearchBot. You stop contributing to Amazon's AI training datasets while keeping your content visible to Rufus users and Alexa queries.

Verify Your Block

# Check nginx access logs for Amazon bots
grep "Amazonbot" /var/log/nginx/access.log | tail -20
grep "Amzn-SearchBot" /var/log/nginx/access.log | tail -10

# Confirm Amazonbot fetched robots.txt (then stopped)
grep "Amazonbot" /var/log/nginx/access.log | grep "robots.txt"

# If server-level blocked — confirm 403s
grep "Amazonbot" /var/log/nginx/access.log | grep " 403 "

# Check published IPs against your access logs
# Amazon publishes Amazonbot IPs at:
# https://developer.amazon.com/amazonbot/live-ip-addresses/

Seeing Amazonbot fetch /robots.txt followed by no content requests confirms the block is working. Amazon updates its robots.txt cache every 30 days — allow up to 24 hours for changes to take effect.

Frequently Asked Questions

Related Guides