Skip to content
FastAPIPythonNew8 min read

How to Block AI Bots on FastAPI

FastAPI has no built-in robots.txt — you add a GET /robots.txt route that returns a PlainTextResponse. For hard blocking, a Starlette BaseHTTPMiddleware intercepts requests before any route handler runs and returns 403 for matched AI bots.

Quick fix — robots.txt endpoint

Add to your main.py. FastAPI does not serve robots.txt automatically.

from fastapi import FastAPI
from fastapi.responses import PlainTextResponse

app = FastAPI()

@app.get("/robots.txt", response_class=PlainTextResponse, include_in_schema=False)
def robots_txt():
    return """User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
"""

All Methods

robots.txt endpoint (Recommended)

Easy

All deployments

GET /robots.txt route

A FastAPI GET route that returns a PlainTextResponse listing all AI bots with Disallow: /. FastAPI has no built-in robots.txt — you must add this route explicitly. Works on any deployment.

Place this route before any catch-all route. Use StaticFiles mount alternatively if you prefer a static file.

BaseHTTPMiddleware — hard blocking

Easy

All deployments

app.add_middleware(BlockAiBotsMiddleware)

A Starlette BaseHTTPMiddleware subclass that returns Response(status_code=403) when the User-Agent matches an AI bot pattern. Intercepts requests before any route handler runs.

Middleware is applied in reverse order. Register bot-blocking middleware last (or first in the add_middleware call sequence) so it runs first.

X-Robots-Tag response header

Easy

All deployments

middleware or FastAPI dependency

Add X-Robots-Tag: noai, noimageai to every response via middleware. Signals AI training crawlers not to use the content — complementary to robots.txt, not a replacement.

HTTP response headers work for any content type (HTML, JSON, XML) — useful for API responses that robots.txt cannot target.

Jinja2 template noai meta tag

Easy

HTML-rendering FastAPI apps

templates/base.html

If rendering HTML with Jinja2Templates, add <meta name="robots" content="noai, noimageai"> to your base template. All pages inherit it automatically.

Applicable only when FastAPI serves HTML. Pure API projects (returning JSON) should use X-Robots-Tag header instead.

nginx — server-level block

Intermediate

nginx + uvicorn deployments

nginx server block config

Match AI bot user agents in nginx and return 403 before uvicorn and FastAPI receive the request. Most efficient — zero Python overhead for blocked bots.

Requires server access (VPS). Not available on Railway, Render, or other PaaS platforms without custom Dockerfile.

Method 1: robots.txt Endpoint

Unlike Django (which has a static files convention) or Rails (which ships with public/robots.txt), FastAPI does not serve any file at /robots.txt by default. Add a GET route that returns a PlainTextResponse — or mount a StaticFiles directory.

# main.py
from fastapi import FastAPI
from fastapi.responses import PlainTextResponse

app = FastAPI()

AI_BOTS = [
    "GPTBot", "ChatGPT-User", "OAI-SearchBot",
    "ClaudeBot", "anthropic-ai", "Google-Extended",
    "Bytespider", "CCBot", "PerplexityBot",
    "meta-externalagent", "Amazonbot", "Applebot-Extended",
    "xAI-Bot", "DeepSeekBot", "MistralBot", "Diffbot",
    "cohere-ai", "AI2Bot", "Ai2Bot-Dolma", "YouBot",
    "DuckAssistBot", "omgili", "omgilibot",
    "webzio-extended", "gemini-deep-research",
]

@app.get("/robots.txt", response_class=PlainTextResponse, include_in_schema=False)
def robots_txt(request: Request):
    lines = []
    for bot in AI_BOTS:
        lines += [f"User-agent: {bot}", "Disallow: /", ""]
    lines += ["User-agent: *", "Allow: /", ""]
    lines.append(f"Sitemap: {request.base_url}sitemap.xml")
    return "\n".join(lines)

Alternatively, serve a static file. Create static/robots.txt and mount the directory:

# main.py — static file approach
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles

app = FastAPI()

# Mount at root so /robots.txt is served from static/robots.txt
app.mount("/", StaticFiles(directory="static", html=True), name="static")
include_in_schema=False — add this to the route decorator to exclude /robots.txt from your OpenAPI docs (/docs and /redoc). It is a utility endpoint, not part of your API surface.

Method 2: BaseHTTPMiddleware (Hard Blocking)

FastAPI is built on Starlette. The BaseHTTPMiddleware class lets you intercept every request before any route handler runs. Return Response(status_code=403) for matched bot user agents — no route handler, no response body, no wasted CPU:

# middleware.py
import re
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response

BLOCKED_UA_PATTERN = re.compile(
    r"GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|anthropic-ai"
    r"|Google-Extended|Bytespider|CCBot|PerplexityBot"
    r"|meta-externalagent|Amazonbot|Applebot-Extended|xAI-Bot"
    r"|DeepSeekBot|MistralBot|Diffbot|cohere-ai|AI2Bot|Ai2Bot-Dolma"
    r"|YouBot|DuckAssistBot|omgili|omgilibot|webzio-extended"
    r"|gemini-deep-research",
    re.IGNORECASE,
)

class BlockAiBotsMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        ua = request.headers.get("user-agent", "")
        if BLOCKED_UA_PATTERN.search(ua):
            return Response(content="Forbidden", status_code=403)
        return await call_next(request)

Register the middleware in main.py:

# main.py
from fastapi import FastAPI
from middleware import BlockAiBotsMiddleware

app = FastAPI()
app.add_middleware(BlockAiBotsMiddleware)

Middleware registration order

Starlette applies middleware in reverse registration order. The last add_middleware() call runs first. Register bot-blocking middleware as the last add_middleware call so it intercepts requests before CORS, authentication, and other middleware execute.

Method 3: X-Robots-Tag Response Header

The X-Robots-Tag HTTP response header is the server-side equivalent of the noai meta tag. It works for any content type — HTML, JSON, XML — making it ideal for API responses where meta tags are not applicable. Add it via middleware:

# middleware.py — add X-Robots-Tag to every response
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request

class RobotsHeaderMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        response = await call_next(request)
        response.headers["X-Robots-Tag"] = "noai, noimageai"
        return response

Or set it as a default response header on the FastAPI app:

# main.py — per-route header via dependency
from fastapi import FastAPI, Response

app = FastAPI()

@app.middleware("http")
async def add_robots_header(request, call_next):
    response = await call_next(request)
    response.headers["X-Robots-Tag"] = "noai, noimageai"
    return response

Method 4: noai Meta Tag in Jinja2 Templates

If your FastAPI app renders HTML with Jinja2Templates, add the noai meta tag to your base template. All pages that extend it inherit the tag automatically:

{# templates/base.html #}
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{% block title %}My App{% endblock %}</title>

    {# Block AI training crawlers on every page #}
    {% block robots_meta %}
    <meta name="robots" content="noai, noimageai">
    {% endblock %}
</head>
<body>
    {% block content %}{% endblock %}
</body>
</html>

Use Jinja2Templates in your route:

# main.py
from fastapi import FastAPI, Request
from fastapi.templating import Jinja2Templates

app = FastAPI()
templates = Jinja2Templates(directory="templates")

@app.get("/")
def index(request: Request):
    return templates.TemplateResponse("index.html", {"request": request})

Method 5: nginx (Server-Level Blocking)

Production FastAPI deployments typically run uvicorn behind nginx as a reverse proxy. Adding user agent blocking to nginx means matched bots never reach Python — no uvicorn worker processes the request, no FastAPI overhead:

# /etc/nginx/sites-available/yourapp.conf
server {
    listen 80;
    server_name yourdomain.com;

    # Block AI training crawlers at the edge
    if ($http_user_agent ~* "(GPTBot|ClaudeBot|anthropic-ai|CCBot|Bytespider|Google-Extended|PerplexityBot|Diffbot|DeepSeekBot|MistralBot|cohere-ai|meta-externalagent|Amazonbot|xAI-Bot|AI2Bot|omgili|webzio-extended|gemini-deep-research|OAI-SearchBot|ChatGPT-User)") {
        return 403;
    }

    location / {
        proxy_pass http://127.0.0.1:8000;  # uvicorn
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Run FastAPI with uvicorn and reload nginx:

# Start uvicorn (production)
uvicorn main:app --host 127.0.0.1 --port 8000 --workers 4

# Or with gunicorn + uvicorn workers
gunicorn main:app -w 4 -k uvicorn.workers.UvicornWorker --bind 127.0.0.1:8000

# Reload nginx after config change
sudo nginx -t && sudo systemctl reload nginx

PaaS deployments (Railway, Render, Fly.io)

On platform-as-a-service hosts without nginx access, use the Python-layer methods:

  • ✓ BlockAiBotsMiddleware — 403 response before any route handler
  • ✓ GET /robots.txt endpoint — polite opt-out for compliant crawlers
  • ✓ Cloudflare in front of your PaaS domain — WAF custom rules for edge blocking
  • ✓ Fly.io Machines — supports custom Dockerfile with nginx included

AI Bots to Block

25 user agents covering AI training crawlers and AI search bots. The endpoint and middleware patterns above include all of them.

GPTBotChatGPT-UserOAI-SearchBotClaudeBotanthropic-aiGoogle-ExtendedBytespiderCCBotPerplexityBotmeta-externalagentAmazonbotApplebot-ExtendedxAI-BotDeepSeekBotMistralBotDiffbotcohere-aiAI2BotAi2Bot-DolmaYouBotDuckAssistBotomgiliomgilibotwebzio-extendedgemini-deep-research

Frequently Asked Questions

How do I serve robots.txt from a FastAPI app?

Add a GET route for '/robots.txt' that returns a PlainTextResponse. FastAPI does not serve a robots.txt automatically — you must add the route explicitly. Use a list of AI bot user agent names to build the Disallow blocks dynamically, or return a static string. Place this route before any catch-all routes in your router.

What is BaseHTTPMiddleware in FastAPI?

BaseHTTPMiddleware is a Starlette class (FastAPI is built on Starlette) that lets you inspect and modify requests and responses. Override the dispatch() method to check the User-Agent header and return a Response(status_code=403) for matched bots — before any route handler runs. Add it to your FastAPI app with app.add_middleware(BlockAiBotsMiddleware). Multiple middleware are applied in reverse order of registration.

Should I use BaseHTTPMiddleware or a pure ASGI middleware for bot blocking?

BaseHTTPMiddleware is easier to write and works perfectly for bot blocking. Pure ASGI middleware (implementing __call__ with scope/receive/send) has slightly lower overhead but the difference is negligible for bot filtering. Use BaseHTTPMiddleware unless you are building a high-throughput proxy or have specific ASGI streaming needs.

Does blocking AI bots affect FastAPI's OpenAPI docs?

Blocking is based on User-Agent matching. FastAPI's built-in docs (/docs, /redoc) are accessed by browsers, not AI crawlers — so they are unaffected. If you want to block AI bots from crawling your API schema specifically, add '/openapi.json' to the robots.txt Disallow list alongside '/'. This tells compliant crawlers to skip your schema endpoint.

How do I add noai meta tags in a FastAPI HTML response?

FastAPI is primarily used for APIs, but if you are rendering HTML with Jinja2Templates, add the noai meta tag to your base template: <meta name="robots" content="noai, noimageai">. Place it in the <head> section of your base.html template. All templates that extend it inherit the tag. For JSON API responses, use the X-Robots-Tag response header instead — add it in middleware or as a dependency.

How do I deploy FastAPI with nginx and block AI bots at the server level?

Run FastAPI with uvicorn behind nginx as a reverse proxy. In the nginx server block, add an if block that checks $http_user_agent against AI bot names and returns 403 before uvicorn is invoked. This is the most efficient approach — no Python code runs for blocked bots. Requires server access (VPS, EC2, Hetzner). Not available on Railway, Render, or other PaaS platforms.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.

Related Guides

How to Block AI Bots on Django: Complete 2026 GuideHow to Block AI Bots on Flask: Complete 2026 Guiderobots.txt for AI Bots: Complete 2026 GuideHow to Block GPTBot (OpenAI)How to Block CCBot (Common Crawl)