Skip to content
Guides/Starlette

How to Block AI Bots on Starlette: Complete 2026 Guide

Starlette is the ASGI toolkit that powers FastAPI. It provides BaseHTTPMiddleware for simple Request/Response-based middleware and a pure ASGI scope/receive/send interface for maximum control. Block AI bots at the middleware layer before any route handler executes.

Starlette = FastAPI's foundation

FastAPI is a Starlette subclass — every Starlette middleware works identically in FastAPI. If you already have the FastAPI guide's BlockAiBotsMiddleware, it runs unchanged in a raw Starlette app. The app.add_middleware() call is the same. The only difference is from starlette.applications import Starlette vs from fastapi import FastAPI.

Protection layers

1
robots.txt — PlainTextResponse route at /robots.txt — or StaticFiles mount from a static/ directory
2
noai meta tag — Jinja2Templates base.html with {{ robots | default("noai, noimageai") }} variable
3
X-Robots-Tag header — response.headers["X-Robots-Tag"] = "noai, noimageai" after await call_next(request)
4
Hard 403 block — return Response("Forbidden", status_code=403) from dispatch() — do not call call_next()

Layer 1: robots.txt

Starlette does not serve a robots.txt automatically. Add a route that returns a PlainTextResponse, or mount a StaticFiles app for the entire static directory. Register this route before applying bot middleware so crawlers can always reach it.

Option A — explicit route (simplest)

from starlette.applications import Starlette
from starlette.requests import Request
from starlette.responses import PlainTextResponse
from starlette.routing import Route

ROBOTS_TXT = """
User-agent: *
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Diffbot
User-agent: cohere-ai
User-agent: FacebookBot
User-agent: omgili
User-agent: omgilibot
User-agent: Amazonbot
User-agent: DeepSeekBot
User-agent: MistralBot
User-agent: xAI-Bot
User-agent: AI2Bot
Disallow: /
""".strip()

async def robots_txt(request: Request) -> PlainTextResponse:
    return PlainTextResponse(ROBOTS_TXT)

app = Starlette(routes=[
    Route("/robots.txt", robots_txt),
    # ... your other routes
])

Option B — StaticFiles mount (if you have a static/ directory)

from starlette.staticfiles import StaticFiles

# Mount /static to serve ./static directory
# Place robots.txt at static/robots.txt — accessible at /static/robots.txt
# For /robots.txt directly, use the route approach above or nginx

app.mount("/static", StaticFiles(directory="static"), name="static")

Layer 2: noai meta tag

If your Starlette app renders HTML with Jinja2Templates, add the noai meta tag to your base layout and pass a per-route override variable:

templates/base.html

<!DOCTYPE html>
<html>
<head>
  <meta name="robots" content="{{ robots | default('noai, noimageai') }}">
  <title>{% block title %}My Site{% endblock %}</title>
</head>
<body>{% block content %}{% endblock %}</body>
</html>

Route handler — per-page override

from starlette.templating import Jinja2Templates

templates = Jinja2Templates(directory="templates")

async def home(request: Request):
    # No 'robots' key → base.html uses default "noai, noimageai"
    return templates.TemplateResponse(request, "home.html")

async def public_blog(request: Request):
    # Explicitly allow a public page for SEO
    return templates.TemplateResponse(
        request, "blog.html",
        {"robots": "index, follow"}
    )

If your Starlette app is a JSON API with a separate SPA frontend (React, Vue, SvelteKit), add the noai meta tag in the frontend's base layout instead.

Layers 3 & 4: BaseHTTPMiddleware

BaseHTTPMiddleware is the simplest way to write Starlette middleware — override dispatch() and work with Request and Response objects directly.

middleware/ai_bot_blocker.py

from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response

AI_BOT_PATTERNS = [
    "gptbot", "chatgpt-user", "oai-searchbot",
    "claudebot", "anthropic-ai", "claude-web",
    "google-extended", "ccbot", "bytespider",
    "applebot-extended", "perplexitybot", "diffbot",
    "cohere-ai", "facebookbot", "meta-externalagent",
    "omgili", "omgilibot", "amazonbot",
    "deepseekbot", "mistralbot", "xai-bot", "ai2-bot",
]

EXEMPT_PATHS = {"/robots.txt", "/sitemap.xml", "/favicon.ico"}


class AiBotBlocker(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next) -> Response:
        # Always pass through exempt paths
        if request.url.path in EXEMPT_PATHS:
            return await call_next(request)

        ua = request.headers.get("user-agent", "").lower()

        for pattern in AI_BOT_PATTERNS:
            if pattern in ua:
                # Layer 4: hard block — return 403, do NOT call call_next()
                return Response("Forbidden", status_code=403)

        # Layer 3: call next, then add X-Robots-Tag to the response
        response = await call_next(request)
        response.headers["X-Robots-Tag"] = "noai, noimageai"
        return response

Key points

  • Blocking: return Response("Forbidden", status_code=403) — returning a Response from dispatch() short-circuits the chain. The route handler never executes. Do not call await call_next(request).
  • X-Robots-Tag after call_next: Unlike Go's net/http (where you must set headers before calling next), Starlette's call_next() returns a Response object whose headers you can mutate before returning. Safe to set after.
  • Reading request headers: request.headers.get("user-agent", "")— returns an empty string if absent, avoiding AttributeError on .lower().
  • Writing response headers: response.headers["X-Robots-Tag"] = "noai, noimageai"— mutable dict-like object on the Response.

Registering the middleware

from starlette.applications import Starlette
from starlette.routing import Route
from starlette.responses import PlainTextResponse
from middleware.ai_bot_blocker import AiBotBlocker

async def robots_txt(request):
    return PlainTextResponse(ROBOTS_TXT)

async def home(request):
    return PlainTextResponse("Hello, World!")

app = Starlette(routes=[
    Route("/robots.txt", robots_txt),
    Route("/", home),
])

# add_middleware() — applied in LIFO order
# (last added = outermost wrapper)
app.add_middleware(AiBotBlocker)

add_middleware() wraps the application in LIFO order — the last middleware added runs first (outermost). To run bot blocking before auth middleware, add the bot blocker after auth middleware.

LIFO registration — the Starlette gotcha

Starlette's add_middleware() is LIFO (last in, first out) — the opposite of Express and Gin which are FIFO. If you add bot blocking first and then add auth middleware, auth runs before bot blocking. To ensure bots are rejected before any auth logic, add AiBotBlocker last. The Starlette Starlette(middleware=[]) constructor list is processed in order (first = outermost), which is the inverse — see route-scoped section below.

Pure ASGI middleware (advanced)

For lower overhead or when handling WebSocket connections, implement the raw ASGI interface with scope, receive, and send:

from typing import Callable

AI_BOT_PATTERNS = [
    "gptbot", "chatgpt-user", "claudebot", "anthropic-ai",
    "google-extended", "ccbot", "bytespider", "perplexitybot",
    # ... full list
]
EXEMPT_PATHS = {b"/robots.txt", b"/sitemap.xml", b"/favicon.ico"}


class AiBotBlockerASGI:
    def __init__(self, app: Callable) -> None:
        self.app = app

    async def __call__(self, scope, receive, send) -> None:
        # Only process HTTP/HTTPS requests (not websocket, lifespan)
        if scope["type"] not in ("http", "https"):
            await self.app(scope, receive, send)
            return

        # Check exempt paths
        path = scope.get("path", "").encode() or scope.get("raw_path", b"")
        if path in EXEMPT_PATHS:
            await self.app(scope, receive, send)
            return

        # Read User-Agent from headers (list of (name_bytes, value_bytes) tuples)
        ua = ""
        for name, value in scope.get("headers", []):
            if name.lower() == b"user-agent":
                ua = value.decode("utf-8", errors="replace").lower()
                break

        for pattern in AI_BOT_PATTERNS:
            if pattern in ua:
                # Send 403 response without calling the inner app
                await send({
                    "type": "http.response.start",
                    "status": 403,
                    "headers": [(b"content-type", b"text/plain")],
                })
                await send({
                    "type": "http.response.body",
                    "body": b"Forbidden",
                })
                return

        # Inject X-Robots-Tag by wrapping the send callable
        async def send_with_header(message):
            if message["type"] == "http.response.start":
                headers = list(message.get("headers", []))
                headers.append((b"x-robots-tag", b"noai, noimageai"))
                message = {**message, "headers": headers}
            await send(message)

        await self.app(scope, receive, send_with_header)

Register as a standard ASGI wrapper: app = AiBotBlockerASGI(app) — works with any ASGI server (uvicorn, hypercorn, daphne).

Route-scoped blocking

Apply middleware to a sub-application using the Middleware() wrapper in a Mount. Middleware passed to Mount applies only to routes within that mount — the root router is unaffected:

from starlette.applications import Starlette
from starlette.middleware import Middleware
from starlette.routing import Route, Mount

from middleware.ai_bot_blocker import AiBotBlocker

# Public routes — no bot blocking
async def home(request):
    from starlette.responses import HTMLResponse
    return HTMLResponse("<h1>Welcome</h1>")

# API routes — bot blocking applied only here
async def api_products(request):
    from starlette.responses import JSONResponse
    return JSONResponse({"products": []})

async def api_users(request):
    from starlette.responses import JSONResponse
    return JSONResponse({"users": []})

app = Starlette(routes=[
    Route("/", home),
    Mount("/api", routes=[
        Route("/products", api_products),
        Route("/users", api_users),
    ], middleware=[
        Middleware(AiBotBlocker),
    ]),
])

The middleware list on a Mount is processed in order (first = outermost), unlike app.add_middleware() which is LIFO.

Comparison: Starlette vs FastAPI

Starlette and FastAPI share the same middleware code — only the app class differs:

Starlette

from starlette.applications import Starlette
from starlette.routing import Route
from middleware.ai_bot_blocker import AiBotBlocker

app = Starlette(routes=[Route("/", home)])
app.add_middleware(AiBotBlocker)

FastAPI (identical middleware)

from fastapi import FastAPI
from middleware.ai_bot_blocker import AiBotBlocker

app = FastAPI()
app.add_middleware(AiBotBlocker)

@app.get("/")
async def home():
    return {"message": "Hello"}

Pure ASGI (framework-agnostic)

# Works with Starlette, FastAPI, Django (ASGI), Litestar
# — any ASGI-compliant application
app = AiBotBlockerASGI(your_app)

The AiBotBlocker middleware file is completely portable between Starlette and FastAPI. Factor it into a shared package if you run both in the same codebase.

Deployment with uvicorn

# Install
pip install starlette uvicorn

# Run (development)
uvicorn app:app --host 0.0.0.0 --port 8000 --reload

# Run (production — multiple workers)
uvicorn app:app --host 0.0.0.0 --port 8000 --workers 4

# Or with gunicorn + uvicorn workers
gunicorn app:app -w 4 -k uvicorn.workers.UvicornWorker

For server-level blocking (before Python runs), add an nginx map $http_user_agent block in front of uvicorn. See the nginx guide for the configuration.

Verification

# Should return 403 (blocked AI bot)
curl -I -A "GPTBot" http://localhost:8000/

# Should return 200 (regular browser)
curl -I -A "Mozilla/5.0" http://localhost:8000/

# robots.txt must always return 200
curl -I -A "GPTBot" http://localhost:8000/robots.txt

# Check X-Robots-Tag on a legitimate request
curl -si -A "Mozilla/5.0" http://localhost:8000/ | grep -i x-robots

Expected: GPTBot → 403. Mozilla/5.0 → 200 with X-Robots-Tag: noai, noimageai. robots.txt → 200 for any user agent.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.