How to Block AI Bots on Dart Shelf: Complete 2026 Guide
Shelf is Dart's composable web server middleware library — the foundation under Dart Frog and most Dart backend frameworks. Middleware follows a single typedef: Handler Function(Handler). To block a bot: return Response.forbidden() without calling innerHandler. Headers use lowercase keys — always request.headers['user-agent'], not 'User-Agent'.
Shelf Middleware typedef
typedef Middleware = Handler Function(Handler innerHandler);
A middleware is a function that takes the next Handler and returns a new one. Return a Response directly to short-circuit; call innerHandler(request) to continue. Compose with Pipeline().addMiddleware(). Middleware added first runs first on the way in.
Protection layers
1
robots.txt — Cascade().add(staticHandler) before bot-protected app — static files served before middleware runs
2
noai meta tag — In HTML response body — <meta name="robots" content="noai, noimageai"> in <head>
3
X-Robots-Tag (blocked) — headers: {"X-Robots-Tag": "noai, noimageai"} in Response.forbidden() — included in 403
4
X-Robots-Tag (legitimate) — response.change(headers: {"X-Robots-Tag": "noai, noimageai"}) on innerHandler result
5
Hard 403 — Response.forbidden() returned directly — innerHandler never called, no downstream processing
Step 1 — Bot detection (lib/ai_bots.dart)
A top-level const List<String> — compiled into the binary, zero allocation per request. List.any() short-circuits on first match. Caller lowercases the UA before passing.
// lib/ai_bots.dart — bot detection
const List<String> aiBotPatterns = [
// OpenAI
'gptbot', 'chatgpt-user', 'oai-searchbot',
// Anthropic
'claudebot', 'claude-web',
// Common Crawl
'ccbot',
// Bytedance
'bytespider',
// Meta
'meta-externalagent',
// Perplexity
'perplexitybot',
// Google AI
'google-extended', 'googleother',
// Cohere
'cohere-ai',
// Amazon
'amazonbot',
// Diffbot
'diffbot',
// AI2
'ai2bot',
// DeepSeek
'deepseekbot',
// Mistral
'mistralai-user',
// xAI
'xai-bot',
// You.com
'youbot',
// DuckDuckGo AI
'duckassistbot',
];
/// Returns true if [ua] belongs to a known AI bot.
/// [ua] must already be lowercased before calling.
bool isAiBot(String ua) {
if (ua.isEmpty) return false;
return aiBotPatterns.any((pattern) => ua.contains(pattern));
}Step 2 — Middleware function (lib/bot_blocker.dart)
Returns a Middleware getter. The inner closure receives the Request at runtime. response.change() copies the response with additional headers — non-destructive.
// lib/bot_blocker.dart — Shelf middleware
import 'package:shelf/shelf.dart';
import 'ai_bots.dart';
/// botBlockerMiddleware is a Shelf Middleware.
/// Middleware typedef: Handler Function(Handler innerHandler)
///
/// Compose with: Pipeline().addMiddleware(botBlockerMiddleware)
Middleware get botBlockerMiddleware {
return (Handler innerHandler) {
return (Request request) async {
// request.headers is Map<String, String> with lowercase keys.
// Shelf normalises header names to lowercase per HTTP spec.
// Use ?? '' — headers['user-agent'] returns String? (nullable).
final ua = (request.headers['user-agent'] ?? '').toLowerCase();
if (isAiBot(ua)) {
// Short-circuit: return 403 directly.
// innerHandler is never called — no downstream processing.
return Response.forbidden(
'Forbidden',
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'X-Robots-Tag': 'noai, noimageai',
},
);
}
// Pass through: call innerHandler, then add X-Robots-Tag to response.
final response = await innerHandler(request);
return response.change(headers: {'X-Robots-Tag': 'noai, noimageai'});
};
};
}Step 3 — Server setup with Cascade for robots.txt
Cascade tries each handler in order. createStaticHandler returns 404 for missing files — the Cascade falls through to the bot-protected app. robots.txt is served by the static handler before botBlockerMiddleware runs.
// bin/server.dart — pipeline composition and server startup
import 'dart:io';
import 'package:shelf/shelf.dart';
import 'package:shelf/shelf_io.dart' as shelf_io;
import 'package:shelf_router/shelf_router.dart';
import 'package:shelf_static/shelf_static.dart';
import '../lib/bot_blocker.dart';
void main() async {
// robots.txt — static files served from public/ directory.
// createStaticHandler serves public/robots.txt at /robots.txt.
// This handler returns 404 for non-existent paths — Cascade falls through.
final staticHandler = createStaticHandler(
'public',
defaultDocument: null,
);
// Application router — only reached for non-static, non-blocked requests.
final router = Router()
..get('/', _homeHandler)
..get('/health', (Request _) => Response.ok('ok'))
..get('/api/data', _apiDataHandler);
// Pipeline: static files → bot blocker → router.
// Middleware added first = outermost wrapper = runs first.
// staticHandler is a plain Handler, not Middleware — use Cascade instead.
final protectedApp = const Pipeline()
.addMiddleware(botBlockerMiddleware)
.addHandler(router);
// Cascade: try staticHandler first (serves robots.txt, falls through on 404),
// then the bot-protected app.
// AI bots that hit /robots.txt are served BEFORE reaching botBlockerMiddleware.
final handler = Cascade()
.add(staticHandler)
.add(protectedApp)
.handler;
// Wrap with logging for request visibility.
final loggedHandler = logRequests().addHandler(handler);
final server = await shelf_io.serve(
loggedHandler,
InternetAddress.anyIPv4,
int.parse(Platform.environment['PORT'] ?? '8080'),
);
print('Serving on http://${server.address.host}:${server.port}');
}
Future<Response> _homeHandler(Request request) async {
const html = '''<!DOCTYPE html>
<html>
<head>
<meta name="robots" content="noai, noimageai">
<title>My Site</title>
</head>
<body><h1>Welcome</h1></body>
</html>''';
return Response.ok(
html,
headers: {'Content-Type': 'text/html; charset=utf-8'},
);
}
Future<Response> _apiDataHandler(Request request) async {
return Response.ok(
'{"data":"protected"}',
headers: {'Content-Type': 'application/json'},
);
}Step 4 — Dart Frog variant (routes/_middleware.dart)
Dart Frog uses the same Shelf middleware contract. Place a _middleware.dart file in routes/ — Dart Frog applies it to every route in that directory and all subdirectories automatically.
// routes/_middleware.dart — Dart Frog middleware
// Dart Frog is built on Shelf. _middleware.dart applies to all routes
// in the same directory and all subdirectories automatically.
import 'package:dart_frog/dart_frog.dart';
import '../lib/ai_bots.dart';
// Dart Frog middleware signature: Handler Function(Handler handler)
// Identical to Shelf's Middleware typedef.
Handler middleware(Handler handler) {
return (RequestContext context) async {
final request = context.request;
// Dart Frog wraps Shelf's Request — access headers via request.headers.
final ua = (request.headers['user-agent'] ?? '').toLowerCase();
if (isAiBot(ua)) {
return Response(
statusCode: 403,
body: 'Forbidden',
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'X-Robots-Tag': 'noai, noimageai',
},
);
}
final response = await handler(context);
// Dart Frog Response does not have .change() — copy headers manually.
return response.copyWith(
headers: {...response.headers, 'X-Robots-Tag': 'noai, noimageai'},
);
};
}
// ----------------------------------------------------------------
// routes/robots.txt.dart — serve robots.txt outside _middleware.dart scope
// Place robots.txt route in the routes/ root to bypass the middleware.
// Dart Frog applies _middleware.dart from the nearest ancestor — a route
// at the same level is still wrapped. For true bypass, serve it statically
// via a public/ directory or Dart Frog's static file serving.pubspec.yaml dependencies
# pubspec.yaml
name: my_app
description: Shelf app with AI bot blocking
version: 1.0.0
environment:
sdk: '>=3.0.0 <4.0.0'
dependencies:
shelf: ^1.4.0
shelf_router: ^1.1.0
shelf_static: ^1.1.0
dev_dependencies:
lints: ^3.0.0
test: ^1.24.0
# For Dart Frog instead:
# dependencies:
# dart_frog: ^1.1.0
# dev_dependencies:
# dart_frog_dev: ^1.1.0Dart Shelf vs Dart Frog vs Express vs Go chi
| Feature | Dart Shelf | Dart Frog | Node Express | Go chi |
|---|---|---|---|---|
| Middleware model | Middleware typedef: Handler Function(Handler) — composed via Pipeline.addMiddleware() | Same Shelf typedef wrapped by Dart Frog. _middleware.dart auto-applied per directory. | app.use(fn(req, res, next)) — imperative, call next() or send response directly | r.Use(func(next http.Handler) http.Handler) — same function-wrapping pattern as Shelf |
| Short-circuit | Return Response directly without calling innerHandler — innerHandler never runs | Return Response(statusCode: 403) without calling handler(context) | res.status(403).send("Forbidden") — do NOT call next() | w.WriteHeader(403); w.Write([]byte("Forbidden")) — do NOT call next.ServeHTTP() |
| UA header access | request.headers["user-agent"] ?? "" — lowercase keys, returns String? (Dart nullable) | context.request.headers["user-agent"] ?? "" — same Shelf header map underneath | req.headers["user-agent"] || "" — lowercase in Node.js http module | r.Header.Get("User-Agent") — Go http.Header.Get normalises case automatically |
| robots.txt | Cascade().add(staticHandler).add(protectedApp) — static files served before bot check | public/ directory served by Dart Frog before middleware chain runs | express.static("public") mounted before bot-blocker middleware | http.FileServer(http.Dir("public")) mounted before chi middleware stack |
| Pipeline composition | const Pipeline().addMiddleware(A).addMiddleware(B).addHandler(router) — first added = outermost | Implicit via _middleware.dart file hierarchy — no explicit Pipeline needed | app.use() call order = execution order — first app.use() = first middleware | r.Use() call order = execution order — same left-to-right execution |
| Typed language | Dart — strong static types, null safety (String?), AOT compilation for production | Dart — same type safety, Dart Frog adds code generation for routes | JavaScript/TypeScript — runtime types, TS adds type safety at build time | Go — strong static types, no null safety needed (zero values instead) |
Summary
- Return without calling innerHandler — returning
Response.forbidden()directly short-circuits the pipeline. The next handler and all downstream middleware never run. - Lowercase header keys —
request.headers['user-agent'](lowercase). Shelf normalises all header names. Using'User-Agent'returns null. Cascadefor robots.txt — static file handler first, bot-protected app second. Static handler returns 404 on miss; Cascade falls through. robots.txt is always accessible.response.change()— immutable response modification. Adds headers to legitimate responses without mutating the original.- Dart Frog — same Shelf middleware contract. Drop
_middleware.dartin your routes directory; Dart Frog wires it automatically.
Is your site protected from AI bots?
Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.