How to Block AI Bots on Fastify: Complete 2026 Guide

Methods at a glance

1. robots.txt — @fastify/static

Register @fastify/static pointing at your public/ directory. Fastify serves every file under that directory at the root path — public/robots.txt becomes /robots.txt with no additional routing.

public/robots.txt

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

server.ts — static file serving

import Fastify from 'fastify';
import fastifyStatic from '@fastify/static';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';

const __dirname = fileURLToPath(new URL('.', import.meta.url));
const fastify = Fastify({ logger: true });

// Register static serving — public/robots.txt → /robots.txt
// Register BEFORE the bot-blocking hook so crawlers can always fetch it
await fastify.register(fastifyStatic, {
  root: join(__dirname, 'public'),
  prefix: '/',
  // Disable index.html serving if you want explicit routes only
  index: false,
});

// ... rest of app
await fastify.listen({ port: 3000, host: '0.0.0.0' });

Alternative: dynamic /robots.txt route

const ROBOTS_PROD = `User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: *
Allow: /`;

// Allow everything in development/staging
const ROBOTS_DEV = 'User-agent: *\nDisallow: /';

fastify.get('/robots.txt', async (request, reply) => {
  const content = process.env.NODE_ENV === 'production' ? ROBOTS_PROD : ROBOTS_DEV;
  return reply.type('text/plain').send(content);
});

2. Fastify lifecycle — why onRequest

Fastify has a strict request lifecycle with named hooks at each stage. The correct hook for bot blocking is onRequest — it fires first, before routing, serialization, and body parsing. Using preHandler wastes cycles parsing the request body before rejecting a bot.

3. Global bot blocking — fastify-plugin

Fastify uses plugin encapsulation — a hook registered inside a plugin only applies to routes within that plugin's scope. Wrap your bot-blocking plugin with fp() (from fastify-plugin) to break encapsulation and apply the hook globally.

plugins/block-ai-bots.ts

import fp from 'fastify-plugin';
import type { FastifyPluginAsync } from 'fastify';

// Compile once at module load — not per request
const BLOCKED_UAS =
  /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|Google-Extended|Bytespider|CCBot|PerplexityBot|Applebot-Extended/i;

const blockAiBotsPlugin: FastifyPluginAsync = async (fastify) => {
  fastify.addHook('onRequest', async (request, reply) => {
    // Always allow robots.txt — exempt before UA check
    if (request.url === '/robots.txt') return;

    const ua = request.headers['user-agent'] ?? '';
    if (BLOCKED_UAS.test(ua)) {
      return reply.code(403).type('text/plain').send('Forbidden');
    }
  });
};

// fp() breaks encapsulation — hook applies to root instance + all plugins
export default fp(blockAiBotsPlugin, {
  name: 'block-ai-bots',
  fastify: '>=4',
});

server.ts — register in correct order

import Fastify from 'fastify';
import fastifyStatic from '@fastify/static';
import { join } from 'node:path';
import { fileURLToPath } from 'node:url';
import blockAiBots from './plugins/block-ai-bots.js';

const __dirname = fileURLToPath(new URL('.', import.meta.url));
const fastify = Fastify({ logger: true });

// 1. Static files first — robots.txt must be reachable by all crawlers
await fastify.register(fastifyStatic, {
  root: join(__dirname, 'public'),
  prefix: '/',
  index: false,
});

// 2. Bot blocking — global via fp(), applies to all routes
await fastify.register(blockAiBots);

// 3. Application routes
fastify.get('/', async () => ({ hello: 'world' }));

await fastify.listen({ port: 3000, host: '0.0.0.0' });

4. Per-route blocking — preHandler

To block AI bots only on specific routes (e.g. your API but not your marketing pages), use preHandler on individual route definitions or within a scoped plugin without fp().

Inline preHandler on a single route

const BLOCKED_UAS =
  /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-Web|anthropic-ai|Google-Extended|Bytespider|CCBot|PerplexityBot|Applebot-Extended/i;

fastify.get('/api/content', {
  preHandler: async (request, reply) => {
    const ua = request.headers['user-agent'] ?? '';
    if (BLOCKED_UAS.test(ua)) {
      return reply.code(403).type('text/plain').send('Forbidden');
    }
  },
  handler: async (request, reply) => {
    return { data: 'protected content' };
  },
});

Scoped plugin — block only /api/* routes

// No fp() — hook only applies to routes inside this plugin
fastify.register(async (scope) => {
  scope.addHook('onRequest', async (request, reply) => {
    const ua = request.headers['user-agent'] ?? '';
    if (BLOCKED_UAS.test(ua)) {
      return reply.code(403).type('text/plain').send('Forbidden');
    }
  });

  scope.get('/users', async () => [{ id: 1 }]);
  scope.get('/posts', async () => [{ id: 1 }]);
}, { prefix: '/api' });

// Public routes — bot blocking does NOT apply here
fastify.get('/', async () => 'Hello');

5. noai meta tag — @fastify/view

Use @fastify/view with a template engine (Eta, Handlebars, Pug, EJS) to render HTML with conditional noai meta tags. Pass a blockAiTraining flag per route.

views/layout.eta

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <% if (it.blockAiTraining) { %>
  <meta name="robots" content="noai, noimageai">
  <% } %>
  <title><%= it.title %></title>
</head>
<body>
  <%~ it.body %>
</body>
</html>

Route with blockAiTraining flag

import view from '@fastify/view';
import { Eta } from 'eta';

await fastify.register(view, {
  engine: { eta: new Eta() },
  root: './views',
});

// Block AI training on article pages
fastify.get('/articles/:id', async (request, reply) => {
  return reply.view('layout', {
    title: 'Article',
    body: '<h1>Protected content</h1>',
    blockAiTraining: true,
  });
});

// Allow AI training on public pages
fastify.get('/', async (request, reply) => {
  return reply.view('layout', {
    title: 'Home',
    body: '<h1>Hello</h1>',
    blockAiTraining: false,
  });
});

6. X-Robots-Tag — onSend hook

Set X-Robots-Tag on the way out using the onSend hook — it fires after the handler has run and the response is ready to be serialized. Wrap in fp() for global application.

// In plugins/block-ai-bots.ts — add alongside the onRequest hook
fastify.addHook('onSend', async (request, reply) => {
  // Don't set X-Robots-Tag on robots.txt itself
  if (request.url !== '/robots.txt') {
    reply.header('X-Robots-Tag', 'noai, noimageai');
  }
});

onSend fires after serialization — the response body is finalized. Setting headers here is safe and applies to all content types (HTML, JSON, binary).

7. TypeScript + ESM setup

Fastify ships first-class TypeScript types. Use ESM ("type": "module" in package.json) for the cleanest imports.

package.json

{
  "type": "module",
  "scripts": {
    "dev": "tsx watch server.ts",
    "build": "tsc",
    "start": "node dist/server.js"
  },
  "dependencies": {
    "fastify": "^5",
    "@fastify/static": "^8",
    "@fastify/view": "^10",
    "fastify-plugin": "^5"
  },
  "devDependencies": {
    "@types/node": "^22",
    "typescript": "^5",
    "tsx": "^4"
  }
}

tsconfig.json

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "outDir": "dist",
    "strict": true,
    "esModuleInterop": true
  }
}

8. nginx — reverse proxy hard block

nginx in front of Fastify blocks AI bots before any request reaches Node.js. This is the most efficient approach — bots get a 403 at the network edge with zero Node.js overhead.

map $http_user_agent $block_ai_bot {
    default              0;
    "~*GPTBot"           1;
    "~*ChatGPT-User"     1;
    "~*OAI-SearchBot"    1;
    "~*ClaudeBot"        1;
    "~*Claude-Web"       1;
    "~*anthropic-ai"     1;
    "~*Google-Extended"  1;
    "~*Bytespider"       1;
    "~*CCBot"            1;
    "~*PerplexityBot"    1;
    "~*Applebot-Extended" 1;
}

server {
    listen 80;
    server_name yourdomain.com;

    # Serve robots.txt directly — exempt from bot blocking
    location = /robots.txt {
        alias /var/www/public/robots.txt;
        add_header Content-Type "text/plain";
    }

    location / {
        if ($block_ai_bot) {
            return 403;
        }
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # Fastify needs the original protocol for redirect logic
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

9. Docker deployment

FROM node:22-alpine AS base
WORKDIR /app

# Install dependencies
COPY package.json package-lock.json ./
RUN npm ci --omit=dev

# Build TypeScript
FROM base AS builder
RUN npm ci
COPY . .
RUN npm run build

# Production image
FROM node:22-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production

COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY public ./public

EXPOSE 3000
CMD ["node", "dist/server.js"]

Copy public/ to the runner stage if using @fastify/static for file-based robots.txt. If you embed robots.txt as a string constant in the route handler, skip the COPY.

Fastify vs Express — key differences

FAQ