Skip to content
React · RedwoodJS · GraphQL · Full-Stack

How to Block AI Bots on RedwoodJS

RedwoodJS is a full-stack React framework built for startups and product teams — React frontend, GraphQL API, Prisma ORM, and first-class support for auth and testing. Unlike Next.js, Redwood has a strict web/api split: the web side is a React SPA (or RSC) built as static files, while the api side runs GraphQL services on Fastify. This split shapes how AI bot protection works — web-side blocking uses CDN headers and edge middleware, api-side blocking uses Fastify hooks or serverless function logic. This guide covers all four layers across both sides of a Redwood application.

8 min readUpdated April 2026RedwoodJS v8+
Redwood's web/api split: A Redwood project has two sides: web/ (React SPA — deployed to CDN or SSR) and api/ (GraphQL + Fastify — deployed to serverless functions or a Node server). Sections 1–4 cover the web side. Section 5 covers the api side.

1. robots.txt

Static files in web/public/ are copied to the build output directory by Redwood's build process and served at the root URL by your deployment platform. Place robots.txt here for zero-config static serving.

Static robots.txt

Create web/public/robots.txt:

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Diffbot
Disallow: /

# Allow standard search crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

Redwood copies web/public/ to the output at build time. When deployed to Netlify or Vercel, the file is available at yoursite.com/robots.txt without any routing configuration.

Dynamic robots.txt via Redwood Function

For environment-aware robots.txt, create a Redwood serverless function at api/src/functions/robots.ts:

// api/src/functions/robots.ts
import type { APIGatewayEvent, Context } from 'aws-lambda'

const AI_BOTS_DISALLOW = `User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /
`

export const handler = async (_event: APIGatewayEvent, _context: Context) => {
  const isProduction = process.env.NODE_ENV === 'production'

  return {
    statusCode: 200,
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
    body: isProduction
      ? AI_BOTS_DISALLOW
      : 'User-agent: *
Disallow: /
',
  }
}
Function URL path: Redwood functions are served at /.redwood/functions/robots by default (or /api/robots depending on your proxy config). This is not the same as /robots.txt. For crawlers to read robots.txt at the correct path, use the static file approach or configure your Netlify/Vercel redirects to map /robots.txt to the function URL.

Redirect robots.txt to function (Netlify)

Add to web/public/_redirects:

/robots.txt /.redwood/functions/robots 200

2. noai meta via Metadata component

RedwoodJS provides the Metadata component (from @redwoodjs/web) for managing document head tags. Add it to your layout for site-wide coverage.

In a layout

Redwood layouts live in web/src/layouts/. Add the Metadata component to your main layout:

// web/src/layouts/MainLayout/MainLayout.tsx
import { Metadata } from '@redwoodjs/web'

type MainLayoutProps = {
  children?: React.ReactNode
}

const MainLayout = ({ children }: MainLayoutProps) => {
  return (
    <>
      <Metadata
        title="My Site"
        description="Site description"
        robots="noai, noimageai"
      />
      <header>{/* nav */}</header>
      <main>{children}</main>
      <footer>{/* footer */}</footer>
    </>
  )
}

export default MainLayout

The Metadata component renders a <meta name="robots" content="noai, noimageai" /> tag in the document <head>. It uses Redwood's head management system internally.

Per-page override

In individual page components, render Metadata to override the layout default:

// web/src/pages/BlogPostPage/BlogPostPage.tsx
import { Metadata } from '@redwoodjs/web'

const BlogPostPage = ({ id }: { id: number }) => {
  return (
    <>
      {/* Allow AI indexing for public blog posts */}
      <Metadata
        title="Blog Post Title"
        robots="index, follow"
      />
      <article>{/* post content */}</article>
    </>
  )
}

export default BlogPostPage
Head merging: Redwood's head management merges tags from layouts and pages. When the same meta name appears in both a layout and a page, the page-level value takes precedence — the last-rendered tag wins. Verify the output with browser DevTools to confirm the correct value is applied.

MetaTags component (alternative)

Older Redwood projects may use MetaTags instead of Metadata. The pattern is the same:

import { MetaTags } from '@redwoodjs/web'

// In layout or page:
<MetaTags
  title="My Site"
  description="Description"
  robots="noai, noimageai"
/>

3. X-Robots-Tag via platform headers

Since Redwood's web side is typically static files on a CDN, X-Robots-Tag headers are set at the hosting platform level — not in application code.

Netlify — _headers file

Create web/public/_headers. Netlify reads this file and applies headers to matching paths:

/*
  X-Robots-Tag: noai, noimageai
  X-Content-Type-Options: nosniff

Redwood copies web/public/_headers to the build output automatically. After deploying, every response from your Netlify site includes the header — HTML, JS bundles, and API function responses.

Netlify — netlify.toml

Add to netlify.toml at the project root:

[build]
  command = "yarn rw deploy netlify"
  publish = "web/dist"
  functions = "api/dist/functions"

[[headers]]
  for = "/*"
  [headers.values]
    X-Robots-Tag = "noai, noimageai"

Vercel — vercel.json

Add to vercel.json at the project root:

{
  "headers": [
    {
      "source": "/(.*)",
      "headers": [
        {
          "key": "X-Robots-Tag",
          "value": "noai, noimageai"
        }
      ]
    }
  ]
}

Cloudflare Pages — _headers

Same as Netlify — create web/public/_headers with the same format. Cloudflare Pages natively reads _headers files from the publish directory.

4. Hard 403 via edge middleware

For the web side, hard 403 blocking requires an edge function or platform middleware — the React SPA itself can't intercept crawler requests.

Netlify Edge Function

Create netlify/edge-functions/bot-block.ts:

// netlify/edge-functions/bot-block.ts
import type { Context } from 'netlify:edge'

const AI_BOT_PATTERNS = [
  'GPTBot', 'ClaudeBot', 'Claude-Web', 'anthropic-ai',
  'CCBot', 'Google-Extended', 'PerplexityBot', 'Applebot-Extended',
  'Amazonbot', 'meta-externalagent', 'Bytespider', 'Diffbot',
]

const EXEMPT_PATHS = ['/robots.txt', '/sitemap.xml', '/favicon.ico']

export default async function botBlock(
  request: Request,
  context: Context
) {
  const url = new URL(request.url)

  if (EXEMPT_PATHS.some((p) => url.pathname === p)) {
    return context.next()
  }

  const ua = request.headers.get('user-agent') ?? ''
  const isBot = AI_BOT_PATTERNS.some((p) =>
    ua.toLowerCase().includes(p.toLowerCase())
  )

  if (isBot) {
    return new Response('Forbidden', { status: 403 })
  }

  const response = await context.next()
  response.headers.set('X-Robots-Tag', 'noai, noimageai')
  return response
}

export const config = { path: '/*' }

Register in netlify.toml:

[[edge_functions]]
  path = "/*"
  function = "bot-block"

Vercel middleware

Create middleware.ts at the project root (not inside web/ or api/):

// middleware.ts (project root)
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'

const AI_BOT_PATTERNS = [
  'GPTBot', 'ClaudeBot', 'Claude-Web', 'anthropic-ai',
  'CCBot', 'Google-Extended', 'PerplexityBot', 'Applebot-Extended',
  'Amazonbot', 'meta-externalagent', 'Bytespider', 'Diffbot',
]

export function middleware(request: NextRequest) {
  const { pathname } = request.nextUrl
  const exempt = ['/robots.txt', '/sitemap.xml', '/favicon.ico']

  if (exempt.includes(pathname)) {
    return NextResponse.next()
  }

  const ua = request.headers.get('user-agent') ?? ''
  const isBot = AI_BOT_PATTERNS.some((p) =>
    ua.toLowerCase().includes(p.toLowerCase())
  )

  if (isBot) {
    return new NextResponse('Forbidden', { status: 403 })
  }

  const response = NextResponse.next()
  response.headers.set('X-Robots-Tag', 'noai, noimageai')
  return response
}

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
}

5. API side — Fastify hooks and functions

The api/ side runs on Fastify (for self-hosted Redwood) or as serverless functions (for Netlify/Vercel). Protect it separately from the web side.

Fastify preHandler hook (self-hosted)

Edit api/server.config.ts to add a Fastify hook that runs before every request:

// api/server.config.ts
import type { FastifyInstance } from 'fastify'

const AI_BOT_PATTERNS = [
  'GPTBot', 'ClaudeBot', 'Claude-Web', 'anthropic-ai',
  'CCBot', 'Google-Extended', 'PerplexityBot', 'Applebot-Extended',
  'Amazonbot', 'meta-externalagent', 'Bytespider', 'Diffbot',
]

function isAIBot(ua: string): boolean {
  const lower = ua.toLowerCase()
  return AI_BOT_PATTERNS.some((p) => lower.includes(p.toLowerCase()))
}

export const config = {
  fastifyConfig: async (fastify: FastifyInstance) => {
    fastify.addHook('preHandler', async (request, reply) => {
      const ua = request.headers['user-agent'] ?? ''

      if (isAIBot(ua)) {
        fastify.log.warn({ ua, path: request.url }, 'AI bot blocked')
        return reply.code(403).send('Forbidden')
      }

      reply.header('X-Robots-Tag', 'noai, noimageai')
    })
  },
}

Serverless function check (Netlify/Vercel)

Add a UA check at the top of critical serverless functions in api/src/functions/:

// api/src/functions/graphql.ts (or any function)
import type { APIGatewayEvent } from 'aws-lambda'

const AI_BOT_PATTERNS = [
  'GPTBot', 'ClaudeBot', 'CCBot', 'Google-Extended',
  'PerplexityBot', 'Bytespider', 'Diffbot',
]

function isAIBot(ua: string): boolean {
  const lower = ua.toLowerCase()
  return AI_BOT_PATTERNS.some((p) => lower.includes(p.toLowerCase()))
}

export const handler = async (event: APIGatewayEvent) => {
  const ua = event.headers['user-agent'] ?? ''

  if (isAIBot(ua)) {
    return { statusCode: 403, body: 'Forbidden' }
  }

  // ... rest of function handler
}

Redwood middleware (api side)

Redwood v8+ supports an api/src/middleware/ directory for middleware that runs before all functions. Create api/src/middleware/botBlock.ts:

// api/src/middleware/botBlock.ts
import type { MiddlewareRequest, MiddlewareResponse } from '@redwoodjs/vite/middleware'

const AI_BOT_PATTERNS = [
  'GPTBot', 'ClaudeBot', 'Claude-Web', 'anthropic-ai',
  'CCBot', 'Google-Extended', 'PerplexityBot', 'Applebot-Extended',
  'Amazonbot', 'meta-externalagent', 'Bytespider', 'Diffbot',
]

export const botBlockMiddleware = (
  req: MiddlewareRequest,
  res: MiddlewareResponse,
  next: () => void
) => {
  const ua = req.headers.get('user-agent') ?? ''
  const isBot = AI_BOT_PATTERNS.some((p) =>
    ua.toLowerCase().includes(p.toLowerCase())
  )

  if (isBot) {
    res.status = 403
    res.body = 'Forbidden'
    return
  }

  next()
}

6. Deployment comparison

Redwood is designed for serverless deployment. The web side is static; the api side runs as functions. AI bot protection requirements differ by layer and platform.

Platformrobots.txtMeta tagsX-Robots-TagHard 403
Netlify✓✓✓ (_headers)✓ (Edge Function)
Vercel✓✓✓ (vercel.json)✓ (middleware.ts)
Cloudflare Pages✓✓✓ (_headers)✓ (Pages Function)
Render✓✓✓ (custom headers)✓ (Fastify hook)
Self-hosted (Fastify)✓✓✓ (Fastify hook)✓ (Fastify hook)

Recommended setup (Netlify)

The most common Redwood deployment is Netlify. Combine these layers for comprehensive protection:

# Layers for Netlify Redwood deployment
1. web/public/robots.txt          — Disallow rules (CDN-served, zero latency)
2. netlify/edge-functions/        — Hard 403 + X-Robots-Tag on all requests
3. web/src/layouts/MainLayout.tsx — noai meta via <Metadata robots="noai, noimageai" />
4. api/src/functions/graphql.ts   — UA check before GraphQL processing

FAQ

Where do I put robots.txt in a Redwood project?

Place robots.txt in web/public/. Redwood's build process copies everything in web/public/ to the build output directory. When deployed to Netlify or Vercel, it's served at yoursite.com/robots.txt with no configuration needed. For dynamic content, create a function in api/src/functions/robots.ts and add a redirect in web/public/_redirects: /robots.txt /.redwood/functions/robots 200.

How do I add noai meta tags in RedwoodJS?

Use the Metadata component from @redwoodjs/web in your layout: <Metadata robots="noai, noimageai" />. This injects a robots meta tag into the document head for every page using the layout. For per-page overrides, render Metadata in the individual page component with a different value — the page-level tag takes precedence.

How do I set X-Robots-Tag headers in RedwoodJS on Netlify?

Create web/public/_headers with:

/*
  X-Robots-Tag: noai, noimageai

Netlify reads this file and applies the header to all matching responses. Redwood copies it to the build output automatically — no manual upload needed.

How is RedwoodJS different from Next.js for AI bot blocking?

Redwood has a strict web/api split — the web side is static files on a CDN, the api side is GraphQL on Fastify or serverless functions. Bot blocking on the web side requires platform features (edge functions, _headers files). Next.js has a unified runtime where middleware runs for all requests in one place. The Redwood approach requires configuring protection at multiple layers — web platform + api server — rather than a single middleware file.

Can I block AI bots in a RedwoodJS serverless function?

Yes. Check the user-agent header at the top of your function handler and return a 403 before processing: if (isAIBot(event.headers['user-agent'] ?? '')) return { statusCode: 403, body: "Forbidden" }. For broader coverage, use Redwood's middleware in api/src/middleware/ (v8+) to protect all functions from one place.

How do I block AI bots in self-hosted RedwoodJS using Fastify?

Add a preHandler hook in api/server.config.ts: fastify.addHook("preHandler", async (request, reply) => { ... }). Check request.headers['user-agent'] and call reply.code(403).send("Forbidden") for matched bots. The preHandler hook runs before every route handler on the API side.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.