Skip to content
Node.js · AdonisJS v6 · TypeScript MVC

How to Block AI Bots on AdonisJS

AdonisJS is a batteries-included Node.js MVC framework — TypeScript-first, with its own ORM (Lucid), Edge templating engine, IoC container, and structured middleware system. It's closer to Laravel than Express: opinionated, convention-driven, and fast to build with. This guide covers all four layers of AI bot protection on AdonisJS v6: robots.txt via the public/ directory, noai meta tags in Edge layouts, X-Robots-Tag via global middleware, and hard 403 blocking — with both global and named middleware patterns.

8 min readUpdated April 2026AdonisJS v6

1. robots.txt

AdonisJS automatically serves all files in the public/ directory as static assets. Place your robots.txt there — it's available at yoursite.com/robots.txt with no configuration.

Static robots.txt

Create public/robots.txt in your project root:

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Diffbot
Disallow: /

# Allow standard search crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

AdonisJS's static file server reads from public/ by default (configured in config/static.ts). No extra packages or routes needed.

Dynamic robots.txt via route

For environment-aware content — blocking all crawlers on staging — add a route to start/routes.ts:

// start/routes.ts
import router from '@adonisjs/core/services/router'
import env from '#start/env'

router.get('/robots.txt', async ({ response }) => {
  response.type('text/plain')

  if (env.get('NODE_ENV') !== 'production') {
    return 'User-agent: *
Disallow: /
'
  }

  return `User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /
`
})
Route vs static file priority: AdonisJS routes take priority over static files when there is a path conflict. If you define a /robots.txt route and also have public/robots.txt, the route wins. Remove one or the other to avoid ambiguity.

2. noai meta tags in Edge templates

AdonisJS uses the Edge templating engine — a fast, TypeScript-aware template language with {{ }} for interpolation and @if, @each, @component directives. Add the noai meta tag to your base layout so every page is protected.

Base layout

The standard AdonisJS project has a layout at resources/views/layouts/main.edge:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">

  {{-- AI bot protection — applies to every page --}}
  <meta name="robots" content="{{ robots ?? 'noai, noimageai' }}">

  <title>{{ title ?? 'My App' }}</title>

  @vite(['resources/css/app.css', 'resources/js/app.js'])
</head>
<body>
  @!section('body')
</body>
</html>

The {{ robots ?? 'noai, noimageai' }} expression uses Edge's nullish coalescing: if the controller passes a robots value, use it; otherwise default to noai, noimageai.

Page template inheriting the layout

{{-- resources/views/pages/home.edge --}}
@layout('layouts/main')

@section('body')
  <h1>Welcome</h1>
  <p>Your homepage content here.</p>
@end

Controller — passing robots value

To allow a specific page to opt in to AI indexing, pass robots from the controller:

// app/controllers/home_controller.ts
import type { HttpContext } from '@adonisjs/core/http'

export default class HomeController {
  async index({ view }: HttpContext) {
    // Default — noai applied by layout fallback
    return view.render('pages/home', { title: 'Home' })
  }

  async blogPost({ view, params }: HttpContext) {
    const post = await Post.findOrFail(params.id)

    // Explicitly allow AI indexing for public blog posts
    return view.render('pages/blog/show', {
      post,
      robots: 'index, follow',
    })
  }
}
Edge vs other engines: Edge uses {{ }} (double braces) for escaped output and {{{ }}} (triple braces) for raw HTML. The ?? operator is standard JavaScript nullish coalescing — Edge evaluates template expressions as JavaScript, so all JS operators work.

3. X-Robots-Tag via global middleware

HTTP response headers are more reliable than meta tags for crawlers that parse headers without rendering HTML. In AdonisJS, create a middleware class and register it globally in start/kernel.ts.

Create the middleware

Generate the middleware with the AdonisJS CLI:

node ace make:middleware AiBotHeaders

This creates app/middleware/ai_bot_headers_middleware.ts. Edit it:

// app/middleware/ai_bot_headers_middleware.ts
import type { HttpContext } from '@adonisjs/core/http'
import type { NextFn } from '@adonisjs/core/types/http'

export default class AiBotHeadersMiddleware {
  async handle(ctx: HttpContext, next: NextFn) {
    await next()

    // Add X-Robots-Tag to HTML responses only
    const contentType = ctx.response.getHeader('Content-Type') as string | undefined
    if (!contentType || contentType.includes('text/html')) {
      ctx.response.header('X-Robots-Tag', 'noai, noimageai')
    }
  }
}

Register as global middleware

Add it to the server middleware array in start/kernel.ts:

// start/kernel.ts
import router from '@adonisjs/core/services/router'
import server from '@adonisjs/core/services/server'

server.use([
  () => import('#middleware/container_bindings_middleware'),
  () => import('@adonisjs/static/static_middleware'),
  () => import('#middleware/ai_bot_headers_middleware'),  // Add here
])

router.use([
  () => import('@adonisjs/core/bodyparser_middleware'),
  () => import('@adonisjs/session/session_middleware'),
  () => import('@adonisjs/shield/shield_middleware'),
  () => import('@adonisjs/auth/initialize_auth_middleware'),
])
server.use vs router.use: Middleware in server.use runs on every HTTP request — including static file requests, before routing. Middleware in router.use runs only after the router matches a route. Place bot-blocking middleware in server.use for the earliest possible interception.

4. Hard 403 via middleware

To reject AI crawlers before any application code runs, create a blocking middleware that checks the User-Agent and aborts the request.

node ace make:middleware AiBotBlock

Edit app/middleware/ai_bot_block_middleware.ts:

// app/middleware/ai_bot_block_middleware.ts
import type { HttpContext } from '@adonisjs/core/http'
import type { NextFn } from '@adonisjs/core/types/http'

const AI_BOT_PATTERNS = [
  'GPTBot',
  'ClaudeBot',
  'Claude-Web',
  'anthropic-ai',
  'CCBot',
  'Google-Extended',
  'PerplexityBot',
  'Applebot-Extended',
  'Amazonbot',
  'meta-externalagent',
  'Bytespider',
  'Diffbot',
  'YouBot',
  'cohere-ai',
]

// Paths exempt from blocking (crawlers need these)
const EXEMPT_PATHS = ['/robots.txt', '/sitemap.xml', '/favicon.ico']

function isAiBot(userAgent: string): boolean {
  const ua = userAgent.toLowerCase()
  return AI_BOT_PATTERNS.some((pattern) => ua.includes(pattern.toLowerCase()))
}

export default class AiBotBlockMiddleware {
  async handle(ctx: HttpContext, next: NextFn) {
    const { request, response, logger } = ctx
    const path = request.url()

    // Skip blocking for essential crawler files
    if (EXEMPT_PATHS.some((p) => path.startsWith(p))) {
      return next()
    }

    const ua = request.header('user-agent') ?? ''

    if (isAiBot(ua)) {
      logger.warn({ ua, path }, 'AI bot blocked')
      return response.status(403).send('Forbidden')
    }

    await next()

    // Add X-Robots-Tag on legitimate requests
    ctx.response.header('X-Robots-Tag', 'noai, noimageai')
  }
}

Register in start/kernel.ts (replaces the separate headers middleware):

server.use([
  () => import('#middleware/container_bindings_middleware'),
  () => import('@adonisjs/static/static_middleware'),
  () => import('#middleware/ai_bot_block_middleware'),  // Combined block + headers
])
Middleware order: Place the bot-blocking middleware after static_middleware so static files (robots.txt, favicon, CSS, JS) are served without going through the bot check. This ensures crawlers can always read robots.txt even when all other paths are blocked.

Logging with AdonisJS logger

AdonisJS uses Pino under the hood. The logger.warn() call above produces structured JSON logs compatible with most log aggregation platforms (Datadog, Logtail, CloudWatch). In development, logs are pretty-printed to the terminal.

5. Named middleware (per-route)

If you want bot protection on specific routes only — for example, protecting your API endpoints but not your public marketing pages — use AdonisJS named middleware.

Register as named middleware

In start/kernel.ts, add to the router.named object:

// start/kernel.ts
export const middleware = router.named({
  auth: () => import('#middleware/auth_middleware'),
  guest: () => import('#middleware/guest_middleware'),
  botBlock: () => import('#middleware/ai_bot_block_middleware'),  // Add this
})

Apply to routes

// start/routes.ts
import router from '@adonisjs/core/services/router'
import { middleware } from '#start/kernel'

// Public routes — no bot blocking
router.get('/', [HomeController, 'index'])

// Protected routes — block AI bots
router.group(() => {
  router.get('/api/content', [ContentController, 'index'])
  router.get('/api/articles/:id', [ArticleController, 'show'])
}).use(middleware.botBlock())

// Or apply to a single route
router.get('/premium-content', [PremiumController, 'index'])
  .use(middleware.botBlock())
Named vs global: For most sites, applying bot blocking globally (via server.use) is simpler and more comprehensive. Use named middleware when you need fine-grained control — for example, blocking AI bots from API endpoints that return structured content, while allowing them to crawl your public pages for legitimate indexing.

6. Deployment

AdonisJS is a Node.js application server — it needs a runtime environment. All four protection layers work on any Node.js-capable platform.

Production build

# Build TypeScript to JavaScript
node ace build

# The output is in build/ — run it with Node.js
node build/bin/server.js

Docker

# Dockerfile
FROM node:22-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN node ace build

FROM node:22-alpine AS production
WORKDIR /app
COPY --from=build /app/build ./
COPY --from=build /app/package*.json ./
RUN npm ci --omit=dev
EXPOSE 3333
CMD ["node", "bin/server.js"]

Fly.io

# fly.toml
app = 'my-adonis-app'
primary_region = 'ams'

[build]
  dockerfile = 'Dockerfile'

[env]
  NODE_ENV = 'production'
  PORT = '3333'
  HOST = '0.0.0.0'

[[services]]
  internal_port = 3333
  protocol = 'tcp'

  [[services.ports]]
    handlers = ['http']
    port = 80

  [[services.ports]]
    handlers = ['tls', 'http']
    port = 443

Railway

Railway auto-detects Node.js projects. Set these environment variables in the Railway dashboard:

NODE_ENV=production
PORT=3333
HOST=0.0.0.0
APP_KEY=<generate with: node ace generate:key>

Railway runs npm start by default — update your package.json:

{
  "scripts": {
    "start": "node build/bin/server.js",
    "build": "node ace build"
  }
}
Platformrobots.txtMeta tagsX-Robots-TagHard 403
Docker / self-hosted✓✓✓✓
Fly.io✓✓✓✓
Railway✓✓✓✓
Heroku✓✓✓✓
Render✓✓✓✓

Because AdonisJS is a runtime server (not a static site generator), all four protection layers work on every deployment target. No platform-specific configuration is required — the middleware handles everything.

FAQ

How do I serve robots.txt in AdonisJS?

Place robots.txt in the public/ directory at the root of your project. AdonisJS automatically serves all files in public/ as static assets via the @adonisjs/static middleware — no configuration needed. For dynamic content (environment-specific rules), define a route in start/routes.ts: router.get("/robots.txt", async ({ response }) => { response.type("text/plain"); return "..."; }. Routes take priority over static files.

How does AdonisJS middleware work for bot blocking?

AdonisJS v6 uses a structured middleware system in start/kernel.ts. Global middleware (in server.use) runs on every request. Named middleware (in router.named) is applied per-route with .use(middleware.botBlock()). Middleware classes live in app/middleware/ and export a handle(ctx, next) method. Call await next() to continue, or return a response early to block.

How do I add noai meta tags in AdonisJS Edge templates?

In your base layout (resources/views/layouts/main.edge), add <meta name="robots" content="{{ robots ?? 'noai, noimageai' }}"> inside <head>. The ?? operator uses noai, noimageai when the controller doesn't pass a robots value. Pass it explicitly when a page should allow indexing: return view.render("pages/home", { robots: "index, follow" }).

What is the difference between global and named middleware in AdonisJS?

Global middleware (server.use in start/kernel.ts) runs on every HTTP request — ideal for site-wide headers and broad bot blocking. Named middleware is registered in the named object and applied selectively with .use(middleware.name()) on specific routes or route groups. Use global middleware for comprehensive protection; named middleware for routes where you need different behavior.

How is AdonisJS different from Express for bot blocking?

Express registers middleware ad-hoc with app.use() — no formal structure. AdonisJS has a typed, declarative middleware system in start/kernel.ts with a clear separation between server-level and router-level middleware. AdonisJS middleware classes receive the full HttpContext (ctx.request, ctx.response, ctx.logger) — no need to pass req, res, and next separately. Static file serving from public/ is built-in via @adonisjs/static, and Edge templates use {{ }} with JavaScript expressions instead of Handlebars or EJS.

Will blocking AI bots affect Googlebot crawling?

No. The AI bot patterns in this guide (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, etc.) are separate user agents from Googlebot and Bingbot. Standard search engine crawlers are not affected by the User-Agent checks above. Always include explicit Allow rules for Googlebot and Bingbot in your robots.txt to confirm your intent.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.