How to Block AI Bots in Nim Jester

1. Bot pattern module

A separate ai_bots.nim module keeps patterns reusable across routes and tests. toLowerAscii() from strutils normalises the header; contains() does plain-text substring matching — no regex overhead needed.

# src/ai_bots.nim
import strutils

const AiBotPatterns* = [
  "gptbot",
  "chatgpt-user",
  "claudebot",
  "anthropic-ai",
  "ccbot",
  "google-extended",
  "cohere-ai",
  "meta-externalagent",
  "bytespider",
  "omgili",
  "diffbot",
  "imagesiftbot",
  "magpie-crawler",
  "amazonbot",
  "dataprovider",
  "netcraft",
]

proc isAiBot*(userAgent: string): bool =
  let ua = userAgent.toLowerAscii()
  for pattern in AiBotPatterns:
    if ua.contains(pattern):
      return true
  false

2. Router with before:/after: hooks

The before: block runs before route matching. halt() sends a response immediately and skips everything else — routes, after:, and further middleware. The after: block adds X-Robots-Tag to all responses that reach a route handler.

# src/server.nim
import jester, strutils
import ai_bots

router myRouter:
  # Runs before every route — check for AI bots first
  before:
    if request.path != "/robots.txt":
      let ua = request.headers.getOrDefault("User-Agent", "")
      if isAiBot(ua):
        # halt() skips routes AND after: — include X-Robots-Tag here
        halt(
          Http403,
          @[("X-Robots-Tag", "noai, noimageai"),
            ("Content-Type", "text/plain")],
          "Forbidden"
        )

  # Runs after every matched route — add X-Robots-Tag to passing responses
  after:
    response.headers["X-Robots-Tag"] = "noai, noimageai"

  get "/":
    resp "<h1>Hello</h1>"

  get "/health":
    resp Http200, "OK"

  # Explicit route: robots.txt bypasses before: check above,
  # but staticDir also serves it automatically from public/
  get "/robots.txt":
    resp Http200, readFile("public/robots.txt")

let settings = newSettings(
  port        = 8080.Port,
  staticDir   = getCurrentDir() / "public",  # serves public/robots.txt before routes
  reusePort   = true,
)

runForever(myRouter, settings)

3. Nimble dependency and compilation

# myapp.nimble
requires "nim    >= 2.0.0"
requires "jester >= 0.5.0"

# Compile:
#   nimble build -d:release
#
# Or run directly:
#   nim c -d:ssl -d:release -r src/server.nim

4. public/robots.txt

Setting staticDir in newSettings() causes Jester to serve files from public/ before route matching. public/robots.txt is always reachable regardless of User-Agent. The explicit path check in before: is a belt-and-suspenders guard for the route handler variant.

# public/robots.txt
User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

5. Async routes

Jester runs on asyncdispatch by default. The before: hook and halt() are async-safe — no changes needed when route handlers use await.

# Async variant using asyncjester / asynchttpserver style
import asyncdispatch, jester, strutils
import ai_bots

# Jester natively uses asyncdispatch — the before: block is async-safe.
# No changes needed: halt() works identically in async context.

router asyncRouter:
  before:
    if request.path != "/robots.txt":
      let ua = request.headers.getOrDefault("User-Agent", "")
      if isAiBot(ua):
        halt(Http403,
             @[("X-Robots-Tag", "noai, noimageai")],
             "Forbidden")

  after:
    response.headers["X-Robots-Tag"] = "noai, noimageai"

  get "/api/data":
    let data = await fetchSomeData()  # await works inside route blocks
    resp $(%* {"data": data})

runForever(asyncRouter)

Key points

• halt() skips after: — include X-Robots-Tag in the halt() call itself for blocked responses. The after: block only runs for requests that reach a route handler.
• Header access: request.headers.getOrDefault("User-Agent", "") — Jester uses Nim's stdlib HttpHeaders type. Header names are case-insensitive.
• Case normalisation: toLowerAscii() from strutils — prefer this over toLower() for ASCII-only input (faster, no Unicode overhead).
• halt() signature: halt(code, headers, body) where headers is seq[tuple[key, val: string]]. Use @[("Key", "Val")] literal syntax.
• staticDir: Files in public/ are served before before: runs — use this for robots.txt, favicons, and other assets that should always be public.
• Compilation flags: Always use -d:release for production — enables optimisations and removes bounds-checking overhead.

Framework comparison — compiled/systems languages

Nim Jester's before:/after: pair is conceptually closest to Crystal Kemal's before_all/after_all. Both are compiled languages with async runtimes and macro-based DSLs. The key Jester-specific detail is that halt() completely bypasses after:, unlike Kemal where halt() skips after_all but not filter inheritance.

Dependencies

Only jester and Nim's stdlib strutils. No regex library needed — plain contains() substring matching is sufficient and eliminates a dependency. Install via Nimble:

nimble install jester