How to Block AI Bots in Ruby Grape

1. Bot detection module

A plain Ruby module with no gem dependencies. Uses String#include? for literal substring matching — no regex overhead. Lowercase once with downcase before iterating.

# lib/ai_bot_detector.rb
# Shared bot detection — no dependencies, no gems required.
module AiBotDetector
  # All lowercase — matched against ua.downcase
  PATTERNS = %w[
    gptbot
    chatgpt-user
    claudebot
    anthropic-ai
    ccbot
    google-extended
    cohere-ai
    meta-externalagent
    bytespider
    omgili
    diffbot
    imagesiftbot
    magpie-crawler
    amazonbot
    dataprovider
    netcraft
  ].freeze

  def self.ai_bot?(ua)
    return false if ua.nil? || ua.empty?
    lower = ua.downcase
    # String#include? — literal substring, no regex engine
    PATTERNS.any? { |pattern| lower.include?(pattern) }
  end
end

2. Global before block — error!() with headers

Add a before block at the top of the API class to intercept every request. error!(body, status, headers) accepts an optional third argument — a hash of response headers. Pass X-Robots-Tag there to set it on the blocked response without a separate header() call. For passing requests, header 'X-Robots-Tag', '...' sets the header on the normal response.

# api/my_api.rb — Grape API class
require 'grape'
require_relative '../lib/ai_bot_detector'

class MyApi < Grape::API
  format :json

  # ── Global before block — runs before every endpoint in this API ─────────────
  before do
    # Path guard: let robots.txt through.
    # In most deployments, robots.txt is served by Nginx before reaching
    # this Rack application — this guard handles the edge case where it
    # reaches Grape (e.g., when running standalone without a reverse proxy).
    pass if request.path == '/robots.txt'

    ua = headers['User-Agent'] || ''

    if AiBotDetector.ai_bot?(ua)
      # error!() raises Grape::Exceptions::Base — caught by Grape, rendered
      # as the response. Stops all further before blocks and the route handler.
      # First argument: response body (string or hash for JSON).
      # Second argument: HTTP status code.
      error!('Forbidden', 403, { 'X-Robots-Tag' => 'noai, noimageai' })
    end

    # Pass-through: set X-Robots-Tag on all non-blocked responses.
    # header() sets a response header — distinct from headers[] which reads requests.
    header 'X-Robots-Tag', 'noai, noimageai'
  end

  # ── Routes ───────────────────────────────────────────────────────────────────
  get '/' do
    { message: 'Hello' }
  end

  namespace :api do
    get '/data' do
      { data: 'value' }
    end

    get '/status' do
      { status: 'ok' }
    end
  end
end

3. config.ru — standalone Rack

Run Grape as a standalone Rack application with rackup config.ru or any Rack-compatible server (Puma, Falcon, Unicorn). The Grape API class is itself a valid Rack application — no wrapper needed.

# config.ru — standalone Rack/Grape application
require_relative 'api/my_api'

# Optional: add Rack middleware BEFORE Grape (more efficient for blocking)
# use AiBotRackMiddleware   # see Rack middleware section below

run MyApi

4. Namespace scoping — protect specific endpoints only

Place the before block inside a namespace block to scope it to specific routes. Endpoints outside the namespace (health check, robots.txt) bypass the filter. This is cleaner than path guards inside a global before block when you have multiple unprotected endpoints.

# Namespace-scoped before block — protect specific endpoints only
class MyApi < Grape::API
  format :json

  # Public endpoints — no bot filter
  get '/health' do
    { status: 'ok' }
  end

  get '/robots.txt' do
    content_type 'text/plain'
    <<~ROBOTS
      User-agent: *
      Allow: /
      User-agent: GPTBot
      Disallow: /
    ROBOTS
  end

  # Protected namespace — bot filter applies only here
  namespace :api do
    before do
      ua = headers['User-Agent'] || ''
      error!('Forbidden', 403, { 'X-Robots-Tag' => 'noai, noimageai' }) if AiBotDetector.ai_bot?(ua)
      header 'X-Robots-Tag', 'noai, noimageai'
    end

    get '/data' do
      { data: 'value' }
    end

    get '/users' do
      { users: [] }
    end
  end
end

5. Base API inheritance — shared filter across multiple API classes

Define the before block in a BaseApi < Grape::API class and inherit from it. All sub-APIs automatically include the filter. Mount them all in a root API. This is the recommended pattern for large Grape applications with multiple resource APIs.

# Shared base API class — bot filter inherited by all sub-APIs
# Use this pattern when you have multiple Grape API classes.

class BaseApi < Grape::API
  before do
    ua = headers['User-Agent'] || ''
    error!('Forbidden', 403, { 'X-Robots-Tag' => 'noai, noimageai' }) if AiBotDetector.ai_bot?(ua)
    header 'X-Robots-Tag', 'noai, noimageai'
  end
end

class UsersApi < BaseApi
  get '/users' do
    { users: [] }
  end
end

class ProductsApi < BaseApi
  get '/products' do
    { products: [] }
  end
end

# Mount both in a root API:
class RootApi < Grape::API
  mount UsersApi
  mount ProductsApi
end

6. Rack middleware variant — block before Grape runs

For maximum efficiency, block at the Rack layer before Grape processes the request. In Rack middleware, the header key is the CGI-style HTTP_USER_AGENT from the env hash — not the normalized headers['User-Agent'] that Grape exposes. Pass-through requests have X-Robots-Tag injected into the response headers array before returning.

# middleware/ai_bot_rack_middleware.rb
# Rack middleware variant — runs BEFORE Grape processes the request.
# More efficient: Grape never instantiates anything for blocked requests.
# Use this when you want to block at the Rack layer rather than inside Grape.

require_relative '../lib/ai_bot_detector'

class AiBotRackMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    # Rack env uses CGI-style header names: HTTP_USER_AGENT
    ua = env['HTTP_USER_AGENT'] || ''
    path = env['PATH_INFO'] || ''

    # Path guard
    return @app.call(env) if path == '/robots.txt'

    if AiBotDetector.ai_bot?(ua)
      return [
        403,
        {
          'Content-Type'  => 'text/plain',
          'X-Robots-Tag'  => 'noai, noimageai',
        },
        ['Forbidden']
      ]
    end

    # Pass through — call the next Rack app (Grape)
    status, headers, body = @app.call(env)
    headers['X-Robots-Tag'] = 'noai, noimageai'
    [status, headers, body]
  end
end

# config.ru with Rack middleware:
# require_relative 'middleware/ai_bot_rack_middleware'
# require_relative 'api/my_api'
# use AiBotRackMiddleware
# run MyApi

7. Rails integration — mount + ActionController filter

In a Rails + Grape stack, mount the Grape API in routes.rb. The Grape before block covers /api routes; Rails ApplicationController before_action covers HTML routes. The cleanest approach is a Rack middleware in config/application.rb that covers both layers.

# config/routes.rb — mount Grape API inside Rails
Rails.application.routes.draw do
  # Mount the Grape API at /api
  # The Grape before block fires for all routes under /api.
  # Rails routes outside /api use ActionController filters instead.
  mount MyApi => '/api'
end

# app/controllers/application_controller.rb — Rails filter for HTML routes
class ApplicationController < ActionController::Base
  before_action :block_ai_bots

  private

  def block_ai_bots
    ua = request.headers['User-Agent'].to_s
    if AiBotDetector.ai_bot?(ua)
      response.headers['X-Robots-Tag'] = 'noai, noimageai'
      render plain: 'Forbidden', status: :forbidden
    end
  end
end

# Note: In a Rails + Grape stack, block bots at BOTH layers:
# 1. Rails ApplicationController filter (for HTML/controller routes)
# 2. Grape before block (for /api routes)
# Or: use Rack middleware in config/application.rb to cover both at once.

8. public/robots.txt

In Rails, public/robots.txt is served as a static file before ActionDispatch and Grape run. In a standalone Grape Rack app, add a get '/robots.txt' route or serve it via Nginx upstream. The pass guard in the before block handles the edge case where the file reaches Grape.

# public/robots.txt (Rails) or public/robots.txt (standalone Rack)
# Served by Nginx/Rails static file handler before Grape runs.

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Key points

• headers[] reads requests; header() writes responses: headers['User-Agent'] reads the incoming request header. header 'X-Robots-Tag', '...' sets an outgoing response header. The asymmetry is intentional in Grape's DSL — headers is a hash accessor, header is a helper method.
• error!() accepts headers as third argument: error!('Forbidden', 403, { 'X-Robots-Tag' => 'noai, noimageai' }) sets headers on the error response in one call. This is more concise than calling header() before error!().
• error!() stops all further processing: No return or next needed after error!(). It raises an exception that Grape catches internally. Any code after error!() in the same block does not run.
• pass in before blocks: Grape's pass keyword inside a before block skips the current block and continues to the next. Use it for the robots.txt path guard as a clean alternative to return.
• Rack env vs Grape headers[]: In Rack middleware, use env['HTTP_USER_AGENT'] — the CGI-style key. Inside Grape, use headers['User-Agent'] — Grape normalises HTTP headers by stripping the HTTP_ prefix and converting underscores to hyphens.
• Namespace scoping vs global before: A before block inside a namespace fires only for routes in that namespace. A before block at the class level fires for all routes. Both run in order when a route matches — namespace before blocks run after class-level ones.

Framework comparison — Ruby web frameworks

Grape and Sinatra share the before do block syntax but diverge on short-circuiting: Grape uses error!() (exception-based), Sinatra uses halt (throw/catch-based). Both stop execution without a separate return. Grape's error!() is more expressive — it accepts a hash body for JSON error responses and a headers hash in one call.

Dependencies

# Gemfile
gem 'grape'
gem 'puma'            # Rack server for standalone deployment
gem 'rack'            # Rack (usually transitive)

# Optional for Rails integration:
gem 'rails'
gem 'grape-entity'    # serialization (optional)
gem 'grape-swagger'   # OpenAPI docs (optional)

# Install
bundle install

# Run standalone
bundle exec rackup config.ru -p 8080

# Run with Puma
bundle exec puma config.ru -p 8080 -w 4