How to Block AI Bots in C++ Drogon

1. Filter header

Declare the filter as a subclass of drogon::HttpFilter<AiBotBlocker>. The METHOD_LIST_BEGIN / METHOD_LIST_END block is required even when empty — it tells Drogon which HTTP methods this filter applies to. An empty block means all methods.

// filters/AiBotBlocker.h
#pragma once
#include <drogon/HttpFilter.h>

class AiBotBlocker : public drogon::HttpFilter<AiBotBlocker> {
public:
    METHOD_LIST_BEGIN
    // No method restrictions — apply to every HTTP method
    METHOD_LIST_END

    void doFilter(const drogon::HttpRequestPtr &req,
                  drogon::FilterChainCallback &&fcb,
                  drogon::FilterChainCutoffCallback &&fccb) override;
};

2. Filter implementation — doFilter()

req->getHeader("user-agent") returns an empty string when the header is absent — no null check needed. Build a lowercase copy once, then use std::string::find() for literal substring matching with no regex overhead. Call fccb(resp) to block; call fcb() to continue.

// filters/AiBotBlocker.cc
#include "AiBotBlocker.h"
#include <drogon/drogon.h>
#include <algorithm>
#include <string>
#include <string_view>
#include <array>

// All lowercase — matched against tolower(ua)
static constexpr std::array<std::string_view, 16> kAiBotPatterns = {{
    "gptbot",
    "chatgpt-user",
    "claudebot",
    "anthropic-ai",
    "ccbot",
    "google-extended",
    "cohere-ai",
    "meta-externalagent",
    "bytespider",
    "omgili",
    "diffbot",
    "imagesiftbot",
    "magpie-crawler",
    "amazonbot",
    "dataprovider",
    "netcraft",
}};

static bool isAiBot(const std::string &ua) {
    if (ua.empty()) return false;
    // Build lowercase copy once
    std::string lower = ua;
    std::transform(lower.begin(), lower.end(), lower.begin(),
                   [](unsigned char c) { return std::tolower(c); });
    for (const auto &pattern : kAiBotPatterns) {
        if (lower.find(pattern) != std::string::npos) {
            return true;
        }
    }
    return false;
}

void AiBotBlocker::doFilter(const drogon::HttpRequestPtr &req,
                             drogon::FilterChainCallback &&fcb,
                             drogon::FilterChainCutoffCallback &&fccb) {
    // Path guard: Drogon's static file handler serves document_root files
    // before filters run — this guard is a safety net for dynamic routes.
    if (req->getPath() == "/robots.txt") {
        fcb();  // Let it through
        return;
    }

    // req->getHeader() returns "" when the header is absent — no null check needed.
    // Header names are case-insensitive; Drogon normalises to lowercase.
    const std::string ua = req->getHeader("user-agent");

    if (isAiBot(ua)) {
        auto resp = drogon::HttpResponse::newHttpResponse();
        resp->setStatusCode(drogon::k403Forbidden);
        resp->setBody("Forbidden");
        resp->addHeader("X-Robots-Tag", "noai, noimageai");
        // fccb(resp) sends this response and stops the filter chain.
        // The route handler is never called.
        fccb(resp);
        return;
    }

    // Pass-through: add X-Robots-Tag and advance the chain.
    // fcb() calls the next filter or the route handler.
    // There is no global "after" hook in Drogon — set headers here
    // or use a post-handling interceptor if you need them on every response.
    req->addHeader("X-Forwarded-BotCheck", "pass");  // optional debug header
    fcb();
}

3. Coroutine filter variant (Drogon >= 1.8)

Drogon 1.8+ supports coroutine-based filters via HttpCoroFilter<T>. The doFilter() method is a drogon::Task<HttpResponsePtr>. Return nullptr to pass through; return a non-null response to block. The callback pattern disappears entirely — the coroutine return value replaces both fcb and fccb. Requires C++20 and a compiler with coroutine support (GCC 11+, Clang 14+, MSVC 19.28+).

// filters/AiBotBlockerCoro.h — coroutine variant (Drogon >= 1.8)
#pragma once
#include <drogon/HttpCoroFilter.h>

class AiBotBlockerCoro : public drogon::HttpCoroFilter<AiBotBlockerCoro> {
public:
    METHOD_LIST_BEGIN
    METHOD_LIST_END

    drogon::Task<drogon::HttpResponsePtr> doFilter(
        const drogon::HttpRequestPtr &req) override;
};

// filters/AiBotBlockerCoro.cc
#include "AiBotBlockerCoro.h"
#include "BotUtils.h"  // isAiBot() from shared header

drogon::Task<drogon::HttpResponsePtr> AiBotBlockerCoro::doFilter(
    const drogon::HttpRequestPtr &req) {
    if (req->getPath() == "/robots.txt") {
        co_return nullptr;  // nullptr = pass through
    }

    const std::string ua = req->getHeader("user-agent");
    if (isAiBot(ua)) {
        auto resp = drogon::HttpResponse::newHttpResponse();
        resp->setStatusCode(drogon::k403Forbidden);
        resp->setBody("Forbidden");
        resp->addHeader("X-Robots-Tag", "noai, noimageai");
        co_return resp;  // non-null response = block (same as fccb(resp))
    }

    co_return nullptr;  // pass through
}

4. Applying the filter per-controller

Pass the filter class name as a string argument to ADD_METHOD_TO(). Multiple filters chain left-to-right. This approach applies the filter only to routes that declare it — useful when some routes (health checks, internal endpoints) should bypass the bot blocker.

// controllers/ApiController.h
#pragma once
#include <drogon/HttpController.h>

class ApiController : public drogon::HttpController<ApiController> {
public:
    METHOD_LIST_BEGIN
    // Apply AiBotBlocker filter to all routes in this controller
    ADD_METHOD_TO(ApiController::getData, "/api/data", drogon::Get,
                  "AiBotBlocker");
    // Or apply to a specific route only:
    // ADD_METHOD_TO(ApiController::getData, "/api/data", drogon::Get,
    //               "AiBotBlocker", "RateLimiter");
    METHOD_LIST_END

    void getData(const drogon::HttpRequestPtr &req,
                 std::function<void(const drogon::HttpResponsePtr &)> &&callback);
};

// Apply AiBotBlocker globally via config.json — see below.
// Per-controller annotation is useful when some routes should bypass the filter.

5. Global filter via config.json

Add the filter to the top-level "filters" array in config.json. Global filters run before per-controller filters and before routing. Static files served from document_root bypass all filters — Drogon's static handler intercepts those requests before the filter chain fires, so public/robots.txt is always accessible.

// config.json — register filter globally or per-path
{
  "listeners": [
    {
      "address": "0.0.0.0",
      "port": 8080
    }
  ],
  "document_root": "./public",
  "static_files_request_path": "/static",

  // Global filters — applied to every request before routing
  // List filter class names in order. Filters execute left-to-right.
  "filters": [
    "AiBotBlocker"
  ],

  // Alternatively, per-path filter configuration:
  // "custom_404_page": "./public/404.html",

  "app": {
    "threads_num": 4,
    "enable_session": false,
    "use_implicit_page": false
  }
}

6. main.cc

Drogon's entry point is minimal. Load config.json and call run(). Drogon auto-discovers filters and controllers that were compiled into the binary — there is no explicit registration step in code when using the ADD_METHOD_TO / global config approach.

// main.cc
#include <drogon/drogon.h>

int main() {
    drogon::app()
        .loadConfigFile("config.json")
        // Or configure programmatically:
        // .setDocumentRoot("./public")
        // .addListener("0.0.0.0", 8080)
        // .setThreadNum(4)
        .run();
    return 0;
}

7. CMakeLists.txt

Drogon is distributed as a CMake package. C++20 is required for coroutine filters. Add filter and controller source files to the executable target — Drogon discovers the classes at link time via static initialisation (the METHOD_LIST_BEGIN macro registers the controller with a static object).

# CMakeLists.txt (relevant section)
cmake_minimum_required(VERSION 3.14)
project(myapp CXX)

set(CMAKE_CXX_STANDARD 20)  # C++20 for coroutines

find_package(Drogon CONFIG REQUIRED)

# Drogon auto-discovers filter and controller source files in these directories
drogon_create_views(${PROJECT_NAME}
    ${CMAKE_CURRENT_SOURCE_DIR}/views
    ${CMAKE_CURRENT_BINARY_DIR}
)

add_executable(${PROJECT_NAME}
    main.cc
    filters/AiBotBlocker.cc
    controllers/ApiController.cc
)

target_link_libraries(${PROJECT_NAME} PRIVATE Drogon::Drogon)

8. X-Robots-Tag on all passing responses

Drogon filters run before the route handler and cannot modify the response after the handler runs without additional wiring. The cleanest approach is to set X-Robots-Tag directly in each controller action, or chain a second filter that tags passing requests for downstream handling. Drogon's interceptors (added in newer versions) can also modify responses post-handler.

// Middleware for X-Robots-Tag on all passing responses
// Drogon doesn't have a global "after" hook, but you can add a second filter
// that runs after AiBotBlocker and injects the header unconditionally.

// filters/XRobotsTagFilter.h
#pragma once
#include <drogon/HttpFilter.h>

class XRobotsTagFilter : public drogon::HttpFilter<XRobotsTagFilter> {
public:
    METHOD_LIST_BEGIN
    METHOD_LIST_END
    void doFilter(const drogon::HttpRequestPtr &req,
                  drogon::FilterChainCallback &&fcb,
                  drogon::FilterChainCutoffCallback &&fccb) override;
};

// filters/XRobotsTagFilter.cc
#include "XRobotsTagFilter.h"

void XRobotsTagFilter::doFilter(const drogon::HttpRequestPtr &req,
                                  drogon::FilterChainCallback &&fcb,
                                  drogon::FilterChainCutoffCallback &&fccb) {
    // Store a flag on the request so the controller can add the header.
    // Drogon doesn't expose the response object in filters before the handler
    // runs. Best approach: set in controller or use a post-routing interceptor.
    // For demonstration: tag the request for downstream handling.
    req->addHeader("X-Internal-BotCheck", "clear");
    fcb();
}

// In config.json, chain them in order:
// "filters": ["AiBotBlocker", "XRobotsTagFilter"]

9. public/robots.txt

Place robots.txt in the document_root directory (configured in config.json). Drogon's static file handler serves it before the filter chain runs — AI crawlers can always fetch it and discover they are disallowed. No path guard is strictly necessary when using document_root, but the guard in the filter handles edge cases if static serving is disabled.

# public/robots.txt
# Placed in document_root — served by Drogon's static handler
# before filters run. Accessible to all crawlers.

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Key points

• fccb(resp) is the block call: Passing a non-nullHttpResponsePtr to fccb sends it to the client and terminates the filter chain. Do not call fcb() after calling fccb(resp) — only one callback should be invoked per doFilter() call.
• fcb() is the pass-through call: Advances to the next filter in the chain, or to the route handler if this is the last filter. Never call both fcb and fccb in the same invocation — the framework will invoke one handler twice, which is undefined behaviour.
• Header names are case-insensitive: req->getHeader("user-agent") and req->getHeader("User-Agent") return the same value. Drogon normalises header names to lowercase internally. Always use lowercase to be explicit.
• Static files bypass filters: Drogon's static file handler intercepts requests that match files in document_root before the filter chain fires. A public/robots.txt file is always accessible regardless of filter configuration — no special path guard needed for static deployments.
• Coroutine filters eliminate callback hell: The HttpCoroFilter variant replaces both callbacks with a co_return value — nullptr passes through, non-null blocks. This is the recommended approach for new code on Drogon 1.8+ where async operations (database lookups, Redis checks) are needed inside the filter.
• Filter discovery is static: Drogon registers filters via static initialisation at startup (the METHOD_LIST_BEGIN macro). Filters must be compiled into the binary — there is no runtime plugin loading. Add filter source files to your CMake target explicitly.

Framework comparison — C++ HTTP frameworks

Drogon's callback-pair pattern (fcb / fccb) is unique among C++ frameworks and reflects its async-first design — the callbacks decouple the filter from the response path, enabling async filter logic without blocking threads. The coroutine variant makes this even cleaner with direct return semantics.

Dependencies

# Install Drogon via vcpkg (recommended)
vcpkg install drogon

# Or via homebrew (macOS)
brew install drogon

# Build the project
cmake -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=20
cmake --build build -j$(nproc)
./build/myapp

# Drogon dependencies (auto-resolved via vcpkg)
# trantor (event loop), jsoncpp, uuid, zlib, openssl