How to Block AI Bots in Go Gorilla Mux

1. Bot detection package

A standalone Go package with no external dependencies. strings.Contains() performs literal substring matching. strings.ToLower() applied once before iteration. Package-private aiPatterns slice — unexported to prevent external mutation.

// internal/botdetect/botdetect.go
package botdetect

import "strings"

// All lowercase — matched against strings.ToLower(ua)
var aiPatterns = []string{
	"gptbot",
	"chatgpt-user",
	"claudebot",
	"anthropic-ai",
	"ccbot",
	"google-extended",
	"cohere-ai",
	"meta-externalagent",
	"bytespider",
	"omgili",
	"diffbot",
	"imagesiftbot",
	"magpie-crawler",
	"amazonbot",
	"dataprovider",
	"netcraft",
}

// IsAIBot returns true if the User-Agent matches a known AI crawler.
func IsAIBot(ua string) bool {
	if ua == "" {
		return false
	}
	lower := strings.ToLower(ua)
	// strings.Contains — literal substring, no regex
	for _, p := range aiPatterns {
		if strings.Contains(lower, p) {
			return true
		}
	}
	return false
}

2. Middleware — func(http.Handler) http.Handler

The standard Go middleware pattern. Set X-Robots-Tag on w.Header() before http.Error() (blocked) or next.ServeHTTP() (passing) — headers are locked after the first write. Never call next.ServeHTTP() after http.Error().

// internal/middleware/botblock.go
package middleware

import (
	"net/http"
	"strings"

	"github.com/example/myapp/internal/botdetect"
)

// AiBotBlocker returns a Gorilla Mux-compatible middleware that blocks
// known AI crawlers. Uses the standard func(http.Handler) http.Handler
// signature — compatible with router.Use() and subrouter.Use().
func AiBotBlocker(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// Path guard: let robots.txt through.
		// If served via Nginx upstream, this guard is a no-op.
		// If served via a Go route or http.FileServer, this guard fires.
		if strings.EqualFold(r.URL.Path, "/robots.txt") {
			next.ServeHTTP(w, r)
			return
		}

		// r.Header.Get() returns "" when absent — no nil check needed.
		// Header names are canonicalised by net/http (Title-Case).
		ua := r.Header.Get("User-Agent")

		if botdetect.IsAIBot(ua) {
			// Set X-Robots-Tag before writing the status — headers must
			// be set before http.Error() or w.WriteHeader() is called.
			w.Header().Set("X-Robots-Tag", "noai, noimageai")
			http.Error(w, "Forbidden", http.StatusForbidden)
			// Do NOT call next.ServeHTTP() after http.Error() —
			// the response has already been written.
			return
		}

		// Pass-through: set X-Robots-Tag on all non-blocked responses.
		// Must be called before next.ServeHTTP() to ensure the header
		// is sent before the response body is written.
		w.Header().Set("X-Robots-Tag", "noai, noimageai")
		next.ServeHTTP(w, r)
	})
}

3. main.go — router.Use() global registration

Register the middleware globally with router.Use(). It fires for every request on the router, including static file handlers registered with PathPrefix. The path guard in the middleware lets /robots.txt through.

// main.go — Gorilla Mux application
package main

import (
	"encoding/json"
	"log"
	"net/http"

	"github.com/gorilla/mux"
	"github.com/example/myapp/internal/middleware"
)

func main() {
	r := mux.NewRouter()

	// Apply AiBotBlocker globally — fires for every request on this router.
	// router.Use() wraps all routes registered on r, including subrouters.
	r.Use(middleware.AiBotBlocker)

	// Static file serving — robots.txt served here, middleware fires first.
	// The path guard in AiBotBlocker lets /robots.txt through.
	r.PathPrefix("/static/").Handler(
		http.StripPrefix("/static/", http.FileServer(http.Dir("./static"))),
	)

	// Routes
	r.HandleFunc("/", indexHandler).Methods(http.MethodGet)
	r.HandleFunc("/api/data", apiDataHandler).Methods(http.MethodGet)
	r.HandleFunc("/health", healthHandler).Methods(http.MethodGet)

	log.Println("Listening on :8080")
	log.Fatal(http.ListenAndServe(":8080", r))
}

func indexHandler(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(map[string]string{"message": "Hello"})
}

func apiDataHandler(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(map[string]string{"data": "value"})
}

func healthHandler(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
}

4. Subrouter scoping — protect only /api routes

Create a subrouter with r.PathPrefix("/api").Subrouter() and call api.Use() to apply middleware only to routes under /api. Routes on the parent router (health, robots.txt) bypass the filter entirely — no path guards needed.

// Subrouter scoping — apply bot blocker only to /api routes.
// Routes outside the subrouter (health, public pages) bypass the filter.

r := mux.NewRouter()

// Health check — no bot filter
r.HandleFunc("/health", healthHandler).Methods(http.MethodGet)

// Serve robots.txt as a static file — no bot filter for this path
r.HandleFunc("/robots.txt", func(w http.ResponseWriter, r *http.Request) {
    http.ServeFile(w, r, "./static/robots.txt")
}).Methods(http.MethodGet)

// API subrouter — bot filter applied only here
api := r.PathPrefix("/api").Subrouter()
api.Use(middleware.AiBotBlocker)

api.HandleFunc("/data",  apiDataHandler).Methods(http.MethodGet)
api.HandleFunc("/users", apiUsersHandler).Methods(http.MethodGet)

5. ResponseWriter wrapper — header injection after handler

Go's http.ResponseWriter locks headers once the first byte is written. If you need to inject X-Robots-Tag regardless of when the handler writes its response, wrap the writer and intercept WriteHeader(). This is rarely needed — setting headers before next.ServeHTTP() is simpler and sufficient in most cases.

// ResponseWriter wrapper — add X-Robots-Tag after the handler runs.
// Standard http.ResponseWriter doesn't allow modifying headers after
// WriteHeader() or Write() is called. Use a wrapper to intercept.
// This is needed only if you want to add headers AFTER next.ServeHTTP().

type responseCapture struct {
	http.ResponseWriter
	wroteHeader bool
}

func (rc *responseCapture) WriteHeader(code int) {
	if !rc.wroteHeader {
		rc.Header().Set("X-Robots-Tag", "noai, noimageai")
		rc.wroteHeader = true
	}
	rc.ResponseWriter.WriteHeader(code)
}

func (rc *responseCapture) Write(b []byte) (int, error) {
	if !rc.wroteHeader {
		rc.WriteHeader(http.StatusOK)
	}
	return rc.ResponseWriter.Write(b)
}

// Usage in middleware — set header regardless of when handler writes:
func AiBotBlockerWithCapture(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ua := r.Header.Get("User-Agent")
		if IsAIBot(ua) {
			w.Header().Set("X-Robots-Tag", "noai, noimageai")
			http.Error(w, "Forbidden", http.StatusForbidden)
			return
		}
		rc := &responseCapture{ResponseWriter: w}
		next.ServeHTTP(rc, r)
	})
}

6. static/robots.txt

# static/robots.txt
User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Key points

• Set headers before the first write: Go's http.ResponseWriter sends headers the moment WriteHeader() or Write() is called. Call w.Header().Set() before http.Error() (blocked) and before next.ServeHTTP() (passing). Any w.Header().Set() call after the first write is a no-op — the headers are already on the wire.
• Never call next.ServeHTTP() after http.Error(): http.Error() writes the status code and body to the wire. Calling next.ServeHTTP() afterwards causes a superfluous response write and a http: superfluous response.WriteHeader call log warning. Always return immediately after http.Error().
• r.Header.Get() returns "" — no nil check needed: Go's http.Header.Get() returns an empty string for missing headers. No nil pointer dereference is possible — skip the nil check.
• Middleware signature is stdlib-compatible: func(next http.Handler) http.Handler is the same pattern used by net/http, Chi, Alice, and many other Go libraries. Gorilla Mux middleware is portable — it can be used with http.Handle() directly or wrapped with libraries like justinas/alice for chaining.
• router.Use() vs subrouter.Use(): router.Use() applies middleware to all routes on the router. subrouter.Use() applies it only to routes registered on that subrouter. Middleware registered on a parent router runs before subrouter middleware.
• Header canonicalisation: Go's net/http canonicalises header names to Title-Case (via textproto.CanonicalMIMEHeaderKey). Both r.Header.Get("user-agent") and r.Header.Get("User-Agent") return the same value. Use Title-Case by convention.

Framework comparison — Go HTTP routers

Gorilla Mux and Chi share an identical middleware signature — middleware written for one works on the other with no changes. Gin and Echo use framework-specific context types, making their middleware non-portable. If you're migrating from Gorilla Mux to Chi (a common migration path since Gorilla Mux entered maintenance mode in 2022), your middleware requires zero changes.

Dependencies

# go.mod
module github.com/example/myapp

go 1.21

require (
    github.com/gorilla/mux v1.8.1
)

# Install
go get github.com/gorilla/mux@v1.8.1

# Run
go run main.go

# Build
go build -o myapp .
./myapp

# Note: Gorilla Mux v1.8.1 is the final release — the project is in
# maintenance mode (security fixes only). Fully stable and production-ready.
# Consider Chi for new projects that need active development.