Skip to content
Guides/Gin (Go)

How to Block AI Bots on Gin (Go): Complete 2026 Guide

Gin is the most popular Go web framework — built on net/http, not fasthttp. Bot blocking uses a gin.HandlerFunc middleware: func(c *gin.Context). Call c.AbortWithStatus(403) to block (Layer 4), or call c.Next() and set c.Header() to add X-Robots-Tag (Layer 3). Unlike Fiber's fasthttp context, *gin.Context wraps *http.Request and http.ResponseWriter — no pooling gotchas.

Gin runs on net/http

Unlike Fiber, Gin uses the standard library net/http under the hood. *gin.Context wraps *http.Request (via c.Request) and http.ResponseWriter (via c.Writer). No context pooling, no sync.Pool recycling — it is safe to pass c.GetHeader() results to goroutines without copying. The key difference from raw net/http: Gin uses a chain model (call c.Next() to continue, c.Abort() to stop) instead of the wrapper pattern.

Protection layers

1
robots.txt — r.StaticFile("/robots.txt", "./public/robots.txt") — registered before middleware
2
noai meta tag — html/template layout — {{ .Robots }} variable with default
3
X-Robots-Tag header — c.Next() → c.Header("X-Robots-Tag", "noai, noimageai")
4
Hard 403 block — c.AbortWithStatus(403) — skips remaining handlers in the chain

Layer 1: robots.txt

Create a public/ directory at your project root (alongside main.go and go.mod) and place robots.txt there. Register the static file route before the bot blocker middleware so robots.txt is served without hitting the middleware at all.

# public/robots.txt

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Diffbot
User-agent: cohere-ai
User-agent: FacebookBot
User-agent: omgili
User-agent: omgilibot
User-agent: Amazonbot
User-agent: DeepSeekBot
User-agent: MistralBot
User-agent: xAI-Bot
User-agent: AI2Bot
Disallow: /

Register the static file route before middleware in your router setup:

r := gin.Default()

// robots.txt BEFORE bot middleware — Gin matches routes in order
r.StaticFile("/robots.txt", "./public/robots.txt")

// Now register bot blocker
r.Use(AiBotBlocker())

Alternatively, generate robots.txt dynamically from a route handler:

r.GET("/robots.txt", func(c *gin.Context) {
    c.String(http.StatusOK, `User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /`)
})

Layer 2: noai meta tag

If your Gin app renders HTML via html/template, add the noai meta tag to your base layout with a per-page override variable:

Go html/template layout (templates/layout.html)

<!-- templates/layout.html -->
{{- if .Robots }}
<meta name="robots" content="{{ .Robots }}">
{{- else }}
<meta name="robots" content="noai, noimageai">
{{- end }}

Render with override in Gin route handler

r.GET("/public-page", func(c *gin.Context) {
    c.HTML(http.StatusOK, "layout.html", gin.H{
        "Robots": "index, follow",
    })
})

// Pages that don't pass "Robots" get the default noai value

If your Gin app is a JSON API and a separate SPA handles rendering (React, Vue, etc.), add the noai meta tag in the frontend's base layout instead.

Layers 3 & 4: gin.HandlerFunc middleware

Gin middleware is a factory that returns a gin.HandlerFunc: func(c *gin.Context). Call c.AbortWithStatus(403) to block. Call c.Next() to pass through to the next handler.

middleware/aibot.go

package middleware

import (
	"net/http"
	"strings"

	"github.com/gin-gonic/gin"
)

var aiBotPatterns = []string{
	"gptbot", "chatgpt-user", "oai-searchbot",
	"claudebot", "anthropic-ai", "claude-web",
	"google-extended", "ccbot", "bytespider",
	"applebot-extended", "perplexitybot", "diffbot",
	"cohere-ai", "facebookbot", "meta-externalagent",
	"omgili", "omgilibot", "amazonbot",
	"deepseekbot", "mistralbot", "xai-bot", "ai2-bot",
}

var exemptPaths = map[string]bool{
	"/robots.txt":  true,
	"/sitemap.xml": true,
	"/favicon.ico": true,
}

// AiBotBlocker returns a gin.HandlerFunc that blocks known AI training crawlers.
func AiBotBlocker() gin.HandlerFunc {
	return func(c *gin.Context) {
		// Always pass through exempt paths
		if exemptPaths[c.Request.URL.Path] {
			c.Next()
			return
		}

		ua := strings.ToLower(c.GetHeader("User-Agent"))

		for _, pattern := range aiBotPatterns {
			if strings.Contains(ua, pattern) {
				// Layer 4: hard block — Abort() prevents remaining handlers from running
				c.AbortWithStatus(http.StatusForbidden)
				return
			}
		}

		// Layer 3: pass through, then add X-Robots-Tag to the response
		c.Next()
		c.Header("X-Robots-Tag", "noai, noimageai")
	}
}

Key points

  • Blocking: c.AbortWithStatus(403) writes the status code AND marks the context as aborted — remaining handlers in the chain are skipped. Always call return after it to exit the current middleware function. c.Abort() alone skips the chain but writes no status; always prefer AbortWithStatus.
  • Pass-through: call c.Next() to run downstream handlers and the route handler, then set c.Header(key, value). Headers set after c.Next() are still written to the response — Gin buffers the response writer.
  • Reading headers: c.GetHeader("User-Agent") is the Gin shorthand for c.Request.Header.Get("User-Agent"). Returns an empty string if the header is absent (never nil).
  • Writing headers: c.Header(key, value) sets a response header. Do not confuse with c.GetHeader()which reads a request header. Same naming pattern as Fiber's c.Get()/c.Set() split, different method names.
  • Factory pattern: AiBotBlocker() returns a handler — the factory wrapper lets you pass config options in the future (allowlist, custom patterns, per-environment toggle) without changing the call site.

Registering the middleware

package main

import (
	"log"
	"net/http"

	"github.com/gin-gonic/gin"
	"yourapp/middleware"
)

func main() {
	r := gin.Default() // includes Logger + Recovery middlewares

	// robots.txt BEFORE bot middleware
	r.StaticFile("/robots.txt", "./public/robots.txt")

	// Global bot blocker — runs on every route after static files
	r.Use(middleware.AiBotBlocker())

	// Your routes
	r.GET("/", func(c *gin.Context) {
		c.String(http.StatusOK, "Hello, World!")
	})
	r.GET("/api/data", func(c *gin.Context) {
		c.JSON(http.StatusOK, gin.H{"status": "ok"})
	})

	log.Fatal(r.Run(":8080"))
}

Gin runs middleware in FIFO order — the first r.Use() call runs first. Register the bot blocker before auth, CORS, and body parsing so blocked requests are rejected before any expensive middleware executes.

Route-group blocking

To protect only API routes while leaving the frontend indexable:

// Public routes — no bot blocking (search engines can index)
r.GET("/", homeHandler)
r.GET("/about", aboutHandler)

// API routes — bot blocker applied to this group
api := r.Group("/api")
api.Use(middleware.AiBotBlocker())
{
    api.GET("/products", productsHandler)
    api.GET("/users", usersHandler)
    api.POST("/orders", ordersHandler)
}

Group middleware only applies to routes registered under that group. r.Group("/api") creates a sub-router; calling group.Use() adds middleware only for that sub-router. You can chain multiple: api.Use(cors.Default(), middleware.AiBotBlocker()).

Abort() vs AbortWithStatus() — pick the right one

Gin has three abort methods. Only one is correct for bot blocking:

  • c.Abort() — skips remaining handlers but writes no status code. The client gets a 200 with no body. Incorrect for bot blocking.
  • c.AbortWithStatus(403) — skips remaining handlers AND writes 403 Forbidden. Correct.
  • c.AbortWithStatusJSON(403, gin.H{"error": "Forbidden"}) — same as above but with a JSON body. Use if your API always returns JSON.

Always call return after any Abort* call. Abort signals the engine to stop the handler chain but does not stop the current middleware function from running further.

Gin vs Fiber vs net/http

All three are Go, but the middleware patterns differ:

Gin (net/http, chain model)

func AiBotBlocker() gin.HandlerFunc {
    return func(c *gin.Context) {
        if isBot(c.GetHeader("User-Agent")) {
            c.AbortWithStatus(403)
            return
        }
        c.Next()
        c.Header("X-Robots-Tag", "noai, noimageai")
    }
}

Fiber (fasthttp, single context)

func AiBotBlocker(c *fiber.Ctx) error {
    if isBot(c.Get("User-Agent")) {
        return c.Status(403).SendString("Forbidden")
    }
    err := c.Next()
    c.Set("X-Robots-Tag", "noai, noimageai")
    return err
}

net/http (wrapper pattern)

func AiBotBlocker(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if isBot(r.UserAgent()) {
            http.Error(w, "Forbidden", 403)
            return
        }
        w.Header().Set("X-Robots-Tag", "noai, noimageai")
        next.ServeHTTP(w, r)
    })
}

Gin and Fiber both use a single context object, but Gin's context wraps net/http (heap-allocated per request, safe to reference later) while Fiber's context is backed by fasthttp's sync.Pool (recycled — do not retain after handler). net/http uses the wrapper pattern — a function that takes and returns http.Handler.

Verification

# Layer 1 — robots.txt (served before middleware runs)
curl http://localhost:8080/robots.txt

# Layer 3 — X-Robots-Tag on legitimate request
curl -I http://localhost:8080/api/products
# Expected: X-Robots-Tag: noai, noimageai

# Layer 4 — hard block on bot user-agent
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
  -I http://localhost:8080/api/products
# Expected: HTTP/1.1 403 Forbidden

# robots.txt must pass through even for bot UAs
curl -A "GPTBot" -I http://localhost:8080/robots.txt
# Expected: HTTP/1.1 200 OK

FAQ

How does Gin middleware differ from Fiber middleware?

Gin runs on net/http. Gin middleware is func(c *gin.Context) — void return. Block with c.AbortWithStatus(403) + return. Pass through with c.Next(). Fiber uses fasthttp and func(c *fiber.Ctx) error — return an error or response to block, call c.Next() to pass through. Both use a single context object, but Fiber's context is pooled (do not retain references) while Gin's is heap-allocated per request (safe to reference in goroutines).

What is the difference between c.Abort() and c.AbortWithStatus()?

c.Abort() marks the context as aborted (skips remaining handlers) but writes no HTTP status code. c.AbortWithStatus(403) does both: sets the status and aborts. For bot blocking, always use AbortWithStatus so the client receives a 403. Always follow with return to exit the current middleware function — Abort() does not stop the current function from running.

Where does robots.txt go in a Gin project?

Register it before the bot blocker: r.StaticFile("/robots.txt", "./public/robots.txt"). Gin matches routes in order, so the static file handler runs before the bot blocker middleware. For dynamic generation, r.GET("/robots.txt", func(c *gin.Context) { c.String(200, robotsTxt) }) also works. Avoid r.Static("/", "./public") unless you intend to serve the entire public directory.

Does Gin middleware run in FIFO order?

Yes. Gin runs middleware FIFO — first r.Use() call runs first. Register the bot blocker before auth, CORS, rate limiting, and body parsing. Blocked requests are rejected before any expensive middleware runs. Route-level middleware (passed as parameters to r.GET(), r.POST(), etc.) runs after router-level middleware but before the route handler.

Should I use r.Use() or group.Use() for bot blocking in Gin?

r.Use() for global protection — every route gets the bot blocker. Create a group for selective protection: api := r.Group("/api"); api.Use(middleware.AiBotBlocker()). This lets your public frontend stay fully indexable while the API is protected. You can also split: global r.Use() for X-Robots-Tag only (Layer 3), plus group.Use() for the hard 403 (Layer 4) on sensitive routes.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.