Skip to content
Guides/Fiber (Go)

How to Block AI Bots on Fiber (Go): Complete 2026 Guide

Fiber is a Go web framework built on fasthttp — not net/http. Bot blocking uses a fiber.Handler middleware: func(c *fiber.Ctx) error. Return a 403 response to block (Layer 4), or call c.Next() and add the X-Robots-Tag header to the response (Layer 3). All request and response methods live on a single *fiber.Ctx — there is no separate http.ResponseWriter.

fasthttp — not net/http

Fiber uses fasthttp under the hood, not the standard library net/http. This means no *http.Request, no http.ResponseWriter, and no http.Handler interface. Everything goes through *fiber.Ctx — read headers with c.Get(key), write headers with c.Set(key, value), set status with c.Status(code). Because fasthttp recycles context objects via a pool, do not retain references to c or its data after the handler returns.

Protection layers

1
robots.txt — app.Static("/", "./public") — served before middleware runs
2
noai meta tag — html/template layout — {{ .Robots }} variable with default
3
X-Robots-Tag header — c.Next() → c.Set("X-Robots-Tag", "noai, noimageai")
4
Hard 403 block — return c.Status(fiber.StatusForbidden).SendString("Forbidden")

Layer 1: robots.txt

Create a public/ directory at your project root (alongside main.go and go.mod) and place robots.txt there. Register the static directory before the bot blocker middleware so robots.txt is served without hitting the middleware.

# public/robots.txt

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Diffbot
User-agent: cohere-ai
User-agent: FacebookBot
User-agent: omgili
User-agent: omgilibot
User-agent: Amazonbot
User-agent: DeepSeekBot
User-agent: MistralBot
User-agent: xAI-Bot
User-agent: AI2Bot
Disallow: /

Register in your app setup:

// Serve static files BEFORE bot middleware
app.Static("/", "./public")

// Then register bot blocker
app.Use(aiBotMiddleware)

Alternatively, define a route for dynamic generation:

app.Get("/robots.txt", func(c *fiber.Ctx) error {
    robotsTxt := `User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /`
    c.Set("Content-Type", "text/plain")
    return c.SendString(robotsTxt)
})

Layer 2: noai meta tag

If your Fiber app renders HTML (via html/template or a template engine), add the noai meta tag to your base layout with a per-page override variable:

Go html/template layout (views/layout.html)

<!-- views/layout.html -->
{{- if .Robots }}
<meta name="robots" content="{{ .Robots }}">
{{- else }}
<meta name="robots" content="noai, noimageai">
{{- end }}

Fiber template engine (with fiber-template)

<!-- views/layout.html (fiber-template) -->
<meta name="robots" content="{{default .Robots "noai, noimageai"}}">

Render with override in route handler

app.Get("/public-page", func(c *fiber.Ctx) error {
    return c.Render("page", fiber.Map{
        "Robots": "index, follow",
    })
})

If your Fiber app serves a JSON API and a separate frontend handles rendering (React, Vue, etc.), add the noai meta tag in the frontend's base layout instead — Fiber never returns HTML in that case.

Layers 3 & 4: fiber.Handler middleware

Fiber middleware is a fiber.Handler: func(c *fiber.Ctx) error. Return a response to block. Call c.Next() to pass through to the next handler.

middleware/aibot.go

package middleware

import (
	"strings"

	"github.com/gofiber/fiber/v2"
)

var aiBotPatterns = []string{
	"gptbot", "chatgpt-user", "oai-searchbot",
	"claudebot", "anthropic-ai", "claude-web",
	"google-extended", "ccbot", "bytespider",
	"applebot-extended", "perplexitybot", "diffbot",
	"cohere-ai", "facebookbot", "meta-externalagent",
	"omgili", "omgilibot", "amazonbot",
	"deepseekbot", "mistralbot", "xai-bot", "ai2-bot",
}

var exemptPaths = map[string]bool{
	"/robots.txt":  true,
	"/sitemap.xml": true,
	"/favicon.ico": true,
}

func AiBotBlocker(c *fiber.Ctx) error {
	path := c.Path()

	// Always pass through exempt paths
	if exemptPaths[path] {
		return c.Next()
	}

	ua := strings.ToLower(c.Get("User-Agent"))

	for _, pattern := range aiBotPatterns {
		if strings.Contains(ua, pattern) {
			// Layer 4: hard 403 — do NOT call c.Next()
			return c.Status(fiber.StatusForbidden).SendString("Forbidden")
		}
	}

	// Layer 3: pass through, then add X-Robots-Tag
	err := c.Next()
	if err != nil {
		return err
	}
	c.Set("X-Robots-Tag", "noai, noimageai")
	return nil
}

Key points

  • Blocking: c.Status(fiber.StatusForbidden).SendString("Forbidden") writes the response and returns — c.Next() is never called, so the route handler and downstream middleware do not execute.
  • Pass-through: call c.Next() first, then set headers with c.Set(). Unlike PSR-7 or net/http, Fiber's response headers are mutable — you modify them in place, no new object returned.
  • c.Get() vs c.Set(): c.Get(key) reads a request header. c.Set(key, value) writes a response header. Same *fiber.Ctx, different directions.
  • Error handling: check the error returned by c.Next() — if the downstream handler failed, propagate the error instead of adding headers to a broken response.
  • Context pooling: fasthttp recycles *fiber.Ctx after each request. Do not store references to c, c.Body(), or c.Get(...) results in goroutines or package variables — copy the data if you need it beyond the handler.

Registering the middleware

package main

import (
	"log"

	"github.com/gofiber/fiber/v2"
	"yourapp/middleware"
)

func main() {
	app := fiber.New()

	// Serve static files FIRST — robots.txt bypasses all middleware
	app.Static("/", "./public")

	// Global bot blocker — runs on every route after static files
	app.Use(middleware.AiBotBlocker)

	// Your routes
	app.Get("/", func(c *fiber.Ctx) error {
		return c.SendString("Hello, World!")
	})
	app.Get("/api/data", func(c *fiber.Ctx) error {
		return c.JSON(fiber.Map{"status": "ok"})
	})

	log.Fatal(app.Listen(":3000"))
}

Fiber runs middleware in FIFO order — the first app.Use() call runs first. Register the bot blocker early (before auth, CORS, rate limiting) so blocked requests are rejected before any other middleware runs.

Route-group blocking

To protect only API routes and leave the frontend unaffected:

// Public routes — no bot blocking
app.Get("/", homeHandler)
app.Get("/about", aboutHandler)

// API routes — bot blocker applied
api := app.Group("/api", middleware.AiBotBlocker)
api.Get("/products", productsHandler)
api.Get("/users", usersHandler)
api.Post("/orders", ordersHandler)

Group middleware only runs for routes registered under that group. app.Group("/api", middleware.AiBotBlocker) passes the middleware as a variadic argument — you can chain multiple: app.Group("/api", cors, auth, middleware.AiBotBlocker).

Fiber vs net/http vs Gin

All three are Go, but the middleware patterns are different:

Fiber (fasthttp)

func AiBotBlocker(c *fiber.Ctx) error {
    if isBot(c.Get("User-Agent")) {
        return c.Status(403).SendString("Forbidden")
    }
    err := c.Next()
    c.Set("X-Robots-Tag", "noai, noimageai")
    return err
}

net/http (standard library)

func AiBotBlocker(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if isBot(r.UserAgent()) {
            http.Error(w, "Forbidden", 403)
            return
        }
        w.Header().Set("X-Robots-Tag", "noai, noimageai")
        next.ServeHTTP(w, r)
    })
}

Gin (net/http under the hood)

func AiBotBlocker() gin.HandlerFunc {
    return func(c *gin.Context) {
        if isBot(c.GetHeader("User-Agent")) {
            c.AbortWithStatus(403)
            return
        }
        c.Header("X-Robots-Tag", "noai, noimageai")
        c.Next()
    }
}

Fiber and Gin both use a single context object, but Fiber's context is backed by fasthttp (pooled, recycled) while Gin's wraps net/http (allocated per request). net/http uses the wrapper pattern — a function that takes and returns http.Handler.

Verification

# Layer 1 — robots.txt (served from ./public, bypasses middleware)
curl http://localhost:3000/robots.txt

# Layer 3 — X-Robots-Tag on legitimate request
curl -I http://localhost:3000/api/products
# Expected: X-Robots-Tag: noai, noimageai

# Layer 4 — hard block on bot user-agent
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
  -I http://localhost:3000/api/products
# Expected: HTTP/1.1 403 Forbidden

# robots.txt must pass through even for bot UAs
curl -A "GPTBot" -I http://localhost:3000/robots.txt
# Expected: HTTP/1.1 200 OK

FAQ

How does Fiber middleware differ from net/http middleware?

Fiber is built on fasthttp, not net/http, so middleware uses fiber.Handler — func(c *fiber.Ctx) error — instead of the net/http func(http.Handler) http.Handler wrapper pattern. There is no http.ResponseWriter or *http.Request. Fiber passes a single *fiber.Ctx that holds both request and response. Call c.Next() to invoke the next handler. Return a response (c.Status(403).SendString(...)) to short-circuit. Gin also uses a single context but runs on net/http under the hood.

Where does robots.txt go in a Fiber project?

Create a ./public directory at your project root (alongside main.go and go.mod) and place robots.txt there. Register with app.Static("/", "./public") before any middleware — Fiber serves static files first, bypassing the bot blocker entirely. For dynamic generation, define app.Get("/robots.txt", handler) before the bot blocker middleware.

Does Fiber middleware run in registration order?

Yes. Fiber uses FIFO — the first app.Use() call runs first. Register the bot blocker early (before CORS, auth, body parsing) so blocked requests are rejected immediately. This is different from Slim (LIFO) but the same as Express, Koa, and most Node.js frameworks.

Why can't I store *fiber.Ctx in a goroutine?

fasthttp recycles *fiber.Ctx objects via a sync.Pool — after the handler returns, the Ctx is zeroed and reused for the next request. If you store a reference to c in a goroutine, you will read stale or corrupted data. Copy what you need: userAgent := string(c.Request().Header.UserAgent()) or body := make([]byte, len(c.Body())); copy(body, c.Body()). Then pass the copy to the goroutine.

Should I use app.Use() or app.Group() for bot blocking?

app.Use() for global protection — every route gets the bot blocker. app.Group("/api", middleware.AiBotBlocker) if only API routes should be protected while the frontend stays indexable. You can also split concerns: global app.Use() that only adds X-Robots-Tag (Layer 3), plus a group-scoped middleware that adds the hard 403 block (Layer 4) only for API routes.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.