How to Block AI Bots on ZIO HTTP: Complete 2026 Guide
ZIO HTTP is Scala's effect-based HTTP framework built on ZIO — distinct from http4s (Cats Effect). Routes use the Method.GET / "path" -> handler DSL. To block a bot: return ZIO.succeed(Response.status(Status.Forbidden)) — the inner handler is never called. request.header(Header.UserAgent) returns Option[Header.UserAgent] — type-safe, no string keys.
ZIO HTTP vs http4s — two different Scala ecosystems
ZIO HTTP uses ZIO[R, E, A] natively. http4s uses Cats Effect F[_] (typically IO). The middleware concepts are similar — both functional, both composable — but the effect types are incompatible. Use ZIO HTTP if your project already uses ZIO; use http4s for the Typelevel/Cats ecosystem.
Protection layers
1
robots.txt — publicRoutes ++ protectedRoutes — public Routes combined first; robots.txt served before any bot check
2
noai meta tag — In HTML body string — <meta name="robots" content="noai, noimageai"> in <head>
3
X-Robots-Tag (blocked) — .addHeader(Header.Custom("X-Robots-Tag", "noai, noimageai")) on the 403 Response
4
X-Robots-Tag (legitimate) — innerHandler(request).map(_.addHeader(xRobotsTag)) — piped onto pass-through responses
5
Hard 403 — ZIO.succeed(Response.status(Status.Forbidden)) — innerHandler never called, no downstream ZIO effects run
Step 1 — Bot detection (AiBots.scala)
A pure Scala object — no ZIO effects. List.exists short-circuits on first match. String.contains for literal substring matching — no regex, no escaping.
// AiBots.scala — bot detection
object AiBots {
// Known AI bot UA substrings — lowercase for comparison.
private val patterns: List[String] = List(
// OpenAI
"gptbot", "chatgpt-user", "oai-searchbot",
// Anthropic
"claudebot", "claude-web",
// Common Crawl
"ccbot",
// Bytedance
"bytespider",
// Meta
"meta-externalagent",
// Perplexity
"perplexitybot",
// Google AI
"google-extended", "googleother",
// Cohere
"cohere-ai",
// Amazon
"amazonbot",
// Diffbot
"diffbot",
// AI2
"ai2bot",
// DeepSeek
"deepseekbot",
// Mistral
"mistralai-user",
// xAI
"xai-bot",
// You.com
"youbot",
// DuckDuckGo AI
"duckassistbot",
)
/** Returns true if ua contains any known AI bot substring.
* ua must already be lowercased before calling. */
def isAiBot(ua: String): Boolean =
ua.nonEmpty && patterns.exists(ua.contains)
}Step 2 — Handler wrapper (BotBlocker.scala)
Handler.fromFunctionZIO lifts a Request => ZIO[...] into a Handler. request.header(Header.UserAgent) returns the typed header — no string key needed. .renderedValue gives the raw string.
// BotBlocker.scala — ZIO HTTP middleware
import zio._
import zio.http._
object BotBlocker {
private val xRobotsTag = Header.Custom("X-Robots-Tag", "noai, noimageai")
/** withBotCheck wraps a Handler with AI bot detection.
*
* ZIO HTTP Handler type: Handler[Env, Err, Request, Response]
* Handler.fromFunctionZIO lifts a Request => ZIO[...] into a Handler.
*
* To short-circuit: return ZIO.succeed(forbiddenResponse)
* — innerHandler is never called.
* To pass through: call innerHandler(request) and map to add headers.
*/
def withBotCheck(
innerHandler: Handler[Any, Response, Request, Response]
): Handler[Any, Response, Request, Response] =
Handler.fromFunctionZIO { (request: Request) =>
// request.header(Header.UserAgent) :: Option[Header.UserAgent]
// .renderedValue is the raw string value of the header.
// getOrElse("") gives safe default if the header is absent.
val ua = request
.header(Header.UserAgent)
.map(_.renderedValue)
.getOrElse("")
.toLowerCase
if (AiBots.isAiBot(ua))
// Short-circuit: 403, no call to innerHandler.
ZIO.succeed(
Response
.status(Status.Forbidden)
.addHeader(xRobotsTag)
.copy(body = Body.fromString("Forbidden"))
)
else
// Pass through: call innerHandler, add X-Robots-Tag to response.
innerHandler(request).map(_.addHeader(xRobotsTag))
}
}Step 3 — Routes with public/protected split
++ combines two Routes values. Routes in publicRoutes are matched first — robots.txt is served to all crawlers before withBotCheck ever runs.
// Routes.scala — ZIO HTTP route definitions
import zio._
import zio.http._
object AppRoutes {
private val robotsTxt = """User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
"""
// Public routes — no bot check. Combined first with ++.
// ZIO HTTP tries routes in the order they are defined.
val publicRoutes: Routes[Any, Response] = Routes(
Method.GET / "robots.txt" ->
Handler.fromResponse(
Response
.text(robotsTxt)
.addHeader(Header.ContentType(MediaType.text.plain))
),
Method.GET / "health" ->
Handler.fromResponse(Response.text("ok")),
)
// Protected handlers — wrapped with BotBlocker.withBotCheck
private val homeHandler: Handler[Any, Response, Request, Response] =
BotBlocker.withBotCheck(
Handler.fromResponse(
Response
.html("""<!DOCTYPE html>
<html>
<head>
<meta name="robots" content="noai, noimageai">
<title>My Site</title>
</head>
<body><h1>Welcome</h1></body>
</html>""")
)
)
private val apiDataHandler: Handler[Any, Response, Request, Response] =
BotBlocker.withBotCheck(
Handler.fromResponse(
Response
.json("""{"data":"protected"}""")
)
)
// Protected routes
val protectedRoutes: Routes[Any, Response] = Routes(
Method.GET / "" -> homeHandler,
Method.GET / "api" / "data" -> apiDataHandler,
)
// Final app: public routes first, then protected.
// ++ combines two Routes values; earlier routes take priority on match.
val app: Routes[Any, Response] = publicRoutes ++ protectedRoutes
}Step 4 — HandlerAspect via @@ (alternative)
HandlerAspect applies to a whole Routes collection via @@. Use ZIO.fail with a Response to short-circuit — the failure is caught and converted to the HTTP response.
// Alternative: HandlerAspect via @@ for route-level middleware
// Apply to all routes in a Routes collection at once.
import zio._
import zio.http._
// HandlerAspect transforms each handler in a Routes value.
// Use @@ to apply: routes @@ botBlockerAspect
val botBlockerAspect: HandlerAspect[Any, Unit] =
HandlerAspect.interceptIncomingHandler(
Handler.fromFunctionZIO { (request: Request) =>
val ua = request
.header(Header.UserAgent)
.map(_.renderedValue)
.getOrElse("")
.toLowerCase
if (AiBots.isAiBot(ua))
ZIO.fail(
Response
.status(Status.Forbidden)
.addHeader(Header.Custom("X-Robots-Tag", "noai, noimageai"))
.copy(body = Body.fromString("Forbidden"))
)
else
ZIO.succeed((request, ()))
}
)
// Apply to protected routes only:
val publicRoutes = Routes(Method.GET / "robots.txt" -> robotsTxtHandler)
val protected_ = Routes(Method.GET / "" -> homeHandler) @@ botBlockerAspect
val app = publicRoutes ++ protected_Step 5 — Server startup (Main.scala)
// Main.scala — ZIO HTTP server startup
import zio._
import zio.http._
object Main extends ZIOAppDefault {
override def run: ZIO[Any, Throwable, Unit] =
Server
.serve(AppRoutes.app)
.provide(
// Server.defaultWithPort: binds to 0.0.0.0 on the specified port
Server.defaultWithPort(8080),
)
}
// build.sbt
// scalaVersion := "3.3.3"
// libraryDependencies += "dev.zio" %% "zio-http" % "3.0.0"ZIO HTTP vs http4s vs Play Framework vs Akka HTTP
| Feature | ZIO HTTP | http4s | Play Framework | Akka HTTP |
|---|---|---|---|---|
| Effect type | ZIO[R, E, A] — environment R, error E, value A. Native ZIO ecosystem. | F[_] tagless final (typically IO from Cats Effect). Cats/Typelevel ecosystem. | Future[Result] (imperative async) or Action[A] blocks. Akka/Play ecosystem. | Future[HttpResponse] or Source[ByteString, _] for streaming. Akka streams. |
| Route DSL | Method.GET / "path" -> handler — path literal syntax, type-safe segments | HttpRoutes.of { case GET -> Root / "path" => ... } — pattern matching | conf/routes file or Action { request => ... } in controller | path("path") { get { complete { ... } } } — directive DSL |
| Short-circuit | ZIO.succeed(Response.status(Status.Forbidden)) — inner handler never called | OptionT.some(Response[F](Forbidden)) — inner routes never called | Results.Forbidden("Forbidden") — return from Action without calling next | complete(StatusCodes.Forbidden, "Forbidden") — directive completes early |
| UA header access | request.header(Header.UserAgent) :: Option[Header.UserAgent] — typed, .renderedValue for string | req.headers.get(CIString("User-Agent")) :: Option[Header.Raw] — CIString for CI lookup | request.headers.get("User-Agent") :: Option[String] — Map-based access | request.header[`User-Agent`] :: Option[User-Agent] — typed header model |
| Middleware / aspect | @@ HandlerAspect or withBotCheck wrapper function on handlers | Kleisli middleware: HttpRoutes[F] => HttpRoutes[F] | ActionBuilder or filter in application.conf filter chain | Directive composition or mapRequest / mapResponse directives |
| robots.txt | Routes(Method.GET / "robots.txt" -> handler) ++ protectedRoutes — public first | (robotsRoutes <+> BotBlockerMiddleware(appRoutes)).orNotFound | Public assets route in conf/routes above controller routes | path("robots.txt") { getFromFile("robots.txt") } before bot-check route |
Summary
ZIO.succeed(Response.status(Status.Forbidden))— return directly fromHandler.fromFunctionZIOto short-circuit. The inner handler effect is never executed.request.header(Header.UserAgent)— typed header access, returnsOption[Header.UserAgent]. Use.renderedValuefor the string. No string key needed.publicRoutes ++ protectedRoutes— combine with++. Public routes match first; robots.txt is always accessible.@@ HandlerAspect— apply middleware to an entireRoutescollection. UseZIO.fail(response)to short-circuit from inside an Aspect.- Choose the right Scala framework — ZIO HTTP for ZIO codebases; http4s for Cats Effect / Typelevel. Both functional, incompatible effect types.
Is your site protected from AI bots?
Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.