Skip to content
Guides/Play Framework (Scala/Java)

How to Block AI Bots on Play Framework (Scala/Java): Complete 2026 Guide

Play Framework is a reactive web framework built on Pekko (formerly Akka) — used by Twitter, LinkedIn, and enterprise Java/Scala shops. Bot blocking uses EssentialFilter for global interception (before body parsing) or Action composition for per-route control.

EssentialFilter vs Filter

EssentialFilter: accesses RequestHeader + raw byte stream. Can short-circuit before body parsing — zero overhead for bot blocking.
Filter: always parses the body first, then gives you Future[Result] to transform.
For bot blocking, always use EssentialFilter — you only need the User-Agent header, not the body.

Protection layers

1
robots.txt — Assets controller route — served before filters fire
2
noai meta tag — Twirl template variable set via request attrs
3
X-Robots-Tag header — .map(_.withHeaders(...)) on the result Future — all responses
4
Hard 403 block — Accumulator.done(Results.Forbidden) — before body parsing

Layer 1: robots.txt

Place in public/robots.txt and add an Assets route. The Assets controller serves static files before filters run:

# public/robots.txt

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: CCBot
User-agent: cohere-ai
User-agent: Bytespider
User-agent: Amazonbot
User-agent: PerplexityBot
User-agent: YouBot
User-agent: Diffbot
User-agent: DeepSeekBot
User-agent: MistralBot
User-agent: xAI-Bot
User-agent: AI2Bot
Disallow: /
# conf/routes
GET  /robots.txt  controllers.Assets.at(path="/public", file="robots.txt")

Layers 2, 3 & 4: EssentialFilter (Scala)

The EssentialFilter receives a RequestHeader and returns an Accumulator[ByteString, Result]. Block immediately with Accumulator.done() — the request body is never read:

// app/filters/AiBotFilter.scala
package filters

import javax.inject.Inject
import org.apache.pekko.stream.Materializer
import play.api.mvc._
import play.api.mvc.Results._

class AiBotFilter @Inject()(implicit val mat: Materializer)
    extends EssentialFilter {

  private val aiBots = Seq(
    "gptbot", "chatgpt-user", "claudebot", "anthropic-ai",
    "ccbot", "cohere-ai", "bytespider", "amazonbot",
    "applebot-extended", "perplexitybot", "youbot", "diffbot",
    "google-extended", "deepseekbot", "mistralbot", "xai-bot",
    "ai2bot", "oai-searchbot", "duckassistbot"
  )

  override def apply(next: EssentialAction): EssentialAction = {
    EssentialAction { requestHeader =>
      val ua = requestHeader.headers
        .get("User-Agent")
        .map(_.toLowerCase)
        .getOrElse("")

      // Layer 4: hard 403 for AI bots — before body parsing
      if (aiBots.exists(ua.contains)) {
        Accumulator.done(
          Forbidden("Forbidden: AI crawlers are not permitted.")
            .withHeaders("X-Robots-Tag" -> "noai, noimageai")
        )
      } else {
        // Layer 3: X-Robots-Tag on all legitimate responses
        next(requestHeader).map { result =>
          result.withHeaders("X-Robots-Tag" -> "noai, noimageai")
        }
      }
    }
  }
}

Register in application.conf:

# conf/application.conf
play.filters.enabled += "filters.AiBotFilter"

Accumulator.done — zero-body short-circuit

Accumulator.done(result) returns an already-completed accumulator — Play never reads the request body, never routes to a controller, never allocates body parsing resources. This is the most efficient blocking point in Play's pipeline. Compare to Spring Boot's doFilter() which always reads request metadata, or Vert.x which similarly short-circuits with ctx.response().end() before handler execution.

Action composition (per-route blocking)

For per-route control instead of global filtering, use Action composition — wrap individual controller actions:

// app/actions/BotBlockAction.scala
package actions

import javax.inject.Inject
import play.api.mvc._
import play.api.mvc.Results._
import scala.concurrent.{ExecutionContext, Future}

class BotBlockAction @Inject()(parser: BodyParsers.Default)(
    implicit ec: ExecutionContext
) extends ActionBuilderImpl(parser) {

  private val aiBots = Seq(
    "gptbot", "chatgpt-user", "claudebot", "anthropic-ai",
    "ccbot", "cohere-ai", "bytespider", "amazonbot",
    "applebot-extended", "perplexitybot", "youbot", "diffbot",
    "google-extended", "deepseekbot", "mistralbot", "xai-bot",
    "ai2bot", "oai-searchbot", "duckassistbot"
  )

  override def invokeBlock[A](
      request: Request[A],
      block: Request[A] => Future[Result]
  ): Future[Result] = {
    val ua = request.headers
      .get("User-Agent")
      .map(_.toLowerCase)
      .getOrElse("")

    if (aiBots.exists(ua.contains)) {
      Future.successful(
        Forbidden("Forbidden: AI crawlers are not permitted.")
          .withHeaders("X-Robots-Tag" -> "noai, noimageai")
      )
    } else {
      block(request).map(_.withHeaders("X-Robots-Tag" -> "noai, noimageai"))
    }
  }
}

Use in controllers:

// app/controllers/ContentController.scala
package controllers

import javax.inject.Inject
import play.api.mvc._
import actions.BotBlockAction

class ContentController @Inject()(
    cc: ControllerComponents,
    botBlock: BotBlockAction
) extends AbstractController(cc) {

  // Protected — AI bots get 403
  def articles = botBlock { implicit request =>
    Ok("Articles content")
  }

  def posts = botBlock { implicit request =>
    Ok("Posts content")
  }

  // Unprotected — no bot blocking
  def health = Action { Ok("healthy") }
  def publicApi = Action { Ok("public data") }
}

Play Java — @With annotation

In Play Java, use play.mvc.Action with the @With annotation for per-controller blocking:

// app/actions/AiBotBlockerAction.java
package actions;

import play.mvc.Action;
import play.mvc.Http;
import play.mvc.Result;
import java.util.List;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.CompletableFuture;

public class AiBotBlockerAction extends Action.Simple {

    private static final List<String> AI_BOTS = List.of(
        "gptbot", "chatgpt-user", "claudebot", "anthropic-ai",
        "ccbot", "cohere-ai", "bytespider", "amazonbot",
        "applebot-extended", "perplexitybot", "youbot", "diffbot",
        "google-extended", "deepseekbot", "mistralbot", "xai-bot",
        "ai2bot", "oai-searchbot", "duckassistbot"
    );

    @Override
    public CompletionStage<Result> call(Http.Request request) {
        String ua = request.header("User-Agent")
            .map(String::toLowerCase)
            .orElse("");

        if (AI_BOTS.stream().anyMatch(ua::contains)) {
            return CompletableFuture.completedFuture(
                forbidden("Forbidden: AI crawlers are not permitted.")
                    .withHeader("X-Robots-Tag", "noai, noimageai")
            );
        }

        return delegate.call(request).thenApply(result ->
            result.withHeader("X-Robots-Tag", "noai, noimageai")
        );
    }
}

Apply to controllers:

// app/controllers/ContentController.java
package controllers;

import play.mvc.*;
import actions.AiBotBlockerAction;

@With(AiBotBlockerAction.class)  // Applies to ALL actions in this controller
public class ContentController extends Controller {

    public Result articles(Http.Request request) {
        return ok("Articles content");
    }

    public Result posts(Http.Request request) {
        return ok("Posts content");
    }
}

// For per-action blocking instead of per-controller:
public class MixedController extends Controller {

    @With(AiBotBlockerAction.class)  // Only this action is protected
    public Result protectedContent(Http.Request request) {
        return ok("Protected");
    }

    public Result publicContent(Http.Request request) {
        return ok("Public — no bot blocking");
    }
}

Play Java — global Filter

For global blocking in Play Java, implement play.mvc.EssentialFilter:

// app/filters/AiBotFilter.java
package filters;

import javax.inject.Inject;
import org.apache.pekko.stream.Materializer;
import play.mvc.EssentialAction;
import play.mvc.EssentialFilter;
import play.mvc.Results;
import play.libs.streams.Accumulator;
import java.util.List;

public class AiBotFilter extends EssentialFilter {

    private static final List<String> AI_BOTS = List.of(
        "gptbot", "chatgpt-user", "claudebot", "anthropic-ai",
        "ccbot", "cohere-ai", "bytespider", "amazonbot",
        "applebot-extended", "perplexitybot", "youbot", "diffbot",
        "google-extended", "deepseekbot", "mistralbot", "xai-bot",
        "ai2bot", "oai-searchbot", "duckassistbot"
    );

    private final Materializer materializer;

    @Inject
    public AiBotFilter(Materializer materializer) {
        this.materializer = materializer;
    }

    @Override
    public EssentialAction apply(EssentialAction next) {
        return EssentialAction.of(requestHeader -> {
            String ua = requestHeader.header("User-Agent")
                .map(String::toLowerCase)
                .orElse("");

            if (AI_BOTS.stream().anyMatch(ua::contains)) {
                return Accumulator.done(
                    Results.forbidden("Forbidden: AI crawlers are not permitted.")
                        .withHeader("X-Robots-Tag", "noai, noimageai")
                );
            }

            return next.apply(requestHeader).map(result ->
                result.withHeader("X-Robots-Tag", "noai, noimageai"),
                materializer.executionContext()
            );
        });
    }
}
# conf/application.conf (Java)
play.filters.enabled += "filters.AiBotFilter"

Play vs Vert.x vs Spring Boot vs Akka HTTP — comparison

Play Framework — EssentialFilter (Scala)

// Short-circuits before body parsing
class AiBotFilter @Inject()(implicit val mat: Materializer)
    extends EssentialFilter {
  override def apply(next: EssentialAction) = EssentialAction { rh =>
    if (isAiBot(rh.headers.get("User-Agent")))
      Accumulator.done(Forbidden("Blocked"))
    else next(rh).map(_.withHeaders("X-Robots-Tag" -> "noai"))
  }
}

Vert.x — Handler<RoutingContext>

// Non-blocking handler with explicit ordering
router.route().order(-1).handler(ctx -> {
    String ua = ctx.request().getHeader("User-Agent");
    if (isAiBot(ua)) {
        ctx.response().setStatusCode(403).end("Blocked");
        return;
    }
    ctx.next();
});

Spring Boot — OncePerRequestFilter

// Blocking Servlet filter (thread-per-request)
@Component @Order(Ordered.HIGHEST_PRECEDENCE)
public class AiBotFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest req,
        HttpServletResponse res, FilterChain chain) {
        if (isAiBot(req.getHeader("User-Agent"))) {
            res.setStatus(403); res.getWriter().write("Blocked");
            return;
        }
        chain.doFilter(req, res);
    }
}

Akka HTTP — Directive

// Functional directive composition
val blockAiBots: Directive0 = extractRequest.flatMap { req =>
  val ua = req.header[headers.`User-Agent`]
    .map(_.value.toLowerCase).getOrElse("")
  if (aiBots.exists(ua.contains))
    complete(StatusCodes.Forbidden, "Blocked")
  else pass
}
// Usage: val route = blockAiBots { path("api") { ... } }

Play's EssentialFilter is unique: it can reject before body parsing (like Vert.x's handler short-circuit) but operates on a stream accumulator (like Akka HTTP's directive model). Spring Boot always parses the full request before the filter chain runs.

noai meta tag (Twirl templates)

Pass the robots directive to Twirl templates via request attributes:

<!-- app/views/main.scala.html -->
@(title: String)(content: Html)(implicit request: RequestHeader)

<!DOCTYPE html>
<html>
<head>
  <title>@title</title>
  <meta name="robots" content="noai, noimageai" />
</head>
<body>@content</body>
</html>

Since the EssentialFilter adds X-Robots-Tag to every response header, the meta tag is a belt-and-suspenders approach — crawlers that respect HTML meta directives will see it even if they ignore HTTP headers.

Testing

Use Play's WithApplication and FakeRequest for filter testing:

// test/filters/AiBotFilterSpec.scala
package filters

import org.scalatestplus.play._
import play.api.test._
import play.api.test.Helpers._

class AiBotFilterSpec extends PlaySpec with GuiceOneAppPerSuite {

  "AiBotFilter" must {

    "block GPTBot with 403" in {
      val request = FakeRequest(GET, "/api/articles")
        .withHeaders("User-Agent" -> "GPTBot/1.0")
      val result = route(app, request).get

      status(result) mustBe FORBIDDEN
      header("X-Robots-Tag", result) mustBe Some("noai, noimageai")
    }

    "block ClaudeBot with 403" in {
      val request = FakeRequest(GET, "/api/articles")
        .withHeaders("User-Agent" -> "ClaudeBot/2.0 (+https://anthropic.com)")
      val result = route(app, request).get

      status(result) mustBe FORBIDDEN
    }

    "allow browser with X-Robots-Tag" in {
      val request = FakeRequest(GET, "/api/articles")
        .withHeaders("User-Agent" -> "Mozilla/5.0 (compatible browser)")
      val result = route(app, request).get

      status(result) mustBe OK
      header("X-Robots-Tag", result) mustBe Some("noai, noimageai")
    }

    "serve robots.txt to all UAs" in {
      val request = FakeRequest(GET, "/robots.txt")
        .withHeaders("User-Agent" -> "GPTBot/1.0")
      val result = route(app, request).get

      status(result) mustBe OK
      contentAsString(result) must include("Disallow: /")
    }

    "handle missing User-Agent gracefully" in {
      val request = FakeRequest(GET, "/api/articles")
      val result = route(app, request).get

      status(result) mustBe OK // No UA = not a bot
    }
  }
}

AI bot User-Agent strings (2026)

GPTBotChatGPT-UserClaudeBotanthropic-aiCCBotcohere-aiBytespiderAmazonbotApplebot-ExtendedPerplexityBotYouBotDiffbotGoogle-ExtendedFacebookBotomgiliomgilibotDeepSeekBotMistralBotxAI-BotAI2Bot

Play Scala: requestHeader.headers.get("User-Agent") returns Option[String] — always pattern match or .getOrElse(""). Play Java: request.header("User-Agent") returns Optional<String>. Both are case-insensitive header lookups — "user-agent" works too.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.