How to Block AI Bots on Yii2 (PHP): Complete 2026 Guide

Layer 1: robots.txt

Yii2's document root is web/ — the folder containing index.php. Place robots.txt there:

# web/robots.txt

User-agent: *
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: CCBot
User-agent: cohere-ai
User-agent: Bytespider
User-agent: Amazonbot
User-agent: PerplexityBot
User-agent: YouBot
User-agent: Diffbot
User-agent: DeepSeekBot
User-agent: MistralBot
User-agent: xAI-Bot
User-agent: AI2Bot
Disallow: /

Primary approach: ActionFilter behavior

Create components/AiBotFilter.php extending yii\base\ActionFilter:

<?php
// components/AiBotFilter.php
namespace app\components;

use Yii;
use yii\base\ActionFilter;
use yii\base\Action;

class AiBotFilter extends ActionFilter
{
    private const AI_BOTS = [
        'gptbot', 'chatgpt-user', 'claudebot', 'anthropic-ai',
        'ccbot', 'cohere-ai', 'bytespider', 'amazonbot',
        'applebot-extended', 'perplexitybot', 'youbot', 'diffbot',
        'google-extended', 'deepseekbot', 'mistralbot', 'xai-bot',
        'ai2bot', 'oai-searchbot', 'duckassistbot',
    ];

    private const EXEMPT_PATHS = [
        '/robots.txt', '/sitemap.xml', '/favicon.ico',
    ];

    public function beforeAction(Action $action): bool
    {
        // Set noai meta for all requests (bots and browsers)
        Yii::$app->params['robots'] = 'noai, noimageai';

        // Exempt paths bypass bot blocking
        $path = '/' . Yii::$app->request->getPathInfo();
        if (in_array($path, self::EXEMPT_PATHS, true)) {
            return parent::beforeAction($action);
        }

        $ua = strtolower(Yii::$app->request->getUserAgent() ?? '');
        foreach (self::AI_BOTS as $bot) {
            if (str_contains($ua, $bot)) {
                // Set 403 response, send it, and exit
                $response = Yii::$app->response;
                $response->statusCode = 403;
                $response->content = 'Forbidden: AI crawlers are not permitted.';
                $response->send();
                Yii::$app->end();
                return false;
            }
        }

        return parent::beforeAction($action);
    }

    public function afterFilter(Action $action, mixed $result): mixed
    {
        // Add X-Robots-Tag to every outgoing response
        Yii::$app->response->headers->add('X-Robots-Tag', 'noai, noimageai');
        return parent::afterFilter($action, $result);
    }
}

Global registration — base controller

Yii2 doesn't support application-level behaviors directly. The cleanest global approach is a base controller that all controllers extend:

<?php
// controllers/BaseController.php (all your controllers extend this)
namespace app\controllers;

use app\components\AiBotFilter;
use yii\web\Controller;

class BaseController extends Controller
{
    public function behaviors(): array
    {
        return array_merge(parent::behaviors(), [
            'aiBotFilter' => [
                'class' => AiBotFilter::class,
            ],
        ]);
    }
}

// All controllers extend BaseController instead of yii\web\Controller:
// class SiteController extends BaseController { ... }
// class ArticleController extends BaseController { ... }

Alternatively, attach globally via the application's on beforeRequest event in config/web.php:

<?php
// config/web.php
use Yii;

return [
    // ...
    'on beforeRequest' => function () {
        $exemptPaths = ['/robots.txt', '/sitemap.xml', '/favicon.ico'];
        $aiBots = [
            'gptbot', 'chatgpt-user', 'claudebot', 'anthropic-ai',
            'ccbot', 'cohere-ai', 'bytespider', 'amazonbot',
            'perplexitybot', 'youbot', 'diffbot', 'google-extended',
            'deepseekbot', 'mistralbot', 'xai-bot', 'ai2bot',
        ];

        $path = '/' . Yii::$app->request->getPathInfo();
        if (in_array($path, $exemptPaths, true)) {
            return;
        }

        $ua = strtolower(Yii::$app->request->getUserAgent() ?? '');
        foreach ($aiBots as $bot) {
            if (str_contains($ua, $bot)) {
                Yii::$app->response->statusCode = 403;
                Yii::$app->response->content = 'Forbidden: AI crawlers are not permitted.';
                Yii::$app->response->send();
                Yii::$app->end();
            }
        }
    },
    'on afterRequest' => function () {
        Yii::$app->response->headers->add('X-Robots-Tag', 'noai, noimageai');
    },
];

The on beforeRequest event fires before routing — even earlier than beforeAction().

Controller-scoped blocking

To block only on specific controllers (e.g., API controllers), attach the filter only there. You can also restrict to specific actions using only and except:

<?php
// controllers/ApiController.php
class ApiController extends \yii\rest\Controller
{
    public function behaviors(): array
    {
        return array_merge(parent::behaviors(), [
            'aiBotFilter' => [
                'class' => AiBotFilter::class,
                // Optional: apply only to specific actions
                // 'only' => ['index', 'view'],
                // 'except' => ['health'],
            ],
        ]);
    }
}

Layer 2: noai meta tag

The filter sets Yii::$app->params['robots']. Read it in your Yii2 layout view:

<!-- views/layouts/main.php -->
<?php
$robots = Yii::$app->params['robots'] ?? 'noai, noimageai';
?>
<!DOCTYPE html>
<html>
<head>
    <meta name="robots" content="<?= \yii\helpers\Html::encode($robots) ?>">
    <!-- rest of head -->
</head>

Override per action in any controller:

// In a controller action — allow indexing for public content
public function actionIndex(): string
{
    Yii::$app->params['robots'] = 'index, follow';
    return $this->render('index');
}

Yii2 vs Laravel vs Symfony — PHP comparison

public function beforeAction(Action $action): bool
{
    if ($this->isAiBot(Yii::$app->request->getUserAgent())) {
        Yii::$app->response->statusCode = 403;
        Yii::$app->response->send();
        Yii::$app->end();
        return false;
    }
    return parent::beforeAction($action);
}

public function handle(Request $request, Closure $next): Response
{
    if ($this->isAiBot($request->userAgent())) {
        return response('Forbidden', 403);
    }
    return $next($request);
}

public function onKernelRequest(RequestEvent $event): void
{
    $ua = $event->getRequest()->headers->get('User-Agent', '');
    if ($this->isAiBot($ua)) {
        $event->setResponse(new Response('Forbidden', 403));
    }
}

public function process(
    ServerRequestInterface $request,
    RequestHandlerInterface $handler
): ResponseInterface {
    if ($this->isAiBot($request->getHeaderLine('User-Agent')))
        return (new Response())->withStatus(403)->withStringBody('Forbidden');
    return $handler->handle($request);
}

Testing

Use Yii2's built-in yii\codeception or plain PHPUnit with yii\web\Application:

<?php
// tests/unit/components/AiBotFilterTest.php
use yii\codeception\TestCase;

class AiBotFilterTest extends TestCase
{
    public $appConfig = '@tests/config/unit.php';

    public function testBlocksAiBot(): void
    {
        \Yii::$app->request->headers->set('User-Agent', 'GPTBot/1.0');
        // Test via controller action — expect 403
        $this->assertEquals(403, \Yii::$app->response->statusCode);
    }
}

// Or with Codeception functional test:
// $I->haveHttpHeader('User-Agent', 'GPTBot/1.0');
// $I->sendGET('/api/articles');
// $I->seeResponseCodeIs(403);

AI bot User-Agent strings (2026)