How to Block AI Bots in Supabase Edge Functions: Complete 2026 Guide

Step 1 — Shared AI bot helper (_shared/ai-bots.ts)

Create supabase/functions/_shared/ai-bots.ts. Supabase bundles the _shared/ directory at deploy time — it is not deployed as its own function. Import it with a relative path from any function.

// supabase/functions/_shared/ai-bots.ts
export const AI_BOTS = [
  // OpenAI
  'gptbot', 'chatgpt-user', 'oai-searchbot',
  // Anthropic
  'claudebot', 'claude-web',
  // Common Crawl / CCBot
  'ccbot',
  // Bytedance
  'bytespider',
  // Meta
  'meta-externalagent',
  // Perplexity
  'perplexitybot',
  // Google AI
  'google-extended', 'googleother',
  // Cohere
  'cohere-ai',
  // Amazon
  'amazonbot',
  // Diffbot
  'diffbot',
  // AI2
  'ai2bot',
  // DeepSeek
  'deepseekbot',
  // Mistral
  'mistralai-user',
  // xAI
  'xai-bot',
  // You.com
  'youbot',
  // DuckDuckGo AI
  'duckassistbot',
  // Webzio
  'webzio',
] as const;

export function isAiBot(userAgent: string | null): boolean {
  if (!userAgent) return false;
  const ua = userAgent.toLowerCase();
  return AI_BOTS.some((bot) => ua.includes(bot));
}

Step 2 — Block AI bots in your edge function

The check runs at the top of Deno.serve() before any database access. A 403 response costs zero Supabase reads.

// supabase/functions/my-api/index.ts
import { isAiBot } from '../_shared/ai-bots.ts';

const ROBOTS_TXT = `User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /`;

Deno.serve(async (req: Request) => {
  // 1. Serve robots.txt (if this function handles the /robots.txt path)
  if (new URL(req.url).pathname.endsWith('/robots.txt')) {
    return new Response(ROBOTS_TXT, {
      headers: { 'Content-Type': 'text/plain; charset=utf-8' },
    });
  }

  // 2. Block AI bots — before any DB call or business logic
  if (isAiBot(req.headers.get('user-agent'))) {
    return new Response('Forbidden', { status: 403 });
  }

  // 3. Handle CORS preflight
  if (req.method === 'OPTIONS') {
    return new Response(null, {
      headers: {
        'Access-Control-Allow-Origin': '*',
        'Access-Control-Allow-Headers': 'authorization, x-client-info, apikey, content-type',
      },
    });
  }

  // 4. Your normal business logic
  const data = { message: 'Hello, architect.' };

  return new Response(JSON.stringify(data), {
    headers: {
      'Content-Type': 'application/json',
      // 5. X-Robots-Tag on all successful responses
      'X-Robots-Tag': 'noai, noimageai',
    },
  });
});

Step 3 — Dedicated robots function

Supabase Edge Functions have no filesystem at runtime — you cannot read a robots.txt file. Embed it as a string constant and serve it from a dedicated robots function. Route GET /robots.txt to this function via your CDN, Vercel rewrite, or Nginx proxy rule.

// supabase/functions/robots/index.ts
// Deploy and route GET /robots.txt → this function via your proxy/CDN

const ROBOTS_TXT = `User-agent: *
Allow: /

# AI training bots — blocked
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: AmazonBot
Disallow: /

User-agent: Diffbot
Disallow: /`;

Deno.serve((_req: Request) => {
  return new Response(ROBOTS_TXT, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=86400',
    },
  });
});

Step 4 — Dynamic block-list from a Supabase table

For rule updates without a redeploy, store UA patterns in a blocked_bots table and cache per isolate instance. The module-level cache avoids a DB hit on every request; a TTL forces a re-fetch after 5 minutes.

// supabase/functions/my-api/index.ts — dynamic block-list from Supabase table
import { createClient } from 'npm:@supabase/supabase-js@2';
import { AI_BOTS } from '../_shared/ai-bots.ts'; // fallback

const supabaseAdmin = createClient(
  Deno.env.get('SUPABASE_URL')!,
  Deno.env.get('SUPABASE_SERVICE_ROLE_KEY')!,
);

// Module-level cache — persists for the lifetime of this isolate instance
let cachedBotList: string[] | null = null;
let cacheExpiresAt = 0;
const CACHE_TTL_MS = 5 * 60 * 1000; // 5 minutes

async function getBotList(): Promise<string[]> {
  const now = Date.now();
  if (cachedBotList && now < cacheExpiresAt) return cachedBotList;

  try {
    const { data, error } = await supabaseAdmin
      .from('blocked_bots')
      .select('ua_pattern');

    if (error || !data?.length) {
      // Fall back to hardcoded list on DB error
      return [...AI_BOTS];
    }

    cachedBotList = data.map((r: { ua_pattern: string }) => r.ua_pattern.toLowerCase());
    cacheExpiresAt = now + CACHE_TTL_MS;
    return cachedBotList;
  } catch {
    return [...AI_BOTS];
  }
}

Deno.serve(async (req: Request) => {
  const ua = req.headers.get('user-agent')?.toLowerCase() ?? '';
  const botList = await getBotList();

  if (botList.some((pattern) => ua.includes(pattern))) {
    return new Response('Forbidden', { status: 403 });
  }

  // ... rest of your handler
  return new Response(JSON.stringify({ ok: true }), {
    headers: { 'Content-Type': 'application/json', 'X-Robots-Tag': 'noai, noimageai' },
  });
});

create table blocked_bots (
  id serial primary key,
  ua_pattern text not null unique,
  created_at timestamptz default now()
);
insert into blocked_bots (ua_pattern) values
  ('gptbot'), ('claudebot'), ('ccbot'), ('bytespider'),
  ('google-extended'), ('perplexitybot'), ('amazonbot');

Step 5 — noai meta tag + X-Robots-Tag for SSR functions

For edge functions that return HTML, add both the meta tag and the response header. Crawlers that bypass your UA check will still see the directive.

// supabase/functions/ssr-page/index.ts — X-Robots-Tag for every HTML response
Deno.serve(async (req: Request) => {
  if (isAiBot(req.headers.get('user-agent'))) {
    return new Response('Forbidden', { status: 403 });
  }

  const html = `<!DOCTYPE html>
<html>
  <head>
    <!-- noai meta tag for crawlers that reach this far -->
    <meta name="robots" content="noai, noimageai">
  </head>
  <body>...</body>
</html>`;

  return new Response(html, {
    headers: {
      'Content-Type': 'text/html; charset=utf-8',
      // Belt-and-suspenders: header + meta tag
      'X-Robots-Tag': 'noai, noimageai',
    },
  });
});

Supabase Edge vs Cloudflare Workers vs Vercel Edge vs Lambda@Edge

FAQ