Skip to content
Deno · Fresh Framework · Island Architecture

How to Block AI Bots on Fresh (Deno)

Fresh is Deno's official web framework. It uses Preact for rendering, island architecture for partial hydration (only interactive components ship JavaScript), and file-based routing under routes/. Fresh runs a server at runtime — unlike static site generators, you have full middleware control over every request. This makes AI bot blocking straightforward: robots.txt, noai meta tags in layouts, response headers via middleware, and hard 403 rejection — all without relying on hosting platform features.

8 min readUpdated April 2026Fresh 1.x / 2.x

1. robots.txt

Fresh serves all files in the static/ directory at the root path. Place your robots.txt here for the simplest approach.

Static robots.txt

Create static/robots.txt:

# Block all AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow legitimate search crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

Fresh serves this directly from static/ — no build step required, no configuration needed. The file is available at yoursite.com/robots.txt immediately.

Dynamic robots.txt via route handler

For environment-aware content (e.g., blocking all crawlers on staging), create a route handler. Fresh route handlers take priority over static files at the same path.

Create routes/robots.txt.ts:

import { Handlers } from "$fresh/server.ts";

export const handler: Handlers = {
  GET(_req) {
    const isProduction = Deno.env.get("DENO_DEPLOYMENT_ID") !== undefined;

    if (!isProduction) {
      return new Response(
        "# Staging — block all crawlers\nUser-agent: *\nDisallow: /\n",
        { headers: { "content-type": "text/plain; charset=utf-8" } },
      );
    }

    const robotsTxt = `# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow search crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /
`;

    return new Response(robotsTxt, {
      headers: { "content-type": "text/plain; charset=utf-8" },
    });
  },
};
Route vs static priority: If both routes/robots.txt.ts and static/robots.txt exist, the route handler wins. Remove the static file to avoid confusion, or keep the static version as a fallback and rely on the route handler for dynamic logic.
DENO_DEPLOYMENT_ID: This environment variable is automatically set on Deno Deploy. Checking for its presence is the idiomatic way to detect a production deployment in Fresh — no need to set custom env vars.

2. noai meta tags in Preact layouts

The noai and noimageai meta values signal to AI crawlers that page content and images should not be used for training. Add them to your app layout so every page is covered.

App-wide layout

Fresh uses routes/_app.tsx as the root layout for all pages. This is where you define the <html>, <head>, and <body> structure:

// routes/_app.tsx
import { type PageProps } from "$fresh/server.ts";

export default function App({ Component }: PageProps) {
  return (
    <html lang="en">
      <head>
        <meta charset="utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />

        {/* AI bot protection — applies to all pages */}
        <meta name="robots" content="noai, noimageai" />

        <title>My Fresh Site</title>
      </head>
      <body>
        <Component />
      </body>
    </html>
  );
}

Per-route override with Head component

To allow individual pages to override the robots meta, use the <Head> component from $fresh/runtime.ts:

// routes/blog/[slug].tsx
import { Head } from "$fresh/runtime.ts";
import { type PageProps } from "$fresh/server.ts";

export default function BlogPost({ data }: PageProps) {
  return (
    <>
      <Head>
        {/* Override: allow indexing for this page */}
        <meta name="robots" content="index, follow" />
        <title>{data.title}</title>
      </Head>
      <article>
        <h1>{data.title}</h1>
        {/* ... */}
      </article>
    </>
  );
}
Head merging behaviour: When a route component renders a <Head> component, Fresh injects those tags into the <head> of the response. If both _app.tsx and a route define a robots meta tag, both will appear in the HTML. Browsers use the last matching meta tag, so the per-route override works — but validate with View Source to confirm the tag order is correct.

Data-driven meta from handler

For pages where the robots value comes from a database or CMS, use a route handler to pass data to the component:

// routes/page/[id].tsx
import { Head } from "$fresh/runtime.ts";
import { type Handlers, type PageProps } from "$fresh/server.ts";

interface PageData {
  title: string;
  robots: string;
  content: string;
}

export const handler: Handlers<PageData> = {
  async GET(_req, ctx) {
    const page = await fetchPage(ctx.params.id);
    if (!page) return ctx.renderNotFound();
    return ctx.render(page);
  },
};

export default function DynamicPage({ data }: PageProps<PageData>) {
  return (
    <>
      <Head>
        <meta name="robots" content={data.robots || "noai, noimageai"} />
        <title>{data.title}</title>
      </Head>
      <article dangerouslySetInnerHTML={{ __html: data.content }} />
    </>
  );
}

3. X-Robots-Tag via middleware

HTTP response headers are the most reliable signal for crawlers that ignore meta tags. Fresh middleware runs server-side on every request — ideal for setting headers globally.

Global middleware

Create routes/_middleware.ts:

// routes/_middleware.ts
import { FreshContext } from "$fresh/server.ts";

export async function handler(
  req: Request,
  ctx: FreshContext,
): Promise<Response> {
  const resp = await ctx.next();
  resp.headers.set("X-Robots-Tag", "noai, noimageai");
  return resp;
}

This adds the X-Robots-Tag header to every response — HTML pages, API routes, and static file requests that pass through the Fresh server.

Middleware scope: A _middleware.ts file applies to all routes in its directory and all subdirectories. Place it in routes/ for global coverage, or in routes/blog/ to only affect blog routes.

Selective headers — exclude API routes

If you have API routes that should not receive the header (e.g., JSON endpoints consumed by your own frontend):

// routes/_middleware.ts
import { FreshContext } from "$fresh/server.ts";

export async function handler(
  req: Request,
  ctx: FreshContext,
): Promise<Response> {
  const resp = await ctx.next();

  // Only add header to HTML responses
  const contentType = resp.headers.get("content-type") || "";
  if (contentType.includes("text/html")) {
    resp.headers.set("X-Robots-Tag", "noai, noimageai");
  }

  return resp;
}

4. Hard 403 via middleware

For aggressive blocking — reject known AI bot requests before they reach your route handlers. This is the most effective layer because the bot receives no content at all.

// routes/_middleware.ts
import { FreshContext } from "$fresh/server.ts";

const AI_BOT_PATTERNS = [
  /GPTBot/i,
  /ClaudeBot/i,
  /Claude-Web/i,
  /anthropic-ai/i,
  /CCBot/i,
  /Google-Extended/i,
  /PerplexityBot/i,
  /Applebot-Extended/i,
  /Amazonbot/i,
  /meta-externalagent/i,
  /Bytespider/i,
  /Diffbot/i,
  /YouBot/i,
  /cohere-ai/i,
];

function isAIBot(userAgent: string): boolean {
  return AI_BOT_PATTERNS.some((pattern) => pattern.test(userAgent));
}

export async function handler(
  req: Request,
  ctx: FreshContext,
): Promise<Response> {
  const ua = req.headers.get("user-agent") || "";

  if (isAIBot(ua)) {
    return new Response("Forbidden", { status: 403 });
  }

  const resp = await ctx.next();
  resp.headers.set("X-Robots-Tag", "noai, noimageai");
  return resp;
}
Combined middleware: This example combines the hard 403 block with the X-Robots-Tag header in a single middleware file. The 403 check runs first — if the request is from an AI bot, it returns immediately without calling ctx.next(). Legitimate requests continue through the chain and receive the header on the response.

Middleware chain order

Fresh supports multiple middleware files. A routes/_middleware.ts runs before routes/blog/_middleware.ts, which runs before the route handler. Place the bot-blocking middleware at the root level (routes/_middleware.ts) so it intercepts all requests before any other middleware or handler executes.

Logging blocked requests

To monitor how many AI bots are hitting your site, add logging before the 403 response:

if (isAIBot(ua)) {
  console.log(`[blocked] ${new Date().toISOString()} ${ua} ${req.url}`);
  return new Response("Forbidden", { status: 403 });
}

On Deno Deploy, these logs appear in the project dashboard under Logs. Locally, they print to the terminal running deno task start.

5. Islands and client-side considerations

Fresh's island architecture means most of your page is server-rendered HTML with zero JavaScript. Only components in the islands/ directory ship JavaScript to the client. This has implications for AI bot protection:

SSR advantage: Unlike client-side frameworks (React SPA, vanilla Preact), Fresh always sends complete HTML. This means robots meta tags and content are visible to crawlers immediately — no rendering pipeline to worry about.

6. Deno Deploy configuration

Deno Deploy is the primary deployment target for Fresh. All middleware, route handlers, and static file serving work identically — no adapter or special configuration needed.

Deploy setup

  1. Push your Fresh project to GitHub
  2. Connect the repository in the Deno Deploy dashboard (dash.deno.com)
  3. Set the entry point to main.ts
  4. Deploy — middleware runs on every request at the edge

Environment variables

If your robots.txt route handler checks for environment variables, set them in the Deno Deploy dashboard under Settings → Environment Variables. The DENO_DEPLOYMENT_ID variable is set automatically on every deployment.

Custom headers via deployctl

If deploying via deployctl (CI/CD), the same middleware applies. No additional header configuration is needed at the platform level — Fresh middleware handles everything:

# .github/workflows/deploy.yml
- name: Deploy to Deno Deploy
  uses: denoland/deployctl@v1
  with:
    project: my-fresh-site
    entrypoint: main.ts

7. Deployment comparison

Fresh runs a Deno server — it needs a runtime environment, not just static file hosting. Here's how AI bot protection features work across deployment targets:

Platformrobots.txtMeta tagsX-Robots-TagHard 403
Deno Deploy✓✓✓✓
Docker / self-hosted✓✓✓✓
Fly.io✓✓✓✓
Railway✓✓✓✓
AWS Lambda✓✓✓✓

Because Fresh includes a runtime server, all four protection layers work on every platform that can run Deno. There is no “static hosting” limitation like with SSGs — middleware executes on every request regardless of the host.

FAQ

How do I add robots.txt to a Fresh project?

Place robots.txt in the static/ directory. Fresh serves all files in static/ at the root path automatically — no configuration needed. For dynamic content (environment-specific rules), create routes/robots.txt.ts with a GET handler that returns a new Response() with text/plain content type.

How do I add noai meta tags in Fresh?

In routes/_app.tsx, add <meta name="robots" content="noai, noimageai" /> inside the <head> element. This applies to every page. For per-route overrides, use the <Head> component from $fresh/runtime.ts in individual route components.

How does Fresh middleware work for AI bot blocking?

Create routes/_middleware.ts and export an async handler function. The function receives the Request and a FreshContext with ctx.next() to continue the middleware chain. For headers: call ctx.next(), get the response, set headers, return it. For blocking: check the User-Agent and return new Response("Forbidden", { status: 403 }) before calling ctx.next() — the route handler never executes.

Does Fresh have a build step that affects robots.txt?

Fresh 1.x had no build step — files in static/ were served directly at runtime. Fresh 2.x added an optional build step (deno task build), but static/ files are still served from the directory directly in both modes. Your robots.txt works without any build configuration in either version.

How is Fresh different from Lume for AI bot blocking?

Lume is a static site generator — it outputs HTML files with no runtime server. AI bot blocking in Lume depends on hosting platform features (Netlify Edge Functions, Vercel middleware, etc.). Fresh has a runtime server with built-in middleware, so all blocking happens in your application code. You don't need platform-specific configuration — the same _middleware.ts works on Deno Deploy, Docker, Fly.io, or any Deno runtime.

Will blocking AI bots affect my SEO?

Blocking AI-specific crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) does not affect standard search engine indexing. Googlebot and Bingbot are separate user agents from Google-Extended and are not blocked by the configurations in this guide. Always include explicit Allow rules for Googlebot and Bingbot in your robots.txt to make your intent unambiguous.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.