Skip to content
.NET · Umbraco CMS · Razor Views

How to Block AI Bots on Umbraco

Umbraco is the most popular open-source .NET CMS — used by over 750,000 websites across enterprise, government, and agency projects. Built on ASP.NET Core, it combines a flexible content editing backoffice with Razor-templated front-end delivery and an optional headless Content Delivery API. Because Umbraco sits on ASP.NET Core, all standard ASP.NET Core middleware applies — but Umbraco's pipeline builder and composition system change how you register it. This guide covers robots.txt, noai meta in Razor layouts with per-document property overrides, X-Robots-Tag via middleware in the Umbraco program pipeline, hard 403 blocking, and Content Delivery API protection.

9 min readUpdated April 2026Umbraco 13 / 14

1. robots.txt

ASP.NET Core serves static files from the wwwroot/ directory via UseStaticFiles(), which is included in Umbraco's default pipeline. Place robots.txt there for zero-configuration static serving.

Option A: Static file in wwwroot/

Create wwwroot/robots.txt:

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Diffbot
Disallow: /

# Allow standard search crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

ASP.NET Core's static file middleware serves this at yoursite.com/robots.txt before Umbraco's routing layer is invoked — zero CMS overhead.

Option B: Surface Controller (dynamic, environment-aware)

For per-environment rules or multi-site Umbraco installations that need different robots.txt per domain, use an Umbraco Surface Controller:

// Controllers/Surface/RobotsController.cs
using Microsoft.AspNetCore.Mvc;
using Umbraco.Cms.Web.Website.Controllers;

namespace MyProject.Controllers.Surface;

public class RobotsController : SurfaceController
{
    private readonly IWebHostEnvironment _env;

    public RobotsController(
        IUmbracoContextAccessor umbracoContextAccessor,
        IUmbracoDatabaseFactory databaseFactory,
        ServiceContext services,
        AppCaches appCaches,
        IProfilingLogger profilingLogger,
        IPublishedUrlProvider publishedUrlProvider,
        IWebHostEnvironment env)
        : base(umbracoContextAccessor, databaseFactory, services,
               appCaches, profilingLogger, publishedUrlProvider)
    {
        _env = env;
    }

    [Route("/robots.txt")]
    [HttpGet]
    public ContentResult RobotsTxt()
    {
        if (!_env.IsProduction())
        {
            return Content(
                "User-agent: *\nDisallow: /\n",
                "text/plain",
                System.Text.Encoding.UTF8);
        }

        var content = @"User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /
";
        return Content(content, "text/plain", System.Text.Encoding.UTF8);
    }
}
Static file vs controller priority: If both wwwroot/robots.txt and a Surface Controller route exist, the static file wins — UseStaticFiles() runs before Umbraco routing. Remove wwwroot/robots.txt when using the controller approach, or use only the static file.

2. noai meta in Razor layouts

Umbraco pages are rendered by Razor templates. The base layout is typically at Views/Shared/_Layout.cshtml — add the noai meta tag here for site-wide coverage.

Hardcoded site-wide tag

@* Views/Shared/_Layout.cshtml *@
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">

    @* AI bot protection — applies to every page *@
    <meta name="robots" content="noai, noimageai">

    <title>@ViewBag.Title</title>

    @* Stylesheet links *@
</head>
<body>
    @RenderBody()
</body>
</html>

With per-document override

When a robotsTag Document Type property is available on the current page (see Section 3):

@* Views/Shared/_Layout.cshtml *@
@inherits Umbraco.Cms.Web.Common.Views.UmbracoViewPage

@{
    var robotsValue = Model.HasValue("robotsTag")
        ? Model.Value<string>("robotsTag")
        : "noai, noimageai";
}

<meta name="robots" content="@robotsValue">

Model.HasValue() checks whether the property has a value (not empty/null). If the editor left the field blank, the fallback noai, noimageai applies.

UmbracoViewPage: The @inherits Umbraco.Cms.Web.Common.Views.UmbracoViewPage directive gives your Razor view access to Model as an IPublishedContent instance — with HasValue(), Value(), and all Umbraco property accessors.

3. Per-document robots control

To give content editors per-page control over AI bot indexing, add a property to your Document Types in the Umbraco backoffice.

Adding the property

  1. Go to Settings → Document Types in the Umbraco backoffice
  2. Open your base Document Type (the one all pages inherit from)
  3. Add a new property: Name = “Robots Tag”, Alias = robotsTag, Editor = Text Box
  4. Set Description: “robots meta tag value. Leave blank for site default (noai, noimageai). Example: index, follow”
  5. Move it to the SEO tab if one exists, or create a new “SEO” tab

In view templates

In Razor templates (not the layout), you can also output this for specific page types:

@* Views/HomePage.cshtml *@
@inherits Umbraco.Cms.Web.Common.Views.UmbracoViewPage<ContentModels.HomePage>

@{
    Layout = "_Layout.cshtml";
    // robotsTag is read in _Layout.cshtml from Model — nothing needed here
}

Because Model in _Layout.cshtml refers to the current page, the robotsTag property is automatically available across all Document Types that have the property defined — editors can control it from the SEO tab.

4. X-Robots-Tag via middleware

Umbraco runs on ASP.NET Core — standard middleware works. The key difference from a plain ASP.NET Core app is where you register it in Program.cs.

Program.cs — Umbraco pipeline

// Program.cs
var builder = WebApplication.CreateBuilder(args);

builder
    .CreateUmbracoBuilder()
    .AddBackOffice()
    .AddWebsite()
    .AddDeliveryApi()   // if using headless API
    .Build();

var app = builder.Build();

await app.BootUmbracoAsync();

app.UseUmbraco()
    .WithMiddleware(u =>
    {
        u.UseBackOffice();
        u.UseWebsite();
    })
    .WithEndpoints(u =>
    {
        u.UseInstallerEndpoints();
        u.UseBackOfficeEndpoints();
        u.UseWebsiteEndpoints();
    });

await app.RunAsync();

Add your AI bot middleware after UseStaticFiles() (which is called inside Umbraco's pipeline) but using app.Use() before app.UseUmbraco(), or via the Umbraco composition system:

// Program.cs — add middleware before app.UseUmbraco()
app.Use(async (context, next) =>
{
    var path = context.Request.Path.Value ?? "";

    // Always allow robots.txt, sitemap.xml, backoffice
    if (path.StartsWith("/robots.txt") ||
        path.StartsWith("/sitemap") ||
        path.StartsWith("/umbraco"))
    {
        await next(context);
        return;
    }

    // Add X-Robots-Tag to HTML responses
    context.Response.OnStarting(() =>
    {
        var contentType = context.Response.ContentType ?? "";
        if (contentType.Contains("text/html"))
        {
            context.Response.Headers["X-Robots-Tag"] = "noai, noimageai";
        }
        return Task.CompletedTask;
    });

    await next(context);
});

app.UseUmbraco()
    // ...

Composition-based registration (reusable)

Umbraco's composition system lets you register middleware from anywhere in your project without editing Program.cs. Create a composer:

// Composers/AiBotComposer.cs
using Umbraco.Cms.Core.Composing;
using Umbraco.Cms.Core.DependencyInjection;

namespace MyProject.Composers;

public class AiBotComposer : IComposer
{
    public void Compose(IUmbracoBuilder builder)
    {
        builder.Services.AddTransient<AiBotMiddleware>();
    }
}

// Middleware/AiBotMiddleware.cs
namespace MyProject.Middleware;

public class AiBotMiddleware
{
    private readonly RequestDelegate _next;

    private static readonly string[] AiBotPatterns =
    [
        "GPTBot", "ClaudeBot", "Claude-Web", "anthropic-ai",
        "CCBot", "Google-Extended", "PerplexityBot", "Applebot-Extended",
        "Amazonbot", "meta-externalagent", "Bytespider", "Diffbot",
    ];

    private static readonly string[] ExemptPaths =
        ["/robots.txt", "/sitemap.xml", "/favicon.ico", "/umbraco"];

    public AiBotMiddleware(RequestDelegate next) => _next = next;

    public async Task InvokeAsync(HttpContext context)
    {
        var path = context.Request.Path.Value ?? "";

        if (ExemptPaths.Any(p => path.StartsWith(p, StringComparison.OrdinalIgnoreCase)))
        {
            await _next(context);
            return;
        }

        context.Response.OnStarting(() =>
        {
            if ((context.Response.ContentType ?? "").Contains("text/html"))
                context.Response.Headers["X-Robots-Tag"] = "noai, noimageai";
            return Task.CompletedTask;
        });

        await _next(context);
    }
}

Register in Program.cs:

// Program.cs — after app.BootUmbracoAsync()
app.UseMiddleware<MyProject.Middleware.AiBotMiddleware>();
app.UseUmbraco()
    // ...

5. Hard 403 blocking

Extend the middleware to reject AI crawlers before Umbraco renders the page:

// Middleware/AiBotMiddleware.cs
public async Task InvokeAsync(HttpContext context)
{
    var path = context.Request.Path.Value ?? "";

    if (ExemptPaths.Any(p => path.StartsWith(p, StringComparison.OrdinalIgnoreCase)))
    {
        await _next(context);
        return;
    }

    var ua = context.Request.Headers.UserAgent.ToString();

    if (IsAiBot(ua))
    {
        context.Response.StatusCode = 403;
        await context.Response.WriteAsync("Forbidden");
        return;   // Do NOT call _next — Umbraco never executes
    }

    context.Response.OnStarting(() =>
    {
        if ((context.Response.ContentType ?? "").Contains("text/html"))
            context.Response.Headers["X-Robots-Tag"] = "noai, noimageai";
        return Task.CompletedTask;
    });

    await _next(context);
}

private static bool IsAiBot(string ua)
{
    if (string.IsNullOrEmpty(ua)) return false;
    var lower = ua.ToLowerInvariant();
    return AiBotPatterns.Any(p => lower.Contains(p.ToLowerInvariant()));
}
Exempt the backoffice: The Umbraco backoffice runs at /umbraco by default. Always include /umbraco in your exempt paths list — otherwise editors could be blocked from logging into the CMS. If you've changed the backoffice path via appsettings.json (Umbraco:CMS:Path), update the exempt path accordingly.

6. Content Delivery API

Umbraco 12+ includes a built-in Content Delivery API (headless) that serves JSON at /umbraco/delivery/api/v2/. The middleware approach above also protects this endpoint — the middleware runs before all routing, including the Delivery API.

Exempt the Delivery API for legitimate use

If you want AI indexing tools (like AI search engines) to access your content via the Delivery API while blocking training crawlers from the HTML site:

private static readonly string[] ExemptPaths =
[
    "/robots.txt",
    "/sitemap.xml",
    "/favicon.ico",
    "/umbraco",                    // backoffice + delivery API
    "/umbraco/delivery/api",       // explicitly exempt Delivery API
];

Block the Delivery API separately

Conversely, to block AI bots from the Delivery API while leaving HTML pages accessible, add a specific check:

if (path.StartsWith("/umbraco/delivery/api", StringComparison.OrdinalIgnoreCase))
{
    var ua = context.Request.Headers.UserAgent.ToString();
    if (IsAiBot(ua))
    {
        context.Response.StatusCode = 403;
        await context.Response.WriteAsync("Forbidden");
        return;
    }
}
API Key protection: The Umbraco Content Delivery API supports API key authentication (Umbraco:CMS:DeliveryApi:PublicAccess: false). Requiring an API key already prevents anonymous crawlers from accessing content — a complementary approach to UA-based blocking.

7. Deployment

Umbraco runs on .NET 8+ and deploys to any platform that supports ASP.NET Core. All middleware runs at the application layer — no platform-specific configuration needed.

Platformrobots.txtMeta tagsX-Robots-TagHard 403
IIS (Windows)✓✓✓✓
Azure App Service✓✓✓✓
Umbraco Cloud✓✓✓✓
Docker / Linux✓✓✓✓
Nginx reverse proxy✓✓✓✓

Umbraco Cloud

Umbraco Cloud is the managed hosting platform for Umbraco. It runs on Azure and supports all ASP.NET Core middleware. Deploy your code via Git — the middleware is active immediately after deployment. Use Umbraco Cloud's environment-specific settings to block all crawlers on development/staging environments.

IIS — additional web.config header

For IIS deployments, you can also add the header via web.config as a belt-and-suspenders approach:

<!-- web.config -->
<system.webServer>
  <httpProtocol>
    <customHeaders>
      <add name="X-Robots-Tag" value="noai, noimageai" />
    </customHeaders>
  </httpProtocol>
</system.webServer>

FAQ

How do I add robots.txt to Umbraco?

Place robots.txt in wwwroot/ — ASP.NET Core's static file middleware (UseStaticFiles()) serves it at /robots.txt before Umbraco routing runs. For dynamic content (per-environment or per-site rules), create an Umbraco Surface Controller with a [Route("/robots.txt")] action that returns a ContentResult. Remove wwwroot/robots.txt when using the controller — static files take priority over routes.

How do I add noai meta tags to every Umbraco page?

Edit Views/Shared/_Layout.cshtml and add <meta name="robots" content="noai, noimageai"> inside <head>. For per-document control, add a Text Box property named robotsTag to your Document Types, then use Model.HasValue("robotsTag") ? Model.Value<string>("robotsTag") : "noai, noimageai" in the layout. Editors leave the field blank (default applies) or enter a specific value.

How do I register ASP.NET Core middleware in Umbraco?

In Program.cs, call app.UseMiddleware<YourMiddleware>() after await app.BootUmbracoAsync() and before app.UseUmbraco(). For reusable or package-style middleware, implement IComposer and register your middleware via builder.Services.AddTransient<YourMiddleware>() — Umbraco discovers composers automatically via reflection.

Does the AI bot middleware affect the Umbraco backoffice?

Only if you don't exempt it. Add /umbraco to your ExemptPaths array — the backoffice and Content Delivery API both live under this path by default. If you've customised the backoffice path in appsettings.json (Umbraco:CMS:Path), update the exempt path to match.

How do I protect the Umbraco Content Delivery API from AI bots?

The middleware approach in this guide runs before all routing — including the Delivery API at /umbraco/delivery/api/. AI bot requests to the API are blocked automatically if their UA matches. To allow the API while blocking the HTML site, add /umbraco/delivery/api to your exempt paths. For maximum control, enable API key authentication in appsettings.json: Umbraco:CMS:DeliveryApi:PublicAccess: false.

What is an Umbraco Surface Controller?

A Surface Controller inherits from SurfaceController and has access to Umbraco context — UmbracoContext, the current page, and Umbraco services. Use it when you need Umbraco data for a dynamic robots.txt (e.g., reading content from a global settings node, or serving per-site rules based on UmbracoContext.PublishedRequest). For static robots.txt, wwwroot/robots.txt is simpler.

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.