How to Block AI Bots on .NET Minimal API: Complete 2026 Guide
.NET Minimal API (introduced in .NET 6) replaces Startup.cs and controllers with a single Program.csand lambda route handlers. It has two blocking mechanisms: app.Use() middleware for global coverage and IEndpointFilter (.NET 7+) for per-endpoint or per-group control.
Middleware vs IEndpointFilter — when to use which
app.Use() middleware
- ✓ Runs on ALL requests (incl. 404 probes)
- ✓ Fires before routing
- ✓ Catches bots probing non-existent paths
- ✗ Runs even for static file requests
IEndpointFilter
- ✓ Only runs on matched endpoints
- ✓ Has access to typed endpoint args
- ✓ Per-group via MapGroup()
- ✗ Doesn't catch 404 probing
Recommended: use both — middleware for global coverage, endpoint filter for scoped reinforcement.
Protection layers
1
robots.txt — app.UseStaticFiles() with wwwroot/robots.txt — call before bot-blocker middleware
2
noai meta tag — In Results.Content() HTML responses, Razor Pages layout, or Blazor HeadContent
3
X-Robots-Tag header — context.Response.OnStarting() in middleware — fires before headers are sent on all responses
4
Hard 403 — app.Use() (global) — Inline lambda or AiBotMiddleware class — fires on all requests before routing
5
Hard 403 — IEndpointFilter (scoped) — MapGroup("/api").AddEndpointFilter<AiBotEndpointFilter>() — only matched /api/* endpoints
Project setup (.csproj + full Program.cs)
No extra NuGet packages needed — bot blocking uses only the built-in ASP.NET Core middleware pipeline.
<!-- MyApp.csproj — .NET 8 Minimal API, no extra packages needed -->
<Project Sdk="Microsoft.NET.Sdk.Web">
<PropertyGroup>
<TargetFramework>net8.0</TargetFramework>
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
</PropertyGroup>
</Project>
// Program.cs — full minimal structure
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
app.UseStaticFiles(); // robots.txt from wwwroot/
app.UseMiddleware<AiBotMiddleware>(); // global bot blocker
var api = app.MapGroup("/api")
.AddEndpointFilter<AiBotEndpointFilter>(); // belt + suspenders on /api
api.MapGet("/data", () => Results.Json(new { data = "ok" }));
app.MapGet("/", () => Results.Content("<html>...</html>", "text/html"));
app.MapGet("/health", () => "ok");
app.Run();Step 1 — Shared bot list (AiBots.cs)
A static utility class with a readonly array — allocated once at startup. ToLowerInvariant() handles all User-Agent capitalisation variants. LINQ Any() short-circuits on first match.
// AiBots.cs — shared bot detection utility
namespace MyApp;
public static class AiBots
{
private static readonly string[] Patterns =
[
// OpenAI
"gptbot", "chatgpt-user", "oai-searchbot",
// Anthropic
"claudebot", "claude-web",
// Common Crawl
"ccbot",
// Bytedance
"bytespider",
// Meta
"meta-externalagent",
// Perplexity
"perplexitybot",
// Google AI
"google-extended", "googleother",
// Cohere
"cohere-ai",
// Amazon
"amazonbot",
// Diffbot
"diffbot",
// AI2
"ai2bot",
// DeepSeek
"deepseekbot",
// Mistral
"mistralai-user",
// xAI
"xai-bot",
// You.com
"youbot",
// DuckDuckGo AI
"duckassistbot",
];
public static bool IsAiBot(string? userAgent)
{
if (string.IsNullOrEmpty(userAgent)) return false;
var ua = userAgent.ToLowerInvariant();
return Patterns.Any(p => ua.Contains(p));
}
}Step 2 — Global middleware via app.Use() (recommended)
The short-circuit pattern: write the response and return without calling await next(context). The entire downstream pipeline — routing, endpoint filters, handlers — never runs. Use Response.OnStarting() to append X-Robots-Tag on all legitimate responses without duplicating the header on blocked responses.
// Program.cs — inline middleware (global, all requests)
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
// 1. Static files FIRST — robots.txt served before bot-blocker middleware
// Place robots.txt in wwwroot/robots.txt
app.UseStaticFiles();
// 2. Inline middleware — runs on EVERY request including unmatched routes
app.Use(async (context, next) =>
{
var ua = context.Request.Headers["User-Agent"].ToString();
if (AiBots.IsAiBot(ua))
{
// Short-circuit — don't call next(context)
// Handler, downstream middleware, and endpoint routing never run.
context.Response.StatusCode = 403;
context.Response.Headers.Append("Content-Type", "text/plain; charset=utf-8");
context.Response.Headers.Append("X-Robots-Tag", "noai, noimageai");
await context.Response.WriteAsync("Forbidden");
return; // ← do not call next
}
// Add X-Robots-Tag to all legitimate responses
context.Response.OnStarting(() =>
{
context.Response.Headers.Append("X-Robots-Tag", "noai, noimageai");
return Task.CompletedTask;
});
await next(context);
});
// 3. Route endpoints — only reached for legitimate requests
app.MapGet("/", () => "Welcome");
app.MapGet("/health", () => Results.Ok("ok"));
app.MapGet("/api/data", () => Results.Json(new { data = "protected" }));
app.Run();Step 3 — Class-based middleware (reusable)
For cross-project reuse, extract the logic into a class with InvokeAsync(HttpContext). Use primary constructor syntax ((RequestDelegate next)) for the concise .NET 8 style. Register with app.UseMiddleware<AiBotMiddleware>().
// AiBotMiddleware.cs — class-based middleware (reusable across projects)
namespace MyApp;
public class AiBotMiddleware(RequestDelegate next)
{
public async Task InvokeAsync(HttpContext context)
{
var ua = context.Request.Headers["User-Agent"].ToString();
if (AiBots.IsAiBot(ua))
{
context.Response.StatusCode = StatusCodes.Status403Forbidden;
context.Response.Headers.Append("X-Robots-Tag", "noai, noimageai");
context.Response.ContentType = "text/plain; charset=utf-8";
await context.Response.WriteAsync("Forbidden");
return; // short-circuit — don't call _next
}
// X-Robots-Tag on all legitimate responses
context.Response.OnStarting(() =>
{
context.Response.Headers.Append("X-Robots-Tag", "noai, noimageai");
return Task.CompletedTask;
});
await next(context);
}
}
// Program.cs — register the class-based middleware
// app.UseMiddleware<AiBotMiddleware>();
// Must be after app.UseStaticFiles() and before app.MapGet() calls.Step 4 — IEndpointFilter for per-group blocking
IEndpointFilter is Minimal API-specific — it only runs on matched endpoints and has access to the endpoint's parsed argument list. Apply to an entire route group with MapGroup().AddEndpointFilter<T>(). Combine with the global middleware for belt-and-suspenders coverage.
// AiBotEndpointFilter.cs — IEndpointFilter (.NET 7+, Minimal API only)
// Unlike middleware, filters only run on MATCHED endpoints.
// They don't run for 404s, static files, or unmatched paths.
namespace MyApp;
public class AiBotEndpointFilter : IEndpointFilter
{
public async ValueTask<object?> InvokeAsync(
EndpointFilterInvocationContext context,
EndpointFilterDelegate next)
{
var ua = context.HttpContext.Request.Headers["User-Agent"].ToString();
if (AiBots.IsAiBot(ua))
{
// Return a result directly — endpoint handler never runs
context.HttpContext.Response.Headers.Append("X-Robots-Tag", "noai, noimageai");
return Results.StatusCode(403);
}
// Pass through to the endpoint handler
return await next(context);
}
}
// Program.cs — apply the filter:
// Option A: Per-endpoint
app.MapGet("/sensitive", () => "Secret data")
.AddEndpointFilter<AiBotEndpointFilter>();
// Option B: MapGroup — applies to all routes in the group
var apiGroup = app.MapGroup("/api")
.AddEndpointFilter<AiBotEndpointFilter>();
apiGroup.MapGet("/data", () => Results.Json(new { data = "protected" }));
apiGroup.MapGet("/users", () => Results.Json(Array.Empty<object>()));
apiGroup.MapPost("/upload", (HttpContext ctx) => Results.Ok());
// Public routes outside the group are unaffected:
app.MapGet("/health", () => "ok");
app.MapGet("/", () => "Welcome");Step 5 — robots.txt
Call app.UseStaticFiles() before app.UseMiddleware<AiBotMiddleware>() in Program.cs. Static file requests are handled and short-circuited before the bot-blocker middleware fires.
// wwwroot/robots.txt — place in your project's wwwroot directory
// app.UseStaticFiles() serves it automatically at GET /robots.txt.
// UseStaticFiles() must be called BEFORE UseMiddleware or app.Use() for the
// bot-blocker so robots.txt is served without hitting the blocker.
User-agent: *
Allow: /
# AI training bots — blocked
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: YouBot
Disallow: /
User-agent: AmazonBot
Disallow: /
User-agent: Diffbot
Disallow: /
// Alternative: explicit endpoint if you don't want wwwroot
app.MapGet("/robots.txt", () =>
Results.Content(
content: """
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
""",
contentType: "text/plain; charset=utf-8"));Step 6 — noai meta tag
// noai meta tag in .NET Minimal API HTML responses
// Option A: String literal return
app.MapGet("/", () => Results.Content(
"""
<!DOCTYPE html>
<html>
<head>
<meta name="robots" content="noai, noimageai">
<title>My Site</title>
</head>
<body><h1>Welcome</h1></body>
</html>
""",
"text/html; charset=utf-8"));
// Option B: Razor Pages (if added to the project)
// In _Layout.cshtml or Layout.razor:
// <meta name="robots" content="noai, noimageai">
// The X-Robots-Tag middleware handles the HTTP header layer.
// Option C: Blazor Server/WASM — add to App.razor head section:
// <HeadContent>
// <meta name="robots" content="noai, noimageai">
// </HeadContent>.NET Minimal API vs ASP.NET Core MVC vs FastEndpoints vs Carter
| Feature | Minimal API | ASP.NET Core MVC | FastEndpoints | Carter |
|---|---|---|---|---|
| Middleware model | app.Use() inline lambda or class with InvokeAsync(HttpContext, RequestDelegate) | Same middleware pipeline — IMiddleware or convention-based middleware class | Pre/post-processor pipeline: IPreProcessor<TRequest> per endpoint | Same ASP.NET Core middleware — Carter is a thin routing layer over Minimal API |
| Can abort request? | Yes — omit await next(context); write response and return | Yes — same middleware short-circuit pattern | Yes — return StopProcessing in IPreProcessor to short-circuit | Yes — same middleware short-circuit |
| Global vs scoped | app.Use() = global. MapGroup().AddEndpointFilter() = scoped to group | app.Use() = global. [ServiceFilter] attribute on controller = scoped | Global pre-processors via AddGlobalPreProcessor or per-endpoint | app.Use() = global. Carter modules can have per-module middleware |
| Minimal API-specific | IEndpointFilter — .NET 7+, only runs on matched endpoints, not 404s | IActionFilter — MVC-specific, only runs on matched controller actions | IPreProcessor<T> — typed, per-endpoint, has access to parsed request | No unique filter — uses standard IEndpointFilter or middleware |
| UA header access | context.Request.Headers["User-Agent"].ToString() | Request.Headers["User-Agent"].ToString() in controller or middleware | ctx.HttpContext.Request.Headers["User-Agent"].ToString() in pre-processor | context.Request.Headers["User-Agent"].ToString() — identical |
| Hard 403 | context.Response.StatusCode = 403; await WriteAsync(); return | context.Result = new StatusCodeResult(403); in action filter | await ctx.Response.SendForbiddenAsync() in pre-processor | Results.StatusCode(403) or context.Response.StatusCode = 403 |
| robots.txt | app.UseStaticFiles() + wwwroot/robots.txt or explicit MapGet | Same UseStaticFiles() approach — identical | Same UseStaticFiles() — FastEndpoints wraps Minimal API | Same UseStaticFiles() — Carter wraps Minimal API routing |
Summary
- app.Use() + return (no next) — global, catches all requests including 404 probing. Use as primary blocker.
- IEndpointFilter + MapGroup — scoped to matched routes. Belt-and-suspenders on protected groups like
/api. - UseStaticFiles() before middleware — robots.txt must be served before the bot-blocker sees the request.
- Response.OnStarting() — the right place to append X-Robots-Tag to all responses; fires just before headers are flushed.
- .ToString() on StringValues — always call it when reading headers; avoids silent type comparison bugs.
Is your site protected from AI bots?
Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.