How is this different from Google Analytics?

Google Analytics shows you traffic. Shadow shows you traffic, AI bot activity, what AI platforms say about your brand, AND tells you what to do about all of it. It's analytics + AI intelligence + action steps in one tool.

Do I need to install anything?

For basic monitoring (bot detection, AI perception, readiness score) — nope, just enter your URL. For full visitor analytics (clicks, behavior, sessions), add one script tag. One-click integrations for Vercel, Shopify, WordPress, and more.

Will it slow down my site?

No. The script is under 5KB and loads async. Zero impact on page speed or Core Web Vitals. External monitoring has literally no impact — it watches from the outside.

What AI bots does Shadow detect?

All of them. GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Amazonbot, and dozens more. The Shadow Network means new bots get identified across all users instantly.

What do you mean by "actionable steps"?

Shadow doesn't just show you graphs. It says things like: "ChatGPT has your pricing wrong — add structured data to /pricing to fix it" or "Your bounce rate on /features is 68% — here's why and what to change." Specific, do-it-today recommendations.

Can Shadow block bots?

Shadow is a telescope, not a shield. It shows you who's visiting and what AI says about you. It generates block rules and robots.txt configs you can apply — but it doesn't intercept traffic.

Yes. Shadow never collects PII. IP addresses are hashed after classification. No cookies on your visitors. All Shadow Network data is anonymized. GDPR compliant by design.

What is Gemini-Deep-Research and what does it do?

Gemini-Deep-Research is a web crawler operated by Google to power the Deep Research feature available to Gemini Advanced subscribers. When a user runs Deep Research, the system autonomously browses the web — reading multiple pages, following links, and synthesizing content into a multi-page report. The crawler fetches your pages on demand to gather information relevant to a specific research query. It reads content in real time for individual user requests, not for bulk training data collection.

What's the difference between Gemini-Deep-Research and Google-Extended?

Google-Extended is Google's training data crawler — it harvests web content to train and improve Gemini AI models. Blocking Google-Extended prevents your content from being used in future Gemini model training. Gemini-Deep-Research is a retrieval crawler — it reads your pages on demand when a Gemini Advanced user runs a research query. Blocking Gemini-Deep-Research prevents your pages from appearing in Gemini Deep Research reports, but has no effect on model training. They are separate bots with separate user agent strings and separate robots.txt directives. You can block one without affecting the other.

Does blocking Gemini-Deep-Research affect my Google Search rankings?

No. Gemini-Deep-Research is not the Googlebot that crawls your site for Google Search. Your search rankings, indexing, and organic traffic are completely unaffected by blocking Gemini-Deep-Research. The only effect is on Gemini Advanced's Deep Research feature — your pages won't be fetched for user research queries.

What is Gemini Advanced Deep Research?

Deep Research is a feature in Google's Gemini Advanced (the paid $19.99/month tier). It allows users to give Gemini a complex research question and have it autonomously browse the web for several minutes — reading dozens of pages, following relevant links, and compiling a structured multi-page report with citations. It's positioned as an AI research assistant for competitive analysis, academic research, market studies, and detailed topic investigations. Gemini-Deep-Research is the crawler that powers this autonomous browsing.

Does Gemini-Deep-Research respect robots.txt?

Yes. Google documents Gemini-Deep-Research and it is designed to respect robots.txt directives. Google's general commitment to respecting robots.txt across all its documented crawlers applies here. Because it's a Google crawler with documented user agent tokens, robots.txt opt-out is a reliable and recognized mechanism.

Should I block Gemini-Deep-Research?

It depends on your content strategy. If you want to appear as a cited source in Gemini Deep Research reports — which are visible only to the individual user running the query — blocking means you're excluded from those reports. If you have paywalled content, proprietary research, or content you don't want summarized in AI research tools, blocking is appropriate. Unlike blocking GPTBot (which affects model training) or blocking Googlebot (which kills search rankings), blocking Gemini-Deep-Research has a limited, targeted effect: it only removes you from Gemini Advanced research queries.

What is the Gemini-Deep-Research user agent string?

The robots.txt token is: Gemini-Deep-Research. The full observed user agent string is: Mozilla/5.0 (compatible; Gemini-Deep-Research/1.0; +https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers). In robots.txt, use: User-agent: Gemini-Deep-Research followed by Disallow: /.

How many Google AI crawlers are there?

Google operates several distinct AI-related crawlers, each with a different purpose: Googlebot (main search indexer — do not block), Google-Extended (AI training for Gemini models — block if you want to opt out of training data), Gemini-Deep-Research (live retrieval for Deep Research queries — block to exclude from research reports), Google-NotebookLM (retrieval for NotebookLM AI notebook feature), GoogleAgent-Mariner (Google's AI agent/computer use feature), Google-Agent (general AI agent browsing). Each has its own robots.txt directive and blocking one has no effect on the others.

GoogleRespects robots.txtAI Research

How to Block Gemini-Deep-Research: Google's AI Research Crawler

Gemini-Deep-Research crawls your site when a Gemini Advanced subscriber runs a Deep Research query. It reads dozens of pages to compile multi-page research reports. Not the training crawler — that's Google-Extended.

Updated March 2026

Google's AI Crawler Family

Google operates multiple AI-specific crawlers. Each serves a different purpose and can be blocked independently:

GooglebotMain search indexer. Never block this unless you want to disappear from Google Search.

Google-ExtendedAI training crawler. Collects data to train and improve Gemini models.

Gemini-Deep-ResearchDeep Research retrieval. Fetches pages for multi-page research reports in Gemini Advanced.

Google-NotebookLMNotebookLM retrieval. Fetches pages when users add URLs as sources in NotebookLM.

What Is Deep Research?

Deep Research is a feature available to Gemini Advanced subscribers ($19.99/month). When a user submits a complex research question, Gemini autonomously browses the web for several minutes — reading dozens of pages, following relevant links, and synthesizing findings into a structured multi-page report with citations.

Common use cases include competitive analysis, academic literature reviews, market studies, and detailed topic investigations. The output is a comprehensive report visible only to the user who requested it.

Gemini-Deep-Research is the crawler that powers this feature. It fetches pages on demand during a research session, reading content in real-time to gather information. It is not a bulk training data harvester.

How to Block Gemini-Deep-Research

Add this to your robots.txt:

robots.txtBlock Deep Research only

User-agent: Gemini-Deep-Research
Disallow: /

To block all Google AI crawlers while keeping your search rankings:

robots.txtBlock all Google AI activity

# Keep Google Search indexing
User-agent: Googlebot
Allow: /

# Block AI training
User-agent: Google-Extended
Disallow: /

# Block Deep Research
User-agent: Gemini-Deep-Research
Disallow: /

# Block NotebookLM
User-agent: Google-NotebookLM
Disallow: /

Your Google Search rankings are safe

Blocking Gemini-Deep-Research, Google-Extended, or Google-NotebookLM has zero effect on your Google Search rankings. Only Googlebot controls search indexing.

Who Should Block Gemini-Deep-Research?

📊

Premium research publishers

If you sell research reports, market analysis, or proprietary data, Deep Research can compile your findings into a free report for the user — bypassing your paywall.

🎯

Competitive intelligence sites

Deep Research is specifically designed for competitive analysis. If your content is a competitive moat, it's a prime target for Deep Research queries.

⚖️

Legal or compliance-sensitive content

Regulated content that shouldn't be processed by external AI systems. Deep Research compiles content across sources, potentially combining sensitive information in new ways.

What Blocking Does (and Doesn't) Do

What it stops

• Your pages being read during Deep Research sessions
• Your content appearing in Deep Research reports
• Gemini Advanced users sourcing your site for research

What it doesn't stop

• Google-Extended training crawls (block separately)
• NotebookLM crawls (block Google-NotebookLM separately)
• Google Search rankings (completely unaffected)
• Gemini's existing knowledge from training data

Frequently Asked Questions

How is Deep Research different from a regular Gemini query?

A regular Gemini query uses the model's training data to answer. Deep Research actively browses the web in real-time — reading dozens of pages over several minutes — to compile a comprehensive report. The Gemini-Deep-Research crawler is what performs this live browsing.

Does Deep Research crawl my entire site?

Not systematically. It reads specific pages relevant to the user's research query, following links where relevant. It's not an index builder — it's a targeted reader. But for a single research session, it may read multiple pages from your site if they're all relevant to the query.

Is Deep Research content used for model training?

Google has not stated that content retrieved by Gemini-Deep-Research feeds back into Gemini model training. That's Google-Extended's purpose. However, Google's internal data practices are not fully transparent, so publishers who want certainty should block both crawlers.

Can I block Deep Research for specific paths only?

Yes. Instead of Disallow: / you can target specific paths: Disallow: /research/ or Disallow: /premium/ — allowing public content to be used in research reports while protecting gated content.

Related Guides

How to Block Google-Extended

Google's AI training crawler

How to Block Google NotebookLM

Google's notebook AI crawler

How to Block GPTBot

OpenAI's training crawler

robots.txt for AI Bots (Complete Guide)

51+ crawlers, full reference table

Is your site protected from AI bots?

Run a free scan to check your robots.txt, meta tags, and overall AI readiness score.