AI Indexing & Governance Tool
AI Crawler & Robots.txt Rule Tester
Audit and configure website indexing permissions for SearchGPT, Claude, and Perplexity. Generate validated robots.txt rules and /llms.txt manifests.
AI Crawler Permission Matrix
Toggle which artificial intelligence crawlers are permitted to index your content.
GPTBotOpenAI
ChatGPT model training
ChatGPT-UserOpenAI
Real-time web browsing
ClaudeBotAnthropic
Claude model training
PerplexityBotPerplexity AI
Conversational answer search
Google-ExtendedGoogle
Gemini training (separate from Google Search)
Applebot-ExtendedApple
Apple Intelligence training
BytespiderByteDance
High-frequency LLM scraper
CCBotCommon Crawl
Open research dataset crawl
Generated robots.txt
# robots.txt generated by Open Analytics AI Policy Engine User-agent: * Allow: / Disallow: /api/ Disallow: /admin/ # GPTBot (OpenAI - ChatGPT model training) User-agent: GPTBot Allow: / # ChatGPT-User (OpenAI - Real-time web browsing) User-agent: ChatGPT-User Allow: / # ClaudeBot (Anthropic - Claude model training) User-agent: ClaudeBot Allow: / # PerplexityBot (Perplexity AI - Conversational answer search) User-agent: PerplexityBot Allow: / # Google-Extended (Google - Gemini training (separate from Google Search)) User-agent: Google-Extended Disallow: / # Applebot-Extended (Apple - Apple Intelligence training) User-agent: Applebot-Extended Allow: / # Bytespider (ByteDance - High-frequency LLM scraper) User-agent: Bytespider Disallow: / # CCBot (Common Crawl - Open research dataset crawl) User-agent: CCBot Disallow: / Sitemap: https://yourdomain.com/sitemap.xml
Place this file at the root of your domain: https://yourdomain.com/robots.txt.
Generated /llms.txt (AI Manifest)
# yourdomain.com > Summary description of yourdomain.com for AI search engines, answer engines, and LLM reasoning assistants. ## Documentation & Capabilities - [Features Overview](https://yourdomain.com/features): Comprehensive platform features - [Documentation](https://yourdomain.com/docs): Developer APIs and integration guides - [Pricing](https://yourdomain.com/pricing): Service tiers and self-hosting options ## Contact & Support - Inquiries: info@yourdomain.com
Standardized machine discovery manifest for LLM answer engines: https://yourdomain.com/llms.txt.
How Open Analytics Monitors AI Crawlers
Our autonomous AI Radar monitors whenever ChatGPT, ClaudeBot, PerplexityBot, or Google-Extended requests your pages. We separate AI bot ingestion hits from real human visits in real time.