+-----------------------------------------------------------------------------------+
|                        RFC 9309 AI PARSING PRECEDENCE PIPELINE                    |
|                                                                                   |
|  [ 1. USER-AGENT GROUP MATCHING ] ─────────────────────────────────────────────── |
|  * Crawler checks for an EXACT named block matching its User-Agent (e.g. GPTBot).  |
|  * IF MATCH FOUND ──> Crawler executes ONLY THAT BLOCK! Ignores User-agent: *!    |
|  * IF NO MATCH FOUND ──> Crawler falls back to the generic User-agent: * block.   |
|                                                                                   |
|  [ 2. RFC 9309 LONGEST-MATCH PATH PRECEDENCE RULE ] ───────────────────────────── |
|  * When Allow and Disallow directives conflict for a given URL path:             |
|  * THE DIRECTIVE WITH THE LONGEST PATH STRING ALWAYS WINS!                        |
|                                                                                   |
|  Example:                                                                         |
|    Disallow: /blog/                         (Path Length: 6 characters)           |
|    Allow: /blog/javascript-seo-guide        (Path Length: 25 characters - WINS!)  |
+-----------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------+
|                        THE 2026 AI CRAWLER TAXONOMY                               |
|                                                                                   |
|  [ GROUP A: REAL-TIME CONVERSATIONAL SEARCH RETRIEVAL BOTS (KEEP OPEN!) ]        |
|  * ChatGPT-User        ──> OpenAI live web search retrieval for user prompts.     |
|  * PerplexityBot       ──> Perplexity AI search retrieval and citation indexer.  |
|  * ClaudeBot / Anthropic-ai ──> Anthropic search retrieval and document citation.|
|  * Applebot-Extended   ──> Apple Intelligence & Siri web answer retrieval.       |
|                                                                                   |
|  [ GROUP B: FOUNDATION MODEL TRAINING SCRAPERS (OPTIONAL GOVERNANCE) ]            |
|  * GPTBot              ──> OpenAI training data ingestion crawler.                |
|  * Google-Extended     ──> Google Gemini & Vertex AI model training scraper.      |
|  * CCBot               ──> Common Crawl multi-LLM public training repository.     |
|  * Bytespider          ──> ByteDance / TikTok AI foundation model scraper.        |
+-----------------------------------------------------------------------------------+

+-----------------------------------------------------------------------------------+
|                        THE 4 FATAL ROBOTS.TXT SYNTAX TRAPS                        |
|                                                                                   |
|  1. THE EMPTY SPECIFIC BLOCK ────> Adding User-agent: GPTBot with no Allow rule.  |
|  2. THE OVERZEALOUS API BLOCK ───> Disallow: /api/ blocking SSR data endpoints.   |
|  3. MISSING SITEMAP DECLARATION ─> Omitting Sitemap: link chokes AI discovery.    |
|  4. LEADING WILDCARD MISMATCHES ─> Disallow: /*?* blocking clean canonical URLs. |
+-----------------------------------------------------------------------------------+

# BROKEN: Leaves GPTBot with no rules while blocking everyone else
User-agent: *
Disallow: /

User-agent: GPTBot
# (Empty - behavior is ambiguous and unpredictable across parsers!)

# FIXED: Explicit permissions for search bots
User-agent: *
Disallow: /admin/
Disallow: /checkout/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/
Disallow: /checkout/

# FIXED: Allow public asset endpoints while securing private mutations
User-agent: *
Allow: /_next/static/
Allow: /api/public/
Disallow: /api/auth/
Disallow: /api/user/

# scripts/audit_robots_txt.py
import urllib.robotparser
import requests

def audit_robots_permissions(domain: str, test_paths: list[str]):
    robots_url = f"{domain.rstrip('/')}/robots.txt"
    resp = requests.get(robots_url, timeout=5)
    
    if resp.status_code != 200:
        raise ValueError(f"CRITICAL: Failed to fetch {robots_url} (HTTP {resp.status_code})")
        
    ai_bots = ['GPTBot', 'ChatGPT-User', 'ClaudeBot', 'PerplexityBot', 'Applebot-Extended']
    results = {}
    
    for bot in ai_bots:
        parser = urllib.robotparser.RobotFileParser()
        parser.parse(resp.text.splitlines())
        results[bot] = {}
        
        for path in test_paths:
            full_url = f"{domain.rstrip('/')}{path}"
            is_allowed = parser.can_fetch(bot, full_url)
            results[bot][path] = "ALLOWED" if is_allowed else "BLOCKED"
            
    return results

if __name__ == "__main__":
    test_urls = ["/", "/blog/nextjs-15-seo-guide", "/pricing", "/api/auth"]
    audit = audit_robots_permissions("https://example.com", test_urls)
    for bot, paths in audit.items():
        print(f"\n[USER-AGENT: {bot}]")
        for path, status in paths.items():
            print(f"  {path}: {status}")

+-----------------------------------------------------------------------------------+
|                        ROBOTS.TXT VS X-ROBOTS-TAG INTERACTION                     |
|                                                                                   |
|  [ SCENARIO: BOT BLOCKED IN ROBOTS.TXT (Disallow: /private) ]                     |
|  * Crawler never fetches the URL; never reads HTML or HTTP response headers.       |
|  * If page has <meta name="robots" content="noindex">, BOT CANNOT READ IT!        |
|  * Result: Google might still index the bare URL as an un-described link!        |
|                                                                                   |
|  [ CORRECT REMOVAL PATTERN: ALLOW CRAWL + EMIT NOINDEX HEADER ]                   |
|  * robots.txt: Allow: /private                                                    |
|  * HTTP Response Header: X-Robots-Tag: noindex, nofollow                          |
|  * Crawler reads noindex instruction and purges URL from search index cleanly!    |
+-----------------------------------------------------------------------------------+

# Template A: Maximum AI Citability & Traffic
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /dashboard/
Disallow: /checkout/
Disallow: /private/

# Explicitly Permit AI Answer Engines
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Applebot-Extended
Allow: /
Disallow: /admin/
Disallow: /dashboard/

Sitemap: https://example.com/sitemap.xml

# Template B: AI Search Allowed, Training Blocked
User-agent: *
Allow: /
Disallow: /admin/

# Allow Real-Time Search Retrieval Bots
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: ClaudeBot
Allow: /

# Block Offline Foundation Model Training Scrapers
User-agent: GPTBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Anthropic-ai
User-agent: Bytespider
Disallow: /

Sitemap: https://example.com/sitemap.xml

+-----------------------------------------------------------------------------------+
|                        BUGVISO ROBOTS.TXT AUDITING PIPELINE                       |
|                                                                                   |
|  [ Web Application Submitted ] ──> [ FastAPI + ARQ Redis Worker Cluster ]         |
|                                         │                                         |
|                                         ▼                                         |
|  [ RFC 9309 COMPLIANT PARSING ENGINE ] ────────────────────────────────────────── |
|  * Simulates Googlebot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot in parallel|
|  * Tests 500+ Canonical URLs against Longest-Match Path Precedence rules          |
|  * Flags Accidental Disallows on High-Traffic Organic Landing Pages               |
|  * Validates Sitemap Header Directives and Syntax Validity                        |
|                                         │                                         |
|                                         ▼                                         |
|  [ 0-100 GEO CITABILITY SCORE + ACTIONABLE DEVELOPER REMEDIATION PLAYBOOK ]       |
+-----------------------------------------------------------------------------------+