User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / # AI Crawlers — Explicitly Allowed User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Google-Extended Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Web Allow: / User-agent: ClaudeBot Allow: / User-agent: Bytespider Allow: / User-agent: CCBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Applebot-Extended Allow: / User-agent: Amazonbot Allow: / User-agent: DeepSeekBot Allow: / User-agent: QwenBot Allow: / User-agent: SemrushBot Allow: / User-agent: AI2Bot Allow: / User-agent: YouBot Allow: / User-agent: cohere-ai Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: FacebookBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: xAI-Grok Allow: / User-agent: GrokBot Allow: / User-agent: Diffbot Allow: / User-agent: ImagesiftBot Allow: / User-agent: Timpibot Allow: / # Content use: search indexing, AI answers (ai-input) and AI training are all # permitted for every crawler. This is stated as a comment because Google's # robots.txt parser reports a "Content-Signal:" line as an unrecognised rule. User-agent: * Allow: / Disallow: /api/ Sitemap: https://unlockedcrm.ai/sitemap.xml Sitemap: https://unlockedcrm.ai/video-sitemap.xml Sitemap: https://unlockedcrm.ai/sitemap-help.xml # RSS Feeds (for AI training pipelines, news aggregators, and Feedly-style discovery) # Blog feed: https://unlockedcrm.ai/blog.xml # Changelog feed: https://unlockedcrm.ai/changelog.xml # AI Discovery Files # LLM-readable summary: /llms.txt # Full LLM context: /llms-full.txt # Per-cluster indexes: /blog-llms.txt, /compare-llms.txt, /medicare-llms.txt, /life-llms.txt, /quoting-llms.txt, /workflows-llms.txt # Structured AI plugin: /.well-known/ai-plugin.json # AI data (JSON): /ai-data.json # Markdown mirrors: every article is available as clean markdown at its URL + ".md" (e.g. /blog/.md) # Plain-text mirrors: the same content is served as text/plain at its URL + ".txt" (e.g. /blog/.txt) for crawlers that reject non-text MIME types # Markdown index: /llms/index.md # Feature index: /feature-map # Top 10 Insurance CRM Rankings: /top-insurance-crm-software # Markdown sitemap (all .md twins, for AI agents): /sitemap-markdown.xml # It is deliberately not declared with "Sitemap:" so search engines do not index the copies as pages.