Mit dem Seobility Widget kannst Du Besuchern Deiner Website zeigen, wie gut Deine Seite SEO-optimiert ist. Das Widget zeigt immer den aktuellen SEO-Score Deiner Homepage/Startseite an.
# ============================================ # IdeaProof.io - Robots.txt # Last Updated: October 2026 # ============================================ # Allow all crawlers by default User-agent: * Allow: / Allow: /lists/ Allow: /guides/ Allow: /tools/ Allow: /versus/ Allow: /questions/ Allow: /calculators/ Allow: /blog/ Allow: /assets/ Allow: /fundraising-benchmarks Allow: /*.js Allow: /*.css # Protected areas - block from all crawlers # Aug 2026: /auth was still burning ~950 crawls/mo because the old rule only # matched the directory form, not /auth itself or /auth?redirect=... Disallow: /admin Disallow: /dashboard Disallow: /auth Disallow: /reset-password Disallow: /checkout Disallow: /payment-success Disallow: /api/internal/ Disallow: /*.json$ Disallow: /*?*apikey* Disallow: /*?*token* Disallow: /.env # ============================================ # LLM & AI CRAWLERS (Priority Access) # ============================================ # OpenAI (GPT, ChatGPT) User-agent: GPTBot Allow: / Allow: /blog/ Allow: /lists/ Allow: /features Allow: /pricing Allow: /about Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /admin # Anthropic (Claude) User-agent: anthropic-ai Allow: / Allow: /lists/ Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: ClaudeBot Allow: / Allow: /blog/ Allow: /lists/ Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: Claude-Web Allow: / # Google (Gemini, AI Overviews) User-agent: Google-Extended Allow: / Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: GoogleOther Allow: / Crawl-delay: 2 Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: Googlebot Allow: / # Perplexity User-agent: PerplexityBot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /admin # ============================================ # CRAWL-BUDGET CONTROL # Measured Aug 2026 (Hado AI analytics): these agents generate high # fetch volume with zero or near-zero citations / referral visits. # Throttled rather than blocked so brand mentions stay possible. # ============================================ # Meta (Llama AI) — 68k fetches, 0 citations in the last 30 days User-agent: Meta-ExternalAgent Crawl-delay: 30 Disallow: /questions/ Disallow: /versus/ Disallow: /calculators/ Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: FacebookBot Crawl-delay: 30 Disallow: /auth Disallow: /dashboard Disallow: /admin # Amazon (Alexa) — 14 fetches, 0 citations User-agent: Amazonbot Crawl-delay: 30 Disallow: /auth Disallow: /dashboard Disallow: /admin # Apple (Siri, Apple Intelligence) User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / Disallow: /admin/ Disallow: /auth Disallow: /dashboard Disallow: /admin # Microsoft (Copilot, Bing) — Copilot cites, keep fast User-agent: Bingbot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: msnbot Allow: / Crawl-delay: 5 # Cohere AI User-agent: cohere-ai Allow: / Crawl-delay: 10 Disallow: /auth Disallow: /dashboard Disallow: /admin # You.com — 486 pages indexed, 0 citations User-agent: YouBot Allow: / Crawl-delay: 10 Disallow: /auth Disallow: /dashboard Disallow: /admin # Common Crawl — training corpus, no referral traffic User-agent: CCBot Allow: /blog/ Allow: /lists/ Allow: /features Allow: /pricing Allow: /about Disallow: /questions/ Disallow: /versus/ Crawl-delay: 10 Disallow: /auth Disallow: /dashboard Disallow: /admin # Diffbot — extraction-only, no citation surface User-agent: Diffbot Disallow: / # xAI (Grok) User-agent: Grok Allow: / Crawl-delay: 2 Disallow: /auth Disallow: /dashboard Disallow: /admin # DeepSeek User-agent: DeepSeekBot Allow: / Crawl-delay: 2 Disallow: /auth Disallow: /dashboard Disallow: /admin # Mistral AI User-agent: MistralBot Allow: / Crawl-delay: 2 Disallow: /auth Disallow: /dashboard Disallow: /admin # ============================================ # SEO CRAWLERS (Rate Limited) # ============================================ User-agent: SemrushBot Allow: / Crawl-delay: 10 Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: AhrefsBot Allow: / Crawl-delay: 10 Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: MJ12bot Crawl-delay: 10 User-agent: DotBot Crawl-delay: 10 # ============================================ # AI / LLM INDEXES # ============================================ # Machine-readable site description and canonical URL index for AI agents: # https://ideaproof.io/llms.txt # https://ideaproof.io/llms-full.txt # Versioned per-block index endpoints (canonical): https://ideaproof.io/v5/llms-index-<block>.txt # Unversioned aliases stay live and always mirror the latest version. # Open datasets (CC BY 4.0): https://ideaproof.io/datasets # ============================================ # SITEMAPS # ============================================ # Primary sitemap (served via rewrite from ideaproof.io/sitemap.xml) Sitemap: https://ideaproof.io/sitemap.xml # LLM index endpoints (llms.txt + per-block subsets) Sitemap: https://ideaproof.io/sitemap-llms.xml # Backup sitemap (direct storage access) Sitemap: https://aioenivvgvtwyhpdioue.supabase.co/storage/v1/object/public/sitemaps/sitemap.xml
Folgende Keywords wurden erkannt. Überprüfe die Optimierung dieser Keywords für Deine Seite.
Bitte wähle eine der Optionen.
Weitere Hilfe findest Du im Erste Schritte Guide,
oder besuche unsere FAQ, unser Wiki oder unseren Blog.
Wir verwenden Cookies, damit unsere Website funktioniert und auch für Analyse- und Werbezwecke. Du kannst optionale Cookies selbstverständlich auch deaktivieren, siehe die folgenden Links für weitere Informationen.
Meta Title
(Nice to have)