Muestra a tus visitantes lo bien optimizado que está tu sitio web con el widget de Seobility. La herramienta presenta la puntuación SEO actual de tu Homepage.
# ============================================ # IdeaProof.io - Robots.txt # Last Updated: October 2026 # ============================================ # Allow all crawlers by default User-agent: * Allow: / Allow: /lists/ Allow: /guides/ Allow: /tools/ Allow: /versus/ Allow: /questions/ Allow: /calculators/ Allow: /blog/ Allow: /assets/ Allow: /fundraising-benchmarks Allow: /*.js Allow: /*.css # Protected areas - block from all crawlers # Aug 2026: /auth was still burning ~950 crawls/mo because the old rule only # matched the directory form, not /auth itself or /auth?redirect=... Disallow: /admin Disallow: /dashboard Disallow: /auth Disallow: /reset-password Disallow: /checkout Disallow: /payment-success Disallow: /api/internal/ Disallow: /*.json$ Disallow: /*?*apikey* Disallow: /*?*token* Disallow: /.env # ============================================ # LLM & AI CRAWLERS (Priority Access) # ============================================ # OpenAI (GPT, ChatGPT) User-agent: GPTBot Allow: / Allow: /blog/ Allow: /lists/ Allow: /features Allow: /pricing Allow: /about Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /admin # Anthropic (Claude) User-agent: anthropic-ai Allow: / Allow: /lists/ Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: ClaudeBot Allow: / Allow: /blog/ Allow: /lists/ Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: Claude-Web Allow: / # Google (Gemini, AI Overviews) User-agent: Google-Extended Allow: / Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: GoogleOther Allow: / Crawl-delay: 2 Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: Googlebot Allow: / # Perplexity User-agent: PerplexityBot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /admin # ============================================ # CRAWL-BUDGET CONTROL # Measured Aug 2026 (Hado AI analytics): these agents generate high # fetch volume with zero or near-zero citations / referral visits. # Throttled rather than blocked so brand mentions stay possible. # ============================================ # Meta (Llama AI) — 68k fetches, 0 citations in the last 30 days User-agent: Meta-ExternalAgent Crawl-delay: 30 Disallow: /questions/ Disallow: /versus/ Disallow: /calculators/ Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: FacebookBot Crawl-delay: 30 Disallow: /auth Disallow: /dashboard Disallow: /admin # Amazon (Alexa) — 14 fetches, 0 citations User-agent: Amazonbot Crawl-delay: 30 Disallow: /auth Disallow: /dashboard Disallow: /admin # Apple (Siri, Apple Intelligence) User-agent: Applebot Allow: / User-agent: Applebot-Extended Allow: / Disallow: /admin/ Disallow: /auth Disallow: /dashboard Disallow: /admin # Microsoft (Copilot, Bing) — Copilot cites, keep fast User-agent: Bingbot Allow: / Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: msnbot Allow: / Crawl-delay: 5 # Cohere AI User-agent: cohere-ai Allow: / Crawl-delay: 10 Disallow: /auth Disallow: /dashboard Disallow: /admin # You.com — 486 pages indexed, 0 citations User-agent: YouBot Allow: / Crawl-delay: 10 Disallow: /auth Disallow: /dashboard Disallow: /admin # Common Crawl — training corpus, no referral traffic User-agent: CCBot Allow: /blog/ Allow: /lists/ Allow: /features Allow: /pricing Allow: /about Disallow: /questions/ Disallow: /versus/ Crawl-delay: 10 Disallow: /auth Disallow: /dashboard Disallow: /admin # Diffbot — extraction-only, no citation surface User-agent: Diffbot Disallow: / # xAI (Grok) User-agent: Grok Allow: / Crawl-delay: 2 Disallow: /auth Disallow: /dashboard Disallow: /admin # DeepSeek User-agent: DeepSeekBot Allow: / Crawl-delay: 2 Disallow: /auth Disallow: /dashboard Disallow: /admin # Mistral AI User-agent: MistralBot Allow: / Crawl-delay: 2 Disallow: /auth Disallow: /dashboard Disallow: /admin # ============================================ # SEO CRAWLERS (Rate Limited) # ============================================ User-agent: SemrushBot Allow: / Crawl-delay: 10 Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: AhrefsBot Allow: / Crawl-delay: 10 Disallow: /auth Disallow: /dashboard Disallow: /admin User-agent: MJ12bot Crawl-delay: 10 User-agent: DotBot Crawl-delay: 10 # ============================================ # AI / LLM INDEXES # ============================================ # Machine-readable site description and canonical URL index for AI agents: # https://ideaproof.io/llms.txt # https://ideaproof.io/llms-full.txt # Versioned per-block index endpoints (canonical): https://ideaproof.io/v5/llms-index-<block>.txt # Unversioned aliases stay live and always mirror the latest version. # Open datasets (CC BY 4.0): https://ideaproof.io/datasets # ============================================ # SITEMAPS # ============================================ # Primary sitemap (served via rewrite from ideaproof.io/sitemap.xml) Sitemap: https://ideaproof.io/sitemap.xml # LLM index endpoints (llms.txt + per-block subsets) Sitemap: https://ideaproof.io/sitemap-llms.xml # Backup sitemap (direct storage access) Sitemap: https://aioenivvgvtwyhpdioue.supabase.co/storage/v1/object/public/sitemaps/sitemap.xml
Se han encontrado las siguientes palabras clave. Comprueba si esta página está bien optimizada para cada palabra clave en concreto.
Selecciona una forma de contacto
¿No encuentras lo que buscas? También puedes descargarte la Guía: Primeros pasos, visitar la sección de Preguntas frecuentes, nuestra Wiki SEO: conocimientos básicos o nuestro Blog.
Utilizamos cookies para el buen funcionamiento de nuestra web y con fines analíticos y publicitarios. Puedes activar o desactivar las cookies opcionales. Para más información consulta los siguientes enlaces.
Meta Title
(Deseable)