# JustClean Robots.txt # Professional cleaning and wellness services platform # https://justclean.com # Default rules for all crawlers User-agent: * Allow: / Crawl-delay: 1 # Sitemap location Sitemap: https://justclean.com/sitemap.xml # Block access to admin/private areas Disallow: /admin/ Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /bookings Disallow: /*.json$ # ============================================ # AI SEARCH ENGINE CRAWLERS - WELCOME! # We want AI assistants to understand our services # ============================================ # OpenAI GPTBot - Powers ChatGPT web browsing User-agent: GPTBot Allow: / Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /bookings # OpenAI ChatGPT-User - ChatGPT browsing plugin User-agent: ChatGPT-User Allow: / Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /bookings # Anthropic ClaudeBot - Powers Claude AI User-agent: ClaudeBot Allow: / Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /bookings # Anthropic Claude-Web User-agent: Claude-Web Allow: / Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /bookings # Google Bard / Gemini User-agent: Google-Extended Allow: / Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /bookings # Perplexity AI User-agent: PerplexityBot Allow: / Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /bookings # Common Crawl - Used for AI training data User-agent: CCBot Allow: / Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /bookings # Cohere AI User-agent: cohere-ai Allow: / Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /bookings # Meta AI User-agent: FacebookBot Allow: / Disallow: /api/ Disallow: /login Disallow: /signup Disallow: /bookings # Microsoft Bing Chat / Copilot User-agent: Bingbot Allow: / # Apple Applebot (Siri, Spotlight) User-agent: Applebot Allow: / # ============================================ # TRADITIONAL SEARCH ENGINES # ============================================ User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Mobile Allow: / User-agent: Yandex Allow: / User-agent: DuckDuckBot Allow: / # ============================================ # BLOCK KNOWN BAD BOTS # ============================================ User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / # Machine-readable brand info for AI # See: https://justclean.com/llms.txt
Se han encontrado las siguientes palabras clave. Comprueba si esta página está bien optimizada para cada palabra clave en concreto.
(Deseable)