# robots.txt for example.com # Webmaster: Full Google access, rate-limited other bots, block bad bots # Googlebot – Full access User-agent: Googlebot Disallow: # Bingbot – Crawl delay of 200 seconds User-agent: Bingbot Crawl-delay: 100 # AI bots – Limit crawl rate to 600 seconds User-agent: GPTBot Crawl-delay: 600 User-agent: ChatGPT-User Crawl-delay: 600 User-agent: ClaudeBot Crawl-delay: 600 User-agent: anthropic-ai Crawl-delay: 600 User-agent: CCBot Crawl-delay: 600 # Ahrefs and SEMrush bots – Limit crawl rate to 300 seconds User-agent: AhrefsBot Crawl-delay: 300 User-agent: SemrushBot Crawl-delay: 300 # Block all known bad bots User-agent: MJ12bot Disallow: / User-agent: BLEXBot Disallow: / User-agent: DotBot Disallow: / User-agent: spbot Disallow: / User-agent: Yandex Disallow: / User-agent: Baiduspider Disallow: / User-agent: Sogou Disallow: / User-agent: 360Spider Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: PetalBot Disallow: / User-agent: MauiBot Disallow: / User-agent: AspiegelBot Disallow: / # Default rule for all other bots User-agent: * Disallow: Crawl-delay: 10
Se han encontrado las siguientes palabras clave. Comprueba si esta página está bien optimizada para cada palabra clave en concreto.
(Deseable)