# General rules for all crawlers User-agent: * # Block access to API endpoints Disallow: /api/ # Block access to premium or user-specific content Disallow: /pro/ # Block access to email template resources Disallow: /public/emailtemplates/ # Block direct access to the email header analysis tool Disallow: /public/tools/emailheaders.aspx # Block access to unsubscribe page to avoid unnecessary indexing Disallow: /public/unsubscribe.aspx # Block access to checkout pages to avoid indexing dynamic purchase flows Disallow: /public/checkout/ # Block AhrefsBot from crawling problem diagnostic tools User-agent: AhrefsBot Disallow: /problem/ # Block dotbot (used by Dotcom-Monitor) from domain-specific health tools User-agent: dotbot Disallow: /domain/ Disallow: /emailhealth/ # Block Meta AI training bots from crawling any content User-agent: meta-externalagent Disallow: /SuperTool.aspx
Se han encontrado las siguientes palabras clave. Comprueba si esta página está bien optimizada para cada palabra clave en concreto.
(Deseable)