Sitemap: https://www.gns.cri.nz/sitemap.xml # ByteSpider (ByteDance/TikTok crawler) User-agent: ByteSpider Disallow: / # PetalBot (Huawei Search crawler) User-agent: PetalBot Disallow: / # All other bots User-agent: * # Admin and CMS areas Disallow: /admin/ Disallow: /dev/ Disallow: /Security/ # Error pages Disallow: /server-error/ Disallow: /page-not-found/ # Crawl-delay for allowed bots (reduce load) # Not all bots respect this Crawl-delay: 5
(Nice to have)