# Generated robots.txt by BlueOnyx # Allow well-known search engines and reputable crawlers User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: DuckDuckBot Allow: / User-agent: Applebot Allow: / User-agent: Yahoo Allow: / User-agent: Yandex Allow: / User-agent: Baiduspider Allow: / # Block known unwanted bots and scrapers User-agent: 360Spider Disallow: / User-agent: AhrefsBot Disallow: / User-agent: aiHitBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Applebot Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: BLEXBot Disallow: / User-agent: BuiBui Disallow: / User-agent: CCBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: Claude Disallow: / User-agent: coccoc Disallow: / User-agent: Crawler Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: datasift Disallow: / User-agent: DeuSu Disallow: / User-agent: DotBot Disallow: / User-agent: EmailCollector Disallow: / User-agent: Exabot Disallow: / User-agent: FacebookBot Disallow: / User-agent: facebookexternalhit Disallow: / User-agent: Finderlein Disallow: / User-agent: FlipboardProxy Disallow: / User-agent: Genieo Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: GRequests Disallow: / User-agent: ia_archiver Disallow: / User-agent: Jimdo_Feed_Fetcher Disallow: / User-agent: linkdexbot Disallow: / User-agent: Mail.RU_Bot Disallow: / User-agent: meanpathbot Disallow: / User-agent: Mediatoolkitbot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: meta-externalfetcher Disallow: / User-agent: MetaURI Disallow: / User-agent: MJ12bot Disallow: / User-agent: msnbot Disallow: / User-agent: NetSeer Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: oBot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Pixray Disallow: / User-agent: proximic Disallow: / User-agent: publiclibraryarchive Disallow: / User-agent: Riddler Disallow: / User-agent: SEMrushBot Disallow: / User-agent: SentiBot Disallow: / User-agent: SEOkicks Disallow: / User-agent: serpstatbot Disallow: / User-agent: SeznamBot Disallow: / User-agent: SiteExplorer Disallow: / User-agent: Sogou Disallow: / User-agent: spbot Disallow: / User-agent: SurveyBot Disallow: / User-agent: TrackBack Disallow: / User-agent: trendiction Disallow: / User-agent: Uptimebot Disallow: / User-agent: WebEMailExtrac Disallow: / User-agent: webmeup Disallow: / User-agent: Scrapy Disallow: / # General restrictions for all other bots User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /tmp/ Disallow: /private/ Disallow: /config/
Following keywords were found. You can check the keyword optimization of this page for each keyword.
(Nice to have)