# === GENERAL SEARCH ENGINES & ALLOWED AI SEARCH/FETCH BOTS === # Applies to general crawlers and AI search/fetch bots that are not # explicitly blocked below. User-agent: * Disallow: /fetch.jsp Disallow: /fetchcap.jsp Disallow: /translation/ Disallow: /access.jsp Disallow: /payment.jsp Disallow: /api/v1 Disallow: /lesson/ # === BLOCK AI MODEL TRAINING BOTS === User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: Claude-Web User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: MistralAI-Training User-agent: Applebot-Extended User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: cohere-ai User-agent: Bytespider User-agent: DeepSeekBot Disallow: / # === We ALLOW AI SEARCH & USER-REQUESTED FETCH BOTS === # # OAI-SearchBot # ChatGPT-User # Claude-SearchBot # Claude-User # # === BLOCK SPECIFIC AGGRESSIVE CRAWLERS & SCRAPERS === User-agent: ExaSearchBot User-agent: SiteCheckerBotCrawler User-agent: Amazonbot User-agent: AmazonAdBot User-agent: AhrefsBot User-agent: 1001FirmsBot User-agent: mojeekbot User-agent: velenpublicwebcrawler User-agent: SerendeputyBot User-agent: MegaIndex.ru User-agent: ias_crawler User-agent: gowikibot User-agent: DomainStatsBot User-agent: YamanaLab-Robot User-agent: ExtLinksBot User-agent: spbot User-agent: SiteExplorer User-agent: Mappy User-agent: JoobleBot User-agent: TurnitinBot User-agent: cxensebot User-agent: JamesBOT User-agent: MJ12bot User-agent: yisouspider User-agent: proximic User-agent: ezooms User-agent: sistrix User-agent: grapeshot User-agent: LinkedInBot User-agent: DeuSu User-agent: Cliqzbot User-agent: MaxPointCrawler User-agent: netseer User-agent: SMTBot User-agent: mediatoolkitbot User-agent: blexbot User-agent: rogerbot User-agent: netestate User-agent: ccbot User-agent: integromedb User-agent: iZSearch.com User-agent: izsearch.com User-agent: ltx71 User-agent: SurdotlyBot User-agent: XoviBot User-agent: yacybot User-agent: experibot_v1 User-agent: wotbox User-agent: SafeDNSBot User-agent: SemrushBot-SA User-agent: SemrushBot User-agent: Linguee Disallow: /
Following keywords were found. You can check the keyword optimization of this page for each keyword.
(Nice to have)