Dwrrah.com - SEO Checker

Meta tags

Media list

Heading structure

Found links on this page

HTTP Response Header

Links from Wikipedia

Robots.txt

# Dwrrah | دورة — robots.txt
#
# ONE group for every crawler. A crawler obeys exactly ONE group — the one
# whose User-agent most specifically names it — and ignores every other group
# (RFC 9309 §2.2.1; Google: "only one group is valid for a particular
# crawler"). This file used to give each named bot its own `Allow: /` group,
# so Googlebot, Bingbot, GPTBot, ClaudeBot … never read the Disallow lines
# under `*`: /login, /admin/, /api/ and ?token= URLs were open to exactly the
# crawlers that matter, and the Content-Signal line reached none of the AI
# bots it was written for. Listing the names as extra User-agent lines of the
# SAME group keeps them explicit (for operators, and for Cloudflare's AI Crawl
# Control robots.txt report) while every one of them gets the same rules.
#
# Keep the group free of BLANK lines: legacy record-based parsers (Python's
# urllib.robotparser among them) treat a blank line as the end of a record and
# would drop the User-agent lines above it. Comment lines are safe.
#
# Search & social crawlers
User-agent: Googlebot
User-agent: Bingbot
User-agent: Twitterbot
User-agent: facebookexternalhit
User-agent: LinkedInBot
User-agent: WhatsApp
# AI / AEO crawlers — explicitly allowed for answer engine visibility
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Google-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: anthropic-ai
User-agent: CCBot
User-agent: Applebot
User-agent: Applebot-Extended
# Assistants that fetch a page because a PERSON asked them to read it — a
# different user agent from the indexing bot (Claude-User is not ClaudeBot;
# missing it was how the blog came back reported as unreadable while Googlebot
# was served the full article).
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-User
User-agent: Bravebot
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: YouBot
User-agent: Google-CloudVertexBot
# Everyone else
User-agent: *
# Content Signals (contentsignals.org / draft-romm-aipref-contentsignals) —
# same syntax Cloudflare's managed robots.txt serves. Google Search Console
# reports this line as "Syntax not understood": EXPECTED and harmless. Google
# ignores fields it does not support and the rest of the group still applies;
# Google's own AI controls are Googlebot + the Google-Extended token above.
# Declared preferences for the public marketing surface:
#   search    = appear in search indexes              → yes
#   ai-input  = be cited/grounded in AI answers (AEO) → yes
#   ai-train  = be used as model training data        → yes (matches the
#               existing posture: /llms-full.txt is published for training
#               and grounding, and training crawlers are named above).
# Flip ai-train to "no" if legal prefers to withhold training rights while
# keeping search + answer-engine visibility.
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Disallow: /instructor/
Disallow: /admin/
Disallow: /dashboard
Disallow: /editor/
Disallow: /api/
Disallow: /*?*token=
# Auth / utility surfaces — not content, must not be indexed (G10). Their
# client-side noindex only applies after JS; this blocks crawl statically.
Disallow: /login
Disallow: /signup
Disallow: /forgot-password
Disallow: /reset-password
Disallow: /post-login
Disallow: /create-academy
Disallow: /checkout
# LAST on purpose: RFC 9309 parsers (Google, Bing) pick the longest match so
# order is irrelevant to them, but first-match parsers (Python's robotparser,
# older bots) would stop at an early `Allow: /` and never reach a Disallow.
Allow: /

Sitemap: https://dwrrah.com/sitemap.xml

Search preview