Whofinance.de - SEO Check

Meta Tags

Medienliste

Überschriftenstruktur

Gefundene Links auf dieser Seite

HTTP-Header

Links von Wikipedia

• Michael Krumtünger

Robots.txt

Sitemap: https://www.whofinance.de/sitemap.xml

# allow googlebot and bingbot to crawl all pages
User-agent: googlebot
Disallow:
User-agent: google
Disallow:
User-agent: bingbot
Disallow:
User-agent: bing
Disallow:
User-agent: LinkedInBot
Disallow:

## Enable robots.txt rules for all other crawlers
User-agent: *
Crawl-Delay: 5
## Do not crawl links with session IDs, like ?page=
## todo: enable later after institute update
## Disallow: /*?

User-agent: Amazonbot
User-agent: PetalBot
User-agent: AppInsights
User-agent: SemanticScholarBot
User-agent: dotbot
User-agent: whatcms
User-agent: rogerbot
User-agent: trendictionbot
User-agent: blexbot
User-agent: linkfluence
User-agent: magpie-crawler
User-agent: MJ12bot
User-agent: mediatoolkitbot
User-agent: aspiegelbot
User-agent: domainstatsbot
User-agent: cincraw
User-agent: nimbostratus
User-agent: httrack
User-agent: serpstatbot
User-agent: omgili
User-agent: omgilibot
User-agent: grapeshotcrawler
User-agent: megaindex
User-agent: semanticbot
User-agent: cocolyzebot
User-agent: domcopbot
User-agent: traackr
User-agent: bomborabot
User-agent: linguee
User-agent: webtechbot
User-agent: domainstatsbot
User-agent: clickagy
User-agent: sqlmap
User-agent: internet-structure-research-project-bot
User-agent: seekport
User-agent: awariosmartbot
User-agent: onalyticabot
User-agent: buck
User-agent: riddler
User-agent: sbl-bot
User-agent: bvbot
User-agent: sogou
User-agent: barkrowler
User-agent: admantx
User-agent: adbeat
User-agent: embed.ly
User-agent: semantic-visions
User-agent: voluumdsp
User-agent: wc-test-dev-bot
User-agent: gulperbot
User-agent: MJ12bot
User-agent: crw
User-agent: PetalBot
User-agent: Lucidworks-Anda
User-agent: Yandex
Disallow: /

User-agent: Baiduspider
User-agent: 360Spider
User-agent: Yisouspider
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Sogou web spider
Disallow: /

User-agent: Sogou inst spider
Disallow: /

## SEO-Bots
# User-agent: SemrushBot
# User-agent: AhrefsBot

Suchvorschau