Auren.com - SEO Checker

Meta tags

Media list

Heading structure

Found links on this page

HTTP Response Header

Links from Wikipedia

Robots.txt

#To block Bytespider  from crawling:
User-agent: Bytespider
Disallow: /

#To block Bytedance  from crawling:
User-agent: Bytedance
Disallow: /

#To block Coccocbot  from crawling:
User-agent: coccocbot-web
Disallow: /

#To block Dotbot from crawling:
User-agent: dotbot
Disallow: /

#To block Common Crawl Bot from crawling:
User-agent: BLEXBot
Disallow: /

#To block Common Crawl Bot from crawling:
User-agent: CCBot
Disallow: /

#To block GeedoBot from crawling:
User-agent: GeedoBot
Disallow: /

# Block netEstate NE Crawler (+http://www.website-datenbank.de/)
User-agent: netEstate NE Crawler (+http://www.website-datenbank.de/)
Disallow: /

#To block PetalBot from crawling:
User-agent: PetalBot
Disallow: /

#To block AhrefsBot from crawling:
User-agent: AhrefsBot
Disallow: /

# Blocked SemrushBot since it is creating lot of requests
User-agent: SemrushBot
Disallow: /

#To block SemrushBot from crawling your site for different SEO and technical issues:
User-agent: SiteAuditBot
Disallow: /

#To block SemrushBot from crawling your site for Backlink Audit tool:
User-agent: SemrushBot-BA
Disallow: /

#To block SemrushBot from crawling your site for On Page SEO Checker tool and similar tools:
User-agent: SemrushBot-SI
Disallow: / 

#To block SemrushBot from checking URLs on your site for SWA tool:
User-agent: SemrushBot-SWA
Disallow: / 

#To block SemrushBot from crawling your site for Content Analyzer and Post Tracking tools:
User-agent: SemrushBot-CT
Disallow: / 

#To block SplitSignalBot from crawling your site for SplitSignal tool:
User-agent: SplitSignalBot
Disallow: / 

#To block SemrushBot-COUB from crawling your site for Content Outline Builder tool:
User-agent: SemrushBot-COUB
Disallow: /

# Block CazoodleBot as it does not present correct accept content headers
User-agent: CazoodleBot
Disallow: /

# Block MJ12bot as it is just noise
User-agent: MJ12bot
Disallow: /

# Block dotbot as it cannot parse base urls properly
User-agent: dotbot/1.1
Disallow: /

# Block Gigabot
User-agent: Gigabot
Disallow: /

# Block Etaospider
User-agent: EtaoSpider
Disallow: /

User-agent: Robozilla
Disallow: /
User-agent: ia_archiver
Disallow: /
User-agent: baiduspider
Disallow: /
User-agent: naverbot
Disallow: /
User-agent: yeti
Disallow: / 

User-Agent: *
Crawl-delay: 30

Search preview