Monitoringresources.org - SEO Checker

Meta tags

Media list

Heading structure

Found links on this page

HTTP Response Header

Links from Wikipedia

Robots.txt

# robots.txt file for MonitoringResources
#
# Some tips:
# * The lines that start with # are comments
# * URLs are case-senitive
# * The /$ means the match is at the end of the URL path
# * $ at the end means exact path match, which we want to avoid having "Index" match "IndexData"
# * Don't have any extra blank lines because "A blank line indicates a new user agent section", use # for comment space
# * There should be one blank line before each User-agent tag
# * Sitemap: directive
# ** Independent of User-agent
# ** Must be a full URL
# ** Can be more than one
#
# ========
# Site Map
# ========
#
Sitemap: https://www.monitoringresources.org/Resources/Home/SiteMap
#
# ===========
# User-Agents
# ===========
#
# All user agents are OK
#

User-agent: *
Crawl-delay: 10
#
# ============
# Basic Allows
# ============
#
# Home Page + Landing Pages Allow 
# -------------------------------
#
Allow: /$
Allow: /Resources/Home/Index$
Allow: /Resources/Home/About$
Allow: /Resources/Home/FAQ$
Allow: /Resources/Glossary/Index$
Allow: /Resources/DataRepository/Index$
Allow: /Resources/Program/Index$
Allow: /Resources/Project/Index$
Allow: /Designer/Home/Index$
Allow: /Designer/Design/Index$
Allow: /Sites/Home/Index$
Allow: /Sites/Master/Index$
Allow: /Sites/Legacy/Index$
#
# Site Map Allow
# --------------
# 
Allow: /Resources/Home/SiteMap$
#
# ==============================================
# Pages that Site Map generates links to go here
# ==============================================
# 
Allow: /Designer/SurveyDesign/Detail
Allow: /Designer/OpportunisticDesign/Detail
Allow: /Sites/Master/Detail
Allow: /Sites/Legacy/Detail
Allow: /Resources/Project/Detail
Allow: /Resources/Program/Detail
Allow: /Resources/DataRepository/Detail
#
# =============================
# Everything else is disallowed
# =============================
#
Disallow: /

Search preview