# Define access-restrictions for robots/spiders # http://www.robotstxt.org/wc/norobots.html User-agent: 008 Disallow: / User-agent: AhrefsBot Disallow: / User-agent: scirus-crawler Disallow: / User-agent: wotbox Disallow: / User-agent: Sosospider Disallow: / User-agent: magpie-crawler Disallow: / User-agent: Blekkobot Disallow: / User-agent: ia_archiver Disallow: / User-agent: Baiduspider Crawl-delay: 90 Disallow: /*-ru/ Disallow: /*-fr/ Disallow: /*-ar/ Disallow: /*-es/ Disallow: /*-sw/ Disallow: /today/ Disallow: /electoral-advice Disallow: /main/ Disallow: /Members/ Disallow: /author/ Disallow: /epic- Disallow: /ero- Disallow: /*CDCountry Disallow: /*CDTable Disallow: /*CDMap Disallow: /*CDBarChart Disallow: /*CDChart Disallow: /*open-flash-chart Disallow: /*world_swf Disallow: /*?searchterm Disallow: /*SearchableText Disallow: /*mail_password_form Disallow: /*sendto_form$ Disallow: /*folder_factories$ Disallow: /*portal_factory$ Disallow: /*search$ Disallow: /*login_form Disallow: /*search_materials_results Disallow: /*onePage Disallow: /ace-en/pdf/ # Yandex bot User-agent: Yandex Crawl-delay: 90 Disallow: /*-en/ Disallow: /*-fr/ Disallow: /*-ar/ Disallow: /*-es/ Disallow: /*-sw/ Disallow: /today/ Disallow: /electoral-advice Disallow: /main/ Disallow: /Members/ Disallow: /author/ Disallow: /epic- Disallow: /ero- Disallow: /*CDCountry Disallow: /*CDTable Disallow: /*CDMap Disallow: /*CDBarChart Disallow: /*CDChart Disallow: /*open-flash-chart Disallow: /*world_swf Disallow: /*?searchterm Disallow: /*SearchableText Disallow: /*mail_password_form Disallow: /*sendto_form$ Disallow: /*folder_factories$ Disallow: /*portal_factory$ Disallow: /*search$ Disallow: /*login_form Disallow: /*search_materials_results Disallow: /ace-en/pdf/ # French natl. library User-agent: bnf.fr_bot Disallow: / # video indexing service User-agent: ShowyouBot Disallow: / User-agent: suggybot Disallow: / User-agent: Speedy Disallow: / # CCbot not obeying wildcards User-agent: ccbot Disallow: / # ezooms not obeying wildcards User-agent: ezooms Disallow: / # MJ12bot not obeying wildcards User-agent: MJ12bot Disallow: / # discobot not obeying wildcards User-agent: discobot Disallow: / # discobot not obeying wildcards User-agent: discoverybot Disallow: / # Exabot not obeying wildcards User-agent: Exabot Disallow: / # archive.org_bot not obeying wildcards User-agent: archive.org_bot Disallow: / # JikeSpider not obeying wildcards User-agent: JikeSpider Disallow: / # WBSearchBot not obeying wildcards User-agent: WBSearchBot Disallow: / # Gigabot not obeying wildcards User-agent: Gigabot Disallow: / # By default we allow robots to access all areas of our site # already accessible to anonymous users User-agent: * Crawl-delay: 30 Disallow: /epic-en/cmfepic/ Disallow: /epic-en/research/ Disallow: /epic-en/countries/ Disallow: /epic-fr/countries/ Disallow: /epic-es/countries/ Disallow: /epic-ar/countries/ Disallow: /epic-ru/countries/ Disallow: /regions-en/archive/ Disallow: /author/ Disallow: /today-ole/ Disallow: /today/feature-articles/portal_factory Disallow: /electoral-advice/ace-workspace/ Disallow: /electoral-advice/ace-workspace/ Disallow: /regions-en/archive/ Disallow: /electoral-advice/dop? Disallow: /translatorsTemplate Disallow: /electoral-advice/ace-workspace/ Disallow: /ace-en/pdf/ # Add Googlebot-specific syntax extension to exclude forms # that are repeated for each piece of content in the site # the wildcard is only supported by Googlebot # http://www.google.com/support/webmasters/bin/answer.py?answer=40367&ctx=sibling User-Agent: * Disallow: /*/topics/onePage Disallow: /ace-es/topics/va/onePage Disallow: /ace-ru/*/onePage Disallow: /ace-sw/*/onePage Disallow: /ace-fr/*/onePage Disallow: /ace-ar/*/onePage Disallow: /main/*/onePage Disallow: /*sendto_form$ Disallow: /*folder_factories$ Disallow: /*portal_factory$ Disallow: /*search$ Disallow: /*login_form Disallow: /*mail_password_form Disallow: /ace-*search_materials_results Disallow: /regions-*search_materials_results Disallow: /regions-en/*CDCountry Disallow: /regions-en/*CDTable Disallow: /regions-en/*CDMap Disallow: /regions-en/*CDChart Disallow: /regions-en/*CDBarChart Disallow: /regions-en/*open-flash-chart Disallow: /regions-en/*world_swf Disallow: /epic-en/feedback Disallow: /epic-es/feedback Disallow: /epic-fr/feedback Disallow: /epic-en/en/CD Disallow: /epic-en/es/CD Disallow: /epic-en/fr/CD Disallow: /epic-en/ar/CD Disallow: /epic-en/ru/CD Disallow: /epic-fr/en/CD Disallow: /epic-fr/es/CD Disallow: /epic-fr/fr/CD Disallow: /epic-fr/ar/CD Disallow: /epic-fr/ru/CD Disallow: /epic-es/en/CD Disallow: /epic-es/es/CD Disallow: /epic-es/fr/CD Disallow: /epic-es/ar/CD Disallow: /epic-es/ru/CD Disallow: /CD Disallow: /epic-*open-flash-chart Disallow: /epic-*world_swf Disallow: /ero-*country= Disallow: /*ero-*/index_html?filter Disallow: /ero-es Disallow: /ero-fr Disallow: /ero-ar Disallow: /ero-ru Disallow: /ace-*CDCountry Disallow: /ace-en/ero- Disallow: /ace-es/ero- Disallow: /ace-ar/ero- Disallow: /ace-ru/ero- Disallow: /ace-fr/ero- Disallow: /ace-*CDTable Disallow: /ace-*CDMap Disallow: /ace-*CDBarChart Disallow: /ace-*CDChart Disallow: /ace-*open-flash-chart Disallow: /ace-*world_swf Disallow: /*?searchterm Disallow: /*SearchableText Disallow: /*mail_password_form Disallow: /*&mission= Disallow: /en/* Disallow: /fr/* Disallow: /es/* Disallow: /ru/* Disallow: /ar/* Disallow: /ace-en/pdf/
Following keywords were found. You can check the keyword optimization of this page for each keyword.
(Nice to have)