Show your SEO score to your visitors using the Seobility widget. The widget always displays the current SEO score of your homepage.
Meta Title
User-agent: *
Disallow: /*?
Disallow: /s/
#
# ── STAGED, do not uncomment yet ─────────────────────────────────────────────
# Read the full rationale and the 🔴 sequencing gate at the FOOT of this file
# before touching these. They live inside this `User-agent: *` group on purpose:
# a second `User-agent: *` group further down would be merged by Google but is
# SILENTLY DISCARDED by other parsers (CPython's urllib.robotparser keeps only
# the first — "the first default entry wins", verified 2026-07-31). Keep them
# here; do not move them next to their explanation.
#
# Disallow: /actualitate/
# Disallow: /*/actualitate/
# Disallow: /actualite/
# Disallow: /*/actualite/
# Disallow: /advertentie/
# Disallow: /*/advertentie/
# Disallow: /annuaire-professionnel/
# Disallow: /*/annuaire-professionnel/
# Disallow: /annunci-classificati/
# Disallow: /*/annunci-classificati/
# Disallow: /anuario-de-empresas/
# Disallow: /*/anuario-de-empresas/
# Disallow: /anuncios-classificados/
# Disallow: /*/anuncios-classificados/
# Disallow: /anunturi-clasificate/
# Disallow: /*/anunturi-clasificate/
# Disallow: /avisos-clasificados/
# Disallow: /*/avisos-clasificados/
# Disallow: /bedrijvengids/
# Disallow: /*/bedrijvengids/
# Disallow: /branchenverzeichnis/
# Disallow: /*/branchenverzeichnis/
# Disallow: /business-directory/
# Disallow: /*/business-directory/
# Disallow: /classified-ads/
# Disallow: /*/classified-ads/
# Disallow: /directorio-empresa/
# Disallow: /*/directorio-empresa/
# Disallow: /directorul-companiei/
# Disallow: /*/directorul-companiei/
# Disallow: /elenco-aziende/
# Disallow: /*/elenco-aziende/
# Disallow: /evenements/
# Disallow: /*/evenements/
# Disallow: /evenimente/
# Disallow: /*/evenimente/
# Disallow: /eventi/
# Disallow: /*/eventi/
# Disallow: /events/
# Disallow: /*/events/
# Disallow: /katalog-firm/
# Disallow: /*/katalog-firm/
# Disallow: /kleinanzeigen/
# Disallow: /*/kleinanzeigen/
# Disallow: /lieux/
# Disallow: /*/lieux/
# Disallow: /news/
# Disallow: /*/news/
# Disallow: /notizia/
# Disallow: /*/notizia/
# Disallow: /ogloszenia-niejawne/
# Disallow: /*/ogloszenia-niejawne/
# Disallow: /petites-annonces/
# Disallow: /*/petites-annonces/
# Disallow: /places/
# Disallow: /*/places/
# ─────────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Blocked crawlers — added 2026-07-26.
#
# Measured from one hour of Cloud Run request logs (2026-07-26 21:00-22:00 UTC):
# of 6,860 HTML page requests — each of which costs an SSR render — 71% came from
# self-declared crawlers. The agents below are blocked because they consume
# rendering budget and return nothing: no search traffic, no citations, no
# referrals. Pages/hour at the time of measurement:
#
# Amazonbot 1,784 AhrefsBot 515 SemrushBot 315 SleepBot 219
# PetalBot 147 Bytespider 89 DotBot 83 SpiderLing 77
# YouBot 15 (+706 asset requests/h) Scrapy 22 ShapBot 10
# VelenPublicWebCrawler 7 AffsignalCrawler 5
# ABEvalBot 2 HanaleiBot 1
#
# Amazonbot alone was ~43k pages/day — five times Googlebot's 360/h.
#
# Deliberately NOT blocked:
# Googlebot (360/h), bingbot (216/h) search revenue.
# Applebot (677/h) heavy, but feeds Siri/Spotlight and
# Safari suggestions: real discovery.
# ClaudeBot (88/h), ChatGPT-User (85/h), PerplexityBot (39/h), OAI-SearchBot,
# Qwantbot (27/h), YandexBot (9/h) answer engines that cite sources and
# can refer traffic.
# CriteoBot, peer39_crawler ad-tech brand-safety verification,
# ~1/h each; blocking them risks
# degrading our AdSense inventory.
# TikTokSpider 1/h, and it backs link previews.
#
# Bytespider IS blocked despite being AI: it trains only and never cites.
#
# The last eight agents below were not seen in the sampled hour — they are
# pre-emptive, and all well-known freeloaders.
#
# To re-measure, pull an hour of request logs and group by user agent:
# gcloud logging read 'resource.type="cloud_run_revision"
# AND resource.labels.service_name="alltrippers"' --project=alltrippers-eu
# ─────────────────────────────────────────────────────────────────────────────
# UPDATE 2026-07-31 — Amazon re-tokenised around the block.
#
# Re-measured over 2026-07-31 07:06-09:00 UTC (20,000 requests, 9,451 of them
# HTML page renders). Self-declared crawlers are now 61% of all page renders.
#
# Amazonbot obeyed the 07-26 block and went to zero — and the same crawl volume
# reappeared under a token this file did not name. Requests/day:
#
# Amazonbot Amzn-SearchBot
# 07-25 29,637 112
# 07-26 38,259 39 <- block added
# 07-27 6,754 8,064
# 07-28 0 8,542
# 07-29 0 11,159
# 07-30 0 18,997
#
# Amzn-SearchBot is now the single largest SSR consumer on the site (~20k
# pages/day), ahead of Googlebot (~9.4k/day). Blocking it is therefore recurring
# maintenance, not a one-off: expect Amazon to appear under a third token.
#
# meta-externalagent (~9k pages/day) was never in this file — it simply was not
# present in the single hour sampled on 07-26. It is Llama training crawl: no
# citations, no referrals.
#
# Deliberately still NOT blocked, on top of the list above:
# facebookexternalhit (~330/day) Facebook/Instagram LINK PREVIEWS. This is a
# different token from meta-externalagent and
# must stay allowed or shared links lose their
# preview cards. Do not merge the two.
# meta-externalfetcher user-initiated link fetches, same rationale
# as ChatGPT-User. Not seen in the sample.
# GoogleOther (~7.9k/day) does not feed Search, so blocking is
# documented-safe — but the risk is asymmetric
# (Google-branded agent vs. ~6% of renders)
# and it honours path rules, so it will follow
# the staged Disallow at the foot of this file
# instead. Revisit only if that lands and it
# is still heavy.
# OAI-SearchBot (~1.9k/day) answer engine that cites.
#
# Ignoring this file entirely as of 07-31 (blocked above, still crawling):
# SleepBot ~4,055/day SpiderLing ~2,312/day ShapBot ~354/day
# Scrapy ~240/day
# robots.txt cannot reach these — they are already listed above and simply do
# not honour it. The only lever left is a UA check in server.ts. NOT Cloud
# Armor: alltrippers.com and www are custom-domain-mapped straight to Cloud Run
# with no load balancer in front (verified 2026-07-31 — zero backend-services,
# zero security-policies in the project), and Cloud Armor requires an external
# HTTPS LB. Adding one to block ~7k renders/day is not worth it.
# ─────────────────────────────────────────────────────────────────────────────
# UPDATE 2026-08-06 — Meta re-tokenised around the block, exactly as predicted.
#
# Every agent blocked on 07-26/07-31 now measures ZERO: Amazonbot 0,
# Amzn-SearchBot 0, meta-externalagent 0. The blocks work. What changed is that
# the volume came back under names this file did not list. Sample of 2,000
# requests, 2026-08-05 09:00-09:30 UTC:
#
# meta-webindexer 423 <- NEW, unblocked
# GeedoShopProductFinder 391 <- NEW, unblocked
# Applebot 86
# bingbot 52
# Googlebot 6
#
# Two unblocked agents were ~41% of requests against Googlebot's 6. Cloud Run
# requests went ~190k/day -> 689k (08-04) -> 773k (08-05); request CPU
# GBP 1.34 -> 2.34/day and intercontinental egress 0.16 -> 0.97/day.
#
# ⚠️ THEY COST COMPLETELY DIFFERENT THINGS — do not reason about them together:
# meta-webindexer 500/500 sampled requests were HTML = SSR RENDERS.
# This is the CPU line. ~20k renders/day.
# GeedoShopProductFinder 494/500 were static assets (JS chunks, images).
# Near-zero CPU; this is the EGRESS line.
#
# WHY meta-webindexer IS BLOCKED DESPITE BEING A CITING ANSWER ENGINE.
# It feeds Meta AI search and CAN cite/link sources, so by the rule used above
# (cites = allowed, trains-only = blocked) it belongs in the allowed group. It
# is blocked anyway because the citation channel measurably pays us nothing.
# GA4, 90 days to 2026-08-05, bot-filtered, by sessionSource:
#
# chatgpt.com 446 perplexity 84 gemini 12 copilot 11 claude.ai 7
# meta.ai 0 <- zero, while meta-externalagent crawled ~9k pages/day
# right up to the 07-31 block
#
# Allowing citing crawlers genuinely pays — ChatGPT sends 446 sessions. It just
# does not pay for Meta. And 79% of meta-webindexer's crawl is
# /fr/*/annuaire-professionnel/ — the business directory this site already
# NOINDEXED on 07-29, so even a future citation would surface a page we have
# decided should not appear in search at all.
#
# 🔁 REVERSE THIS IF: `meta.ai` ever appears in
# npm run ga -- report --days=90 --dimensions=sessionSource --metrics=sessions
# Worth re-checking quarterly. The honest caveat is that meta-webindexer only
# began crawling us ~08-04, so its citations had not had time to appear.
#
# 🔴 DO NOT ADD facebookexternalhit. It is a DIFFERENT agent and it is what the
# 58 Facebook/Instagram sessions in that same 90 days actually come from —
# l.facebook.com and lm.facebook.com are its share-link redirectors. Blocking it
# kills link previews on every shared URL. Same for meta-externalfetcher
# (user-initiated). The warning above about not merging the Meta tokens is now
# backed by measurement, not just caution.
#
# GeedoShopProductFinder is geedo.com, a PRODUCT search engine indexing online
# stores (documented IPs 83.99.206.0/24, reverse-DNS *.geedo.com). We are a
# travel affiliate site, not a shop: it is crawling JS bundles and images
# hunting for products that do not exist. No search value, no referral value.
#
# LESSON, now confirmed twice: blocking by token is RECURRING MAINTENANCE.
# Amazonbot -> Amzn-SearchBot (07-27), meta-externalagent -> meta-webindexer
# (~08-04). Expect a third token each time. The durable fix is the staged
# path-level Disallow at the foot of this file, which stops the FETCH for every
# compliant agent at once — but it is gated on deindexation completing.
# ─────────────────────────────────────────────────────────────────────────────
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: GeedoShopProductFinder
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SleepBot
User-agent: PetalBot
User-agent: Bytespider
User-agent: DotBot
User-agent: SpiderLing
User-agent: YouBot
User-agent: Scrapy
User-agent: ShapBot
User-agent: VelenPublicWebCrawler
User-agent: AffsignalCrawler
User-agent: ABEvalBot
User-agent: HanaleiBot
User-agent: MJ12bot
User-agent: BLEXBot
User-agent: Barkrowler
User-agent: DataForSeoBot
User-agent: ImagesiftBot
User-agent: serpstatbot
User-agent: Diffbot
User-agent: SeekportBot
Disallow: /
# ─────────────────────────────────────────────────────────────────────────────
# STAGED — do not uncomment yet. Gated on deindexation completing.
#
# Why this beats blocking crawlers one token at a time: 76% of Amzn-SearchBot's
# crawl on 2026-07-31 was the country x entity crossproduct, which returns 200
# for every combination:
#
# /en/mongolia/places/the-british-museum -> 200
# /en/guam/business-directory/oakwood-theme-park/avis -> 200
# /en/tunisia/places/national-gallery -> 200
#
# ~200 countries x N entities x sub-pages (/avis, /a-propos, /evenements,
# /annonces, /codes-promo) x 9 locales is an unbounded URL space. Amazon's
# repeat rate across 1,617 requests was 0% — it is not re-crawling, it is
# walking a maze with no end, which is why its volume climbed 8k -> 19k/day.
#
# These pages already serve `X-Robots-Tag: noindex, follow` (shipped 01466-lag,
# 2026-07-29), so they will not be indexed — but noindex still costs a full SSR
# render every time, because the crawler must fetch the page to discover it.
# `follow` also means crawlers keep walking outward from them. Only a path-level
# Disallow stops the fetch, and it applies to every compliant agent — Googlebot
# included, which is the real prize: crawl budget currently spent here is budget
# not spent on the affiliate money pages.
#
# 🔴 SEQUENCING — why this is commented out:
# Disallow prevents Google from RE-CRAWLING, and therefore from ever seeing the
# noindex header. URLs already in the index at the time you block can get stuck
# there indefinitely. The deindex only shipped 2026-07-29. Wait until GSC shows
# these URLs dropped out of the index, THEN uncomment. Blocking too early
# freezes them in the index permanently — the exact opposite of the goal.
#
# This is the same conclusion already recorded at server.ts:966 ("Add the
# Disallow only once Search Console shows these deindexed"). That comment is the
# decision; this block is the prepared payload. Do not enable one without the
# other.
#
# ⚠️ SOURCE OF TRUTH — do not hand-edit this list.
# The segments below are the 28 distinct non-empty values of the five
# NOINDEXED_FUNCTIONALITY_KEYS (server.ts:495 — classifiedAds, placemarks,
# companies, events, news) resolved through FUNCTIONALITY_URIS in Enums.ts. They
# must stay in lockstep with NOINDEXED_PATH_RE, or robots.txt will block a path
# set that differs from the one being noindexed. Regenerate rather than patch:
#
# node -e "const {FUNCTIONALITY_URIS}=require('./src/app/misc/Enums.ts'); \
# console.log([...new Set(['classifiedAds','placemarks','companies', \
# 'events','news'].flatMap(k=>Object.values(FUNCTIONALITY_URIS[k]) \
# .filter(v=>typeof v==='string'&&v&&!v.startsWith('_'))))].sort().join('\n'))"
#
# ⚠️ Empty-segment hazard: placemarks, events and news carry an EMPTY STRING for
# several locales in FUNCTIONALITY_URIS and fall back to `_default_` — which is
# why `places` covers de/es/it/nl/pl/pt/ro while only fr has its own (`lieux`).
# Verified against live traffic 2026-07-31: /de/.../places/, /ro/.../places/ all
# appear. The same empty values are what makes the `v.length > 0` filter at
# server.ts:510 load-bearing; the note there explains what an empty alternative
# does to the regex. Never emit a bare `Disallow: //`.
#
# ⚠️ Two forms per segment on purpose: the country-prefixed one and a root-level
# one, because a handful of URLs are served with no locale prefix (/places/…,
# /lieux/… both seen in logs). Do NOT collapse to `Disallow: /*places/` — a
# bare-substring pattern also swallows unrelated paths.
#
# ⚠️ The rules themselves are NOT here — they sit commented inside the
# `User-agent: *` group at the TOP of this file, because a second `User-agent: *`
# group down here would be silently dropped by strict parsers even though Google
# merges it. To enable: uncomment the 56 `# Disallow:` lines at the top. Nothing
# needs to change in this note.
# ─────────────────────────────────────────────────────────────────────────────
Sitemap: https://alltrippers.com/sitemap/sitemap-index.xmlFollowing keywords were found. You can check the keyword optimization of this page for each keyword.
Please select one of the following options for your request.
None of these options suit your need?
You can also download our Quick Start Guide, visit the FAQ section, our Knowledge Base or Blog.
(Nice to have)