Alltrippers.com - SEO Check

Meta Tags

Medienliste

Überschriftenstruktur

Gefundene Links auf dieser Seite

HTTP-Header

Links von Wikipedia

Robots.txt

User-agent: *
Disallow: /*?
Disallow: /s/
#
# ── STAGED, do not uncomment yet ─────────────────────────────────────────────
# Read the full rationale and the 🔴 sequencing gate at the FOOT of this file
# before touching these. They live inside this `User-agent: *` group on purpose:
# a second `User-agent: *` group further down would be merged by Google but is
# SILENTLY DISCARDED by other parsers (CPython's urllib.robotparser keeps only
# the first — "the first default entry wins", verified 2026-07-31). Keep them
# here; do not move them next to their explanation.
#
# Disallow: /actualitate/
# Disallow: /*/actualitate/
# Disallow: /actualite/
# Disallow: /*/actualite/
# Disallow: /advertentie/
# Disallow: /*/advertentie/
# Disallow: /annuaire-professionnel/
# Disallow: /*/annuaire-professionnel/
# Disallow: /annunci-classificati/
# Disallow: /*/annunci-classificati/
# Disallow: /anuario-de-empresas/
# Disallow: /*/anuario-de-empresas/
# Disallow: /anuncios-classificados/
# Disallow: /*/anuncios-classificados/
# Disallow: /anunturi-clasificate/
# Disallow: /*/anunturi-clasificate/
# Disallow: /avisos-clasificados/
# Disallow: /*/avisos-clasificados/
# Disallow: /bedrijvengids/
# Disallow: /*/bedrijvengids/
# Disallow: /branchenverzeichnis/
# Disallow: /*/branchenverzeichnis/
# Disallow: /business-directory/
# Disallow: /*/business-directory/
# Disallow: /classified-ads/
# Disallow: /*/classified-ads/
# Disallow: /directorio-empresa/
# Disallow: /*/directorio-empresa/
# Disallow: /directorul-companiei/
# Disallow: /*/directorul-companiei/
# Disallow: /elenco-aziende/
# Disallow: /*/elenco-aziende/
# Disallow: /evenements/
# Disallow: /*/evenements/
# Disallow: /evenimente/
# Disallow: /*/evenimente/
# Disallow: /eventi/
# Disallow: /*/eventi/
# Disallow: /events/
# Disallow: /*/events/
# Disallow: /katalog-firm/
# Disallow: /*/katalog-firm/
# Disallow: /kleinanzeigen/
# Disallow: /*/kleinanzeigen/
# Disallow: /lieux/
# Disallow: /*/lieux/
# Disallow: /news/
# Disallow: /*/news/
# Disallow: /notizia/
# Disallow: /*/notizia/
# Disallow: /ogloszenia-niejawne/
# Disallow: /*/ogloszenia-niejawne/
# Disallow: /petites-annonces/
# Disallow: /*/petites-annonces/
# Disallow: /places/
# Disallow: /*/places/
# ─────────────────────────────────────────────────────────────────────────────

# ─────────────────────────────────────────────────────────────────────────────
# Blocked crawlers — added 2026-07-26.
#
# Measured from one hour of Cloud Run request logs (2026-07-26 21:00-22:00 UTC):
# of 6,860 HTML page requests — each of which costs an SSR render — 71% came from
# self-declared crawlers. The agents below are blocked because they consume
# rendering budget and return nothing: no search traffic, no citations, no
# referrals. Pages/hour at the time of measurement:
#
#   Amazonbot 1,784   AhrefsBot 515   SemrushBot 315   SleepBot 219
#   PetalBot 147      Bytespider 89   DotBot 83        SpiderLing 77
#   YouBot 15 (+706 asset requests/h)  Scrapy 22       ShapBot 10
#   VelenPublicWebCrawler 7           AffsignalCrawler 5
#   ABEvalBot 2       HanaleiBot 1
#
# Amazonbot alone was ~43k pages/day — five times Googlebot's 360/h.
#
# Deliberately NOT blocked:
#   Googlebot (360/h), bingbot (216/h)    search revenue.
#   Applebot (677/h)                      heavy, but feeds Siri/Spotlight and
#                                         Safari suggestions: real discovery.
#   ClaudeBot (88/h), ChatGPT-User (85/h), PerplexityBot (39/h), OAI-SearchBot,
#   Qwantbot (27/h), YandexBot (9/h)      answer engines that cite sources and
#                                         can refer traffic.
#   CriteoBot, peer39_crawler             ad-tech brand-safety verification,
#                                         ~1/h each; blocking them risks
#                                         degrading our AdSense inventory.
#   TikTokSpider                          1/h, and it backs link previews.
#
# Bytespider IS blocked despite being AI: it trains only and never cites.
#
# The last eight agents below were not seen in the sampled hour — they are
# pre-emptive, and all well-known freeloaders.
#
# To re-measure, pull an hour of request logs and group by user agent:
#   gcloud logging read 'resource.type="cloud_run_revision"
#     AND resource.labels.service_name="alltrippers"' --project=alltrippers-eu
# ─────────────────────────────────────────────────────────────────────────────
# UPDATE 2026-07-31 — Amazon re-tokenised around the block.
#
# Re-measured over 2026-07-31 07:06-09:00 UTC (20,000 requests, 9,451 of them
# HTML page renders). Self-declared crawlers are now 61% of all page renders.
#
# Amazonbot obeyed the 07-26 block and went to zero — and the same crawl volume
# reappeared under a token this file did not name. Requests/day:
#
#            Amazonbot   Amzn-SearchBot
#   07-25       29,637              112
#   07-26       38,259               39   <- block added
#   07-27        6,754            8,064
#   07-28            0            8,542
#   07-29            0           11,159
#   07-30            0           18,997
#
# Amzn-SearchBot is now the single largest SSR consumer on the site (~20k
# pages/day), ahead of Googlebot (~9.4k/day). Blocking it is therefore recurring
# maintenance, not a one-off: expect Amazon to appear under a third token.
#
# meta-externalagent (~9k pages/day) was never in this file — it simply was not
# present in the single hour sampled on 07-26. It is Llama training crawl: no
# citations, no referrals.
#
# Deliberately still NOT blocked, on top of the list above:
#   facebookexternalhit (~330/day)  Facebook/Instagram LINK PREVIEWS. This is a
#                                   different token from meta-externalagent and
#                                   must stay allowed or shared links lose their
#                                   preview cards. Do not merge the two.
#   meta-externalfetcher            user-initiated link fetches, same rationale
#                                   as ChatGPT-User. Not seen in the sample.
#   GoogleOther (~7.9k/day)         does not feed Search, so blocking is
#                                   documented-safe — but the risk is asymmetric
#                                   (Google-branded agent vs. ~6% of renders)
#                                   and it honours path rules, so it will follow
#                                   the staged Disallow at the foot of this file
#                                   instead. Revisit only if that lands and it
#                                   is still heavy.
#   OAI-SearchBot (~1.9k/day)       answer engine that cites.
#
# Ignoring this file entirely as of 07-31 (blocked above, still crawling):
#   SleepBot ~4,055/day   SpiderLing ~2,312/day   ShapBot ~354/day
#   Scrapy ~240/day
# robots.txt cannot reach these — they are already listed above and simply do
# not honour it. The only lever left is a UA check in server.ts. NOT Cloud
# Armor: alltrippers.com and www are custom-domain-mapped straight to Cloud Run
# with no load balancer in front (verified 2026-07-31 — zero backend-services,
# zero security-policies in the project), and Cloud Armor requires an external
# HTTPS LB. Adding one to block ~7k renders/day is not worth it.
# ─────────────────────────────────────────────────────────────────────────────
# UPDATE 2026-08-06 — Meta re-tokenised around the block, exactly as predicted.
#
# Every agent blocked on 07-26/07-31 now measures ZERO: Amazonbot 0,
# Amzn-SearchBot 0, meta-externalagent 0. The blocks work. What changed is that
# the volume came back under names this file did not list. Sample of 2,000
# requests, 2026-08-05 09:00-09:30 UTC:
#
#   meta-webindexer         423      <- NEW, unblocked
#   GeedoShopProductFinder  391      <- NEW, unblocked
#   Applebot                 86
#   bingbot                  52
#   Googlebot                 6
#
# Two unblocked agents were ~41% of requests against Googlebot's 6. Cloud Run
# requests went ~190k/day -> 689k (08-04) -> 773k (08-05); request CPU
# GBP 1.34 -> 2.34/day and intercontinental egress 0.16 -> 0.97/day.
#
# ⚠️ THEY COST COMPLETELY DIFFERENT THINGS — do not reason about them together:
#   meta-webindexer         500/500 sampled requests were HTML = SSR RENDERS.
#                           This is the CPU line. ~20k renders/day.
#   GeedoShopProductFinder  494/500 were static assets (JS chunks, images).
#                           Near-zero CPU; this is the EGRESS line.
#
# WHY meta-webindexer IS BLOCKED DESPITE BEING A CITING ANSWER ENGINE.
# It feeds Meta AI search and CAN cite/link sources, so by the rule used above
# (cites = allowed, trains-only = blocked) it belongs in the allowed group. It
# is blocked anyway because the citation channel measurably pays us nothing.
# GA4, 90 days to 2026-08-05, bot-filtered, by sessionSource:
#
#   chatgpt.com 446   perplexity 84   gemini 12   copilot 11   claude.ai 7
#   meta.ai       0   <- zero, while meta-externalagent crawled ~9k pages/day
#                        right up to the 07-31 block
#
# Allowing citing crawlers genuinely pays — ChatGPT sends 446 sessions. It just
# does not pay for Meta. And 79% of meta-webindexer's crawl is
# /fr/*/annuaire-professionnel/ — the business directory this site already
# NOINDEXED on 07-29, so even a future citation would surface a page we have
# decided should not appear in search at all.
#
# 🔁 REVERSE THIS IF: `meta.ai` ever appears in
#   npm run ga -- report --days=90 --dimensions=sessionSource --metrics=sessions
# Worth re-checking quarterly. The honest caveat is that meta-webindexer only
# began crawling us ~08-04, so its citations had not had time to appear.
#
# 🔴 DO NOT ADD facebookexternalhit. It is a DIFFERENT agent and it is what the
# 58 Facebook/Instagram sessions in that same 90 days actually come from —
# l.facebook.com and lm.facebook.com are its share-link redirectors. Blocking it
# kills link previews on every shared URL. Same for meta-externalfetcher
# (user-initiated). The warning above about not merging the Meta tokens is now
# backed by measurement, not just caution.
#
# GeedoShopProductFinder is geedo.com, a PRODUCT search engine indexing online
# stores (documented IPs 83.99.206.0/24, reverse-DNS *.geedo.com). We are a
# travel affiliate site, not a shop: it is crawling JS bundles and images
# hunting for products that do not exist. No search value, no referral value.
#
# LESSON, now confirmed twice: blocking by token is RECURRING MAINTENANCE.
# Amazonbot -> Amzn-SearchBot (07-27), meta-externalagent -> meta-webindexer
# (~08-04). Expect a third token each time. The durable fix is the staged
# path-level Disallow at the foot of this file, which stops the FETCH for every
# compliant agent at once — but it is gated on deindexation completing.
# ─────────────────────────────────────────────────────────────────────────────

User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: GeedoShopProductFinder
User-agent: AhrefsBot
User-agent: SemrushBot
User-agent: SleepBot
User-agent: PetalBot
User-agent: Bytespider
User-agent: DotBot
User-agent: SpiderLing
User-agent: YouBot
User-agent: Scrapy
User-agent: ShapBot
User-agent: VelenPublicWebCrawler
User-agent: AffsignalCrawler
User-agent: ABEvalBot
User-agent: HanaleiBot
User-agent: MJ12bot
User-agent: BLEXBot
User-agent: Barkrowler
User-agent: DataForSeoBot
User-agent: ImagesiftBot
User-agent: serpstatbot
User-agent: Diffbot
User-agent: SeekportBot
Disallow: /

# ─────────────────────────────────────────────────────────────────────────────
# STAGED — do not uncomment yet. Gated on deindexation completing.
#
# Why this beats blocking crawlers one token at a time: 76% of Amzn-SearchBot's
# crawl on 2026-07-31 was the country x entity crossproduct, which returns 200
# for every combination:
#
#   /en/mongolia/places/the-british-museum          -> 200
#   /en/guam/business-directory/oakwood-theme-park/avis  -> 200
#   /en/tunisia/places/national-gallery             -> 200
#
# ~200 countries x N entities x sub-pages (/avis, /a-propos, /evenements,
# /annonces, /codes-promo) x 9 locales is an unbounded URL space. Amazon's
# repeat rate across 1,617 requests was 0% — it is not re-crawling, it is
# walking a maze with no end, which is why its volume climbed 8k -> 19k/day.
#
# These pages already serve `X-Robots-Tag: noindex, follow` (shipped 01466-lag,
# 2026-07-29), so they will not be indexed — but noindex still costs a full SSR
# render every time, because the crawler must fetch the page to discover it.
# `follow` also means crawlers keep walking outward from them. Only a path-level
# Disallow stops the fetch, and it applies to every compliant agent — Googlebot
# included, which is the real prize: crawl budget currently spent here is budget
# not spent on the affiliate money pages.
#
# 🔴 SEQUENCING — why this is commented out:
# Disallow prevents Google from RE-CRAWLING, and therefore from ever seeing the
# noindex header. URLs already in the index at the time you block can get stuck
# there indefinitely. The deindex only shipped 2026-07-29. Wait until GSC shows
# these URLs dropped out of the index, THEN uncomment. Blocking too early
# freezes them in the index permanently — the exact opposite of the goal.
#
# This is the same conclusion already recorded at server.ts:966 ("Add the
# Disallow only once Search Console shows these deindexed"). That comment is the
# decision; this block is the prepared payload. Do not enable one without the
# other.
#
# ⚠️ SOURCE OF TRUTH — do not hand-edit this list.
# The segments below are the 28 distinct non-empty values of the five
# NOINDEXED_FUNCTIONALITY_KEYS (server.ts:495 — classifiedAds, placemarks,
# companies, events, news) resolved through FUNCTIONALITY_URIS in Enums.ts. They
# must stay in lockstep with NOINDEXED_PATH_RE, or robots.txt will block a path
# set that differs from the one being noindexed. Regenerate rather than patch:
#
#   node -e "const {FUNCTIONALITY_URIS}=require('./src/app/misc/Enums.ts'); \
#     console.log([...new Set(['classifiedAds','placemarks','companies', \
#     'events','news'].flatMap(k=>Object.values(FUNCTIONALITY_URIS[k]) \
#     .filter(v=>typeof v==='string'&&v&&!v.startsWith('_'))))].sort().join('\n'))"
#
# ⚠️ Empty-segment hazard: placemarks, events and news carry an EMPTY STRING for
# several locales in FUNCTIONALITY_URIS and fall back to `_default_` — which is
# why `places` covers de/es/it/nl/pl/pt/ro while only fr has its own (`lieux`).
# Verified against live traffic 2026-07-31: /de/.../places/, /ro/.../places/ all
# appear. The same empty values are what makes the `v.length > 0` filter at
# server.ts:510 load-bearing; the note there explains what an empty alternative
# does to the regex. Never emit a bare `Disallow: //`.
#
# ⚠️ Two forms per segment on purpose: the country-prefixed one and a root-level
# one, because a handful of URLs are served with no locale prefix (/places/…,
# /lieux/… both seen in logs). Do NOT collapse to `Disallow: /*places/` — a
# bare-substring pattern also swallows unrelated paths.
#
# ⚠️ The rules themselves are NOT here — they sit commented inside the
# `User-agent: *` group at the TOP of this file, because a second `User-agent: *`
# group down here would be silently dropped by strict parsers even though Google
# merges it. To enable: uncomment the 56 `# Disallow:` lines at the top. Nothing
# needs to change in this note.
# ─────────────────────────────────────────────────────────────────────────────

Sitemap: https://alltrippers.com/sitemap/sitemap-index.xml

Suchvorschau