# robots.txt for www.brandsforless.com # --- Sitemap Index Reference --- Sitemap: https://www.brandsforless.com/sitemap-index.xml # --- Allow Google crawlers full access (except user-specific areas) --- User-agent: Googlebot Disallow: /login/ Disallow: /sign-up/ Disallow: */wishlist/ Disallow: */my-cart/ Disallow: */checkout/ Disallow: */my-account/ Allow: / User-agent: Googlebot-Image Allow: / User-agent: AdsBot-Google Allow: / # --- General bot rules --- User-agent: * Disallow: /login/ Disallow: /sign-up/ Disallow: */wishlist/ Disallow: */checkout/ Disallow: */my-cart/ Disallow: */my-account/ Disallow: /customer/ Disallow: /contentful/preview Disallow: /experiments Disallow: /v2/maintenance Disallow: /WebResource.axd Disallow: /ScriptResource.axd Disallow: /index.html Disallow: /styleguide.info Disallow: /submit-review.info Disallow: /products.foundationMatcher Disallow: /components/diversionNotifier/ Disallow: /components/internationalOverlay/ Disallow: /basketinterface.json?productId=* Disallow: /my.basket?*buy=* Disallow: /my.basket?*buylist=* Disallow: /*.location Disallow: /search/* Disallow: /*search=* Disallow: /*sortOrder=* Disallow: /*pagination_sortSelect=* Disallow: /*sessionSettings.overlay Disallow: /*switchLocale.overlay Disallow: /*_escaped_fragment_ Disallow: /*.html?buy=* Disallow: /*helpfulReview.account?*reviewId=* Disallow: /*reportReview.account?*reviewId=* Disallow: /*aggregate* Disallow: /*metrics* Disallow: /*page_view* Disallow: /*jserror* Disallow: /*undefined* Disallow: /*234567890987654323456789876545678* Disallow: /*utm_* Disallow: /*fbclid=* Disallow: /*gclid=* Disallow: /*session* Disallow: /*tracking* # --- Allow rendering-critical static assets --- Allow: /static/ Allow: /js/ Allow: /css/ Allow: /assets/ Allow: /images/ Allow: /fonts/ Allow: /_next/ Allow: /cdn-cgi/ # --- Allow key new arrivals pages --- Allow: */women/new-arrivals/ Allow: */men/new-arrivals/ Allow: */home-and-living/new-arrivals/ Allow: */kids/new-arrivals/ Allow: */sports/new-arrivals/ # --- Allow LLM bots --- User-agent: GPTBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / User-agent: GeminiBot Allow: / # Reference to llms.txt Llm-access: https://www.brandsforless.com/llms.txt # --- Block aggressive/scraping bots --- User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: PetalBot Disallow: / User-agent: Bytespider Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: HTTrack Disallow: / User-agent: Xenu Disallow: / User-agent: Zeus Disallow: / User-agent: RepoMonkey Disallow: / User-agent: Wget Disallow: / User-agent: LinkScan Disallow: / User-agent: NetAnts Disallow: / User-agent: Microsoft URL Control Disallow: / User-agent: CherryPicker Disallow: / User-agent: EmailSiphon Disallow: / User-agent: humanlinks Disallow: / User-agent: lwp-trivial Disallow: / User-agent: spanner Disallow: / User-agent: asterias Disallow: / User-agent: CheeseBot Disallow: / User-agent: turingos Disallow: / User-agent: DittoSpyder Disallow: / User-agent: Mata Hari Disallow: / User-agent: Crescent Disallow: / User-agent: Black Hole Disallow: / User-agent: BunnySlippers Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: /
Following keywords were found. You can check the keyword optimization of this page for each keyword.
(Nice to have)