Di.se - SEO Check

Meta Tags

Medienliste

Überschriftenstruktur

Gefundene Links auf dieser Seite

HTTP-Header

Links von Wikipedia

• Antonio Cesti
• Antonio Salvi
• Dagens Industri
• Francesco Silvani
• Frida Gustavsson
• Hovet
• Lena Micko
• Michael William Balfe
• Rigoletto
• Teatro Malibran

Robots.txt

# All content on this website, including text, graphics, images, web design, film, sound 
# and other content, is protected by the Swedish Copyright Act (1960:729) (hereinafter
# "the Copyright Act"), both in terms of copyright and related rights. Bonnier News AB and
# its group companies (“Bonnier News”/”us”) reserve all rights to the content, including the
# right to use the content and the press publication for text and data extraction in
# accordance with Section 15a of the Copyright Act and Article 4 of the Copyright Directive.
#
# You may not use, nor may you facilitate or authorize the use of, the websites and/or any
# content for any purpose without prior written permission by Bonnier News, including, but
# not limited to, scraping the content or reproducing, modifying, collecting or extracting
# the content for text and data aggregation, analysis or extraction or commercial use.
# However, permission for website scraping and indexing may be granted for inclusion in a
# search engine provided that such authorization has been made explicitly by us through our
# technical instructions or similar specifications. Any form of scraping and indexing not
# expressly authorized by our technical instructions or specifications is strictly
# prohibited.

# Common Crawl robot, the resulting dataset is the primary training corpus in every LLM.
User-agent: CCBot
Disallow: /

# ChatGPT robot, used to improve the ChatGPT LLM. 
User-agent: ChatGPT-User
Disallow: /

# ChatGPT robot, may be used to improve the ChatGPT LLM. 
User-agent: GPTBot
Disallow: /

# Robot used to improve Bard and Vertex AI LLMs.
User-agent: Google-Extended
Disallow: /

# Associated with Google Vertex AI agents
User-agent: Google-CloudVertexBot
Disallow: /

# webz.io robot, the resulting dataset can and is purchased to train LLMs.
User-agent: Omgilibot
Disallow: /

# webz.io robot, the resulting dataset can and is purchased to train LLMs.
User-agent: Omgili
Disallow: /

# FacebookBot crawls public web pages to improve LLMs for Facebook's speech recognition technology. 
User-agent: FacebookBot
Disallow: /

# Another agent used by Anthropic that is more specifically related to Claude
User-agent: ClaudeBot
Disallow: /

# AI search indexing for Claude search capabilities
User-agent: Claude-SearchBot
Disallow: /

# AI agent for real-time web access when Claude users browse
User-agent: Claude-User
Disallow: /

# Diffbot crawls the web in or others to train their LLMs.
User-agent: Diffbot
Disallow: /

# Uses scraped data on-the-fly to create answers for DuckAssist.
User-agent: DuckAssistBot
Disallow: /

# Used by perplexity.ai. Generates text based on scraped material.
User-agent: PerplexityBot
Disallow: /

# AI agent for real-time browsing when Perplexity users request information
User-agent: Perplexity-User
Disallow: /

#  Cohere’s chatbot.
User-agent: cohere-ai
Disallow: /

#  Cohere’s chatbot.
User-agent: cohere-training-data-crawler
Disallow: /

# Use cases such as training AI models or improving products by indexing content directly.
User-agent: Meta-ExternalAgent
Disallow: /

# Crawler performs user-initiated fetches of individual links in support of some AI tools.
User-agent: Meta-ExternalFetcher
Disallow: /

# Used to improve Meta AI search.
User-agent: Meta-WebIndexer
Disallow: /

# Used by Timpi to scrape data for training their Large Language Models.
User-agent: Timpibot
Disallow: /

# Used by Webz.io to indicate that your site should not be included those using it to train AI models.
User-agent: Webzio-Extended
Disallow: /

# Crawler behind You.com’s AI search and browser assistant, indexing content for real-time answers.
User-agent: YouBot
Disallow: /

# Amazonbot is used to train Amazon services such as Alexa.
User-agent: Amazonbot
Disallow: /

# Bytespider is ByteDance's bot and may not respect robots.txt.
User-agent: Bytespider
Disallow: /

# Robot used to improve Anthropic AI LLMs.
User-agent: anthropic-ai
Disallow: /

# OpenAI search bot 
User-agent: OAI-SearchBot
Disallow: /

# Velen.io/Hunter.io "build business datasets and machine learning models to better understand the web" - seems to focus on collecting email adresses for spam though. 
User-agent: VelenPublicWebCrawler
Disallow: /

# AI research agent for Google Gemini’s Deep Research feature
User-agent: Gemini-Deep-Research
Disallow: /

# AI and machine learning data collection
User-agent: ICC-Crawler
Disallow: /

# Mistral’s real-time citation fetcher for “Le Chat” assistant
User-agent: MistralAI-User
Disallow: /

# additional bots and agents
# Training (27)
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: Applebot-Extended
User-agent: Brightbot
User-agent: Brightbot 1.0
User-agent: ChatGLM-Spider   # not documented by its operator
User-agent: CloudVertexBot
User-agent: Cotoyogi
User-agent: DeepSeekBot
User-agent: ERNIEBot   # not documented by its operator
User-agent: Factset_spyderbot
User-agent: FriendlyCrawler
User-agent: img2dataset
User-agent: ISSCyberRiskCrawler
User-agent: Kangaroo Bot
User-agent: KimiBot
User-agent: laion-huggingface-processor   # not documented by its operator
User-agent: LAIONDownloader
User-agent: Linguee Bot
User-agent: MistralAI-Training
User-agent: PanguBot
User-agent: QwenBot   # not documented by its operator
User-agent: Reflectionbot   # not documented by its operator
User-agent: SBIntuitionsBot
User-agent: Sidetrade indexer bot
User-agent: YandexAdditional
User-agent: YandexAdditionalBot
Disallow: /

# AI search (33)
User-agent: AddSearchBot
User-agent: AIWebIndex
User-agent: amazon-kendra
User-agent: Amzn-SearchBot
User-agent: Andibot
User-agent: Anomura
User-agent: atlassian-bot
User-agent: AzureAI-SearchBot   # not documented by its operator
User-agent: Channel3Bot   # not documented by its operator
User-agent: Cloudflare-AutoRAG
User-agent: ExaBot
User-agent: ExaSearchBot
User-agent: Google-NotebookLM
User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: HenkBot
User-agent: iAskBot   # not documented by its operator
User-agent: iaskspider   # not documented by its operator
User-agent: kagi-fetcher
User-agent: Kimi-SearchBot
User-agent: LinerBot
User-agent: LinkupBot   # not documented by its operator
User-agent: MistralAI-Index
User-agent: NotebookLM
User-agent: PetalBot
User-agent: PhindBot
User-agent: Querit-SearchBot
User-agent: QueritBot
User-agent: TavilyBot
User-agent: Terra Cotta
User-agent: TerraCotta
User-agent: ZanistaBot   # not documented by its operator
Disallow: /

# Live fetchers (25)
User-agent: AgentTimes
User-agent: AI2Bot-DeepResearchEval
User-agent: amazon-QBusiness
User-agent: Amzn-User
User-agent: bedrockbot
User-agent: BuddyBot
User-agent: Claude-Web   # not documented by its operator
User-agent: Diffbot-User
User-agent: DoubaoBot   # not documented by its operator
User-agent: GeistHaus-PageFetcher
User-agent: Google-Firebase
User-agent: Kimi-User
User-agent: KlaviyoAIBot
User-agent: NagetBot
User-agent: OpenAI   # not documented by its operator
User-agent: Poggio-Citations
User-agent: QualifiedBot
User-agent: QuillBot
User-agent: quillbot.com
User-agent: Shap-User
User-agent: ShapBot
User-agent: TongyiBot
User-agent: wpbot
User-agent: WRTNBot   # not documented by its operator
User-agent: YiyanBot
Disallow: /

# Resale (31)
User-agent: AgentDataBot
User-agent: aiHitBot
User-agent: ApifyBot
User-agent: ApifyWebsiteContentCrawler
User-agent: Awario   # not documented by its operator
User-agent: bigsur.ai
User-agent: BixelBot
User-agent: CloudflareBrowserRenderingCrawler
User-agent: CragCrawler
User-agent: Crawl4AI   # not documented by its operator
User-agent: Crawlspace
User-agent: Datenbank Crawler   # not documented by its operator
User-agent: Echobot Bot   # not documented by its operator
User-agent: EchoboxBot
User-agent: FirecrawlAgent
User-agent: ImagesiftBot
User-agent: imageSpider   # not documented by its operator
User-agent: LCC   # not documented by its operator
User-agent: MyCentralAIScraperBot   # not documented by its operator
User-agent: netEstate Imprint Crawler
User-agent: newsai
User-agent: Panscient
User-agent: panscient.com
User-agent: Poseidon Research Crawler
User-agent: Scrapy
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA
User-agent: Spider   # not documented by its operator
User-agent: Thinkbot
User-agent: WARDBot   # not documented by its operator
User-agent: YaK
Disallow: /

# Agentic (21)
User-agent: AmazonBuyForMe
User-agent: ChatGPT Agent
User-agent: Claude-Code
User-agent: Code
User-agent: Cursor
User-agent: Devin
User-agent: Google-Agent
User-agent: Google-Gemini-CLI
User-agent: GoogleAgent-Mariner
User-agent: GoogleAgent-URLContext
User-agent: Kimi-Agent   # not documented by its operator
User-agent: Lightpanda
User-agent: Manus-User
User-agent: Mozilla-Tabstack
User-agent: NovaAct
User-agent: opencode
User-agent: Operator
User-agent: qodercli   # not documented by its operator
User-agent: Trae
User-agent: TwinAgent
User-agent: UseAI
Disallow: /

# Purpose unknown (3)
User-agent: Aranet-SearchBot   # not documented by its operator
User-agent: KunatoCrawler   # not documented by its operator
User-agent: TikTokSpider
Disallow: /

User-agent: *
Disallow: /_alive/
Disallow: /_alive
Disallow: /login*
Disallow: /search*
Disallow: /bors/search*
Disallow: /*?tab=
Disallow: /konto/*
Disallow: /insider-bolag/
Disallow: /finansiell-information/pressreleaser-per-foretag/
Disallow: /finansiell-information/pressreleaser/?page=
Disallow: /insider-person/
Disallow: /Comments/WebServices/
Disallow: /*?fhtab=
Disallow: /*&fhtab=
Disallow: /*?allakommentarer=
Disallow: /*&allakommentarer=
Disallow: /*?timestamp=
Disallow: /*&timestamp=
Disallow: /*?t=
Disallow: /*&t=
Disallow: /*?flik=
Disallow: /*&flik=
Disallow: /*?qr=
Disallow: /*&qr=
Disallow: /*?screenwidth
Disallow: /*&screenwidth
Disallow: /*?screenheight
Disallow: /*&screenheight
Disallow: /*?ctl
Disallow: /*&ctl
Disallow: /*?currentIndex=
Disallow: /*?ns_
Disallow: /*&ns_
Disallow: /bip/
Disallow: /bip-callback*
Disallow: /register*
Disallow: /part*
Disallow: /akademi/anmalan/*
Disallow: /akademi/info-utbildningar/
Disallow: /34405621/bn/*
Disallow: /di-fonder/*
Disallow: /_akademi/
Disallow: /_akademi
Disallow: /logout

Sitemap: https://www.di.se/sitemap.xml
Sitemap: https://www.di.se/sitemap-news.xml

Suchvorschau