# =========================================================================== # robots.txt — vindobona.wien # # Context: investigation-report/FINDINGS.md (Action 3 in ACTION-PLAN.md) # # Every crawler blocked below was measured in this site's own access logs # (477,566 requests, 12-31 Jul 2026). Together the blocked set accounted for # 11,440 of 200,459 dynamic HTML page requests, i.e. 5.7% of all page # renders. Each of those is a full, uncached Craft render, so this is real # PHP and MySQL load rather than static file serving. # # Search engines and social link-preview fetchers are deliberately NOT # blocked. See the "deliberately allowed" note at the end of this file # before adding anything. # # Note: robots.txt is advisory. Well-behaved crawlers honour it; scrapers # that ignore it need blocking at the server or CDN instead (which requires # server access this project does not currently have). # # Numbers in [brackets] below are measured HTML page requests in that window. # =========================================================================== Sitemap: https://vindobona.wien/sitemap.xml # --------------------------------------------------------------------------- # All crawlers # --------------------------------------------------------------------------- # /cp control panel; a crawl there is a full Craft boot to reach a # login screen. Also covers /cpresources/. Verified no public # page path begins with "cp". # /export the "Export" single (template _export). A print/export view # that runs an unbounded events query, and is not intended as # a search landing page. User-agent: * Disallow: /cp Disallow: /cpresources/ Disallow: /export # --------------------------------------------------------------------------- # Generative-AI training opt-out tokens # --------------------------------------------------------------------------- # These two are NOT crawlers and cost no traffic. They are control tokens # that opt this site out of AI model training with no effect on search: # # Applebot-Extended # Apple states that disallowing it does not stop Applebot crawling for # search, and content remains discoverable via Spotlight, Siri and # Safari. https://support.apple.com/en-us/HT204683 # # Google-Extended # Google states it does not affect a site's inclusion in Google Search # and is not a ranking signal. It does not crawl; Googlebot fetches. # https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers # # (Content was rephrased from those sources for licensing compliance.) User-agent: Applebot-Extended Disallow: / User-agent: Google-Extended Disallow: / # --------------------------------------------------------------------------- # AI / LLM crawlers — blocked # --------------------------------------------------------------------------- # These crawl for model training or dataset building and send effectively no # referral traffic. meta-externalagent alone was the single largest crawler # cost on the site. # # meta-externalagent [6013] ClaudeBot [1792] GPTBot [186] # Amazonbot [64] Timpibot [34] Bytespider [10] # PerplexityBot [10] CCBot [1] anthropic-ai [1] # # The remainder are pre-emptive: not yet seen in the logs, but same category. User-agent: meta-externalagent User-agent: Meta-ExternalFetcher User-agent: ClaudeBot User-agent: Claude-Web User-agent: anthropic-ai User-agent: GPTBot User-agent: CCBot User-agent: Bytespider User-agent: Amazonbot User-agent: Timpibot User-agent: PerplexityBot User-agent: Diffbot User-agent: omgili User-agent: omgilibot User-agent: ImagesiftBot User-agent: FriendlyCrawler User-agent: Google-CloudVertexBot Disallow: / # --------------------------------------------------------------------------- # SEO / competitive-analysis crawlers — blocked # --------------------------------------------------------------------------- # These exist to sell backlink and keyword data to third parties. They bring # no visitors and no search visibility to this site. # # SemrushBot [2607] DotBot [648] AhrefsBot [540] PetalBot [380] # Barkrowler [167] MJ12bot [26] DataForSeoBot [9] User-agent: SemrushBot User-agent: AhrefsBot User-agent: DotBot User-agent: PetalBot User-agent: Barkrowler User-agent: MJ12bot User-agent: DataForSeoBot User-agent: BLEXBot Disallow: / # =========================================================================== # Deliberately allowed — do not block these without a deliberate decision # =========================================================================== # # Search engines (visibility for a venue depends on these): # Googlebot [.] Google web search # Googlebot-Image image search # AdsBot-Google checks Google Ads landing pages. This site runs # Google Ads (gclid parameters appear throughout # the access logs), so blocking it would affect ad # quality checks. # bingbot [.] Bing, and Bing powers other engines # Applebot [19946] the single largest crawler here. Left allowed # because it powers Spotlight, Siri and Safari # suggestions. AI training is opted out separately # via Applebot-Extended above. # DuckDuckBot, YandexBot, SeznamBot, Baiduspider # # Social link-preview fetchers (needed so shared links render a card): # facebookexternalhit [5191] note: this is the preview fetcher and is a # different agent from meta-externalagent # blocked above. The site links to its own # Facebook profile in the footer. # Twitterbot, LinkedInBot, Slackbot, Discordbot, TelegramBot, Pinterestbot # # AI agents acting on a real user's request, left allowed on purpose because # they represent actual user intent and can send referrals: # OAI-SearchBot [114] powers ChatGPT search citations # ChatGPT-User [393] a person asked ChatGPT to fetch this page # Perplexity-User [2] a person asked Perplexity to fetch this page # If you would rather block these too, move them into the AI crawler group # above; it is a one-line change each. # # Not addressed here: Applebot volume. Applebot is by far the heaviest # crawler at ~20k requests in the window. Crawl-delay is the usual lever, # but it is a non-standard extension with inconsistent support (Google # ignores it outright) and Apple's documentation does not confirm whether # Applebot honours it, so no Crawl-delay has been added rather than adding # a directive that may do nothing. Reducing Applebot load reliably needs # rate limiting at the server or CDN. # ===========================================================================