# # Hello bot! There are some routes that need authentication. You should avoid them :) # # Hilfr, 2026-07-27: ALLE crawlere er tilladt — også trænings-crawlere (GPTBot, ClaudeBot, # Google-Extended, CCBot, Meta-ExternalAgent). Beslutning truffet af Marius: indholdet her er # markedsføring, ikke betalingsindhold, så det er en fordel at ligge i modellernes vægte. # AI-bottene har egne grupper UDEN Crawl-Delay, fordi en bot kun læser sin EGEN gruppe: # uden gruppen ville de arve `User-agent: *` og dermed 5 sekunders forsinkelse pr. side. # # ⚠️ Disallow-listen er gentaget i hver gruppe (robots.txt har ingen arv) OG i # robotsPrivateMarketplace.txt, som robotsTxt.js vælger i stedet, hvis marketplacen er sat # privat i Console. Ret ét sted = ret ALLE steder. robotsTxt.test.js fejler, hvis de driver # fra hinanden. # User-agent: * Disallow: /profile-settings Disallow: /l/new Disallow: /l/*/checkout Disallow: /l/*/draft Disallow: /l/*/pending-approval Disallow: /l/*/new Disallow: /l/*/edit Disallow: /inbox Disallow: /order Disallow: /sale Disallow: /listings Disallow: /account Disallow: /reset-password Disallow: /verify-email Disallow: /preview Crawl-Delay: 5 # --- AI: hente-/søge-bots (dem der henter en side for at kunne citere den i et svar) --- User-agent: OAI-SearchBot Disallow: /profile-settings Disallow: /l/new Disallow: /l/*/checkout Disallow: /l/*/draft Disallow: /l/*/pending-approval Disallow: /l/*/new Disallow: /l/*/edit Disallow: /inbox Disallow: /order Disallow: /sale Disallow: /listings Disallow: /account Disallow: /reset-password Disallow: /verify-email Disallow: /preview User-agent: ChatGPT-User Disallow: /profile-settings Disallow: /l/new Disallow: /l/*/checkout Disallow: /l/*/draft Disallow: /l/*/pending-approval Disallow: /l/*/new Disallow: /l/*/edit Disallow: /inbox Disallow: /order Disallow: /sale Disallow: /listings Disallow: /account Disallow: /reset-password Disallow: /verify-email Disallow: /preview User-agent: Claude-SearchBot Disallow: /profile-settings Disallow: /l/new Disallow: /l/*/checkout Disallow: /l/*/draft Disallow: /l/*/pending-approval Disallow: /l/*/new Disallow: /l/*/edit Disallow: /inbox Disallow: /order Disallow: /sale Disallow: /listings Disallow: /account Disallow: /reset-password Disallow: /verify-email Disallow: /preview User-agent: Claude-User Disallow: /profile-settings Disallow: /l/new Disallow: /l/*/checkout Disallow: /l/*/draft Disallow: /l/*/pending-approval Disallow: /l/*/new Disallow: /l/*/edit Disallow: /inbox Disallow: /order Disallow: /sale Disallow: /listings Disallow: /account Disallow: /reset-password Disallow: /verify-email Disallow: /preview User-agent: PerplexityBot Disallow: /profile-settings Disallow: /l/new Disallow: /l/*/checkout Disallow: /l/*/draft Disallow: /l/*/pending-approval Disallow: /l/*/new Disallow: /l/*/edit Disallow: /inbox Disallow: /order Disallow: /sale Disallow: /listings Disallow: /account Disallow: /reset-password Disallow: /verify-email Disallow: /preview # --- AI: trænings-crawlere (bevidst tilladt, se noten øverst) --- User-agent: GPTBot Disallow: /profile-settings Disallow: /l/new Disallow: /l/*/checkout Disallow: /l/*/draft Disallow: /l/*/pending-approval Disallow: /l/*/new Disallow: /l/*/edit Disallow: /inbox Disallow: /order Disallow: /sale Disallow: /listings Disallow: /account Disallow: /reset-password Disallow: /verify-email Disallow: /preview User-agent: ClaudeBot Disallow: /profile-settings Disallow: /l/new Disallow: /l/*/checkout Disallow: /l/*/draft Disallow: /l/*/pending-approval Disallow: /l/*/new Disallow: /l/*/edit Disallow: /inbox Disallow: /order Disallow: /sale Disallow: /listings Disallow: /account Disallow: /reset-password Disallow: /verify-email Disallow: /preview User-agent: Google-Extended Disallow: /profile-settings Disallow: /l/new Disallow: /l/*/checkout Disallow: /l/*/draft Disallow: /l/*/pending-approval Disallow: /l/*/new Disallow: /l/*/edit Disallow: /inbox Disallow: /order Disallow: /sale Disallow: /listings Disallow: /account Disallow: /reset-password Disallow: /verify-email Disallow: /preview Sitemap: https://hilfr.dk/sitemap-index.xml