# robots.txt for Halal Gourmet Japan # halalgourmet.jp # # This file controls how search engines, AI crawlers, and other bots access the site. # Last updated: 2026-08-20 # # Query-bearing internal search URLs (/search?..., /{lang}/search?...) are # disallowed for bulk crawlers: freeword search is DB-heavy and uncacheable (#678). # The /search landing page itself stays crawlable (sitemap / canonical target). # The wildcard line (/*/search?) auto-covers future locales on wildcard-capable # bots; locales are also listed explicitly for parsers without wildcard support. # --------------------------------------------------------------------------- # Default rules - apply to all bots not explicitly listed below # --------------------------------------------------------------------------- User-agent: * Disallow: /api/ Disallow: /admin/ Disallow: /private/ Disallow: /search? Disallow: /ja/search? Disallow: /id/search? Disallow: /fr/search? Disallow: /de/search? Disallow: /ms/search? Disallow: /tr/search? Disallow: /ar/search? Disallow: /*/search? Allow: / # --------------------------------------------------------------------------- # Major search engines # --------------------------------------------------------------------------- User-agent: Googlebot Disallow: /api/ Disallow: /admin/ Disallow: /private/ Disallow: /search? Disallow: /ja/search? Disallow: /id/search? Disallow: /fr/search? Disallow: /de/search? Disallow: /ms/search? Disallow: /tr/search? Disallow: /ar/search? Disallow: /*/search? Allow: / User-agent: Bingbot Disallow: /api/ Disallow: /admin/ Disallow: /private/ Disallow: /search? Disallow: /ja/search? Disallow: /id/search? Disallow: /fr/search? Disallow: /de/search? Disallow: /ms/search? Disallow: /tr/search? Disallow: /ar/search? Disallow: /*/search? Allow: / User-agent: Slurp Disallow: /api/ Disallow: /admin/ Disallow: /private/ Disallow: /search? Disallow: /ja/search? Disallow: /id/search? Disallow: /fr/search? Disallow: /de/search? Disallow: /ms/search? Disallow: /tr/search? Disallow: /ar/search? Disallow: /*/search? Allow: / User-agent: DuckDuckBot Disallow: /api/ Disallow: /admin/ Disallow: /private/ Disallow: /search? Disallow: /ja/search? Disallow: /id/search? Disallow: /fr/search? Disallow: /de/search? Disallow: /ms/search? Disallow: /tr/search? Disallow: /ar/search? Disallow: /*/search? Allow: / # --------------------------------------------------------------------------- # AI answer / search crawlers (drive referral traffic - welcomed) # --------------------------------------------------------------------------- User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Claude-User User-agent: Claude-SearchBot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: /api/ Disallow: /admin/ Disallow: /private/ Disallow: /search? Disallow: /ja/search? Disallow: /id/search? Disallow: /fr/search? Disallow: /de/search? Disallow: /ms/search? Disallow: /tr/search? Disallow: /ar/search? Disallow: /*/search? Allow: / # --------------------------------------------------------------------------- # AI training crawlers (allowed - see PR notes for trade-offs) # --------------------------------------------------------------------------- User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Amazonbot User-agent: Meta-ExternalAgent Disallow: /api/ Disallow: /admin/ Disallow: /private/ Disallow: /search? Disallow: /ja/search? Disallow: /id/search? Disallow: /fr/search? Disallow: /de/search? Disallow: /ms/search? Disallow: /tr/search? Disallow: /ar/search? Disallow: /*/search? Allow: / # --------------------------------------------------------------------------- # Social media crawlers # --------------------------------------------------------------------------- User-agent: facebookexternalhit User-agent: Twitterbot Disallow: /api/ Disallow: /admin/ Disallow: /private/ Allow: / # --------------------------------------------------------------------------- # Blocked: SEO backlink scanners & aggressive scrapers # --------------------------------------------------------------------------- User-agent: AhrefsBot User-agent: SemrushBot User-agent: DotBot User-agent: MJ12bot User-agent: PetalBot User-agent: Scrapy Disallow: / # --------------------------------------------------------------------------- Sitemap: https://halalgourmet.jp/sitemap.xml