User-agent: * Disallow: /admin/ Disallow: /admin-dashboard/ Disallow: /api/ Disallow: /payment-gateway/ Disallow: /reward/ Disallow: /notice/ Disallow: /inbox/ Disallow: /accounts/ Disallow: /order/ Disallow: /e-wallet/ Disallow: /dashboard Disallow: /item/ Disallow: /yard/ Disallow: /__debug__/ Disallow: /media/ # Block tracking-param duplicates from the index. Canonical tags also # handle this, but Disallow is a cleaner signal for crawlers that ignore # rel=canonical (Yandex sometimes does). Disallow: /*?utm_ Disallow: /*?fbclid= Disallow: /*?gclid= # Link-preview crawlers get their own group. They must be able to fetch URLs # carrying ad parameters: every shared campaign/ad link has ?utm_, which the # Disallow above would block — the share would then render as a bare URL with # no card. These agents build previews, they do not index, so allowing the # parameterised URLs costs nothing. A named group REPLACES the * group for # that agent, so the private paths are restated here; within a group the # longest matching rule wins, so Allow: / does not reopen /admin/. User-agent: facebookexternalhit User-agent: Facebot User-agent: Twitterbot User-agent: WhatsApp User-agent: TelegramBot User-agent: LinkedInBot User-agent: Slackbot-LinkExpanding User-agent: Discordbot Disallow: /admin/ Disallow: /admin-dashboard/ Disallow: /api/ Disallow: /payment-gateway/ Disallow: /reward/ Disallow: /notice/ Disallow: /inbox/ Disallow: /accounts/ Disallow: /order/ Disallow: /e-wallet/ Disallow: /dashboard Disallow: /yard/ Disallow: /__debug__/ Allow: / # AI assistants fetching a URL a USER just gave them. These are not indexers — # they retrieve one page, on demand, because someone pasted or asked about that # exact link. They get the same treatment as the link-preview crawlers above, # and for the same reason: every shared campaign URL carries ?utm_, which the # Disallow rules in the * group would otherwise block, so the assistant would # report the page as inaccessible to the very person who asked about it. # # Same trap as the group above — a named group REPLACES the * group for that # agent, so the private paths must be restated here. Do not "tidy" them away; # within a group the longest matching rule wins, so `Allow: /` does not reopen # /admin/. Verified by tests/test_robots_ai_policy.py. User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: Perplexity-User Disallow: /admin/ Disallow: /admin-dashboard/ Disallow: /api/ Disallow: /payment-gateway/ Disallow: /reward/ Disallow: /notice/ Disallow: /inbox/ Disallow: /accounts/ Disallow: /order/ Disallow: /e-wallet/ Disallow: /dashboard Disallow: /yard/ Disallow: /__debug__/ Allow: / # Indexing / training crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, # Google-Extended, Applebot-Extended, Bytespider, Amazonbot, meta-externalagent) # are DELIBERATELY not named here, which leaves them under the `*` group above: # allowed on the public catalogue, blocked from the private paths, and blocked # from ?utm_ duplicates. That is the intended policy — being readable is how a # parts importer gets recommended when someone asks an assistant where to buy # JDM parts in Malaysia, and the catalogue is public information anyway. # # To opt out of AI TRAINING while staying citable in AI SEARCH, add a group # disallowing GPTBot / CCBot / Google-Extended / Applebot-Extended and leave # the retrieval agents above allowed. That is a business decision, not a # technical one, so it is not made here. Sitemap: https://mobieco.io/sitemap.xml