User-agent: * Allow: / Disallow: /admin/ Disallow: /login Disallow: /test/ Disallow: /old.hassanradhi.com/ # OpenAI Search: allow; model-training crawler: block User-agent: OAI-SearchBot Allow: / Disallow: /admin/ Disallow: /login Disallow: /test/ Disallow: /old.hassanradhi.com/ User-agent: GPTBot Disallow: / # Google Search and Microsoft Copilot use their existing search crawlers User-agent: Googlebot Allow: / Disallow: /admin/ Disallow: /login Disallow: /test/ Disallow: /old.hassanradhi.com/ User-agent: Bingbot Allow: / Disallow: /admin/ Disallow: /login Disallow: /test/ Disallow: /old.hassanradhi.com/ # Perplexity Search User-agent: PerplexityBot Allow: / Disallow: /admin/ Disallow: /login Disallow: /test/ Disallow: /old.hassanradhi.com/ # Claude Search: allow; Claude model-training crawler: block User-agent: Claude-SearchBot Allow: / Disallow: /admin/ Disallow: /login Disallow: /test/ Disallow: /old.hassanradhi.com/ User-agent: ClaudeBot Disallow: / # Apple search: leave allowed by the general rule; # block Applebot-Extended training use User-agent: Applebot-Extended Disallow: / # Block Meta's named external crawler User-agent: meta-externalagent Disallow: / Sitemap: https://hassanradhi.com/sitemap.xml