Prepared with AI assistance and authored under the Sunpersona brand. Source links, test dates and public records are provided for verification; this is not proof of an individual's credentials or client outcomes.
Separate three kinds of access
| Purpose | Examples | Meaning |
|---|---|---|
| Search crawling | OAI-SearchBot / PerplexityBot / Claude-SearchBot | Check access for search discovery |
| Training crawling | GPTBot / ClaudeBot | Training permission is a separate choice |
| User-requested fetch | ChatGPT-User / Perplexity-User / Claude-User | A one-off fetch is not long-term indexing |
OpenAI separates search and training permissions. ChatGPT-User is user initiated; robots rules may not apply and it does not determine Search eligibility.
Inspect robots.txt
Read the actual root-level rules and check specific crawler groups. A general allow rule does not simply replace a specific disallow group. Business pages and required rendering resources should be accessible.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlThis is a minimal example for a public site. Keep appropriate path and training policies; protect private content with access controls.
Inspect network protection
If rules permit access but requests receive a challenge or 403, inspect the matching security event. Verify bot identity using official addresses or verification information, then adjust the affected rule. Do not broadly allow a name alone.
Perplexity distinguishes search crawling from user fetches and provides network-rule guidance. Anthropic says its bots honor robots.txt. Check each platform separately.
What evidence establishes access
- Command checks: status, destination and actual content.
- Platform live tests: fetch result and crawl permission.
- Verified bot logs: authentic requests and responses.
- Indexation: recorded separately from access evidence.
Putting a bot name in curl simulates a request. It does not cause that platform to crawl or index the URL.
How to handle a block
Keep the exact URL, crawl date and error content, fix the relevant rule, then retest. Search Console email may reflect an older version, so retain both the historical report and live result.