Do nowAI crawler access
ClaudeBot and PerplexityBot are disallowed
- What we found
- robots.txt contains Disallow: / for ClaudeBot and PerplexityBot.
- Evidence
- Fetched https://example-b2b.com/robots.txt — lines 14 and 19.
- Why it matters
- Two of the four engines your buyers use cannot fetch a single page, so no content or authority work can reach them.
- What to change
- Replace both blocks with Allow: / , or scope the Disallow to /app/ and /account/ only.
- Owner
- Engineering or whoever owns DNS and hosting
- Effort
- 15 minutes
- How we verify
- We re-fetch robots.txt and confirm both agents are permitted.