Haycion's public pages, the Advisor Network listings included, are written to be read by AI assistants. This page says which crawlers may read them, and on what terms. Our robots.txt says the same thing in a form crawlers read.
1. Welcome: AI search and on-request readers
Crawlers that keep an AI assistant's search up to date, or read a page because someone just asked about an advisor, may read every public page.
OAI-SearchBot(OpenAI)ChatGPT-User(OpenAI)Claude-User(Anthropic)Claude-SearchBot(Anthropic)PerplexityBot(Perplexity)Perplexity-User(Perplexity)meta-externalfetcher(Meta)DuckAssistBot(DuckDuckGo)YouBot(You.com)MistralAI-User(Mistral AI)
2. Allowed on terms: AI training crawlers
Crawlers that collect pages to train AI models may read every public page, as long as they:
- identify themselves with the user agent their operator publishes, and follow robots.txt;
- read at a modest rate;
- use the pages only to train and evaluate models, and don't republish them, or sell or license them as a dataset or through a search or data API.
Anything beyond that needs a licence (section 3). The training crawlers we know of:
GPTBot(OpenAI)ClaudeBot(Anthropic)Claude-Web(Anthropic)anthropic-ai(Anthropic)meta-externalagent(Meta)Bytespider(ByteDance)cohere-ai(Cohere)Amazonbot(Amazon)Google-Extended(a robots.txt setting, not a crawler)
3. Licensed crawlers
To use Haycion's public data in other ways, for example the Advisor Network in bulk or as a regular feed, email hello@haycion.com.
4. Blocked: link indexes
Services that collect pages to resell them through a search or data API may not read Haycion. robots.txt asks them to stay away, and the site answers them with an error (403). If you run one and want access, ask for a licence.
5. Everyone else
People, search engines and crawlers not named here read the site as usual. We update these lists as crawlers come and go; robots.txt always has the current ones.
