HenkBot powers Valyu’s web indexing. It crawls and indexes content to deliver real-time, accurate search results through our APIs.Documentation Index
Fetch the complete documentation index at: https://docs.valyu.ai/llms.txt
Use this file to discover all available pages before exploring further.
User-Agent
Robots.txt Configuration
Allow HenkBot:robots.txt
robots.txt
Crawl Behavior
- Rate limiting - Adaptive and respectful of server load
- Robots.txt - Always honored
- Crawl-delay - Respected when specified
- Meta directives -
nofollowandnoindextags are followed
Partner With Us
If you’re a content owner, we offer revenue share partnerships with full attribution tracking. Your content can power AI agents while you maintain control and earn from every retrieval.Revenue Share Programme
Learn about content partnerships and monetisation