User-Agent
Robots.txt Configuration
Allow HenkBot:robots.txt
robots.txt
Crawl Behavior
- Rate limiting - Adaptive and respectful of server load
- Robots.txt - Always honored
- Crawl-delay - Respected when specified
- Meta directives -
nofollowandnoindextags are followed
Partner With Us
If you’re a content owner, we offer revenue share partnerships with full attribution tracking. Your content can power AI agents while you maintain control and earn from every retrieval.Revenue Share Programme
Learn about content partnerships and monetisation