User agent
Configure robots.txt
Allow HenkBot:robots.txt
robots.txt
Crawl behaviour
- Rate limiting - Adaptive and respectful of server load
- Robots.txt - Always honored
- Crawl-delay - Respected when specified
- Meta directives -
nofollowandnoindextags are followed
Content partnerships
If you own content and want to make it available through Valyu, contact the team to discuss access, attribution and commercial terms.Revenue Share Programme
Learn about content partnerships and monetisation