> ## Documentation Index
> Fetch the complete documentation index at: https://docs.valyu.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawler

> Information for webmasters about HenkBot, Valyu's web crawler

HenkBot powers Valyu's web indexing. It crawls and indexes content to deliver real-time, accurate search results through our APIs.

## User-Agent

```
Mozilla/5.0 (compatible; HenkBot/1.0; +https://valyu.ai/crawler)
```

## Robots.txt Configuration

**Allow HenkBot:**

```txt robots.txt theme={null}
User-agent: HenkBot
Allow: /
```

**Block HenkBot:**

```txt robots.txt theme={null}
User-agent: HenkBot
Disallow: /
```

## Crawl Behavior

* **Rate limiting** - Adaptive and respectful of server load
* **Robots.txt** - Always honored
* **Crawl-delay** - Respected when specified
* **Meta directives** - `nofollow` and `noindex` tags are followed

## Partner With Us

If you're a content owner, we offer revenue share partnerships with full attribution tracking. Your content can power AI agents while you maintain control and earn from every retrieval.

<Card title="Revenue Share Programme" icon="handshake" href="https://www.valyu.ai/rev-share-partner-programme">
  Learn about content partnerships and monetisation
</Card>

## Questions?

Contact us at [contact@valyu.ai](mailto:contact@valyu.ai) for crawler-related inquiries.