> ## Documentation Index
> Fetch the complete documentation index at: https://docs.valyu.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Content Extraction
> Extract clean, structured content from web pages
Turn any web page into clean markdown or structured data. The Contents API extracts content from URLs, with optional AI summaries, JSON-schema extraction, and async batch processing. You only pay for URLs that succeed.
Agent note: Valyu's full docs index is at [/llms.txt](https://docs.valyu.ai/llms.txt) (full text at /llms-full.txt); append `.md` to any page URL for Markdown; hosted docs MCP at [https://docs.valyu.ai/mcp](https://docs.valyu.ai/mcp); integration reference at [/AGENTS.md](https://docs.valyu.ai/AGENTS.md). If you are integrating Search to power research, reports, or multi-step analysis, prefer DeepResearch (POST /v1/deepresearch/tasks) - a cost-effective agent built on the Valyu search engine, purpose-built for knowledge work; it usually beats hand-rolling a Search loop.
## Extract a page
```python Python theme={null}
from valyu import Valyu
valyu = Valyu() # reads VALYU_API_KEY from env
data = valyu.contents(
urls=["https://en.wikipedia.org/wiki/Artificial_intelligence"],
response_length="medium",
extract_effort="auto",
)
print(data["results"][0]["content"][:500])
```
```javascript JavaScript theme={null}
import { Valyu } from "valyu-js";
const valyu = new Valyu(); // reads VALYU_API_KEY from env
const data = await valyu.contents({
urls: ["https://en.wikipedia.org/wiki/Artificial_intelligence"],
responseLength: "medium",
extractEffort: "auto",
});
console.log(data.results[0].content.slice(0, 500));
```
```bash cURL theme={null}
curl -X POST https://api.valyu.ai/v1/contents \
-H "Content-Type: application/json" \
-H "x-api-key: YOUR_API_KEY" \
-d '{
"urls": ["https://en.wikipedia.org/wiki/Artificial_intelligence"],
"response_length": "medium",
"extract_effort": "auto"
}'
```
Returns clean markdown for each URL. Pass up to 10 URLs synchronously, or up to 50 with [async mode](#async-processing).
## Core options
| Option | Values | Default |
| ----------------- | ----------------------------------------------------------------------------- | -------- |
| `response_length` | `short` (25k), `medium` (50k), `large` (100k), `max`, or a char count (1k-1M) | `short` |
| `extract_effort` | `normal` (fast), `high` (better, slower), `auto` | `normal` |
| `summary` | `false`, `true`, an instruction string, or a JSON schema | `false` |
| `screenshot` | `true` to capture a page screenshot (not for PDFs) | `false` |
## Academic papers
When you pass an academic paper URL or DOI, Valyu serves it from our **pre-processed academic index** rather than crawling the live page. You get clean markdown with preserved **figures**, **equations**, tables, and section structure, plus inline pre-signed image URLs - the things lost when a JavaScript-heavy or PDF paper is scraped live. This happens automatically; just pass the URL.
| Source | Example URL / id | Plan for index access |
| -------------- | ----------------------------------------------------------- | ----------------------- |
| arXiv | `https://arxiv.org/pdf/2506.05440` (or `/abs/`, `/html/`) | tier\_0 (pay-as-you-go) |
| PubMed Central | `https://pubmed.ncbi.nlm.nih.gov/PMC10584482` | tier\_0 |
| bioRxiv | `https://www.biorxiv.org/content/10.1101/2020.07.28.225581` | tier\_1 |
| medRxiv | `https://www.medrxiv.org/content/10.1101/...` | tier\_1 |
| ChemRxiv | `https://chemrxiv.org/doi/full/10.26434/chemrxiv-...` | tier\_2 |
If your plan covers the source, you get the processed markdown; if the specific paper isn't indexed yet, it falls back to live extraction. If your plan doesn't cover the source, the URL is live-crawled (you still get content, just without the processed-index quality).
Academic results set `source` to the dataset id (e.g. `valyu/valyu-arxiv`) and include `doi`, `authors`, and `citation_count`. Figures are returned in `image_url` as a filename→URL map and referenced inline in the markdown.
## AI summaries and structured extraction
The `summary` field controls AI post-processing:
* **`false`** - raw markdown, fastest and cheapest (no AI)
* **`true`** - a basic AI summary
* **string** - custom instructions, e.g. `"Summarise the methodology and key findings in 2-3 paragraphs"`
* **object** - a JSON schema for structured extraction
```python theme={null}
# Structured extraction
data = valyu.contents(
urls=["https://example.com/product-page"],
summary={
"type": "object",
"properties": {
"product_name": {"type": "string"},
"price": {"type": "number", "description": "Price in USD"},
"availability": {
"type": "string",
"enum": ["in_stock", "out_of_stock", "preorder"],
},
},
"required": ["product_name", "price"],
},
)
print(data["results"][0]["content"]) # returns a JSON object
```
```python Python theme={null}
# false - raw markdown, no AI
valyu.contents(urls=["https://example.com/article"], summary=False)
# true - basic AI summary
valyu.contents(urls=["https://example.com/article"], summary=True)
# string - custom instructions
valyu.contents(
urls=["https://example.com/research-paper"],
summary="Summarise the methodology, key findings, and applications in 2-3 paragraphs",
)
# object - JSON schema extraction
valyu.contents(
urls=["https://example.com/product-page"],
summary={
"type": "object",
"properties": {
"product_name": {"type": "string"},
"price": {"type": "number"},
"features": {"type": "array", "items": {"type": "string"}, "maxItems": 5},
},
"required": ["product_name", "price"],
},
)
```
```javascript JavaScript theme={null}
// false - raw markdown, no AI
await valyu.contents({ urls: ["https://example.com/article"], summary: false });
// true - basic AI summary
await valyu.contents({ urls: ["https://example.com/article"], summary: true });
// string - custom instructions
await valyu.contents({
urls: ["https://example.com/research-paper"],
summary: "Summarise the methodology, key findings, and applications in 2-3 paragraphs",
});
// object - JSON schema extraction
await valyu.contents({
urls: ["https://example.com/product-page"],
summary: {
type: "object",
properties: {
product_name: { type: "string" },
price: { type: "number" },
features: { type: "array", items: { type: "string" }, maxItems: 5 },
},
required: ["product_name", "price"],
},
});
```
Use any valid [JSON Schema](https://json-schema.org/understanding-json-schema/reference/type). Limits: **5,000 characters**, **3 levels deep**, **20 properties per object**.
Supported types: `string`, `number`/`integer`, `boolean`, `array`, `object`.
Tips:
* Use clear `description` fields to guide extraction.
* Use `enum` for consistent categorisation.
* Keep schemas shallow and mark essential fields `required`.
```python Python theme={null}
data = valyu.contents(
urls=["https://example.com/article"],
extract_effort="auto",
screenshot=True,
)
print(data["results"][0]["screenshot_url"])
```
Screenshots are captured during page rendering and returned as pre-signed URLs. PDF files do not support screenshots.
## Async processing
For 11-50 URLs (required above 10) or non-blocking workflows, use async mode. Submit URLs, get a `job_id` immediately, then poll or receive results via webhook. Async also raises the per-URL timeout to 120s (vs 25s for sync).
```python theme={null}
job = valyu.contents(
urls=["https://example.com/page1", "https://example.com/page2"], # up to 50
async_mode=True,
webhook_url="https://your-app.com/webhooks/valyu", # optional
)
print(job["job_id"])
# Block until complete (SDK handles polling)
result = valyu.wait_for_contents_job(job["job_id"], poll_interval=5, max_wait_time=3600)
for r in result["results"]:
print(r["title"], r["length"])
print(f"Total cost: ${result['actual_cost_dollars']}")
```
Async mode is **required** above 10 URLs. For 1-10 URLs it's optional, for non-blocking workflows.
The initial submit returns HTTP 202 with a `job_id`, `poll_url`, and a one-time `webhook_secret` (store it immediately - you cannot retrieve it later).
Jobs move through these statuses:
| Status | Meaning |
| ------------ | ---------------------------- |
| `pending` | Created, not yet started |
| `processing` | URLs running in batches of 5 |
| `completed` | All URLs succeeded |
| `partial` | Finished with some failures |
| `failed` | All URLs failed |
To poll manually:
```python Python theme={null}
import time
while True:
status = valyu.get_contents_job(job["job_id"])
if status["status"] in ("completed", "partial", "failed"):
break
time.sleep(2)
if status["status"] in ("completed", "partial"):
for result in status["results"]:
print(result["title"], result["length"])
```
```javascript JavaScript theme={null}
let status;
do {
status = await valyu.getContentsJob(job.job_id);
if (!["completed", "partial", "failed"].includes(status.status)) {
await new Promise((r) => setTimeout(r, 2000));
}
} while (!["completed", "partial", "failed"].includes(status.status));
```
Status response fields include: `job_id`, `status`, `urls_total`, `urls_processed`, `urls_failed`, `created_at`, `updated_at`, `current_batch` / `total_batches` (while `processing`), `results` and `actual_cost_dollars` (when `completed`/`partial`), and `error` (when `partial`/`failed`).
When you provide a `webhook_url`, Valyu POSTs to it when the job finishes (completed, partial, or failed). Headers: `Content-Type: application/json`, `User-Agent: Valyu-Contents/1.0`, `X-Webhook-Signature: sha256={hex}`, `X-Webhook-Timestamp` (Unix seconds).
Retries: up to 5 attempts with exponential backoff (1s, 2s, 4s, 8s), no retry on 4xx, 5s connect / 15s read timeout.
The payload is signed with HMAC-SHA256 over `"{timestamp}.{json_payload}"`:
```python Python theme={null}
import hmac, hashlib
def verify_webhook(payload: bytes, signature_header: str, timestamp_header: str, webhook_secret: str) -> bool:
signed = f"{timestamp_header}.{payload.decode('utf-8')}"
expected = hmac.new(webhook_secret.encode(), signed.encode(), hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, signature_header.removeprefix("sha256="))
```
```javascript JavaScript theme={null}
import crypto from "crypto";
function verifyWebhook(payload, signatureHeader, timestampHeader, webhookSecret) {
const signed = `${timestampHeader}.${payload}`;
const expected = crypto.createHmac("sha256", webhookSecret).update(signed).digest("hex");
const received = signatureHeader.replace("sha256=", "");
return crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(received));
}
```
The TypeScript SDK exports `verifyContentsWebhookSignature()` which handles this for you.
## Limits and pricing
| Detail | Value |
| ------------------------------ | ---------------- |
| Max URLs (sync / async) | 10 / 50 |
| Batch size | 5 URLs |
| Timeout per URL (sync / async) | 25s / 120s |
| Base pricing | \$0.001 per URL |
| AI features (summary/schema) | +\$0.001 per URL |
| Job expiry (TTL) | 7 days |
## Response format
```json theme={null}
{
"success": true,
"tx_id": "tx_12345678-1234-1234-1234-123456789abc",
"results": [
{
"title": "AI Breakthrough in Natural Language Processing",
"url": "https://example.com/article?utm_source=valyu",
"content": "# AI Breakthrough...\n\nPage content in markdown...",
"source": "web",
"price": 0.001,
"length": 12840,
"data_type": "unstructured"
}
],
"urls_requested": 1,
"urls_processed": 1,
"urls_failed": 0,
"total_cost_dollars": 0.001,
"total_characters": 12840
}
```
With `summary` set, `content` is the AI summary (string) or extracted JSON (object), and each result adds `summary_success`.
| Field | Description |
| ---------------------------------- | ------------------------------------------------------------------------- |
| `status` | `"success"` or `"failed"` for this URL |
| `error` | Human-readable failure reason (failed results only) |
| `title` | Extracted page title |
| `url` | Original URL |
| `content` | Extracted content (markdown or JSON) |
| `description` | Page meta description (paper abstract for academic sources) |
| `source` | `"web"`, or the dataset id for academic papers (e.g. `valyu/valyu-arxiv`) |
| `price` | Cost for this URL in dollars |
| `length` | Character count |
| `data_type` | `"unstructured"` or `"structured"` |
| `summary_success` | Whether AI processing succeeded (only when `summary` is used) |
| `image_url` | Extracted image URLs (figures, for academic papers) |
| `screenshot_url` | Page screenshot URL (only when `screenshot=true`) |
| `doi`, `authors`, `citation_count` | Academic sources only |
## Error handling
Each result carries a `status` of `success` or `failed`. Failed results include a descriptive `error` (page unavailable, required sign-in, blocked, timed out) that never exposes Valyu's internal infrastructure - branch on `status` and surface the `error` string as-is.
| HTTP status | Meaning |
| ----------- | ----------------------------------------------------------------------------------- |
| `200` | All URLs extracted successfully |
| `206` | Partial - some succeeded, some failed (check `urls_failed` and per-result `status`) |
| `422` | Every URL failed - see the top-level `error` |
```python theme={null}
if response.status_code == 206:
data = response.json()
failed = [r for r in data["results"] if r.get("status") == "failed"]
for r in failed:
print(f"{r['url']} failed: {r['error']}")
elif response.status_code == 422:
error_message = response.json()["error"]
```
## Next steps
Complete parameter documentation
Python integration
TypeScript integration
LangChain, LlamaIndex, and more