Content Endpoint Guide

Turn any web page into clean markdown or structured data. The Contents API extracts content from URLs, with optional AI summaries, JSON-schema extraction, and async batch processing. You only pay for URLs that succeed.

Extract a page

from valyu import Valyu

valyu = Valyu()  # reads VALYU_API_KEY from env

data = valyu.contents(
    urls=["https://en.wikipedia.org/wiki/Artificial_intelligence"],
    response_length="medium",
    extract_effort="auto",
)
print(data["results"][0]["content"][:500])

import { Valyu } from "valyu-js";

const valyu = new Valyu(); // reads VALYU_API_KEY from env

const data = await valyu.contents({
  urls: ["https://en.wikipedia.org/wiki/Artificial_intelligence"],
  responseLength: "medium",
  extractEffort: "auto",
});
console.log(data.results[0].content.slice(0, 500));

curl -X POST https://api.valyu.ai/v1/contents \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{
    "urls": ["https://en.wikipedia.org/wiki/Artificial_intelligence"],
    "response_length": "medium",
    "extract_effort": "auto"
  }'

Returns clean markdown for each URL. Pass up to 10 URLs synchronously, or up to 50 with async mode.

Core options

Option	Values	Default
`response_length`	`short` (25k), `medium` (50k), `large` (100k), `max`, or a char count (1k-1M)	`short`
`extract_effort`	`normal` (fast), `high` (better, slower), `auto`	`normal`
`summary`	`false`, `true`, an instruction string, or a JSON schema	`false`
`screenshot`	`true` to capture a page screenshot (not for PDFs)	`false`

Academic papers

When you pass an academic paper URL or DOI, Valyu serves it from our pre-processed academic index rather than crawling the live page. You get clean markdown with preserved figures, equations, tables, and section structure, plus inline pre-signed image URLs - the things lost when a JavaScript-heavy or PDF paper is scraped live. This happens automatically; just pass the URL.

Source	Example URL / id	Plan for index access
arXiv	`https://arxiv.org/pdf/2506.05440` (or `/abs/`, `/html/`)	tier_0 (pay-as-you-go)
PubMed Central	`https://pubmed.ncbi.nlm.nih.gov/PMC10584482`	tier_0
bioRxiv	`https://www.biorxiv.org/content/10.1101/2020.07.28.225581`	tier_1
medRxiv	`https://www.medrxiv.org/content/10.1101/...`	tier_1
ChemRxiv	`https://chemrxiv.org/doi/full/10.26434/chemrxiv-...`	tier_2

If your plan covers the source, you get the processed markdown; if the specific paper isn’t indexed yet, it falls back to live extraction. If your plan doesn’t cover the source, the URL is live-crawled (you still get content, just without the processed-index quality).

Academic results set source to the dataset id (e.g. valyu/valyu-arxiv) and include doi, authors, and citation_count. Figures are returned in image_url as a filename→URL map and referenced inline in the markdown.

AI summaries and structured extraction

The summary field controls AI post-processing:

false - raw markdown, fastest and cheapest (no AI)
true - a basic AI summary
string - custom instructions, e.g. "Summarise the methodology and key findings in 2-3 paragraphs"
object - a JSON schema for structured extraction

# Structured extraction
data = valyu.contents(
    urls=["https://example.com/product-page"],
    summary={
        "type": "object",
        "properties": {
            "product_name": {"type": "string"},
            "price": {"type": "number", "description": "Price in USD"},
            "availability": {
                "type": "string",
                "enum": ["in_stock", "out_of_stock", "preorder"],
            },
        },
        "required": ["product_name", "price"],
    },
)
print(data["results"][0]["content"])  # returns a JSON object

Summary examples (all four modes)

# false - raw markdown, no AI
valyu.contents(urls=["https://example.com/article"], summary=False)

# true - basic AI summary
valyu.contents(urls=["https://example.com/article"], summary=True)

# string - custom instructions
valyu.contents(
    urls=["https://example.com/research-paper"],
    summary="Summarise the methodology, key findings, and applications in 2-3 paragraphs",
)

# object - JSON schema extraction
valyu.contents(
    urls=["https://example.com/product-page"],
    summary={
        "type": "object",
        "properties": {
            "product_name": {"type": "string"},
            "price": {"type": "number"},
            "features": {"type": "array", "items": {"type": "string"}, "maxItems": 5},
        },
        "required": ["product_name", "price"],
    },
)

// false - raw markdown, no AI
await valyu.contents({ urls: ["https://example.com/article"], summary: false });

// true - basic AI summary
await valyu.contents({ urls: ["https://example.com/article"], summary: true });

// string - custom instructions
await valyu.contents({
  urls: ["https://example.com/research-paper"],
  summary: "Summarise the methodology, key findings, and applications in 2-3 paragraphs",
});

// object - JSON schema extraction
await valyu.contents({
  urls: ["https://example.com/product-page"],
  summary: {
    type: "object",
    properties: {
      product_name: { type: "string" },
      price: { type: "number" },
      features: { type: "array", items: { type: "string" }, maxItems: 5 },
    },
    required: ["product_name", "price"],
  },
});

JSON schema reference and tips

Use any valid JSON Schema. Limits: 5,000 characters, 3 levels deep, 20 properties per object.Supported types: string, number/integer, boolean, array, object.Tips:

Use clear description fields to guide extraction.
Use enum for consistent categorisation.
Keep schemas shallow and mark essential fields required.

Screenshot capture

Python

data = valyu.contents(
    urls=["https://example.com/article"],
    extract_effort="auto",
    screenshot=True,
)
print(data["results"][0]["screenshot_url"])

Screenshots are captured during page rendering and returned as pre-signed URLs. PDF files do not support screenshots.

Async processing

For 11-50 URLs (required above 10) or non-blocking workflows, use async mode. Submit URLs, get a job_id immediately, then poll or receive results via webhook. Async also raises the per-URL timeout to 120s (vs 25s for sync).

job = valyu.contents(
    urls=["https://example.com/page1", "https://example.com/page2"],  # up to 50
    async_mode=True,
    webhook_url="https://your-app.com/webhooks/valyu",  # optional
)
print(job["job_id"])

# Block until complete (SDK handles polling)
result = valyu.wait_for_contents_job(job["job_id"], poll_interval=5, max_wait_time=3600)
for r in result["results"]:
    print(r["title"], r["length"])
print(f"Total cost: ${result['actual_cost_dollars']}")

Async mode is required above 10 URLs. For 1-10 URLs it’s optional, for non-blocking workflows.

Polling, job lifecycle, and status fields

The initial submit returns HTTP 202 with a job_id, poll_url, and a one-time webhook_secret (store it immediately - you cannot retrieve it later).Jobs move through these statuses:

Status	Meaning
`pending`	Created, not yet started
`processing`	URLs running in batches of 5
`completed`	All URLs succeeded
`partial`	Finished with some failures
`failed`	All URLs failed

To poll manually:

import time

while True:
    status = valyu.get_contents_job(job["job_id"])
    if status["status"] in ("completed", "partial", "failed"):
        break
    time.sleep(2)

if status["status"] in ("completed", "partial"):
    for result in status["results"]:
        print(result["title"], result["length"])

let status;
do {
  status = await valyu.getContentsJob(job.job_id);
  if (!["completed", "partial", "failed"].includes(status.status)) {
    await new Promise((r) => setTimeout(r, 2000));
  }
} while (!["completed", "partial", "failed"].includes(status.status));

Status response fields include: job_id, status, urls_total, urls_processed, urls_failed, created_at, updated_at, current_batch / total_batches (while processing), results and actual_cost_dollars (when completed/partial), and error (when partial/failed).

Webhooks and signature verification

When you provide a webhook_url, Valyu POSTs to it when the job finishes (completed, partial, or failed). Headers: Content-Type: application/json, User-Agent: Valyu-Contents/1.0, X-Webhook-Signature: sha256={hex}, X-Webhook-Timestamp (Unix seconds).Retries: up to 5 attempts with exponential backoff (1s, 2s, 4s, 8s), no retry on 4xx, 5s connect / 15s read timeout.The payload is signed with HMAC-SHA256 over "{timestamp}.{json_payload}":

import hmac, hashlib

def verify_webhook(payload: bytes, signature_header: str, timestamp_header: str, webhook_secret: str) -> bool:
    signed = f"{timestamp_header}.{payload.decode('utf-8')}"
    expected = hmac.new(webhook_secret.encode(), signed.encode(), hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature_header.removeprefix("sha256="))

import crypto from "crypto";

function verifyWebhook(payload, signatureHeader, timestampHeader, webhookSecret) {
  const signed = `${timestampHeader}.${payload}`;
  const expected = crypto.createHmac("sha256", webhookSecret).update(signed).digest("hex");
  const received = signatureHeader.replace("sha256=", "");
  return crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(received));
}

The TypeScript SDK exports verifyContentsWebhookSignature() which handles this for you.

Limits and pricing

Detail	Value
Max URLs (sync / async)	10 / 50
Batch size	5 URLs
Timeout per URL (sync / async)	25s / 120s
Base pricing	$0.001 per URL
AI features (summary/schema)	+$0.001 per URL
Job expiry (TTL)	7 days

Response format

{
  "success": true,
  "tx_id": "tx_12345678-1234-1234-1234-123456789abc",
  "results": [
    {
      "title": "AI Breakthrough in Natural Language Processing",
      "url": "https://example.com/article?utm_source=valyu",
      "content": "# AI Breakthrough...\n\nPage content in markdown...",
      "source": "web",
      "price": 0.001,
      "length": 12840,
      "data_type": "unstructured"
    }
  ],
  "urls_requested": 1,
  "urls_processed": 1,
  "urls_failed": 0,
  "total_cost_dollars": 0.001,
  "total_characters": 12840
}

With summary set, content is the AI summary (string) or extracted JSON (object), and each result adds summary_success.

Result fields

Field	Description
`status`	`"success"` or `"failed"` for this URL
`error`	Human-readable failure reason (failed results only)
`title`	Extracted page title
`url`	Original URL
`content`	Extracted content (markdown or JSON)
`description`	Page meta description (paper abstract for academic sources)
`source`	`"web"`, or the dataset id for academic papers (e.g. `valyu/valyu-arxiv`)
`price`	Cost for this URL in dollars
`length`	Character count
`data_type`	`"unstructured"` or `"structured"`
`summary_success`	Whether AI processing succeeded (only when `summary` is used)
`image_url`	Extracted image URLs (figures, for academic papers)
`screenshot_url`	Page screenshot URL (only when `screenshot=true`)
`doi`, `authors`, `citation_count`	Academic sources only

Error handling

Each result carries a status of success or failed. Failed results include a descriptive error (page unavailable, required sign-in, blocked, timed out) that never exposes Valyu’s internal infrastructure - branch on status and surface the error string as-is.

HTTP status	Meaning
`200`	All URLs extracted successfully
`206`	Partial - some succeeded, some failed (check `urls_failed` and per-result `status`)
`422`	Every URL failed - see the top-level `error`

if response.status_code == 206:
    data = response.json()
    failed = [r for r in data["results"] if r.get("status") == "failed"]
    for r in failed:
        print(f"{r['url']} failed: {r['error']}")
elif response.status_code == 422:
    error_message = response.json()["error"]

Next steps

API reference

Complete parameter documentation

Python SDK

Python integration

TypeScript SDK

TypeScript integration

Integrations

LangChain, LlamaIndex, and more

​Extract a page

​Core options

​Academic papers

​AI summaries and structured extraction

​Async processing

​Limits and pricing

​Response format

​Error handling

​Next steps

API reference

Python SDK

TypeScript SDK

Integrations

Extract a page

Core options

Academic papers

AI summaries and structured extraction

Async processing

Limits and pricing

Response format

Error handling

Next steps