Contents API | Python SDK

Extract clean, structured content from web pages, with optional AI summarization and structured data extraction.

Wire up Valyu Contents extraction with the Python SDK.

Basic usage

from valyu import Valyu

valyu = Valyu()

response = valyu.contents(["https://en.wikipedia.org/wiki/Machine_learning"])

print(f"Processed {response.urls_processed} of {response.urls_requested} URLs")
for result in response.results or []:
    print(result.title, "-", result.length, "chars")
    print(result.content[:200], "...")

Common patterns

# Auto AI summary
valyu.contents(urls, summary=True, response_length="medium")

# Custom summary instruction
valyu.contents(urls, summary="Summarize the main trends in 3 bullet points", extract_effort="high")

# Structured extraction - pass a JSON schema, get back structured fields
valyu.contents(
    ["https://www.openai.com"],
    extract_effort="high",
    summary={
        "type": "object",
        "properties": {
            "company_name": {"type": "string"},
            "industry": {"type": "string"},
            "founded_year": {"type": "number"},
        },
        "required": ["company_name"],
    },
)

Set extract_effort="high" for JS-heavy pages or complex layouts, and response_length ("short" 25k, "medium" 50k, "large" 100k, "max", or an int) to control content per URL.

For arXiv, PubMed Central, bioRxiv, medRxiv, and ChemRxiv papers, Valyu serves clean processed markdown (with figures and equations) from its academic index when your plan covers the source - otherwise it uses the live crawler. Pass the paper URL (a /pdf/ arXiv link or a DOI works best) or bare id. See Academic Papers.

Reference

Parameters

urls (List[str], required) - URLs to process (max 10 sync, max 50 async).

Parameter	Type	Description	Default
`summary`	bool \| str \| dict	`False` (none), `True` (auto), a string instruction, or a JSON schema for structured extraction	False
`extract_effort`	`"normal"` \| `"high"` \| `"auto"`	Extraction effort. Use `"high"` for JS-heavy pages	”auto”
`response_length`	str \| int	`"short"` (25k), `"medium"` (50k), `"large"` (100k), `"max"`, or a custom character count	”short”
`screenshot`	bool	Request page screenshots. When `True`, results include `screenshot_url`	False

Response format

class ContentsResponse:
    success: bool
    error: Optional[str]
    tx_id: str
    urls_requested: int
    urls_processed: int
    urls_failed: int
    results: List[ContentsResult]
    total_cost_dollars: float
    total_characters: int

class ContentsResult:
    url: str
    title: str
    content: Union[str, dict]  # string for raw content, dict for structured
    description: Optional[str]
    length: int
    price: float
    source: str
    status: str                # "success" | "failed"
    error: Optional[str]       # Present when status == "failed"
    summary_success: Optional[bool]
    data_type: Optional[str]
    image_url: Optional[Dict[str, str]]
    screenshot_url: Optional[str]  # Only present when screenshot=True
    # Academic-index results (arXiv, PubMed, etc.) also populate:
    doi: Optional[str]
    authors: Optional[List[str]]
    citation_count: Optional[int]
    source_type: Optional[str]  # e.g. "paper"

Async jobs (11-50 URLs)

Async mode is required above 10 URLs. Max 50 URLs per request, processed in batches of 5 with a 120s timeout per URL (vs 25s sync). Jobs expire after 7 days.

# Submit and block until the job completes
result = valyu.contents(
    urls=[...],            # 11-50 URLs
    async_mode=True,
    wait=True,             # SDK handles polling
    poll_interval=5,
    max_wait_time=3600,
)

for r in result["results"]:
    print(r["title"], r["length"])
print(f"Total cost: ${result['actual_cost_dollars']}")

Submit, then wait with progress and webhooks

# Submit - returns immediately
job = valyu.contents(
    urls=[...],
    async_mode=True,
    webhook_url="https://your-app.com/webhooks/valyu",  # optional
)
print(f"Job ID: {job['job_id']}")

# Store the webhook_secret immediately - it is ONLY returned here
if job.get("webhook_secret"):
    save_webhook_secret(job["job_id"], job["webhook_secret"])

# Wait with progress tracking
result = valyu.wait_for_contents_job(
    job["job_id"],
    poll_interval=5,
    on_progress=lambda s: print(f"  {s['status']} - batch {s.get('current_batch', '?')}/{s.get('total_batches', '?')}"),
)

Webhooks are signed with HMAC-SHA256 over "{timestamp}.{json_body}". See the Content Extraction guide for verification.For full control, poll valyu.get_contents_job(job_id) yourself until status is completed, partial, or failed.

Async parameters and response types

Parameter	Type	Description	Default
`async_mode`	bool	Process URLs asynchronously. Required above 10 URLs	False
`webhook_url`	str	HTTPS URL to receive results via webhook POST	None
`wait`	bool	Block until the job completes (SDK handles polling)	False
`poll_interval`	int	Seconds between polls	5
`max_wait_time`	int	Max seconds to wait before timing out	3600
`on_progress`	Callable	Callback invoked on each poll with the current status dict	None

# Initial response (HTTP 202)
class ContentsAsyncResponse:
    success: bool
    job_id: str
    status: str               # Always "pending"
    urls_total: int
    poll_url: str
    tx_id: str
    webhook_secret: Optional[str]  # ONLY returned here - store immediately

# Job status response (polling / wait result)
class ContentsJobResponse:
    success: bool
    job_id: str
    status: str               # "pending" | "processing" | "completed" | "partial" | "failed"
    urls_total: int
    urls_processed: int
    urls_failed: int
    created_at: int           # Milliseconds since epoch
    updated_at: int
    current_batch: Optional[int]
    total_batches: Optional[int]
    results: Optional[List[ContentsResult]]
    actual_cost_dollars: Optional[float]
    error: Optional[str]

Async client (AsyncValyu)

The AsyncValyu client calls Contents with async/await inside your event loop - distinct from the server-side async jobs above (async_mode=True). It’s the natural fit for many single-URL extractions in parallel:

import asyncio
from valyu import AsyncValyu

urls = ["https://arxiv.org/abs/1706.03762", "https://arxiv.org/abs/2005.14165"]

async def main():
    async with AsyncValyu() as valyu:
        responses = await asyncio.gather(*[valyu.contents(urls=[u]) for u in urls])
        for url, r in zip(urls, responses):
            print(url, "->", r.urls_processed, "/", r.urls_requested)

asyncio.run(main())

You can combine both: submit a server-side async job through AsyncValyu and await valyu.wait_for_contents_job(job.job_id) without parking a thread. See the Python SDK overview for constructor options and lifecycle.

​Basic usage

​Common patterns

​Reference

​Async jobs (11-50 URLs)

Basic usage

Common patterns

Reference

Async jobs (11-50 URLs)