Contents API | TypeScript SDK

Extract clean, structured content from web pages, with optional AI summarization and structured data extraction.

Wire up Valyu Contents extraction with the TypeScript SDK.

Basic usage

import { Valyu } from "valyu-js";

const valyu = new Valyu();

const response = await valyu.contents([
  "https://en.wikipedia.org/wiki/Machine_learning"
]);

console.log(`Processed ${response.urls_processed} of ${response.urls_requested} URLs`);
response.results?.forEach((result) => {
  console.log(result.title, "-", result.length, "chars");
  console.log((result.content as string).substring(0, 200), "...");
});

Common patterns

// Auto AI summary
await valyu.contents(urls, { summary: true, responseLength: "medium" });

// Custom summary instruction
await valyu.contents(urls, {
  summary: "Summarize the main trends in 3 bullet points",
  extractEffort: "high",
});

// Structured extraction - pass a JSON schema, get back structured fields
await valyu.contents(["https://www.openai.com"], {
  extractEffort: "high",
  summary: {
    type: "object",
    properties: {
      company_name: { type: "string" },
      industry: { type: "string" },
      founded_year: { type: "number" },
    },
    required: ["company_name"],
  },
});

Set extractEffort: "high" for JS-heavy pages or complex layouts, and responseLength ("short" 25k, "medium" 50k, "large" 100k, "max", or a number) to control content per URL.

For arXiv, PubMed Central, bioRxiv, medRxiv, and ChemRxiv papers, Valyu serves clean processed markdown (with figures and equations) from its academic index when your plan covers the source - otherwise it uses the live crawler. Pass the paper URL (a /pdf/ arXiv link or a DOI works best) or bare id. See Academic Papers.

Reference

Parameters

urls (string[], required) - URLs to process (max 10 sync, max 50 async).

Parameter	Type	Description	Default
`summary`	boolean \| string \| object	`false` (none), `true` (auto), a string instruction, or a JSON schema for structured extraction	false
`extractEffort`	`"normal"` \| `"high"` \| `"auto"`	Extraction effort. Use `"high"` for JS-heavy pages	”auto”
`responseLength`	string \| number	`"short"` (25k), `"medium"` (50k), `"large"` (100k), `"max"`, or a custom character count	”short”
`screenshot`	boolean	Request page screenshots. When `true`, results include `screenshot_url`	false

Response format

interface ContentsResponse {
  success: boolean;
  error?: string | null;
  tx_id?: string;
  urls_requested?: number;
  urls_processed?: number;
  urls_failed?: number;
  results?: ContentResult[];
  total_cost_dollars?: number;
  total_characters?: number;
}

interface ContentResult {
  url: string;
  title: string;
  content: string | object; // string for raw content, object for structured
  description?: string;
  length: number;
  price: number;
  source: string;
  status: "success" | "failed";
  error?: string; // Present when status is "failed"
  summary_success?: boolean;
  data_type?: string;
  image_url?: Record<string, string>;
  screenshot_url?: string; // Only present when screenshot=true
  // Academic-index results (arXiv, PubMed, etc.) also populate:
  doi?: string;
  authors?: string[];
  citation_count?: number;
  source_type?: string; // e.g. "paper"
}

Async jobs (11-50 URLs)

Async mode is required above 10 URLs. Max 50 URLs per request, processed in batches of 5 with a 120s timeout per URL (vs 25s sync). Jobs expire after 7 days.

// Submit - returns immediately
const job = await valyu.contents([...], {  // 11-50 URLs
  asyncMode: true,
  webhookUrl: "https://your-app.com/webhooks/valyu", // optional
});

// Store the webhook_secret immediately - it is ONLY returned here
if (job.webhook_secret) await saveWebhookSecret(job.job_id, job.webhook_secret);

// Wait with progress tracking (SDK handles polling)
const result = await valyu.waitForJob(job.job_id, {
  pollInterval: 5000,
  onProgress: (s) => console.log(`  ${s.status} - batch ${s.current_batch ?? "?"}/${s.total_batches ?? "?"}`),
});

result.results?.forEach((r) => console.log(r.title, r.length));
console.log(`Total cost: $${result.actual_cost_dollars}`);

Manual polling and webhook verification

For full control, poll valyu.getContentsJob(job.job_id) yourself until status is completed, partial, or failed.The SDK exports verifyContentsWebhookSignature():

import { verifyContentsWebhookSignature } from "valyu-js";

const isValid = verifyContentsWebhookSignature(
  rawBody,           // raw request body string
  signatureHeader,   // X-Webhook-Signature header
  timestampHeader,   // X-Webhook-Timestamp header
  webhookSecret      // secret from initial 202 response
);

See the Content Extraction guide for details.

Async parameters and response types

Parameter	Type	Description	Default
`asyncMode`	boolean	Process URLs asynchronously. Required above 10 URLs	false
`webhookUrl`	string	HTTPS URL to receive results via webhook POST	-

waitForJob() options: pollInterval (ms, default 5000), maxWaitTime (ms, default 7200000), onProgress (callback).

// Initial response (HTTP 202)
interface ContentsAsyncResponse {
  success: boolean;
  job_id: string;
  status: "pending";
  urls_total: number;
  poll_url: string;
  tx_id: string;
  webhook_secret?: string; // ONLY returned here - store immediately
}

// Job status response (polling / waitForJob result)
interface ContentsJobResponse {
  success: boolean;
  job_id: string;
  status: "pending" | "processing" | "completed" | "partial" | "failed";
  urls_total: number;
  urls_processed: number;
  urls_failed: number;
  created_at: number;
  updated_at: number;
  current_batch?: number;
  total_batches?: number;
  results?: ContentResult[];
  actual_cost_dollars?: number;
  error?: string;
}

​Basic usage

​Common patterns

​Reference

​Async jobs (11-50 URLs)

Basic usage

Common patterns

Reference

Async jobs (11-50 URLs)