Skip to main content
Extract clean, structured content from web pages, with optional AI summarization and structured data extraction.

Wire up Valyu Contents extraction with the TypeScript SDK.

Open in Cursor

​
Basic usage

import { Valyu } from "valyu-js";

const valyu = new Valyu();

const response = await valyu.contents([
  "https://en.wikipedia.org/wiki/Machine_learning"
]);

console.log(`Processed ${response.urls_processed} of ${response.urls_requested} URLs`);
response.results?.forEach((result) => {
  console.log(result.title, "-", result.length, "chars");
  console.log((result.content as string).substring(0, 200), "...");
});

​
Common patterns

// Auto AI summary
await valyu.contents(urls, { summary: true, responseLength: "medium" });

// Custom summary instruction
await valyu.contents(urls, {
  summary: "Summarize the main trends in 3 bullet points",
  extractEffort: "high",
});

// Structured extraction - pass a JSON schema, get back structured fields
await valyu.contents(["https://www.openai.com"], {
  extractEffort: "high",
  summary: {
    type: "object",
    properties: {
      company_name: { type: "string" },
      industry: { type: "string" },
      founded_year: { type: "number" },
    },
    required: ["company_name"],
  },
});
Set extractEffort: "high" for JS-heavy pages or complex layouts, and responseLength ("short" 25k, "medium" 50k, "large" 100k, "max", or a number) to control content per URL.
For arXiv, PubMed Central, bioRxiv, medRxiv, and ChemRxiv papers, Valyu serves clean processed markdown (with figures and equations) from its academic index when your plan covers the source - otherwise it uses the live crawler. Pass the paper URL (a /pdf/ arXiv link or a DOI works best) or bare id. See Academic Papers.

​
Reference

urls (string[], required) - URLs to process (max 10 sync, max 50 async).
ParameterTypeDescriptionDefault
summaryboolean | string | objectfalse (none), true (auto), a string instruction, or a JSON schema for structured extractionfalse
extractEffort"normal" | "high" | "auto"Extraction effort. Use "high" for JS-heavy pages”auto”
responseLengthstring | number"short" (25k), "medium" (50k), "large" (100k), "max", or a custom character count”short”
screenshotbooleanRequest page screenshots. When true, results include screenshot_urlfalse
interface ContentsResponse {
  success: boolean;
  error?: string | null;
  tx_id?: string;
  urls_requested?: number;
  urls_processed?: number;
  urls_failed?: number;
  results?: ContentResult[];
  total_cost_dollars?: number;
  total_characters?: number;
}

interface ContentResult {
  url: string;
  title: string;
  content: string | object; // string for raw content, object for structured
  description?: string;
  length: number;
  price: number;
  source: string;
  status: "success" | "failed";
  error?: string; // Present when status is "failed"
  summary_success?: boolean;
  data_type?: string;
  image_url?: Record<string, string>;
  screenshot_url?: string; // Only present when screenshot=true
  // Academic-index results (arXiv, PubMed, etc.) also populate:
  doi?: string;
  authors?: string[];
  citation_count?: number;
  source_type?: string; // e.g. "paper"
}

​
Async jobs (11-50 URLs)

Async mode is required above 10 URLs. Max 50 URLs per request, processed in batches of 5 with a 120s timeout per URL (vs 25s sync). Jobs expire after 7 days. 
// Submit - returns immediately
const job = await valyu.contents([...], {  // 11-50 URLs
  asyncMode: true,
  webhookUrl: "https://your-app.com/webhooks/valyu", // optional
});

// Store the webhook_secret immediately - it is ONLY returned here
if (job.webhook_secret) await saveWebhookSecret(job.job_id, job.webhook_secret);

// Wait with progress tracking (SDK handles polling)
const result = await valyu.waitForJob(job.job_id, {
  pollInterval: 5000,
  onProgress: (s) => console.log(`  ${s.status} - batch ${s.current_batch ?? "?"}/${s.total_batches ?? "?"}`),
});

result.results?.forEach((r) => console.log(r.title, r.length));
console.log(`Total cost: $${result.actual_cost_dollars}`);
For full control, poll valyu.getContentsJob(job.job_id) yourself until status is completed, partial, or failed.The SDK exports verifyContentsWebhookSignature():
import { verifyContentsWebhookSignature } from "valyu-js";

const isValid = verifyContentsWebhookSignature(
  rawBody,           // raw request body string
  signatureHeader,   // X-Webhook-Signature header
  timestampHeader,   // X-Webhook-Timestamp header
  webhookSecret      // secret from initial 202 response
);
See the Content Extraction guide for details.
ParameterTypeDescriptionDefault
asyncModebooleanProcess URLs asynchronously. Required above 10 URLsfalse
webhookUrlstringHTTPS URL to receive results via webhook POST-
waitForJob() options: pollInterval (ms, default 5000), maxWaitTime (ms, default 7200000), onProgress (callback).
// Initial response (HTTP 202)
interface ContentsAsyncResponse {
  success: boolean;
  job_id: string;
  status: "pending";
  urls_total: number;
  poll_url: string;
  tx_id: string;
  webhook_secret?: string; // ONLY returned here - store immediately
}

// Job status response (polling / waitForJob result)
interface ContentsJobResponse {
  success: boolean;
  job_id: string;
  status: "pending" | "processing" | "completed" | "partial" | "failed";
  urls_total: number;
  urls_processed: number;
  urls_failed: number;
  created_at: number;
  updated_at: number;
  current_batch?: number;
  total_batches?: number;
  results?: ContentResult[];
  actual_cost_dollars?: number;
  error?: string;
}