@purepageio/fetch-engines 0.5.1-rc.0 → 0.6.0

package/README.md

@@ -3,546 +3,341 @@
 [![npm version](https://img.shields.io/npm/v/@purepageio/fetch-engines.svg)](https://www.npmjs.com/package/@purepageio/fetch-engines)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-Fetching web content can be complex. You need to handle static HTML, dynamic JavaScript-driven sites, network errors, retries, caching, and potential bot detection measures. Managing browser automation tools like Playwright adds another layer of complexity with resource pooling and stealth configurations.
+Web scraping requires handling static HTML, JavaScript-heavy sites, network errors, retries, caching, and bot detection. Managing browser automation tools like Playwright adds complexity with resource pooling and stealth configurations.
 
-`@purepageio/fetch-engines` simplifies this entire process by providing a set of robust, configurable, and easy-to-use engines for retrieving web page content.
+`@purepageio/fetch-engines` provides engines for retrieving web content through a unified API.
 
-**Why use `@purepageio/fetch-engines`?**
+**Key Benefits:**
 
-- **Unified API:** Get content from simple or complex sites using the same `fetchHTML(url, options?)` method for processed content or `fetchContent(url, options?)` for raw content.
-- **Flexible Strategies:** Choose the right tool for the job:
-  - `FetchEngine`: Lightweight and fast for static HTML, using the standard `fetch` API. Ideal for speed and efficiency with content that doesn't require JavaScript rendering. Supports custom headers.
-  - `HybridEngine`: The best of both worlds – tries `FetchEngine` first for speed, automatically falls back to a powerful browser engine (internally, `PlaywrightEngine`) for reliability on complex, JavaScript-heavy pages. Supports custom headers.
-- **Raw Content Support:** Use `fetchContent()` to retrieve any type of content (PDFs, images, APIs, etc.) with the same smart fallback logic as `fetchHTML()`.
-- **Robust & Resilient:** Built-in caching, configurable retries, and standardized error handling make your fetching logic more dependable.
-- **Simplified Automation:** When `HybridEngine` uses its browser capabilities (via the internal `PlaywrightEngine`), it manages browser instances and contexts automatically through efficient pooling and includes integrated stealth measures to bypass common anti-bot systems.
-- **Content Transformation:** Optionally convert fetched HTML directly to clean Markdown content.
-- **TypeScript Ready:** Fully typed for a better development experience.
-
-This package provides a high-level abstraction, letting you focus on using the web content rather than the intricacies of fetching it.
+- **Unified API:** Use `fetchHTML(url, options?)` for processed content or `fetchContent(url, options?)` for raw content
+- **Smart Fallback Strategy:** Tries fast HTTP first, automatically falls back to full browser for complex sites
+- **AI-Powered Data Extraction:** Extract structured data from web pages using OpenAI and Zod schemas
+- **Raw Content Support:** Retrieve PDFs, images, APIs with the same fallback logic
+- **Built-in Resilience:** Caching, retries, and standardised error handling
+- **Browser Management:** Automatic browser pooling and stealth measures for complex sites
+- **Content Transformation:** Convert HTML to clean Markdown
+- **TypeScript Ready:** Fully typed codebase
 
 ## Table of Contents
 
-- [Features](#features)
 - [Installation](#installation)
 - [Engines](#engines)
 - [Basic Usage](#basic-usage)
 - [fetchHTML vs fetchContent](#fetchhtml-vs-fetchcontent)
+- [Structured Content Extraction](#structured-content-extraction)
 - [Configuration](#configuration)
 - [Return Value](#return-value)
 - [API Reference](#api-reference)
-- [Stealth / Anti-Detection (`PlaywrightEngine`)](#stealth--anti-detection-playwrightengine)
+- [Stealth Features](#stealth-features)
 - [Error Handling](#error-handling)
-- [Logging](#logging)
 - [Contributing](#contributing)
-- [License](#license)
-
-## Features
-
-- **Multiple Fetching Strategies:** Choose between `FetchEngine` (lightweight `fetch`) or `HybridEngine` (smart fallback to a full browser engine).
-- **Unified API:** Simple `fetchHTML(url, options?)` interface for processed content and `fetchContent(url, options?)` for raw content across both primary engines.
-- **Raw Content Fetching:** Use `fetchContent()` to retrieve any type of content (PDFs, images, JSON, XML, etc.) without HTML processing or content-type restrictions.
-- **Custom Headers:** Easily provide custom HTTP headers for requests in both `FetchEngine` and `HybridEngine`.
-- **Configurable Retries:** Automatic retries on failure with customizable attempts and delays.
-- **Built-in Caching:** In-memory caching with configurable TTL to reduce redundant fetches.
-- **Playwright Stealth:** When `HybridEngine` utilizes its browser capabilities, it automatically integrates `playwright-extra` and stealth plugins to bypass common bot detection.
-- **Managed Browser Pooling:** Efficient resource management for `HybridEngine`'s browser mode with configurable browser/context limits and lifecycles.
-- **Smart Fallbacks:** `HybridEngine` uses `FetchEngine` first, falling back to its internal browser engine only when needed. The internal browser engine can also optionally use a fast HTTP fetch before launching a full browser.
-- **Content Conversion:** Optionally convert fetched HTML directly to Markdown, preserving `<table>` elements even without header rows.
-- **Standardized Errors:** Custom `FetchError` classes provide context on failures.
-- **TypeScript Ready:** Fully typed codebase for enhanced developer experience.
 
 ## Installation
 
 ```bash
 pnpm add @purepageio/fetch-engines
-# or with npm
-npm install @purepageio/fetch-engines
-# or with yarn
-yarn add @purepageio/fetch-engines
 ```
 
-If you plan to use the `HybridEngine` (which internally uses Playwright for advanced fetching), you also need to install Playwright's browser binaries:
+For `HybridEngine` (uses Playwright), install browser binaries:
 
 ```bash
 pnpm exec playwright install
-# or
-npx playwright install
 ```
 
 ## Engines
 
-- **`FetchEngine`**: Uses the standard `fetch` API. Suitable for simple HTML pages or APIs returning HTML. Lightweight and fast. This is your go-to for speed and efficiency when JavaScript rendering is not required.
-- **`HybridEngine`**: A smart combination. It first attempts to fetch content using the lightweight `FetchEngine`. If that fails for _any_ reason (e.g., network error, non-HTML content, HTTP error like 403), or if `spaMode` is enabled and an SPA shell is detected, it automatically falls back to using an internal, powerful browser engine (based on Playwright). This provides the speed of `FetchEngine` for simple sites while retaining the power of a full browser for complex, dynamic websites. This is recommended for most general-purpose fetching tasks.
-- **`PlaywrightEngine` (Internal Component)**: While not recommended for direct use by most users, `PlaywrightEngine` is the component `HybridEngine` uses internally for its browser-based fetching. It manages Playwright browser instances, contexts, and stealth features. Users needing direct, low-level control over Playwright might consider it, but `HybridEngine` offers a more robust and flexible approach for most scenarios.
+**`HybridEngine`** (recommended): Attempts fast HTTP fetch first, falls back to Playwright browser on failure or when SPA shell detected. Handles both simple and complex sites automatically.
+
+**`FetchEngine`**: Lightweight HTTP-only engine for basic sites without browser fallback.
+
+**`StructuredContentEngine`**: AI-powered engine that combines HybridEngine with OpenAI for structured data extraction.
 
 ## Basic Usage
 
-### FetchEngine
+### Quick Start
 
 ```typescript
-import { FetchEngine } from "@purepageio/fetch-engines";
-
-const engine = new FetchEngine(); // Default: fetches HTML
-
-async function main() {
-  try {
-    const url = "https://example.com";
-    const result = await engine.fetchHTML(url);
-    console.log(`Fetched ${result.url} (ContentType: ${result.contentType})`);
-    console.log(`Title: ${result.title}`);
-    console.log(`Content (HTML): ${result.content.substring(0, 100)}...`);
-
-    // Example fetching Markdown directly via constructor option
-    const markdownEngine = new FetchEngine({ markdown: true });
-    const mdResult = await markdownEngine.fetchHTML(url);
-    console.log(`\nFetched ${mdResult.url} (ContentType: ${mdResult.contentType})`);
-    console.log(`Content (Markdown):\n${mdResult.content.substring(0, 300)}...`);
-  } catch (error) {
-    console.error("Fetch failed:", error);
-  }
-}
-main();
+import { HybridEngine } from "@purepageio/fetch-engines";
+
+const engine = new HybridEngine();
+
+// Simple sites use fast HTTP
+const simple = await engine.fetchHTML("https://example.com");
+console.log(`Title: ${simple.title}`);
+
+// Complex sites automatically use browser
+const complex = await engine.fetchHTML("https://spa-site.com", {
+  markdown: true,
+  spaMode: true,
+});
+
+await engine.cleanup(); // Important: releases browser resources
 ```
 
-### HybridEngine
+### With Custom Headers
 
 ```typescript
-import { HybridEngine } from "@purepageio/fetch-engines";
-
-// Engine configured to fetch HTML by default for its internal engines
-// and provide some custom headers for all requests made by HybridEngine.
 const engine = new HybridEngine({
-  markdown: false,
-  headers: { "X-Global-Custom-Header": "HybridGlobalValue" },
-  // Other PlaywrightEngine specific configs can be set here for the fallback mechanism
-  // e.g., playwrightLaunchOptions: { args: ["--disable-gpu"] }
+  headers: { "X-Custom-Header": "value" },
 });
 
-async function main() {
-  try {
-    const urlSimple = "https://example.com"; // Simple site, likely handled by FetchEngine
-    const urlComplex = "https://quotes.toscrape.com/"; // JS-heavy site, likely requiring Playwright fallback
-
-    // --- Scenario 1: FetchEngine part of HybridEngine handles it ---
-    console.log(`\nFetching simple site (${urlSimple}) with per-request headers...`);
-    const result1 = await engine.fetchHTML(urlSimple, {
-      headers: { "X-Request-Specific": "SimpleRequestValue" },
-    });
-    // FetchEngine (via HybridEngine) will use:
-    // 1. Its base default headers (User-Agent etc.)
-    // 2. Overridden/augmented by HybridEngine's constructor headers ("X-Global-Custom-Header")
-    // 3. Overridden/augmented by per-request headers ("X-Request-Specific")
-    console.log(`Fetched ${result1.url} (ContentType: ${result1.contentType}) - Title: ${result1.title}`);
-    console.log(`Content (HTML): ${result1.content.substring(0, 100)}...`);
-
-    // --- Scenario 2: Playwright part of HybridEngine handles it ---
-    console.log(`\nFetching complex site (${urlComplex}) requesting Markdown and with per-request headers...`);
-    const result2 = await engine.fetchHTML(urlComplex, {
-      markdown: true,
-      headers: { "X-Request-Specific": "ComplexRequestValue", "X-Another": "ComplexAnother" },
-    });
-    // PlaywrightEngine (via HybridEngine) will use:
-    // 1. Its base default headers (User-Agent etc. if doing HTTP fallback, or for page.setExtraHTTPHeaders)
-    // 2. Overridden/augmented by HybridEngine's constructor headers ("X-Global-Custom-Header")
-    // 3. Overridden/augmented by per-request headers ("X-Request-Specific", "X-Another")
-    // The markdown: true option will be respected by the Playwright part.
-    console.log(`Fetched ${result2.url} (ContentType: ${result2.contentType}) - Title: ${result2.title}`);
-    console.log(`Content (Markdown):\n${result2.content.substring(0, 300)}...`);
-  } catch (error) {
-    console.error("Hybrid fetch failed:", error);
-  } finally {
-    await engine.cleanup(); // Important for HybridEngine
-  }
-}
-main();
+const result = await engine.fetchHTML("https://example.com", {
+  headers: { "X-Request-Header": "value" },
+});
 ```
 
-### Raw Content Fetching
+### Raw Content (PDFs, Images, APIs)
 
 ```typescript
-import { HybridEngine } from "@purepageio/fetch-engines";
-
 const engine = new HybridEngine();
 
-async function fetchRawContent() {
-  try {
-    // Fetch a PDF document
-    const pdfResult = await engine.fetchContent("https://example.com/document.pdf");
-    console.log(`PDF Content-Type: ${pdfResult.contentType}`);
-    console.log(
-      `PDF Size: ${Buffer.isBuffer(pdfResult.content) ? pdfResult.content.length : pdfResult.content.length} bytes`
-    );
-
-    // Fetch JSON API
-    const jsonResult = await engine.fetchContent("https://api.example.com/data");
-    console.log(`JSON Content-Type: ${jsonResult.contentType}`);
-    console.log(`JSON Data: ${typeof jsonResult.content === "string" ? jsonResult.content : "Binary data"}`);
-
-    // Fetch with custom headers
-    const customResult = await engine.fetchContent("https://protected-api.example.com/data", {
-      headers: {
-        Authorization: "Bearer your-token",
-        Accept: "application/json",
-      },
-    });
-    console.log(`Custom fetch result: ${customResult.statusCode}`);
-  } catch (error) {
-    console.error("Raw content fetch failed:", error);
-  } finally {
-    await engine.cleanup();
-  }
-}
-fetchRawContent();
+// Fetch PDF
+const pdf = await engine.fetchContent("https://example.com/doc.pdf");
+console.log(`PDF size: ${pdf.content.length} bytes`);
+
+// Fetch JSON API with auth
+const api = await engine.fetchContent("https://api.example.com/data", {
+  headers: { Authorization: "Bearer token" },
+});
+
+await engine.cleanup();
 ```
 
 ## fetchHTML vs fetchContent
 
-Choose the right method for your use case:
-
 ### `fetchHTML(url, options?)`
 
-**Use when:** You want to extract and process web page content.
-
-**Features:**
+**Use for:** Web page content extraction
 
-- Processes HTML content and extracts metadata (title, etc.)
+- Processes HTML and extracts metadata (title, etc.)
 - Supports HTML-to-Markdown conversion
-- Optimized for web page content
 - Content-type restrictions (HTML/XML only)
 - Returns processed content as `string`
 
-**Best for:**
-
-- Web scraping
-- Content extraction
-- Blog/article processing
-- Any scenario where you need structured HTML or Markdown
-
 ### `fetchContent(url, options?)`
 
-**Use when:** You want raw content without processing, mimicking standard `fetch()` behavior.
-
-**Features:**
+**Use for:** Raw content retrieval (like standard `fetch`)
 
 - Retrieves any content type (PDFs, images, JSON, XML, etc.)
 - No content-type restrictions
-- Returns raw content as `Buffer` (binary) or `string` (text)
-- Preserves original MIME type information
-- Minimal processing overhead
-
-**Best for:**
-
-- API consumption
-- File downloads (PDFs, images, etc.)
-- Binary content retrieval
-- Any scenario where you need the raw response
+- Returns `Buffer` (binary) or `string` (text)
+- Preserves original MIME type
 
 ### Example Comparison
 
 ```typescript
-import { HybridEngine } from "@purepageio/fetch-engines";
-
-const engine = new HybridEngine();
-
-// fetchHTML - for web page content
-const htmlResult = await engine.fetchHTML("https://example.com");
-console.log(htmlResult.title); // "Example Domain"
-console.log(htmlResult.contentType); // "html" or "markdown"
-console.log(typeof htmlResult.content); // "string" (processed HTML/Markdown)
-
-// fetchContent - for raw content
-const contentResult = await engine.fetchContent("https://example.com");
-console.log(contentResult.title); // "Example Domain" (extracted but not processed)
-console.log(contentResult.contentType); // "text/html" (original MIME type)
-console.log(typeof contentResult.content); // "string" (raw HTML)
-
-// fetchContent - for non-HTML content
-const pdfResult = await engine.fetchContent("https://example.com/doc.pdf");
-console.log(pdfResult.contentType); // "application/pdf"
-console.log(Buffer.isBuffer(pdfResult.content)); // true (binary content)
+// fetchHTML - processes content
+const html = await engine.fetchHTML("https://example.com");
+console.log(html.title); // "Example Domain"
+console.log(html.contentType); // "html" or "markdown"
+
+// fetchContent - raw content
+const raw = await engine.fetchContent("https://example.com");
+console.log(raw.contentType); // "text/html"
+console.log(typeof raw.content); // "string" (raw HTML)
+
+// Binary content
+const pdf = await engine.fetchContent("https://example.com/doc.pdf");
+console.log(Buffer.isBuffer(pdf.content)); // true
 ```
 
-## Configuration
+## Structured Content Extraction
 
-Engines accept an optional configuration object in their constructor to customise behavior.
+Extract structured data from web pages using AI and Zod schemas.
 
-### FetchEngine
+### Prerequisites
 
-The `FetchEngine` accepts a `FetchEngineOptions` object with the following properties:
+Set environment variable:
 
-| Option     | Type                     | Default | Description                                                                                                                                                                    |
-| ---------- | ------------------------ | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `markdown` | `boolean`                | `false` | If `true`, converts fetched HTML to Markdown. `contentType` in the result will be set to `'markdown'`.                                                                         |
-| `headers`  | `Record<string, string>` | `{}`    | Custom HTTP headers to be sent with the request. These are merged with and can override the engine's default headers. Headers from `fetchHTML` options take higher precedence. |
+```bash
+export OPENAI_API_KEY="your-openai-api-key"
+```
+
+### Basic Usage
 
 ```typescript
-// Example: FetchEngine with custom headers and Markdown conversion
-const customFetchEngine = new FetchEngine({
-  markdown: true,
-  headers: {
-    "User-Agent": "MyCustomFetchAgent/1.0",
-    "X-Api-Key": "your-api-key",
-  },
+import { fetchStructuredContent } from "@purepageio/fetch-engines";
+import { z } from "zod";
+
+const articleSchema = z.object({
+  title: z.string(),
+  author: z.string().optional(),
+  publishDate: z.string().optional(),
+  summary: z.string(),
+  tags: z.array(z.string()),
 });
+
+const result = await fetchStructuredContent("https://example.com/article", articleSchema, {
+  model: "gpt-4.1-mini",
+  customPrompt: "Extract main article information",
+});
+
+console.log("Extracted:", result.data);
+console.log("Token usage:", result.usage);
 ```
 
-#### Header Precedence for `FetchEngine`:
-
-1.  Headers passed in `fetchHTML(url, { headers: { ... } })` (highest precedence).
-2.  Headers passed in the `FetchEngine` constructor `new FetchEngine({ headers: { ... } })`.
-3.  Default headers of the `FetchEngine` (e.g., its default `User-Agent`) (lowest precedence).
-
-### `PlaywrightEngineConfig` (Used by `HybridEngine`)
-
-The `HybridEngine` constructor accepts a `PlaywrightEngineConfig` object. These settings configure the underlying `FetchEngine` and `PlaywrightEngine` (for fallback scenarios) and the hybrid strategy itself. When using `HybridEngine`, you are essentially configuring how it will behave and how its internal Playwright capabilities will operate if needed.
-
-**Key Options for `HybridEngine` (from `PlaywrightEngineConfig`):**
-
-| Option                    | Type                     | Default     | Description                                                                                                                                                                                                                                                           |
-| ------------------------- | ------------------------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `headers`                 | `Record<string, string>` | `{}`        | Custom HTTP headers. For `HybridEngine`, these serve as default headers for both its internal `FetchEngine` (constructor) and `PlaywrightEngine` (constructor). They can be overridden by headers in `HybridEngine.fetchHTML()` options.                              |
-| `markdown`                | `boolean`                | `false`     | Default Markdown conversion. For `HybridEngine`: sets default for internal `FetchEngine` (constructor) and internal `PlaywrightEngine`. Can be overridden per-request for the `PlaywrightEngine` part.                                                                |
-| `useHttpFallback`         | `boolean`                | `true`      | (For Playwright part) If `true`, attempts a fast HTTP fetch before using Playwright. Ineffective if `spaMode` is `true`.                                                                                                                                              |
-| `useHeadedModeFallback`   | `boolean`                | `false`     | (For Playwright part) If `true`, automatically retries specific failed Playwright attempts in headed (visible) mode.                                                                                                                                                  |
-| `defaultFastMode`         | `boolean`                | `true`      | If `true`, initially blocks non-essential resources and skips human simulation. Can be overridden per-request. Effectively `false` if `spaMode` is `true`.                                                                                                            |
-| `simulateHumanBehavior`   | `boolean`                | `true`      | If `true` (and not `fastMode` or `spaMode`), attempts basic human-like interactions.                                                                                                                                                                                  |
-| `concurrentPages`         | `number`                 | `3`         | Max number of pages to process concurrently within the engine queue.                                                                                                                                                                                                  |
-| `maxRetries`              | `number`                 | `3`         | Max retry attempts for a failed fetch (excluding initial try).                                                                                                                                                                                                        |
-| `retryDelay`              | `number`                 | `5000`      | Delay (ms) between retries.                                                                                                                                                                                                                                           |
-| `cacheTTL`                | `number`                 | `900000`    | Cache Time-To-Live (ms). `0` disables caching. (15 mins default)                                                                                                                                                                                                      |
-| `spaMode`                 | `boolean`                | `false`     | If `true`, enables Single Page Application mode. This typically bypasses `useHttpFallback`, effectively sets `fastMode` to `false`, uses more patient load conditions (e.g., network idle), and may apply `spaRenderDelayMs`. Recommended for JavaScript-heavy sites. |
-| `spaRenderDelayMs`        | `number`                 | `0`         | Explicit delay (ms) after page load events in `spaMode` to allow for client-side rendering. Only applies if `spaMode` is `true`.                                                                                                                                      |
-| `playwrightLaunchOptions` | `LaunchOptions`          | `undefined` | (For Playwright part) Optional Playwright launch options (from `playwright` package, e.g., `{ args: ['--some-flag'] }`) passed when a browser instance is created. Merged with internal defaults.                                                                     |
-
-**Browser Pool Options (For `HybridEngine`'s internal `PlaywrightEngine`):**
-
-| Option                     | Type                       | Default     | Description                                                                                 |
-| -------------------------- | -------------------------- | ----------- | ------------------------------------------------------------------------------------------- |
-| `maxBrowsers`              | `number`                   | `2`         | Max concurrent browser instances managed by the pool.                                       |
-| `maxPagesPerContext`       | `number`                   | `6`         | Max pages per browser context before recycling.                                             |
-| `maxBrowserAge`            | `number`                   | `1200000`   | Max age (ms) a browser instance lives before recycling. (20 mins default)                   |
-| `healthCheckInterval`      | `number`                   | `60000`     | How often (ms) the pool checks browser health. (1 min default)                              |
-| `useHeadedMode`            | `boolean`                  | `false`     | Forces the _entire pool_ (for Playwright part) to launch browsers in headed (visible) mode. |
-| `poolBlockedDomains`       | `string[]`                 | `[]`        | List of domain glob patterns to block requests to (for Playwright part).                    |
-| `poolBlockedResourceTypes` | `string[]`                 | `[]`        | List of Playwright resource types (e.g., 'image', 'font') to block (for Playwright part).   |
-| `proxy`                    | `{ server: string, ... }?` | `undefined` | Proxy configuration object (see `PlaywrightEngineConfig` type) (for Playwright part).       |
-
-### `HybridEngine` - Configuration Summary & Header Precedence
-
-When you configure `HybridEngine` using `PlaywrightEngineConfig`:
-
-- **`headers`**: Constructor headers are passed to the internal `FetchEngine`'s constructor and the internal `PlaywrightEngine`'s constructor.
-- **`markdown`**: Sets the default for both internal engines.
-- **`spaMode`**: Sets the default for `HybridEngine`'s SPA shell detection and for the internal `PlaywrightEngine`.
-- Other options primarily configure the internal `PlaywrightEngine` or general retry/caching logic.
-
-**Per-request `options` in `HybridEngine.fetchHTML(url, options)`:**
-
-- **`headers?: Record<string, string>`**:
-  - These headers override any headers set in the `HybridEngine` constructor.
-  - If `FetchEngine` is used: These headers are passed to `FetchEngine.fetchHTML(url, { headers: ... })`. `FetchEngine` then merges them with its constructor headers and base defaults.
-  - If `PlaywrightEngine` (fallback) is used: These headers are merged with `HybridEngine` constructor headers (options take precedence) and the result is passed to `PlaywrightEngine`'s `fetchHTML()`. `PlaywrightEngine` then applies its own logic (e.g., for `page.setExtraHTTPHeaders` or its HTTP fallback).
-- **`markdown?: boolean`**:
-  - If `FetchEngine` is used: This per-request option is **ignored**. `FetchEngine` uses its own constructor `markdown` setting.
-  - If `PlaywrightEngine` (fallback) is used: This overrides `PlaywrightEngine`'s default and determines its output format.
-- **`spaMode?: boolean`**: Overrides `HybridEngine`'s default SPA mode and is passed to `PlaywrightEngine` if used.
-- **`fastMode?: boolean`**: Passed to `PlaywrightEngine` if used; no effect on `FetchEngine`.
+### StructuredContentEngine Class
 
 ```typescript
-// Example: HybridEngine with SPA mode enabled by default
-const spaHybridEngine = new HybridEngine({ spaMode: true, spaRenderDelayMs: 2000 });
-
-async function fetchSpaSite() {
-  try {
-    // This will use PlaywrightEngine directly if smallblackdots is an SPA shell
-    const result = await spaHybridEngine.fetchHTML(
-      "https://www.smallblackdots.net/release/16109/corrina-joseph-wish-tonite-lonely"
-    );
-    console.log(`Title: ${result.title}`);
-  } catch (e) {
-    console.error(e);
-  }
-}
+import { StructuredContentEngine } from "@purepageio/fetch-engines";
+
+const productSchema = z.object({
+  name: z.string(),
+  price: z.number(),
+  inStock: z.boolean(),
+});
+
+const engine = new StructuredContentEngine({
+  spaMode: true,
+  spaRenderDelayMs: 2000,
+});
+
+const result = await engine.fetchStructuredContent("https://shop.com/product", productSchema);
+console.log(`${result.data.name} costs $${result.data.price}`);
+
+await engine.cleanup();
 ```
 
-## Return Value
+### Supported Models
 
-### `fetchHTML()` Result
+- `'gpt-5-mini'` - Latest model, mini version **(default)**
+- `'gpt-5'` - Most capable model
+- `'gpt-4.1-mini'` - Fast and cost-effective
+- `'gpt-4.1'` - More capable GPT-4.1 version
 
-All `fetchHTML()` methods return a Promise that resolves to an `HTMLFetchResult` object:
+## Configuration
 
-- `content` (`string`): The fetched content, either original HTML or converted Markdown.
-- `contentType` (`'html' | 'markdown'`): Indicates the format of the `content` string.
-- `title` (`string | null`): Extracted page title (from original HTML).
-- `url` (`string`): Final URL after redirects.
-- `isFromCache` (`boolean`): True if the result came from cache.
-- `statusCode` (`number | undefined`): HTTP status code.
-- `error` (`Error | undefined`): Error object if the fetch failed after all retries. It's generally recommended to rely on catching thrown errors for failure handling.
+### FetchEngine Options
 
-### `fetchContent()` Result
+| Option     | Type                     | Default | Description              |
+| ---------- | ------------------------ | ------- | ------------------------ |
+| `markdown` | `boolean`                | `false` | Convert HTML to Markdown |
+| `headers`  | `Record<string, string>` | `{}`    | Custom HTTP headers      |
 
-All `fetchContent()` methods return a Promise that resolves to a `ContentFetchResult` object:
+### HybridEngine Configuration
 
-- `content` (`Buffer | string`): The raw fetched content. Binary content (PDFs, images, etc.) is returned as `Buffer`, text content as `string`.
-- `contentType` (`string`): The original MIME type from the server (e.g., `"application/pdf"`, `"text/html"`, `"application/json"`).
-- `title` (`string | null`): Extracted page title if the content is HTML, otherwise `null`.
-- `url` (`string`): Final URL after redirects.
-- `isFromCache` (`boolean`): True if the result came from cache.
-- `statusCode` (`number | undefined`): HTTP status code.
-- `error` (`Error | undefined`): Error object if the fetch failed after all retries. It's generally recommended to rely on catching thrown errors for failure handling.
+| Option             | Type                     | Default  | Description                                  |
+| ------------------ | ------------------------ | -------- | -------------------------------------------- |
+| `headers`          | `Record<string, string>` | `{}`     | Default headers for both engines             |
+| `markdown`         | `boolean`                | `false`  | Default Markdown conversion                  |
+| `useHttpFallback`  | `boolean`                | `true`   | Try HTTP before Playwright                   |
+| `spaMode`          | `boolean`                | `false`  | Enable SPA mode with patient load conditions |
+| `spaRenderDelayMs` | `number`                 | `0`      | Delay after page load in SPA mode            |
+| `maxRetries`       | `number`                 | `3`      | Max retry attempts                           |
+| `cacheTTL`         | `number`                 | `900000` | Cache TTL in ms (15 min default)             |
+| `concurrentPages`  | `number`                 | `3`      | Max concurrent pages                         |
 
-## API Reference
+### Browser Pool Options
 
-### `engine.fetchHTML(url, options?)`
+| Option               | Type     | Default   | Description                        |
+| -------------------- | -------- | --------- | ---------------------------------- |
+| `maxBrowsers`        | `number` | `2`       | Max browser instances              |
+| `maxPagesPerContext` | `number` | `6`       | Pages per context before recycling |
+| `maxBrowserAge`      | `number` | `1200000` | Browser lifetime (20 min)          |
 
-- `url` (`string`): URL to fetch.
-- `options?` (`FetchOptions`): Optional per-request overrides.
-  - `headers?: Record<string, string>`: Custom headers for this specific request.
-  - `markdown?: boolean`: (For `HybridEngine`'s Playwright part) Request Markdown conversion.
-  - `fastMode?: boolean`: (For `HybridEngine`'s Playwright part) Override fast mode.
-  - `spaMode?: boolean`: (For `HybridEngine`) Override SPA mode behavior for this request.
-- **Returns:** `Promise<HTMLFetchResult>`
+### Header Precedence
 
-Fetches content, returning HTML or Markdown based on configuration/options in `result.content` with `result.contentType` indicating the format.
+Headers merge in this order (highest precedence first):
 
-### `engine.fetchContent(url, options?)`
+1. Request-specific headers in `fetchHTML(url, { headers })`
+2. Engine constructor headers
+3. Engine default headers
 
-- `url` (`string`): URL to fetch.
-- `options?` (`ContentFetchOptions`): Optional per-request overrides.
-  - `headers?: Record<string, string>`: Custom headers for this specific request.
-- **Returns:** `Promise<ContentFetchResult>`
+## Return Value
 
-Fetches raw content without processing, mimicking standard `fetch()` behavior. Returns binary content as `Buffer` and text content as `string`. Supports any content type (PDFs, images, JSON, XML, etc.) and uses the same smart fallback logic as `fetchHTML()` but without HTML-specific processing or content-type restrictions.
+### HTMLFetchResult (fetchHTML)
 
-### `engine.cleanup()` (`HybridEngine` and direct `FetchEngine` if no cleanup needed)
+- `content` (`string`): HTML or Markdown content
+- `contentType` (`'html' | 'markdown'`): Content format
+- `title` (`string | null`): Extracted page title
+- `url` (`string`): Final URL after redirects
+- `isFromCache` (`boolean`): Cache hit indicator
+- `statusCode` (`number | undefined`): HTTP status code
 
-- **Returns:** `Promise<void>`
+### ContentFetchResult (fetchContent)
 
-For `HybridEngine`, this gracefully shuts down all browser instances managed by its internal `PlaywrightEngine`. **It is crucial to call `await engine.cleanup()` when you are finished using `HybridEngine`** to release system resources.
-`FetchEngine` has a `cleanup` method for API consistency, but it's a no-op as `FetchEngine` doesn't manage persistent resources.
+- `content` (`Buffer | string`): Raw content (binary as Buffer, text as string)
+- `contentType` (`string`): Original MIME type
+- `title` (`string | null`): Title if HTML content, otherwise null
+- `url` (`string`): Final URL after redirects
+- `isFromCache` (`boolean`): Cache hit indicator
+- `statusCode` (`number | undefined`): HTTP status code
 
-## Stealth / Anti-Detection (via `HybridEngine`)
+## API Reference
 
-When `HybridEngine` uses its internal browser capabilities (via `PlaywrightEngine`), it automatically integrates `playwright-extra` and its powerful stealth plugin ([`puppeteer-extra-plugin-stealth`](https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth)). This plugin applies various techniques to make the headless browser controlled by Playwright appear more like a regular human-operated browser, helping to bypass many common bot detection systems.
+### `engine.fetchHTML(url, options?)`
 
-There are **no manual configuration options** for stealth; it is enabled by default when `HybridEngine` uses its browser functionality.
+- `url` (`string`): Target URL
+- `options?` (`FetchOptions`):
+  - `headers?: Record<string, string>`: Request headers
+  - `markdown?: boolean`: Request Markdown (HybridEngine only)
+  - `fastMode?: boolean`: Override fast mode (HybridEngine only)
+  - `spaMode?: boolean`: Override SPA mode (HybridEngine only)
+- **Returns:** `Promise<HTMLFetchResult>`
 
-While effective, be aware that no stealth technique is foolproof, and sophisticated websites may still detect automated browsing.
+### `engine.fetchContent(url, options?)`
 
-## Error Handling
+- `url` (`string`): Target URL
+- `options?` (`ContentFetchOptions`):
+  - `headers?: Record<string, string>`: Request headers
+- **Returns:** `Promise<ContentFetchResult>`
 
-Errors during fetching are typically thrown as instances of `FetchError` (or its subclasses like `FetchEngineHttpError`), providing more context than standard `Error` objects.
+### `fetchStructuredContent(url, schema, options?)`
 
-- `FetchError` properties:
-  - `message` (`string`): Description of the error.
-  - `code` (`string | undefined`): A specific error code (e.g., `ERR_NAVIGATION_TIMEOUT`, `ERR_HTTP_ERROR`, `ERR_NON_HTML_CONTENT`).
-  - `originalError` (`Error | undefined`): The underlying error that caused this fetch error (e.g., a Playwright error object).
-  - `statusCode` (`number | undefined`): The HTTP status code, if relevant (especially for `FetchEngineHttpError`).
+- `url` (`string`): Target URL
+- `schema` (`z.ZodSchema<T>`): Zod schema for extraction
+- `options?` (`StructuredContentOptions`):
+  - `model?: string`: OpenAI model (default: 'gpt-5-mini')
+  - `customPrompt?: string`: Additional AI context
+  - `engineConfig?: PlaywrightEngineConfig`: HybridEngine config
+- **Returns:** `Promise<StructuredContentResult<T>>`
 
-Common `FetchError` codes and scenarios:
+### `engine.cleanup()`
 
-- **`ERR_HTTP_ERROR`**: Thrown by `FetchEngine` for HTTP status codes >= 400. `error.statusCode` will be set.
-- **`ERR_NON_HTML_CONTENT`**: Thrown by `FetchEngine` if the content type is not HTML and `markdown` conversion is not requested. **Note:** `fetchContent()` does not throw this error as it supports all content types.
-- **`ERR_PLAYWRIGHT_OPERATION`**: A general error from `HybridEngine`'s browser mode indicating a failure during a Playwright operation (e.g., page acquisition, navigation, interaction). The `originalError` property will often contain the specific Playwright error.
-- **`ERR_NAVIGATION`**: Often seen as part of `ERR_PLAYWRIGHT_OPERATION`'s message or in `originalError` when a Playwright navigation (in `HybridEngine`'s browser mode) fails (e.g., timeout, SSL error).
-- **`ERR_MARKDOWN_CONVERSION_NON_HTML`**: Thrown by `HybridEngine` (when its Playwright part is active) if `markdown: true` is requested for a non-HTML content type (e.g., XML, JSON). **Note:** Only applies to `fetchHTML()` as `fetchContent()` doesn't perform markdown conversion.
-- **`ERR_CACHE_ERROR`**: Indicates an issue with cache read/write operations.
-- **`ERR_PROXY_CONFIG_ERROR`**: Problem with proxy configuration (for `HybridEngine`'s browser mode).
-- **`ERR_BROWSER_POOL_EXHAUSTED`**: If `HybridEngine`'s browser pool cannot provide a page.
-- **Other Scenarios (often wrapped by `ERR_PLAYWRIGHT_OPERATION` or a generic `FetchError` when `HybridEngine` uses its browser mode):**
-  - Network issues (DNS resolution, connection refused).
-  - Proxy connection failures.
-  - Page crashes or context/browser disconnections within Playwright.
-  - Failures during browser launch or management by the pool.
+Shuts down browser instances for `HybridEngine` and `StructuredContentEngine`. Call when finished to release resources. No-op for `FetchEngine`.
 
-The `HTMLFetchResult` object may also contain an `error` property if the final fetch attempt failed after all retries but an earlier attempt (within retries) might have produced some intermediate (potentially unusable) result data. It's generally best to rely on the thrown error for failure handling.
+## Stealth Features
 
-**Example:**
+When `HybridEngine` uses Playwright, it automatically applies stealth measures via `playwright-extra` and stealth plugins to bypass common bot detection. No manual configuration required.
 
-```typescript
-import { HybridEngine, FetchError } from "@purepageio/fetch-engines";
-
-// Example using HybridEngine to illustrate error handling
-const engine = new HybridEngine({ useHttpFallback: false, maxRetries: 1 }); // useHttpFallback for Playwright part
-
-async function fetchWithHandling(url: string) {
-  try {
-    // Try fetchHTML first
-    const htmlResult = await engine.fetchHTML(url, { headers: { "X-My-Header": "TestValue" } });
-    if (htmlResult.error) {
-      console.warn(`fetchHTML for ${url} included non-critical error after retries: ${htmlResult.error.message}`);
-    }
-    console.log(`fetchHTML Success for ${url}! Title: ${htmlResult.title}, Content type: ${htmlResult.contentType}`);
-    // Use htmlResult.content
-  } catch (error) {
-    console.error(`fetchHTML failed for ${url}, trying fetchContent...`);
-
-    try {
-      // Fallback to fetchContent for raw content
-      const contentResult = await engine.fetchContent(url, { headers: { "X-My-Header": "TestValue" } });
-      if (contentResult.error) {
-        console.warn(
-          `fetchContent for ${url} included non-critical error after retries: ${contentResult.error.message}`
-        );
-      }
-      console.log(`fetchContent Success for ${url}! Content type: ${contentResult.contentType}`);
-      // Use contentResult.content (could be Buffer or string)
-    } catch (contentError) {
-      console.error(`Both fetchHTML and fetchContent failed for ${url}:`);
-      if (contentError instanceof FetchError) {
-        console.error(`  Error Code: ${contentError.code || "N/A"}`);
-        console.error(`  Message: ${contentError.message}`);
-        if (contentError.statusCode) {
-          console.error(`  Status Code: ${contentError.statusCode}`);
-        }
-        if (contentError.originalError) {
-          console.error(`  Original Error: ${contentError.originalError.name} - ${contentError.originalError.message}`);
-        }
-        // Example of specific handling:
-        if (contentError.code === "ERR_PLAYWRIGHT_OPERATION") {
-          console.error(
-            "  Hint: This was a Playwright operation failure (HybridEngine's browser mode). Check Playwright logs or originalError."
-          );
-        }
-      } else if (contentError instanceof Error) {
-        console.error(`  Generic Error: ${contentError.message}`);
-      } else {
-        console.error(`  Unknown error occurred: ${String(contentError)}`);
-      }
-    }
-  }
-}
+Stealth techniques are not foolproof against sophisticated detection systems.
 
-async function runExamples() {
-  await fetchWithHandling("https://nonexistentdomain.example.com"); // Likely DNS or navigation error
-  await fetchWithHandling("https://example.com/document.pdf"); // PDF content - fetchHTML will fail, fetchContent will succeed
-  await fetchWithHandling("https://example.com/api/data.json"); // JSON content - fetchHTML will fail, fetchContent will succeed
-  await engine.cleanup(); // Important for HybridEngine
-}
+## Error Handling
 
-runExamples();
-```
+Errors are thrown as `FetchError` instances with additional context:
 
-## Logging
+- `message` (`string`): Error description
+- `code` (`string | undefined`): Specific error code
+- `originalError` (`Error | undefined`): Underlying error
+- `statusCode` (`number | undefined`): HTTP status code
 
-Currently, the library uses `console.warn` and `console.error` for internal warnings (like fallback events) and critical errors. More sophisticated logging options may be added in the future.
+Common error codes:
 
-## Testing
+- `ERR_HTTP_ERROR`: HTTP status >= 400
+- `ERR_NON_HTML_CONTENT`: Non-HTML content for HTML request
+- `ERR_FETCH_FAILED`: General fetch operation failure
+- `ERR_PLAYWRIGHT_OPERATION`: Playwright operation failure
+- `ERR_NAVIGATION`: Navigation timeout or failure
+- `ERR_BROWSER_POOL_EXHAUSTED`: No available browser resources
+- `ERR_MAX_RETRIES_REACHED`: All retry attempts exhausted
+- `ERR_MARKDOWN_CONVERSION_NON_HTML`: Markdown conversion on non-HTML content
 
-To run the test suite locally:
+```typescript
+import { HybridEngine } from "@purepageio/fetch-engines";
 
-```bash
-pnpm install
-pnpm exec playwright install
-pnpm test
-```
+const engine = new HybridEngine();
 
-The `pnpm exec playwright install` step downloads the required browser binaries for Playwright.
+try {
+  const result = await engine.fetchHTML(url);
+} catch (error: any) {
+  console.error(`Error: ${error.code || "Unknown"} - ${error.message}`);
+  if (error.statusCode) console.error(`Status: ${error.statusCode}`);
+}
+```
 
 ## Contributing
 
-Contributions are welcome! Please open an issue or submit a pull request on the [GitHub repository](https://github.com/purepageio/fetch-engines).
+Contributions welcome! Open an issue or submit a pull request on [GitHub](https://github.com/purepageio/fetch-engines).
 
 ## License
 

package/dist/StructuredContentEngine.d.ts

@@ -0,0 +1,67 @@
+import type { z } from "zod";
+import type { PlaywrightEngineConfig } from "./types.js";
+/**
+ * Configuration options for structured content fetching
+ */
+export interface StructuredContentOptions {
+    /** OpenAI model to use. Options: 'gpt-4.1-mini', 'gpt-4.1', 'gpt-5', 'gpt-5-mini' */
+    model?: "gpt-4.1-mini" | "gpt-4.1" | "gpt-5" | "gpt-5-mini";
+    /** Custom prompt to provide additional context to the LLM */
+    customPrompt?: string;
+    /** HybridEngine configuration for content fetching */
+    engineConfig?: PlaywrightEngineConfig;
+}
+/**
+ * Result of structured content extraction
+ */
+export interface StructuredContentResult<T> {
+    /** The structured data extracted from the content */
+    data: T;
+    /** The original markdown content that was processed */
+    markdown: string;
+    /** The URL that was processed */
+    url: string;
+    /** The title of the page if available */
+    title: string | null;
+    /** Token usage information */
+    usage: {
+        promptTokens: number;
+        completionTokens: number;
+        totalTokens: number;
+    };
+}
+/**
+ * Engine for fetching web content and extracting structured data using AI
+ */
+export declare class StructuredContentEngine {
+    private hybridEngine;
+    constructor(config?: PlaywrightEngineConfig);
+    /**
+     * Fetches content from a URL and extracts structured data using AI
+     *
+     * @param url The URL to fetch content from
+     * @param schema Zod schema defining the structure of data to extract
+     * @param options Additional options for the extraction process
+     * @returns Promise resolving to structured data and metadata
+     * @throws Error if OPENAI_API_KEY is not set or if extraction fails
+     */
+    fetchStructuredContent<T>(url: string, schema: z.ZodSchema<T>, options?: StructuredContentOptions): Promise<StructuredContentResult<T>>;
+    /**
+     * Get model-specific configuration options
+     */
+    private getModelConfig;
+    /**
+     * Clean up resources
+     */
+    cleanup(): Promise<void>;
+}
+/**
+ * Convenience function for one-off structured content extraction
+ *
+ * @param url The URL to fetch content from
+ * @param schema Zod schema defining the structure of data to extract
+ * @param options Additional options for the extraction process
+ * @returns Promise resolving to structured data and metadata
+ */
+export declare function fetchStructuredContent<T>(url: string, schema: z.ZodSchema<T>, options?: StructuredContentOptions): Promise<StructuredContentResult<T>>;
+//# sourceMappingURL=StructuredContentEngine.d.ts.map

package/dist/StructuredContentEngine.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"StructuredContentEngine.d.ts","sourceRoot":"","sources":["../src/StructuredContentEngine.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAE7B,OAAO,KAAK,EAAE,sBAAsB,EAAE,MAAM,YAAY,CAAC;AAEzD;;GAEG;AACH,MAAM,WAAW,wBAAwB;IACvC,qFAAqF;IACrF,KAAK,CAAC,EAAE,cAAc,GAAG,SAAS,GAAG,OAAO,GAAG,YAAY,CAAC;IAC5D,6DAA6D;IAC7D,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,sDAAsD;IACtD,YAAY,CAAC,EAAE,sBAAsB,CAAC;CACvC;AAED;;GAEG;AACH,MAAM,WAAW,uBAAuB,CAAC,CAAC;IACxC,qDAAqD;IACrD,IAAI,EAAE,CAAC,CAAC;IACR,uDAAuD;IACvD,QAAQ,EAAE,MAAM,CAAC;IACjB,iCAAiC;IACjC,GAAG,EAAE,MAAM,CAAC;IACZ,yCAAyC;IACzC,KAAK,EAAE,MAAM,GAAG,IAAI,CAAC;IACrB,8BAA8B;IAC9B,KAAK,EAAE;QACL,YAAY,EAAE,MAAM,CAAC;QACrB,gBAAgB,EAAE,MAAM,CAAC;QACzB,WAAW,EAAE,MAAM,CAAC;KACrB,CAAC;CACH;AAED;;GAEG;AACH,qBAAa,uBAAuB;IAClC,OAAO,CAAC,YAAY,CAAe;gBAEvB,MAAM,GAAE,sBAA2B;IAQ/C;;;;;;;;OAQG;IACG,sBAAsB,CAAC,CAAC,EAC5B,GAAG,EAAE,MAAM,EACX,MAAM,EAAE,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,EACtB,OAAO,GAAE,wBAA6B,GACrC,OAAO,CAAC,uBAAuB,CAAC,CAAC,CAAC,CAAC;IAsDtC;;OAEG;IACH,OAAO,CAAC,cAAc;IAiBtB;;OAEG;IACG,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC;CAG/B;AAED;;;;;;;GAOG;AACH,wBAAsB,sBAAsB,CAAC,CAAC,EAC5C,GAAG,EAAE,MAAM,EACX,MAAM,EAAE,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,EACtB,OAAO,GAAE,wBAA6B,GACrC,OAAO,CAAC,uBAAuB,CAAC,CAAC,CAAC,CAAC,CAOrC"}

package/dist/StructuredContentEngine.js

@@ -0,0 +1,116 @@
+import { generateObject } from "ai";
+import { openai } from "@ai-sdk/openai";
+import { HybridEngine } from "./HybridEngine.js";
+/**
+ * Engine for fetching web content and extracting structured data using AI
+ */
+export class StructuredContentEngine {
+    hybridEngine;
+    constructor(config = {}) {
+        // Always enable markdown conversion for structured content
+        this.hybridEngine = new HybridEngine({
+            ...config,
+            markdown: true,
+        });
+    }
+    /**
+     * Fetches content from a URL and extracts structured data using AI
+     *
+     * @param url The URL to fetch content from
+     * @param schema Zod schema defining the structure of data to extract
+     * @param options Additional options for the extraction process
+     * @returns Promise resolving to structured data and metadata
+     * @throws Error if OPENAI_API_KEY is not set or if extraction fails
+     */
+    async fetchStructuredContent(url, schema, options = {}) {
+        // Check for OpenAI API key
+        if (!process.env.OPENAI_API_KEY) {
+            throw new Error("OPENAI_API_KEY environment variable is required for structured content extraction");
+        }
+        const { model = "gpt-5-mini", customPrompt = "", engineConfig = {} } = options;
+        // Fetch content using HybridEngine with markdown enabled
+        const result = await this.hybridEngine.fetchHTML(url, {
+            markdown: true,
+            ...engineConfig,
+        });
+        if (result.contentType !== "markdown") {
+            throw new Error("Failed to convert content to markdown");
+        }
+        // Prepare the prompt for the LLM
+        const systemPrompt = `You are an expert at extracting structured data from web content. 
+Extract the requested information from the provided markdown content accurately and completely.
+${customPrompt ? `\nAdditional context: ${customPrompt}` : ""}
+
+Content to analyze:
+${result.content}`;
+        // Configure model-specific options
+        const modelConfig = this.getModelConfig(model);
+        try {
+            // Generate structured object using AI SDK
+            const aiResult = await generateObject({
+                model: openai(model),
+                schema,
+                prompt: systemPrompt,
+                ...modelConfig,
+            });
+            return {
+                data: aiResult.object,
+                markdown: result.content,
+                url: result.url,
+                title: result.title,
+                usage: {
+                    promptTokens: aiResult.usage?.promptTokens ?? 0,
+                    completionTokens: aiResult.usage?.completionTokens ?? 0,
+                    totalTokens: aiResult.usage?.totalTokens ?? 0,
+                },
+            };
+        }
+        catch (error) {
+            throw new Error(`Failed to extract structured data: ${error instanceof Error ? error.message : String(error)}`);
+        }
+    }
+    /**
+     * Get model-specific configuration options
+     */
+    getModelConfig(model) {
+        if (model.startsWith("gpt-5")) {
+            return {
+                providerOptions: {
+                    openai: {
+                        reasoning_effort: "low",
+                    },
+                },
+            };
+        }
+        else if (model.startsWith("gpt-4.1")) {
+            return {
+                temperature: 0,
+            };
+        }
+        return {};
+    }
+    /**
+     * Clean up resources
+     */
+    async cleanup() {
+        await this.hybridEngine.cleanup();
+    }
+}
+/**
+ * Convenience function for one-off structured content extraction
+ *
+ * @param url The URL to fetch content from
+ * @param schema Zod schema defining the structure of data to extract
+ * @param options Additional options for the extraction process
+ * @returns Promise resolving to structured data and metadata
+ */
+export async function fetchStructuredContent(url, schema, options = {}) {
+    const engine = new StructuredContentEngine(options.engineConfig);
+    try {
+        return await engine.fetchStructuredContent(url, schema, options);
+    }
+    finally {
+        await engine.cleanup();
+    }
+}
+//# sourceMappingURL=StructuredContentEngine.js.map

package/dist/StructuredContentEngine.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"StructuredContentEngine.js","sourceRoot":"","sources":["../src/StructuredContentEngine.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,IAAI,CAAC;AACpC,OAAO,EAAE,MAAM,EAAE,MAAM,gBAAgB,CAAC;AAExC,OAAO,EAAE,YAAY,EAAE,MAAM,mBAAmB,CAAC;AAmCjD;;GAEG;AACH,MAAM,OAAO,uBAAuB;IAC1B,YAAY,CAAe;IAEnC,YAAY,SAAiC,EAAE;QAC7C,2DAA2D;QAC3D,IAAI,CAAC,YAAY,GAAG,IAAI,YAAY,CAAC;YACnC,GAAG,MAAM;YACT,QAAQ,EAAE,IAAI;SACf,CAAC,CAAC;IACL,CAAC;IAED;;;;;;;;OAQG;IACH,KAAK,CAAC,sBAAsB,CAC1B,GAAW,EACX,MAAsB,EACtB,UAAoC,EAAE;QAEtC,2BAA2B;QAC3B,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,cAAc,EAAE,CAAC;YAChC,MAAM,IAAI,KAAK,CAAC,mFAAmF,CAAC,CAAC;QACvG,CAAC;QAED,MAAM,EAAE,KAAK,GAAG,YAAY,EAAE,YAAY,GAAG,EAAE,EAAE,YAAY,GAAG,EAAE,EAAE,GAAG,OAAO,CAAC;QAE/E,yDAAyD;QACzD,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,YAAY,CAAC,SAAS,CAAC,GAAG,EAAE;YACpD,QAAQ,EAAE,IAAI;YACd,GAAG,YAAY;SAChB,CAAC,CAAC;QAEH,IAAI,MAAM,CAAC,WAAW,KAAK,UAAU,EAAE,CAAC;YACtC,MAAM,IAAI,KAAK,CAAC,uCAAuC,CAAC,CAAC;QAC3D,CAAC;QAED,iCAAiC;QACjC,MAAM,YAAY,GAAG;;EAEvB,YAAY,CAAC,CAAC,CAAC,yBAAyB,YAAY,EAAE,CAAC,CAAC,CAAC,EAAE;;;EAG3D,MAAM,CAAC,OAAO,EAAE,CAAC;QAEf,mCAAmC;QACnC,MAAM,WAAW,GAAG,IAAI,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC;QAE/C,IAAI,CAAC;YACH,0CAA0C;YAC1C,MAAM,QAAQ,GAAG,MAAM,cAAc,CAAC;gBACpC,KAAK,EAAE,MAAM,CAAC,KAAK,CAAC;gBACpB,MAAM;gBACN,MAAM,EAAE,YAAY;gBACpB,GAAG,WAAW;aACf,CAAC,CAAC;YAEH,OAAO;gBACL,IAAI,EAAE,QAAQ,CAAC,MAAM;gBACrB,QAAQ,EAAE,MAAM,CAAC,OAAO;gBACxB,GAAG,EAAE,MAAM,CAAC,GAAG;gBACf,KAAK,EAAE,MAAM,CAAC,KAAK;gBACnB,KAAK,EAAE;oBACL,YAAY,EAAG,QAAQ,CAAC,KAAa,EAAE,YAAY,IAAI,CAAC;oBACxD,gBAAgB,EAAG,QAAQ,CAAC,KAAa,EAAE,gBAAgB,IAAI,CAAC;oBAChE,WAAW,EAAG,QAAQ,CAAC,KAAa,EAAE,WAAW,IAAI,CAAC;iBACvD;aACF,CAAC;QACJ,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,MAAM,IAAI,KAAK,CAAC,sCAAsC,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QAClH,CAAC;IACH,CAAC;IAED;;OAEG;IACK,cAAc,CAAC,KAAa;QAClC,IAAI,KAAK,CAAC,UAAU,CAAC,OAAO,CAAC,EAAE,CAAC;YAC9B,OAAO;gBACL,eAAe,EAAE;oBACf,MAAM,EAAE;wBACN,gBAAgB,EAAE,KAAK;qBACxB;iBACF;aACF,CAAC;QACJ,CAAC;aAAM,IAAI,KAAK,CAAC,UAAU,CAAC,SAAS,CAAC,EAAE,CAAC;YACvC,OAAO;gBACL,WAAW,EAAE,CAAC;aACf,CAAC;QACJ,CAAC;QACD,OAAO,EAAE,CAAC;IACZ,CAAC;IAED;;OAEG;IACH,KAAK,CAAC,OAAO;QACX,MAAM,IAAI,CAAC,YAAY,CAAC,OAAO,EAAE,CAAC;IACpC,CAAC;CACF;AAED;;;;;;;GAOG;AACH,MAAM,CAAC,KAAK,UAAU,sBAAsB,CAC1C,GAAW,EACX,MAAsB,EACtB,UAAoC,EAAE;IAEtC,MAAM,MAAM,GAAG,IAAI,uBAAuB,CAAC,OAAO,CAAC,YAAY,CAAC,CAAC;IACjE,IAAI,CAAC;QACH,OAAO,MAAM,MAAM,CAAC,sBAAsB,CAAC,GAAG,EAAE,MAAM,EAAE,OAAO,CAAC,CAAC;IACnE,CAAC;YAAS,CAAC;QACT,MAAM,MAAM,CAAC,OAAO,EAAE,CAAC;IACzB,CAAC;AACH,CAAC"}

package/dist/index.d.ts

@@ -4,4 +4,5 @@ import type { HTMLFetchResult, ContentFetchResult, ContentFetchOptions, BrowserM
 export type { IEngine, HTMLFetchResult, ContentFetchResult, ContentFetchOptions, BrowserMetrics };
 export { FetchEngine };
 export * from "./HybridEngine.js";
+export * from "./StructuredContentEngine.js";
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAE/C,OAAO,KAAK,EAAE,eAAe,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AAE3G,YAAY,EAAE,OAAO,EAAE,eAAe,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,cAAc,EAAE,CAAC;AAClG,OAAO,EAAE,WAAW,EAAE,CAAC;AACvB,cAAc,mBAAmB,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAE/C,OAAO,KAAK,EAAE,eAAe,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC;AAE3G,YAAY,EAAE,OAAO,EAAE,eAAe,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,cAAc,EAAE,CAAC;AAClG,OAAO,EAAE,WAAW,EAAE,CAAC;AACvB,cAAc,mBAAmB,CAAC;AAClC,cAAc,8BAA8B,CAAC"}

package/dist/index.js

@@ -1,4 +1,5 @@
 import { FetchEngine } from "./FetchEngine.js";
 export { FetchEngine };
 export * from "./HybridEngine.js"; // Export the new engine
+export * from "./StructuredContentEngine.js"; // Export structured content functionality
 //# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAK/C,OAAO,EAAE,WAAW,EAAE,CAAC;AACvB,cAAc,mBAAmB,CAAC,CAAC,wBAAwB"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAK/C,OAAO,EAAE,WAAW,EAAE,CAAC;AACvB,cAAc,mBAAmB,CAAC,CAAC,wBAAwB;AAC3D,cAAc,8BAA8B,CAAC,CAAC,0CAA0C"}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@purepageio/fetch-engines",
-  "version": "0.5.1-rc.0",
+  "version": "0.6.0",
   "type": "module",
   "description": "A collection of configurable engines for fetching HTML content using fetch or Playwright.",
   "main": "dist/index.js",
@@ -11,6 +11,8 @@
     "LICENSE"
   ],
   "dependencies": {
+    "@ai-sdk/openai": "^2.0.30",
+    "ai": "^5.0.44",
     "axios": "^1.6.8",
     "node-html-parser": "^7.0.1",
     "p-queue": "^7.4.1",
@@ -21,7 +23,8 @@
     "turndown": "^7.2.0",
     "turndown-plugin-gfm": "^1.0.2",
     "user-agents": "^1.1.208",
-    "uuid": "^11.1.0"
+    "uuid": "^11.1.0",
+    "zod": "^4.1.8"
   },
   "devDependencies": {
     "@types/axios": "^0.14.0",
@@ -76,6 +79,9 @@
     "test": "vitest run",
     "test:unit": "vitest run",
     "test:live": "LIVE_NETWORK=1 vitest run test/live/*.test.ts",
-    "examples:hybrid-md": "node scripts/hybrid-md-dump.mjs"
+    "examples:hybrid-md": "node scripts/hybrid-md-dump.mjs",
+    "simple-scraping": "tsx examples/simple-scraping.ts",
+    "smart-scraping": "tsx examples/smart-scraping.ts",
+    "ai-extraction": "tsx examples/ai-extraction.ts"
   }
 }