scrapex 0.5.3 → 1.0.0-beta.1

package/README.md

@@ -1,157 +1,563 @@
-# Overview
-
-ScrapeX provides a node library to scrape basic details of a url using the [metascraper](https://metascraper.js.org/#/), [firefox readability](https://github.com/mozilla/readability), [page-metadata-parser](https://github.com/mozilla/page-metadata-parser) libraries.
-
-## Getting Started
-
-### Install
-
-Install
-
-```javascript
-npm i scrapex --save or yarn add scrapex
-```
-
-### Usage
-
-```javascript
-import { scrape, scrapeHtml } from 'scrapex';
-// or
-const { scrape, scrapeHtml } = require('scrapex');
-// usage
-scrape(url, options)
-options = { metascraperRules?, timeout?, sanitizeOptions?}
-metascraperRules = optional array of 'audio'
-  | 'amazon'
-  | 'iframe'
-  | 'media-provider'
-  | 'soundcloud'
-  | 'uol'
-  | 'spotify'
-  | 'video'
-  | 'youtube'
-timeout: number = default to 60 seconds
-sanitizeOptions: sanitize-html options
-
-```
-
-```javascript
-// define a url
-const url = "https://appleinsider.com/articles/19/08/22/like-apple-music-spotify-now-offers-a-three-month-premium-trial"
-const data = await scrape(url, {metascraperRules:['youtube']})
-// or
-const html = `html string of the webpage extracted separately`
-const data = await scrape(url, html, {metascraperRules:['youtube']})
-
-// scrape result
-{
-  audio: undefined,
-  author: 'Amber Neely',
-  logo: 'https://logo.clearbit.com/appleinsider.com',
-  favicon: 'https://photos5.appleinsider.com/v9/images/apple-touch-icon-72.png',
-  image: 'https://photos5.appleinsider.com/gallery/32489-55660-header-xl.jpg',
-  publisher: 'AppleInsider',
-  date: '2019-08-22T12:14:35.000Z',
-  description: "Spotify has extended the free-trial period it offers for Spotify Premium from one month to three, the default length of Apple's free trial for Apple Music.",
-  lang: 'en',
-  url: 'https://appleinsider.com/articles/19/08/22/like-apple-music-spotify-now-offers-a-three-month-premium-trial',
-  text: "Spotify has extended the free-trial period it offers for Spotify Premium from one month to three, the default length of Apple's free trial for Apple Music.\n" +
-    'Streaming giant Spotify is now offering three free months to anyone who has yet to try their service, according to a news post on their site.\n' +
-    `"Beginning August 22, eligible users will receive the first three months on us for free when they sign up for any Spotify Premium plan," says Spotify in a statement about the new trial. "You'll unlock a world of on-demand access to millions of hours of audio content—  no matter when you sign up, winter, spring, summer, or fall."\n` +
-    "The trial period currently only extends to individual and student plans and will roll out across Duo and Family in the coming months. The trial doesn't extend to Headspace or anyone who is billed directly through their carrier, with the exception of those in Japan, Australia, China, and Germany. \n" +
-    "Apple has been offering free three-month trials to Apple Music since it's inception, though they may begin limiting their trial to one month. Apple had learned artists are wary of lengthy trial periods when Taylor Swift protested the three-month trial by withholding her album 1989 from the service. The protest earned artists the ability to be paid for track and album streams through the free trial period.\n" +
-    'Like most other paid music subscriptions, Spotify Premium offers users the ability to listen ad-free, download music to their device, create playlists, skip tracks, and toggle between devices when listening. ',
-  video: null,
-  keywords: [
-    'Apple',               'Apple Inc',
-    'iPhone',              'iPad',
-    'iPod touch',          'iPod nano',
-    'Apple TV',            'Apple',
-    'iPod shuffle',        'iphone 6',
-    'iphone 6s',           'ios 9',
-    'ios9',                'iTunes',
-    'i mac',               'mac os x',
-    'mac osx',             'Apple Computer',
-    'Apple Computer Inc.', 'Mac OS X',
-    'iMac',                'iBook',
-    'Mac Pro',             'MacBook Pro',
-    'Magic Pad',           'Magic Mouse',
-    'iPod classic',        'App Store',
-    'iTunes Store',        'iBook Store',
-    'mac book',            'Microsoft',
-    'Adobe',               'Research in Motion',
-    'RIM',                 'Nokia',
-    'Samsung',             'Google',
-    'Nvidia',              'Intel'
-  ],
-  tags: [],
-  embeds: [],
-  source: 'appleinsider.com',
-  twitter: {
-    site: '@appleinsider',
-    creator: '@appleinsider',
-    description: "Spotify has extended the free-trial period it offers for Spotify Premium from one month to three, the default length of Apple's free trial for Apple Music.",
-    title: 'Like Apple Music, Spotify now offers a three month premium trial | AppleInsider',
-    image: 'https://photos5.appleinsider.com/gallery/32489-55660-header-xl.jpg'
+# scrapex
+
+Modern web scraper with LLM-enhanced extraction, extensible pipeline, and pluggable parsers.
+
+> **Beta Release**: v1.0.0 is currently in beta. The API is stable but minor changes may occur before the stable release.
+
+## Features
+
+- **LLM-Ready Output** - Content extracted as Markdown, optimized for AI/LLM consumption
+- **Provider-Agnostic LLM** - Works with OpenAI, Anthropic, Ollama, LM Studio, or any OpenAI-compatible API
+- **Vector Embeddings** - Generate embeddings with OpenAI, Azure, Cohere, HuggingFace, Ollama, or local Transformers.js
+- **Extensible Pipeline** - Pluggable extractors with priority-based execution
+- **Smart Extraction** - Uses Mozilla Readability for content, Cheerio for metadata
+- **Markdown Parsing** - Parse markdown content, awesome lists, and GitHub repos
+- **RSS/Atom Feeds** - Parse RSS 2.0, RSS 1.0 (RDF), and Atom feeds with pagination support
+- **TypeScript First** - Full type safety with comprehensive type exports
+- **Dual Format** - ESM and CommonJS builds
+
+## Installation
+
+```bash
+npm install scrapex@beta
+```
+
+### Optional Peer Dependencies
+
+```bash
+# For LLM features
+npm install openai           # OpenAI/Ollama/LM Studio
+npm install @anthropic-ai/sdk  # Anthropic Claude
+
+# For JavaScript-rendered pages
+npm install puppeteer
+```
+
+## Quick Start
+
+```typescript
+import { scrape } from 'scrapex';
+
+const result = await scrape('https://example.com/article');
+
+console.log(result.title);       // "Article Title"
+console.log(result.content);     // Markdown content
+console.log(result.textContent); // Plain text (lower tokens)
+console.log(result.excerpt);     // First ~300 chars
+```
+
+## API Reference
+
+### `scrape(url, options?)`
+
+Fetch and extract metadata and content from a URL.
+
+```typescript
+import { scrape } from 'scrapex';
+
+const result = await scrape('https://example.com', {
+  timeout: 10000,
+  userAgent: 'MyBot/1.0',
+  extractContent: true,
+  maxContentLength: 50000,
+  respectRobots: false,
+});
+```
+
+### `scrapeHtml(html, url, options?)`
+
+Extract from raw HTML without fetching.
+
+```typescript
+import { scrapeHtml } from 'scrapex';
+
+const html = await fetchSomehow('https://example.com');
+const result = await scrapeHtml(html, 'https://example.com');
+```
+
+### Result Object (`ScrapedData`)
+
+```typescript
+interface ScrapedData {
+  // Identity
+  url: string;
+  canonicalUrl: string;
+  domain: string;
+
+  // Basic metadata
+  title: string;
+  description: string;
+  image?: string;
+  favicon?: string;
+
+  // Content (LLM-optimized)
+  content: string;      // Markdown format
+  textContent: string;  // Plain text
+  excerpt: string;      // ~300 char preview
+  wordCount: number;
+
+  // Context
+  author?: string;
+  publishedAt?: string;
+  modifiedAt?: string;
+  siteName?: string;
+  language?: string;
+
+  // Classification
+  contentType: 'article' | 'repo' | 'docs' | 'package' | 'video' | 'tool' | 'product' | 'unknown';
+  keywords: string[];
+
+  // Structured data
+  jsonLd?: Record<string, unknown>[];
+  links?: ExtractedLink[];
+
+  // LLM Enhancements (when enabled)
+  summary?: string;
+  suggestedTags?: string[];
+  entities?: ExtractedEntities;
+  extracted?: Record<string, unknown>;
+
+  // Meta
+  scrapedAt: string;
+  scrapeTimeMs: number;
+  error?: string;
+}
+```
+
+## LLM Integration
+
+### Using OpenAI
+
+```typescript
+import { scrape } from 'scrapex';
+import { createOpenAI } from 'scrapex/llm';
+
+const llm = createOpenAI({ apiKey: 'sk-...' });
+
+const result = await scrape('https://example.com/article', {
+  llm,
+  enhance: ['summarize', 'tags', 'entities', 'classify'],
+});
+
+console.log(result.summary);       // AI-generated summary
+console.log(result.suggestedTags); // ['javascript', 'web', ...]
+console.log(result.entities);      // { people: [], organizations: [], ... }
+```
+
+### Embeddings
+
+Generate vector embeddings from scraped content for semantic search, RAG, and similarity matching:
+
+```typescript
+import { scrape } from 'scrapex';
+import { createOpenAIEmbedding } from 'scrapex/embeddings';
+
+const result = await scrape('https://example.com/article', {
+  embeddings: {
+    provider: { type: 'custom', provider: createOpenAIEmbedding() },
+    model: 'text-embedding-3-small',
   },
-  title: 'Like Apple Music, Spotify now offers a three month premium trial',
-  links: [
-    {
-      href: 'https://newsroom.spotify.com/2019-08-22/5-ways-to-take-control-of-your-streaming-with-spotify-premium/',
-      text: 'a news post on their site.'
-    },
-    {
-      href: 'https://appleinsider.com/inside/apple-music',
-      text: 'Apple Music'
-    },
-    {
-      href: 'https://appleinsider.com/articles/19/07/25/apple-begins-limiting-apple-music-free-trial-period-to-one-month',
-      text: 'one month.'
-    },
-    {
-      href: 'https://appleinsider.com/articles/15/06/18/apple-music-to-miss-out-on-taylor-swifts-1989-album',
-      text: 'Taylor Swift protested the three-month trial'
-    }
-  ],
-  content:
-  '<div><div><div><div> <p>\n\t\t\tBy <a href="undefined/cdn-cgi/l/email-protection#d6b7bbb4b3a496b7a6a6bab3bfb8a5bfb2b3a4f8b5b9bb">Amber Neely</a>\t\t\t<br />\n\t\t\tThursday, August 22, 2019, 05:14 am PT (08:14 am ET)\n\t\t</p>Spotify has extended the free-trial period it offers for Spotify Premium from one month to three, the default length of Apple\'s free trial for Apple Music.<br /><p>\nStreaming giant Spotify is now offering three free months to anyone who has yet to try their service, according to <a href="https://newsroom.spotify.com/2019-08-22/5-ways-to-take-control-of-your-streaming-with-spotify-premium/">a news post on their site.</a></p><p>\n"Beginning August 22, eligible users will receive the first three months on us for free when they sign up for any Spotify Premium plan," says Spotify in a statement about the new trial. "You\'ll unlock a world of on-demand access to millions of hours of audio content—no matter when you sign up, winter, spring, summer, or fall."</p><p>\nThe trial period currently only extends to individual and student plans and will roll out across Duo and Family in the coming months. The trial doesn\'t extend to Headspace or anyone who is billed directly through their carrier, with the exception of those in Japan, Australia, China, and Germany. </p><p>\nApple has been offering free three-month trials to Apple Music since it\'s inception, though they may begin limiting their trial to <a href="https://appleinsider.com/articles/19/07/25/apple-begins-limiting-apple-music-free-trial-period-to-one-month">one month.</a> Apple had learned artists are wary of lengthy trial periods when <a href="https://appleinsider.com/articles/15/06/18/apple-music-to-miss-out-on-taylor-swifts-1989-album">Taylor Swift protested the three-month trial</a> by withholding her album <em>1989</em> from the service. The protest earned artists the ability to be paid for track and album streams through the free trial period.</p><p>\nStudents who sign up for Apple Music can get a free six-month trial <a href="https://support.apple.com/en-ke/HT205928">by visiting Apple\'s Support Page.</a> After the trial ends, students pay $4.99 a month to continue their subscription until graduation, which works out to be <a href="https://appleinsider.com/articles/16/05/06/apple-begins-offering-half-price-499-apple-music-subscriptions-for-students">about half the price of a standard subscription.</a></p><p>\nLike most other paid music subscriptions, Spotify Premium offers users the ability to listen ad-free, download music to their device, create playlists, skip tracks, and toggle between devices when listening. </p></div></div></div></div>',
-  html: ...
+});
+
+if (result.embeddings?.status === 'success') {
+  console.log(result.embeddings.vector); // [0.023, -0.041, ...]
 }
 ```
 
-### Extracted data elements
+Features include:
+- **Multiple providers** - OpenAI, Azure, Cohere, HuggingFace, Ollama, Transformers.js
+- **PII redaction** - Automatically redact emails, phones, SSNs before sending to APIs
+- **Smart chunking** - Split long content with configurable overlap
+- **Caching** - Content-addressable cache to avoid redundant API calls
+- **Resilience** - Retry, circuit breaker, rate limiting
+
+See the [Embeddings Guide](https://scrapex.dev/guides/embeddings) for full documentation.
+
+## Breaking Changes (Beta)
+
+- LLM provider classes (e.g., `AnthropicProvider`) were removed. Use preset factories like
+  `createOpenAI`, `createAnthropic`, `createOllama`, and `createLMStudio` instead.
+
+### Using Anthropic Claude
+
+```typescript
+import { createAnthropic } from 'scrapex/llm';
+
+const llm = createAnthropic({
+  apiKey: process.env.ANTHROPIC_API_KEY,
+  model: 'claude-3-5-haiku-20241022', // or 'claude-3-5-sonnet-20241022'
+});
+
+const result = await scrape(url, { llm, enhance: ['summarize'] });
+```
+
+### Using Ollama (Local)
+
+```typescript
+import { createOllama } from 'scrapex/llm';
+
+const llm = createOllama({ model: 'llama3.2' });
+
+const result = await scrape(url, { llm, enhance: ['summarize'] });
+```
+
+### Using LM Studio (Local)
+
+```typescript
+import { createLMStudio } from 'scrapex/llm';
+
+const llm = createLMStudio({ model: 'local-model' });
+
+const result = await scrape(url, { llm, enhance: ['summarize'] });
+```
+
+### Structured Extraction
+
+Extract specific data using a schema:
+
+```typescript
+const result = await scrape('https://example.com/product', {
+  llm,
+  extract: {
+    productName: 'string',
+    price: 'number',
+    features: 'string[]',
+    inStock: 'boolean',
+    sku: 'string?', // optional
+  },
+});
+
+console.log(result.extracted);
+// { productName: "Widget", price: 29.99, features: [...], inStock: true }
+```
+
+## Custom Extractors
+
+Create custom extractors to add domain-specific extraction logic:
+
+```typescript
+import { scrape, type Extractor, type ExtractionContext } from 'scrapex';
+
+const recipeExtractor: Extractor = {
+  name: 'recipe',
+  priority: 60, // Higher = runs earlier
+
+  async extract(context: ExtractionContext) {
+    const { $ } = context;
+
+    return {
+      custom: {
+        ingredients: $('.ingredients li').map((_, el) => $(el).text()).get(),
+        cookTime: $('[itemprop="cookTime"]').attr('content'),
+        servings: $('[itemprop="recipeYield"]').text(),
+      },
+    };
+  },
+};
+
+const result = await scrape('https://example.com/recipe', {
+  extractors: [recipeExtractor],
+});
+
+console.log(result.custom?.ingredients);
+```
+
+### Replacing Default Extractors
+
+```typescript
+const result = await scrape(url, {
+  replaceDefaultExtractors: true,
+  extractors: [myCustomExtractor],
+});
+```
 
-This is what `scrapex` will try to grab from a web page:
+## Markdown Parsing
 
-- `audio` — eg. <https://cf-media.sndcdn.com/U78RIfDPV6ok.128.mp3>. A audio URL that best represents the article.
-- `title` - The document's title (from the &lt;title&gt; tag)
-- `date` - The document's publication date
-- `copyright` - The document's copyright line, if present
-- `author` - The document's author
-- `publisher` - The document's publisher (website name)
-- `text` - The main text of the document with all the junk thrown away
-- `image` - The main image for the document (what's used by facebook, etc.)
-- `video` - A video URL that best represents the article.
-- `embeds` - An array of iframe, embed, object, video that were embedded in the article.
-- `tags`- Any tags or keywords that could be found by checking href urls at has the following pattern `a[href*='/t/'],a[href*='/tag/'], a[href*='/tags/'], a[href*='/topic/'],a[href*='/tagged/'], a[href*='?keyword=']`.
-- `keywords`- Any keywords that could be found by checking &lt;rel&gt; tags or by looking at href urls.
-- `lang` - The language of the document, either detected or supplied by you.
-- `description` - The description of the document, from &lt;meta&gt; tags
-- `favicon` - The url of the document's [favicon](http://en.wikipedia.org/wiki/Favicon).
-- `links` - An array of links embedded within the main article text. (text and href for each)
-- `logo` — eg. <https://entrepreneur.com/favicon180x180.png>. An image URL that best represents the publisher brand.
-- `content` — readability view html of the article.
-- `html` — full html of the page.
-- `text` — clear text of the readable html.
-- `code` - code segments defined using pre > code tags
+Parse markdown content:
 
-## Todo
+```typescript
+import { MarkdownParser, extractListLinks, groupByCategory } from 'scrapex/parsers';
 
-## Contributors
+// Parse any markdown
+const parser = new MarkdownParser();
+const result = parser.parse(markdownContent);
 
-<img width=150px src="https://pbs.twimg.com/profile_images/1028292150205661185/TFP8E8Fc_400x400.jpg">
-<p><strong>Rakesh Paul</strong> - <a href="https://xtrios.com">Xtrios</a></p>
+console.log(result.data.title);
+console.log(result.data.sections);
+console.log(result.data.links);
+console.log(result.data.codeBlocks);
+
+// Extract links from markdown lists and group by category
+const links = extractListLinks(markdownContent);
+const grouped = groupByCategory(links);
+
+grouped.forEach((categoryLinks, category) => {
+  console.log(`${category}: ${categoryLinks.length} links`);
+});
+```
+
+### GitHub Utilities
+
+```typescript
+import {
+  isGitHubRepo,
+  parseGitHubUrl,
+  toRawUrl,
+} from 'scrapex/parsers';
+
+isGitHubRepo('https://github.com/owner/repo');
+// true
+
+parseGitHubUrl('https://github.com/facebook/react');
+// { owner: 'facebook', repo: 'react' }
+
+toRawUrl('https://github.com/owner/repo');
+// 'https://raw.githubusercontent.com/owner/repo/main/README.md'
+```
+
+## RSS/Atom Feed Parsing
+
+Parse RSS 2.0, RSS 1.0 (RDF), and Atom 1.0 feeds:
+
+```typescript
+import { RSSParser } from 'scrapex';
+
+const parser = new RSSParser();
+const result = parser.parse(feedXml, 'https://example.com/feed.xml');
+
+console.log(result.data.format);  // 'rss2' | 'rss1' | 'atom'
+console.log(result.data.title);   // Feed title
+console.log(result.data.items);   // Array of feed items
+```
+
+**Supported formats:**
+- `rss2` - RSS 2.0 (most common format)
+- `rss1` - RSS 1.0 (RDF-based, older format)
+- `atom` - Atom 1.0 (modern format with better semantics)
+
+### Feed Item Structure
+
+```typescript
+interface FeedItem {
+  id: string;
+  title: string;
+  link: string;
+  description?: string;
+  content?: string;
+  author?: string;
+  publishedAt?: string;      // ISO 8601
+  rawPublishedAt?: string;   // Original date string
+  updatedAt?: string;        // Atom only
+  categories: string[];
+  enclosure?: FeedEnclosure; // Podcast/media attachments
+  customFields?: Record<string, string>;
+}
+```
+
+### Fetching and Parsing Feeds
+
+```typescript
+import { fetchFeed, paginateFeed } from 'scrapex';
+
+// Fetch and parse in one call
+const result = await fetchFeed('https://example.com/feed.xml');
+console.log(result.data.items);
+
+// Paginate through feeds with rel="next" links (Atom)
+for await (const page of paginateFeed('https://example.com/atom')) {
+  console.log(`Page with ${page.data.items.length} items`);
+}
+```
+
+### Discovering Feeds in HTML
+
+```typescript
+import { discoverFeeds } from 'scrapex';
+
+const html = await fetch('https://example.com').then(r => r.text());
+const feedUrls = discoverFeeds(html, 'https://example.com');
+// ['https://example.com/feed.xml', 'https://example.com/atom.xml']
+```
+
+### Filtering by Date
+
+```typescript
+import { RSSParser, filterByDate } from 'scrapex';
+
+const parser = new RSSParser();
+const result = parser.parse(feedXml);
+
+const recentItems = filterByDate(result.data.items, {
+  after: new Date('2024-01-01'),
+  before: new Date('2024-12-31'),
+  includeUndated: false,
+});
+```
+
+### Converting to Markdown/Text
+
+```typescript
+import { RSSParser, feedToMarkdown, feedToText } from 'scrapex';
+
+const parser = new RSSParser();
+const result = parser.parse(feedXml);
+
+// Convert to markdown (great for LLM consumption)
+const markdown = feedToMarkdown(result.data, { maxItems: 10 });
+
+// Convert to plain text
+const text = feedToText(result.data);
+```
+
+### Custom Fields (Podcast/Media)
+
+Extract custom namespace fields like iTunes podcast tags:
+
+```typescript
+const parser = new RSSParser({
+  customFields: {
+    duration: 'itunes\\:duration',
+    explicit: 'itunes\\:explicit',
+    rating: 'media\\:rating',
+  },
+});
+
+const result = parser.parse(podcastXml);
+const item = result.data.items[0];
+
+console.log(item.customFields?.duration);  // '10:00'
+console.log(item.customFields?.explicit);  // 'no'
+```
+
+### Security
+
+The RSS parser enforces strict URL security:
+
+- **HTTPS-only URLs (RSS parser only)**: The RSS/Atom parser (`RSSParser`) resolves all links to HTTPS only. Non-HTTPS URLs (http, javascript, data, file) are rejected and returned as empty strings. This is specific to feed parsing to prevent malicious links in untrusted feeds.
+- **XML Mode**: Feeds are parsed with Cheerio's `{ xml: true }` mode, which disables HTML entity processing and prevents XSS vectors.
+
+> **Note**: The public URL utilities (`resolveUrl`, `isValidUrl`, etc.) accept both `http:` and `https:` URLs. Protocol-relative URLs (e.g., `//example.com/path`) are resolved against the base URL's protocol by the standard `URL` constructor.
+
+## URL Utilities
+
+```typescript
+import {
+  isValidUrl,
+  normalizeUrl,
+  extractDomain,
+  resolveUrl,
+  isExternalUrl,
+} from 'scrapex';
+
+isValidUrl('https://example.com');
+// true
+
+normalizeUrl('https://example.com/page?utm_source=twitter');
+// 'https://example.com/page' (tracking params removed)
+
+extractDomain('https://www.example.com/path');
+// 'example.com'
+
+resolveUrl('/path', 'https://example.com/page');
+// 'https://example.com/path'
+
+isExternalUrl('https://other.com', 'example.com');
+// true
+```
+
+## Error Handling
+
+```typescript
+import { scrape, ScrapeError } from 'scrapex';
+
+try {
+  const result = await scrape('https://example.com');
+} catch (error) {
+  if (error instanceof ScrapeError) {
+    console.log(error.code);       // 'FETCH_FAILED' | 'TIMEOUT' | 'INVALID_URL' | ...
+    console.log(error.statusCode); // HTTP status if available
+    console.log(error.isRetryable()); // true for network errors
+  }
+}
+```
+
+Error codes:
+- `FETCH_FAILED` - Network request failed
+- `TIMEOUT` - Request timed out
+- `INVALID_URL` - URL is malformed
+- `BLOCKED` - Access denied (403)
+- `NOT_FOUND` - Page not found (404)
+- `ROBOTS_BLOCKED` - Blocked by robots.txt
+- `PARSE_ERROR` - HTML parsing failed
+- `LLM_ERROR` - LLM provider error
+- `VALIDATION_ERROR` - Schema validation failed
+
+## Robots.txt
+
+```typescript
+import { scrape, checkRobotsTxt } from 'scrapex';
+
+// Check before scraping
+const check = await checkRobotsTxt('https://example.com/path');
+if (check.allowed) {
+  const result = await scrape('https://example.com/path');
+}
+
+// Or let scrape() handle it
+const result = await scrape('https://example.com/path', {
+  respectRobots: true, // Throws if blocked
+});
+```
+
+## Built-in Extractors
+
+| Extractor | Priority | Description |
+|-----------|----------|-------------|
+| `MetaExtractor` | 100 | OG, Twitter, meta tags |
+| `JsonLdExtractor` | 80 | JSON-LD structured data |
+| `ContentExtractor` | 50 | Readability + Turndown |
+| `FaviconExtractor` | 40 | Favicon discovery |
+| `LinksExtractor` | 30 | Content link extraction |
+
+## Configuration
+
+### Options
+
+```typescript
+interface ScrapeOptions {
+  timeout?: number;              // Default: 10000ms
+  userAgent?: string;            // Custom user agent
+  extractContent?: boolean;      // Default: true
+  maxContentLength?: number;     // Default: 50000 chars
+  fetcher?: Fetcher;             // Custom fetcher
+  extractors?: Extractor[];      // Additional extractors
+  replaceDefaultExtractors?: boolean;
+  respectRobots?: boolean;       // Check robots.txt
+  llm?: LLMProvider;             // LLM provider
+  enhance?: EnhancementType[];   // LLM enhancements
+  extract?: ExtractionSchema;    // Structured extraction
+}
+```
+
+### Enhancement Types
+
+```typescript
+type EnhancementType =
+  | 'summarize'  // Generate summary
+  | 'tags'       // Extract keywords/tags
+  | 'entities'   // Extract named entities
+  | 'classify';  // Classify content type
+```
+
+## Requirements
+
+- Node.js 20+
+- TypeScript 5.0+ (for type imports)
 
 ## License
 
-This project is licensed under the MIT License.
+MIT
+
+## Author
+
+Rakesh Paul - [binaryroute](https://binaryroute.com/authors/rk-paul/)