scrapex 0.5.2 → 1.0.0-alpha.1

package/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2019 Rakesh Paul
+Copyright (c) 2025 Rakesh Paul
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

package/README.md

@@ -1,157 +1,404 @@
-# Overview
-
-ScrapeX provides a node library to scrape basic details of a url using the [metascraper](https://metascraper.js.org/#/), [firefox readability](https://github.com/mozilla/readability), [page-metadata-parser](https://github.com/mozilla/page-metadata-parser) libraries.
-
-## Getting Started
-
-### Install
-
-Install
-
-```javascript
-npm i scrapex --save or yarn add scrapex
-```
-
-### Usage
-
-```javascript
-import { scrape, scrapeHtml } from 'scrapex';
-// or
-const { scrape, scrapeHtml } = require('scrapex');
-// usage
-scrape(url, options)
-options = { metascraperRules?, timeout?, sanitizeOptions?}
-metascraperRules = optional array of 'audio'
-  | 'amazon'
-  | 'iframe'
-  | 'media-provider'
-  | 'soundcloud'
-  | 'uol'
-  | 'spotify'
-  | 'video'
-  | 'youtube'
-timeout: number = default to 60 seconds
-sanitizeOptions: sanitize-html options
-
-```
-
-```javascript
-// define a url
-const url = "https://appleinsider.com/articles/19/08/22/like-apple-music-spotify-now-offers-a-three-month-premium-trial"
-const data = await scrape(url, {metascraperRules:['youtube']})
-// or
-const html = `html string of the webpage extracted separately`
-const data = await scrape(url, html, {metascraperRules:['youtube']})
-
-// scrape result
-{
-  audio: undefined,
-  author: 'Amber Neely',
-  logo: 'https://logo.clearbit.com/appleinsider.com',
-  favicon: 'https://photos5.appleinsider.com/v9/images/apple-touch-icon-72.png',
-  image: 'https://photos5.appleinsider.com/gallery/32489-55660-header-xl.jpg',
-  publisher: 'AppleInsider',
-  date: '2019-08-22T12:14:35.000Z',
-  description: "Spotify has extended the free-trial period it offers for Spotify Premium from one month to three, the default length of Apple's free trial for Apple Music.",
-  lang: 'en',
-  url: 'https://appleinsider.com/articles/19/08/22/like-apple-music-spotify-now-offers-a-three-month-premium-trial',
-  text: "Spotify has extended the free-trial period it offers for Spotify Premium from one month to three, the default length of Apple's free trial for Apple Music.\n" +
-    'Streaming giant Spotify is now offering three free months to anyone who has yet to try their service, according to a news post on their site.\n' +
-    `"Beginning August 22, eligible users will receive the first three months on us for free when they sign up for any Spotify Premium plan," says Spotify in a statement about the new trial. "You'll unlock a world of on-demand access to millions of hours of audio content—  no matter when you sign up, winter, spring, summer, or fall."\n` +
-    "The trial period currently only extends to individual and student plans and will roll out across Duo and Family in the coming months. The trial doesn't extend to Headspace or anyone who is billed directly through their carrier, with the exception of those in Japan, Australia, China, and Germany. \n" +
-    "Apple has been offering free three-month trials to Apple Music since it's inception, though they may begin limiting their trial to one month. Apple had learned artists are wary of lengthy trial periods when Taylor Swift protested the three-month trial by withholding her album 1989 from the service. The protest earned artists the ability to be paid for track and album streams through the free trial period.\n" +
-    'Like most other paid music subscriptions, Spotify Premium offers users the ability to listen ad-free, download music to their device, create playlists, skip tracks, and toggle between devices when listening. ',
-  video: null,
-  keywords: [
-    'Apple',               'Apple Inc',
-    'iPhone',              'iPad',
-    'iPod touch',          'iPod nano',
-    'Apple TV',            'Apple',
-    'iPod shuffle',        'iphone 6',
-    'iphone 6s',           'ios 9',
-    'ios9',                'iTunes',
-    'i mac',               'mac os x',
-    'mac osx',             'Apple Computer',
-    'Apple Computer Inc.', 'Mac OS X',
-    'iMac',                'iBook',
-    'Mac Pro',             'MacBook Pro',
-    'Magic Pad',           'Magic Mouse',
-    'iPod classic',        'App Store',
-    'iTunes Store',        'iBook Store',
-    'mac book',            'Microsoft',
-    'Adobe',               'Research in Motion',
-    'RIM',                 'Nokia',
-    'Samsung',             'Google',
-    'Nvidia',              'Intel'
-  ],
-  tags: [],
-  embeds: [],
-  source: 'appleinsider.com',
-  twitter: {
-    site: '@appleinsider',
-    creator: '@appleinsider',
-    description: "Spotify has extended the free-trial period it offers for Spotify Premium from one month to three, the default length of Apple's free trial for Apple Music.",
-    title: 'Like Apple Music, Spotify now offers a three month premium trial | AppleInsider',
-    image: 'https://photos5.appleinsider.com/gallery/32489-55660-header-xl.jpg'
+# scrapex
+
+Modern web scraper with LLM-enhanced extraction, extensible pipeline, and pluggable parsers.
+
+> **Alpha Release**: v1.0.0 is currently in alpha. The API may change before the stable release.
+
+## Features
+
+- **LLM-Ready Output** - Content extracted as Markdown, optimized for AI/LLM consumption
+- **Provider-Agnostic LLM** - Works with OpenAI, Anthropic, Ollama, LM Studio, or any OpenAI-compatible API
+- **Extensible Pipeline** - Pluggable extractors with priority-based execution
+- **Smart Extraction** - Uses Mozilla Readability for content, Cheerio for metadata
+- **Markdown Parsing** - Parse markdown content, awesome lists, and GitHub repos
+- **TypeScript First** - Full type safety with comprehensive type exports
+- **Dual Format** - ESM and CommonJS builds
+
+## Installation
+
+```bash
+npm install scrapex@alpha
+```
+
+### Optional Peer Dependencies
+
+```bash
+# For LLM features
+npm install openai           # OpenAI/Ollama/LM Studio
+npm install @anthropic-ai/sdk  # Anthropic Claude
+
+# For JavaScript-rendered pages
+npm install puppeteer
+```
+
+## Quick Start
+
+```typescript
+import { scrape } from 'scrapex';
+
+const result = await scrape('https://example.com/article');
+
+console.log(result.title);       // "Article Title"
+console.log(result.content);     // Markdown content
+console.log(result.textContent); // Plain text (lower tokens)
+console.log(result.excerpt);     // First ~300 chars
+```
+
+## API Reference
+
+### `scrape(url, options?)`
+
+Fetch and extract metadata and content from a URL.
+
+```typescript
+import { scrape } from 'scrapex';
+
+const result = await scrape('https://example.com', {
+  timeout: 10000,
+  userAgent: 'MyBot/1.0',
+  extractContent: true,
+  maxContentLength: 50000,
+  respectRobots: false,
+});
+```
+
+### `scrapeHtml(html, url, options?)`
+
+Extract from raw HTML without fetching.
+
+```typescript
+import { scrapeHtml } from 'scrapex';
+
+const html = await fetchSomehow('https://example.com');
+const result = await scrapeHtml(html, 'https://example.com');
+```
+
+### Result Object (`ScrapedData`)
+
+```typescript
+interface ScrapedData {
+  // Identity
+  url: string;
+  canonicalUrl: string;
+  domain: string;
+
+  // Basic metadata
+  title: string;
+  description: string;
+  image?: string;
+  favicon?: string;
+
+  // Content (LLM-optimized)
+  content: string;      // Markdown format
+  textContent: string;  // Plain text
+  excerpt: string;      // ~300 char preview
+  wordCount: number;
+
+  // Context
+  author?: string;
+  publishedAt?: string;
+  modifiedAt?: string;
+  siteName?: string;
+  language?: string;
+
+  // Classification
+  contentType: 'article' | 'repo' | 'docs' | 'package' | 'video' | 'tool' | 'product' | 'unknown';
+  keywords: string[];
+
+  // Structured data
+  jsonLd?: Record<string, unknown>[];
+  links?: ExtractedLink[];
+
+  // LLM Enhancements (when enabled)
+  summary?: string;
+  suggestedTags?: string[];
+  entities?: ExtractedEntities;
+  extracted?: Record<string, unknown>;
+
+  // Meta
+  scrapedAt: string;
+  scrapeTimeMs: number;
+  error?: string;
+}
+```
+
+## LLM Integration
+
+### Using OpenAI
+
+```typescript
+import { scrape } from 'scrapex';
+import { createOpenAI } from 'scrapex/llm';
+
+const llm = createOpenAI({ apiKey: 'sk-...' });
+
+const result = await scrape('https://example.com/article', {
+  llm,
+  enhance: ['summarize', 'tags', 'entities', 'classify'],
+});
+
+console.log(result.summary);       // AI-generated summary
+console.log(result.suggestedTags); // ['javascript', 'web', ...]
+console.log(result.entities);      // { people: [], organizations: [], ... }
+```
+
+### Using Anthropic Claude
+
+```typescript
+import { AnthropicProvider } from 'scrapex/llm';
+
+const llm = new AnthropicProvider({
+  apiKey: process.env.ANTHROPIC_API_KEY,
+  model: 'claude-3-5-haiku-20241022', // or 'claude-sonnet-4-20250514'
+});
+
+const result = await scrape(url, { llm, enhance: ['summarize'] });
+```
+
+### Using Ollama (Local)
+
+```typescript
+import { createOllama } from 'scrapex/llm';
+
+const llm = createOllama({ model: 'llama3.2' });
+
+const result = await scrape(url, { llm, enhance: ['summarize'] });
+```
+
+### Using LM Studio (Local)
+
+```typescript
+import { createLMStudio } from 'scrapex/llm';
+
+const llm = createLMStudio({ model: 'local-model' });
+
+const result = await scrape(url, { llm, enhance: ['summarize'] });
+```
+
+### Structured Extraction
+
+Extract specific data using a schema:
+
+```typescript
+const result = await scrape('https://example.com/product', {
+  llm,
+  extract: {
+    productName: 'string',
+    price: 'number',
+    features: 'string[]',
+    inStock: 'boolean',
+    sku: 'string?', // optional
   },
-  title: 'Like Apple Music, Spotify now offers a three month premium trial',
-  links: [
-    {
-      href: 'https://newsroom.spotify.com/2019-08-22/5-ways-to-take-control-of-your-streaming-with-spotify-premium/',
-      text: 'a news post on their site.'
-    },
-    {
-      href: 'https://appleinsider.com/inside/apple-music',
-      text: 'Apple Music'
-    },
-    {
-      href: 'https://appleinsider.com/articles/19/07/25/apple-begins-limiting-apple-music-free-trial-period-to-one-month',
-      text: 'one month.'
-    },
-    {
-      href: 'https://appleinsider.com/articles/15/06/18/apple-music-to-miss-out-on-taylor-swifts-1989-album',
-      text: 'Taylor Swift protested the three-month trial'
-    }
-  ],
-  content:
-  '<div><div><div><div> <p>\n\t\t\tBy <a href="undefined/cdn-cgi/l/email-protection#d6b7bbb4b3a496b7a6a6bab3bfb8a5bfb2b3a4f8b5b9bb">Amber Neely</a>\t\t\t<br />\n\t\t\tThursday, August 22, 2019, 05:14 am PT (08:14 am ET)\n\t\t</p>Spotify has extended the free-trial period it offers for Spotify Premium from one month to three, the default length of Apple\'s free trial for Apple Music.<br /><p>\nStreaming giant Spotify is now offering three free months to anyone who has yet to try their service, according to <a href="https://newsroom.spotify.com/2019-08-22/5-ways-to-take-control-of-your-streaming-with-spotify-premium/">a news post on their site.</a></p><p>\n"Beginning August 22, eligible users will receive the first three months on us for free when they sign up for any Spotify Premium plan," says Spotify in a statement about the new trial. "You\'ll unlock a world of on-demand access to millions of hours of audio content—no matter when you sign up, winter, spring, summer, or fall."</p><p>\nThe trial period currently only extends to individual and student plans and will roll out across Duo and Family in the coming months. The trial doesn\'t extend to Headspace or anyone who is billed directly through their carrier, with the exception of those in Japan, Australia, China, and Germany. </p><p>\nApple has been offering free three-month trials to Apple Music since it\'s inception, though they may begin limiting their trial to <a href="https://appleinsider.com/articles/19/07/25/apple-begins-limiting-apple-music-free-trial-period-to-one-month">one month.</a> Apple had learned artists are wary of lengthy trial periods when <a href="https://appleinsider.com/articles/15/06/18/apple-music-to-miss-out-on-taylor-swifts-1989-album">Taylor Swift protested the three-month trial</a> by withholding her album <em>1989</em> from the service. The protest earned artists the ability to be paid for track and album streams through the free trial period.</p><p>\nStudents who sign up for Apple Music can get a free six-month trial <a href="https://support.apple.com/en-ke/HT205928">by visiting Apple\'s Support Page.</a> After the trial ends, students pay $4.99 a month to continue their subscription until graduation, which works out to be <a href="https://appleinsider.com/articles/16/05/06/apple-begins-offering-half-price-499-apple-music-subscriptions-for-students">about half the price of a standard subscription.</a></p><p>\nLike most other paid music subscriptions, Spotify Premium offers users the ability to listen ad-free, download music to their device, create playlists, skip tracks, and toggle between devices when listening. </p></div></div></div></div>',
-  html: ...
+});
+
+console.log(result.extracted);
+// { productName: "Widget", price: 29.99, features: [...], inStock: true }
+```
+
+## Custom Extractors
+
+Create custom extractors to add domain-specific extraction logic:
+
+```typescript
+import { scrape, type Extractor, type ExtractionContext } from 'scrapex';
+
+const recipeExtractor: Extractor = {
+  name: 'recipe',
+  priority: 60, // Higher = runs earlier
+
+  async extract(context: ExtractionContext) {
+    const { $ } = context;
+
+    return {
+      custom: {
+        ingredients: $('.ingredients li').map((_, el) => $(el).text()).get(),
+        cookTime: $('[itemprop="cookTime"]').attr('content'),
+        servings: $('[itemprop="recipeYield"]').text(),
+      },
+    };
+  },
+};
+
+const result = await scrape('https://example.com/recipe', {
+  extractors: [recipeExtractor],
+});
+
+console.log(result.custom?.ingredients);
+```
+
+### Replacing Default Extractors
+
+```typescript
+const result = await scrape(url, {
+  replaceDefaultExtractors: true,
+  extractors: [myCustomExtractor],
+});
+```
+
+## Markdown Parsing
+
+Parse markdown content:
+
+```typescript
+import { MarkdownParser, extractListLinks, groupByCategory } from 'scrapex/parsers';
+
+// Parse any markdown
+const parser = new MarkdownParser();
+const result = parser.parse(markdownContent);
+
+console.log(result.data.title);
+console.log(result.data.sections);
+console.log(result.data.links);
+console.log(result.data.codeBlocks);
+
+// Extract links from markdown lists and group by category
+const links = extractListLinks(markdownContent);
+const grouped = groupByCategory(links);
+
+grouped.forEach((categoryLinks, category) => {
+  console.log(`${category}: ${categoryLinks.length} links`);
+});
+```
+
+### GitHub Utilities
+
+```typescript
+import {
+  isGitHubRepo,
+  parseGitHubUrl,
+  toRawUrl,
+} from 'scrapex/parsers';
+
+isGitHubRepo('https://github.com/owner/repo');
+// true
+
+parseGitHubUrl('https://github.com/facebook/react');
+// { owner: 'facebook', repo: 'react' }
+
+toRawUrl('https://github.com/owner/repo');
+// 'https://raw.githubusercontent.com/owner/repo/main/README.md'
+```
+
+## URL Utilities
+
+```typescript
+import {
+  isValidUrl,
+  normalizeUrl,
+  extractDomain,
+  resolveUrl,
+  isExternalUrl,
+} from 'scrapex';
+
+isValidUrl('https://example.com');
+// true
+
+normalizeUrl('https://example.com/page?utm_source=twitter');
+// 'https://example.com/page' (tracking params removed)
+
+extractDomain('https://www.example.com/path');
+// 'example.com'
+
+resolveUrl('/path', 'https://example.com/page');
+// 'https://example.com/path'
+
+isExternalUrl('https://other.com', 'example.com');
+// true
+```
+
+## Error Handling
+
+```typescript
+import { scrape, ScrapeError } from 'scrapex';
+
+try {
+  const result = await scrape('https://example.com');
+} catch (error) {
+  if (error instanceof ScrapeError) {
+    console.log(error.code);       // 'FETCH_FAILED' | 'TIMEOUT' | 'INVALID_URL' | ...
+    console.log(error.statusCode); // HTTP status if available
+    console.log(error.isRetryable()); // true for network errors
+  }
 }
 ```
 
-### Extracted data elements
+Error codes:
+- `FETCH_FAILED` - Network request failed
+- `TIMEOUT` - Request timed out
+- `INVALID_URL` - URL is malformed
+- `BLOCKED` - Access denied (403)
+- `NOT_FOUND` - Page not found (404)
+- `ROBOTS_BLOCKED` - Blocked by robots.txt
+- `PARSE_ERROR` - HTML parsing failed
+- `LLM_ERROR` - LLM provider error
+- `VALIDATION_ERROR` - Schema validation failed
 
-This is what `scrapex` will try to grab from a web page:
+## Robots.txt
 
-- `audio` — eg. <https://cf-media.sndcdn.com/U78RIfDPV6ok.128.mp3>. A audio URL that best represents the article.
-- `title` - The document's title (from the &lt;title&gt; tag)
-- `date` - The document's publication date
-- `copyright` - The document's copyright line, if present
-- `author` - The document's author
-- `publisher` - The document's publisher (website name)
-- `text` - The main text of the document with all the junk thrown away
-- `image` - The main image for the document (what's used by facebook, etc.)
-- `video` - A video URL that best represents the article.
-- `embeds` - An array of iframe, embed, object, video that were embedded in the article.
-- `tags`- Any tags or keywords that could be found by checking href urls at has the following pattern `a[href*='/t/'],a[href*='/tag/'], a[href*='/tags/'], a[href*='/topic/'],a[href*='/tagged/'], a[href*='?keyword=']`.
-- `keywords`- Any keywords that could be found by checking &lt;rel&gt; tags or by looking at href urls.
-- `lang` - The language of the document, either detected or supplied by you.
-- `description` - The description of the document, from &lt;meta&gt; tags
-- `favicon` - The url of the document's [favicon](http://en.wikipedia.org/wiki/Favicon).
-- `links` - An array of links embedded within the main article text. (text and href for each)
-- `logo` — eg. <https://entrepreneur.com/favicon180x180.png>. An image URL that best represents the publisher brand.
-- `content` — readability view html of the article.
-- `html` — full html of the page.
-- `text` — clear text of the readable html.
-- `code` - code segments defined using pre > code tags
+```typescript
+import { scrape, checkRobotsTxt } from 'scrapex';
 
-## Todo
+// Check before scraping
+const check = await checkRobotsTxt('https://example.com/path');
+if (check.allowed) {
+  const result = await scrape('https://example.com/path');
+}
 
-## Contributors
+// Or let scrape() handle it
+const result = await scrape('https://example.com/path', {
+  respectRobots: true, // Throws if blocked
+});
+```
+
+## Built-in Extractors
+
+| Extractor | Priority | Description |
+|-----------|----------|-------------|
+| `MetaExtractor` | 100 | OG, Twitter, meta tags |
+| `JsonLdExtractor` | 80 | JSON-LD structured data |
+| `ContentExtractor` | 50 | Readability + Turndown |
+| `FaviconExtractor` | 40 | Favicon discovery |
+| `LinksExtractor` | 30 | Content link extraction |
+
+## Configuration
+
+### Options
+
+```typescript
+interface ScrapeOptions {
+  timeout?: number;              // Default: 10000ms
+  userAgent?: string;            // Custom user agent
+  extractContent?: boolean;      // Default: true
+  maxContentLength?: number;     // Default: 50000 chars
+  fetcher?: Fetcher;             // Custom fetcher
+  extractors?: Extractor[];      // Additional extractors
+  replaceDefaultExtractors?: boolean;
+  respectRobots?: boolean;       // Check robots.txt
+  llm?: LLMProvider;             // LLM provider
+  enhance?: EnhancementType[];   // LLM enhancements
+  extract?: ExtractionSchema;    // Structured extraction
+}
+```
+
+### Enhancement Types
+
+```typescript
+type EnhancementType =
+  | 'summarize'  // Generate summary
+  | 'tags'       // Extract keywords/tags
+  | 'entities'   // Extract named entities
+  | 'classify';  // Classify content type
+```
 
-<img width=150px src="https://pbs.twimg.com/profile_images/1028292150205661185/TFP8E8Fc_400x400.jpg">
-<p><strong>Rakesh Paul</strong> - <a href="https://xtrios.com">Xtrios</a></p>
+## Requirements
+
+- Node.js 20+
+- TypeScript 5.0+ (for type imports)
 
 ## License
 
-This project is licensed under the MIT License.
+MIT
+
+## Author
+
+Rakesh Paul - [binaryroute](https://binaryroute.com/authors/rk-paul/)