scrapex 1.0.0-alpha.1 → 1.0.0-beta.2

package/README.md

@@ -2,22 +2,24 @@
 
 Modern web scraper with LLM-enhanced extraction, extensible pipeline, and pluggable parsers.
 
-> **Alpha Release**: v1.0.0 is currently in alpha. The API may change before the stable release.
+> **Beta Release**: v1.0.0 is currently in beta. The API is stable but minor changes may occur before the stable release.
 
 ## Features
 
 - **LLM-Ready Output** - Content extracted as Markdown, optimized for AI/LLM consumption
 - **Provider-Agnostic LLM** - Works with OpenAI, Anthropic, Ollama, LM Studio, or any OpenAI-compatible API
+- **Vector Embeddings** - Generate embeddings with OpenAI, Azure, Cohere, HuggingFace, Ollama, or local Transformers.js
 - **Extensible Pipeline** - Pluggable extractors with priority-based execution
 - **Smart Extraction** - Uses Mozilla Readability for content, Cheerio for metadata
 - **Markdown Parsing** - Parse markdown content, awesome lists, and GitHub repos
+- **RSS/Atom Feeds** - Parse RSS 2.0, RSS 1.0 (RDF), and Atom feeds with pagination support
 - **TypeScript First** - Full type safety with comprehensive type exports
 - **Dual Format** - ESM and CommonJS builds
 
 ## Installation
 
 ```bash
-npm install scrapex@alpha
+npm install scrapex@beta
 ```
 
 ### Optional Peer Dependencies
@@ -142,14 +144,48 @@ console.log(result.suggestedTags); // ['javascript', 'web', ...]
 console.log(result.entities);      // { people: [], organizations: [], ... }
 ```
 
+### Embeddings
+
+Generate vector embeddings from scraped content for semantic search, RAG, and similarity matching:
+
+```typescript
+import { scrape } from 'scrapex';
+import { createOpenAIEmbedding } from 'scrapex/embeddings';
+
+const result = await scrape('https://example.com/article', {
+  embeddings: {
+    provider: { type: 'custom', provider: createOpenAIEmbedding() },
+    model: 'text-embedding-3-small',
+  },
+});
+
+if (result.embeddings?.status === 'success') {
+  console.log(result.embeddings.vector); // [0.023, -0.041, ...]
+}
+```
+
+Features include:
+- **Multiple providers** - OpenAI, Azure, Cohere, HuggingFace, Ollama, Transformers.js
+- **PII redaction** - Automatically redact emails, phones, SSNs before sending to APIs
+- **Smart chunking** - Split long content with configurable overlap
+- **Caching** - Content-addressable cache to avoid redundant API calls
+- **Resilience** - Retry, circuit breaker, rate limiting
+
+See the [Embeddings Guide](https://scrapex.dev/guides/embeddings) for full documentation.
+
+## Breaking Changes (Beta)
+
+- LLM provider classes (e.g., `AnthropicProvider`) were removed. Use preset factories like
+  `createOpenAI`, `createAnthropic`, `createOllama`, and `createLMStudio` instead.
+
 ### Using Anthropic Claude
 
 ```typescript
-import { AnthropicProvider } from 'scrapex/llm';
+import { createAnthropic } from 'scrapex/llm';
 
-const llm = new AnthropicProvider({
+const llm = createAnthropic({
   apiKey: process.env.ANTHROPIC_API_KEY,
-  model: 'claude-3-5-haiku-20241022', // or 'claude-sonnet-4-20250514'
+  model: 'claude-3-5-haiku-20241022', // or 'claude-3-5-sonnet-20241022'
 });
 
 const result = await scrape(url, { llm, enhance: ['summarize'] });
@@ -279,6 +315,129 @@ toRawUrl('https://github.com/owner/repo');
 // 'https://raw.githubusercontent.com/owner/repo/main/README.md'
 ```
 
+## RSS/Atom Feed Parsing
+
+Parse RSS 2.0, RSS 1.0 (RDF), and Atom 1.0 feeds:
+
+```typescript
+import { RSSParser } from 'scrapex';
+
+const parser = new RSSParser();
+const result = parser.parse(feedXml, 'https://example.com/feed.xml');
+
+console.log(result.data.format);  // 'rss2' | 'rss1' | 'atom'
+console.log(result.data.title);   // Feed title
+console.log(result.data.items);   // Array of feed items
+```
+
+**Supported formats:**
+- `rss2` - RSS 2.0 (most common format)
+- `rss1` - RSS 1.0 (RDF-based, older format)
+- `atom` - Atom 1.0 (modern format with better semantics)
+
+### Feed Item Structure
+
+```typescript
+interface FeedItem {
+  id: string;
+  title: string;
+  link: string;
+  description?: string;
+  content?: string;
+  author?: string;
+  publishedAt?: string;      // ISO 8601
+  rawPublishedAt?: string;   // Original date string
+  updatedAt?: string;        // Atom only
+  categories: string[];
+  enclosure?: FeedEnclosure; // Podcast/media attachments
+  customFields?: Record<string, string>;
+}
+```
+
+### Fetching and Parsing Feeds
+
+```typescript
+import { fetchFeed, paginateFeed } from 'scrapex';
+
+// Fetch and parse in one call
+const result = await fetchFeed('https://example.com/feed.xml');
+console.log(result.data.items);
+
+// Paginate through feeds with rel="next" links (Atom)
+for await (const page of paginateFeed('https://example.com/atom')) {
+  console.log(`Page with ${page.data.items.length} items`);
+}
+```
+
+### Discovering Feeds in HTML
+
+```typescript
+import { discoverFeeds } from 'scrapex';
+
+const html = await fetch('https://example.com').then(r => r.text());
+const feedUrls = discoverFeeds(html, 'https://example.com');
+// ['https://example.com/feed.xml', 'https://example.com/atom.xml']
+```
+
+### Filtering by Date
+
+```typescript
+import { RSSParser, filterByDate } from 'scrapex';
+
+const parser = new RSSParser();
+const result = parser.parse(feedXml);
+
+const recentItems = filterByDate(result.data.items, {
+  after: new Date('2024-01-01'),
+  before: new Date('2024-12-31'),
+  includeUndated: false,
+});
+```
+
+### Converting to Markdown/Text
+
+```typescript
+import { RSSParser, feedToMarkdown, feedToText } from 'scrapex';
+
+const parser = new RSSParser();
+const result = parser.parse(feedXml);
+
+// Convert to markdown (great for LLM consumption)
+const markdown = feedToMarkdown(result.data, { maxItems: 10 });
+
+// Convert to plain text
+const text = feedToText(result.data);
+```
+
+### Custom Fields (Podcast/Media)
+
+Extract custom namespace fields like iTunes podcast tags:
+
+```typescript
+const parser = new RSSParser({
+  customFields: {
+    duration: 'itunes\\:duration',
+    explicit: 'itunes\\:explicit',
+    rating: 'media\\:rating',
+  },
+});
+
+const result = parser.parse(podcastXml);
+const item = result.data.items[0];
+
+console.log(item.customFields?.duration);  // '10:00'
+console.log(item.customFields?.explicit);  // 'no'
+```
+
+### Security
+
+The RSS parser enforces strict URL security:
+
+- **HTTPS-only URLs (RSS parser only)**: The RSS/Atom parser (`RSSParser`) resolves all links to HTTPS only. Non-HTTPS URLs (http, javascript, data, file) are rejected and returned as empty strings. This is specific to feed parsing to prevent malicious links in untrusted feeds.
+- **XML Mode**: Feeds are parsed with Cheerio's `{ xml: true }` mode, which disables HTML entity processing and prevents XSS vectors.
+
+> **Note**: The public URL utilities (`resolveUrl`, `isValidUrl`, etc.) accept both `http:` and `https:` URLs. Protocol-relative URLs (e.g., `//example.com/path`) are resolved against the base URL's protocol by the standard `URL` constructor.
+
 ## URL Utilities
 
 ```typescript

package/dist/embeddings/index.cjs

@@ -0,0 +1,52 @@
+const require_http_base = require('../http-base-CHLf-Tco.cjs');
+const require_embeddings = require('../embeddings-BjNTQSG9.cjs');
+
+exports.CircuitBreaker = require_http_base.CircuitBreaker;
+exports.CircuitOpenError = require_http_base.CircuitOpenError;
+exports.HttpEmbeddingProvider = require_embeddings.HttpEmbeddingProvider;
+exports.InMemoryEmbeddingCache = require_embeddings.InMemoryEmbeddingCache;
+exports.RateLimiter = require_http_base.RateLimiter;
+exports.Semaphore = require_http_base.Semaphore;
+exports.TRANSFORMERS_MODELS = require_embeddings.TRANSFORMERS_MODELS;
+exports.aggregateVectors = require_embeddings.aggregateVectors;
+exports.chunkText = require_embeddings.chunkText;
+exports.containsPii = require_embeddings.containsPii;
+exports.cosineSimilarity = require_embeddings.cosineSimilarity;
+exports.createAzureEmbedding = require_embeddings.createAzureEmbedding;
+exports.createCohereEmbedding = require_embeddings.createCohereEmbedding;
+exports.createEmbeddingProvider = require_embeddings.createEmbeddingProvider;
+exports.createHttpEmbedding = require_embeddings.createHttpEmbedding;
+exports.createHuggingFaceEmbedding = require_embeddings.createHuggingFaceEmbedding;
+exports.createNoOpCache = require_embeddings.createNoOpCache;
+exports.createOllamaEmbedding = require_embeddings.createOllamaEmbedding;
+exports.createOpenAIEmbedding = require_embeddings.createOpenAIEmbedding;
+exports.createPiiRedactor = require_embeddings.createPiiRedactor;
+exports.createTimeoutSignal = require_http_base.createTimeoutSignal;
+exports.createTokenizer = require_embeddings.createTokenizer;
+exports.createTransformersEmbedding = require_embeddings.createTransformersEmbedding;
+exports.dotProduct = require_embeddings.dotProduct;
+exports.embed = require_embeddings.embed;
+exports.embedScrapedData = require_embeddings.embedScrapedData;
+exports.estimateTokens = require_embeddings.estimateTokens;
+exports.euclideanDistance = require_embeddings.euclideanDistance;
+exports.generateCacheKey = require_embeddings.generateCacheKey;
+exports.generateChecksum = require_embeddings.generateChecksum;
+exports.generateEmbeddings = require_embeddings.generateEmbeddings;
+exports.getChunkingStats = require_embeddings.getChunkingStats;
+exports.getDefaultCache = require_embeddings.getDefaultCache;
+exports.getDefaultModel = require_embeddings.getDefaultModel;
+exports.getDimensions = require_embeddings.getDimensions;
+exports.heuristicTokenCount = require_embeddings.heuristicTokenCount;
+exports.isEmbeddingProvider = require_embeddings.isEmbeddingProvider;
+exports.isRetryableError = require_http_base.isRetryableError;
+exports.needsChunking = require_embeddings.needsChunking;
+exports.normalizeVector = require_embeddings.normalizeVector;
+exports.previewInput = require_embeddings.previewInput;
+exports.redactPii = require_embeddings.redactPii;
+exports.resetDefaultCache = require_embeddings.resetDefaultCache;
+exports.selectInput = require_embeddings.selectInput;
+exports.validateCachedResult = require_embeddings.validateCachedResult;
+exports.validateInput = require_embeddings.validateInput;
+exports.withResilience = require_http_base.withResilience;
+exports.withRetry = require_http_base.withRetry;
+exports.withTimeout = require_http_base.withTimeout;

package/dist/embeddings/index.d.cts

@@ -0,0 +1,3 @@
+import { A as HttpEmbeddingProvider, B as withResilience, C as RateLimitConfig, D as SafetyConfig, E as RetryConfig, F as CircuitOpenError, H as withTimeout, I as RateLimiter, L as Semaphore, O as TextChunk, P as CircuitBreaker, R as createTimeoutSignal, S as PiiRedactionConfig, T as ResilienceState, V as withRetry, _ as EmbeddingSkipped, a as EmbedRequest, b as EmbeddingSuccessSingle, c as EmbeddingCache, d as EmbeddingInputType, f as EmbeddingMetrics, g as EmbeddingResult, h as EmbeddingProviderConfig, i as CircuitState, j as createHttpEmbedding, k as HttpEmbeddingConfig, l as EmbeddingCacheConfig, m as EmbeddingProvider, n as CircuitBreakerConfig, o as EmbedResponse, p as EmbeddingOptions, r as CircuitBreakerState, s as EmbeddingAggregation, t as ChunkingConfig, u as EmbeddingInputConfig, v as EmbeddingSource, w as ResilienceConfig, x as OutputConfig, y as EmbeddingSuccessMultiple, z as isRetryableError } from "../types-DutdBpqd.cjs";
+import { A as createNoOpCache, B as euclideanDistance, C as createTokenizer, D as needsChunking, E as heuristicTokenCount, F as validateCachedResult, H as normalizeVector, I as AggregationResult, L as aggregateVectors, M as generateChecksum, N as getDefaultCache, O as CacheStats, P as resetDefaultCache, R as cosineSimilarity, S as chunkText, T as getChunkingStats, V as getDimensions, _ as generateEmbeddings, a as createEmbeddingProvider, b as selectInput, c as createAzureEmbedding, d as createOllamaEmbedding, f as createOpenAIEmbedding, g as embedScrapedData, h as embed, i as redactPii, j as generateCacheKey, k as InMemoryEmbeddingCache, l as createCohereEmbedding, m as getDefaultModel, n as containsPii, o as isEmbeddingProvider, p as createTransformersEmbedding, r as createPiiRedactor, s as TRANSFORMERS_MODELS, t as RedactionResult, u as createHuggingFaceEmbedding, v as InputValidation, w as estimateTokens, x as validateInput, y as previewInput, z as dotProduct } from "../index-RFSpP5g8.cjs";
+export { AggregationResult, CacheStats, ChunkingConfig, CircuitBreaker, CircuitBreakerConfig, CircuitBreakerState, CircuitOpenError, CircuitState, EmbedRequest, EmbedResponse, EmbeddingAggregation, EmbeddingCache, EmbeddingCacheConfig, EmbeddingInputConfig, EmbeddingInputType, EmbeddingMetrics, EmbeddingOptions, EmbeddingProvider, EmbeddingProviderConfig, EmbeddingResult, EmbeddingSkipped, EmbeddingSource, EmbeddingSuccessMultiple, EmbeddingSuccessSingle, HttpEmbeddingConfig, HttpEmbeddingProvider, InMemoryEmbeddingCache, InputValidation, OutputConfig, PiiRedactionConfig, RateLimitConfig, RateLimiter, RedactionResult, ResilienceConfig, ResilienceState, RetryConfig, SafetyConfig, Semaphore, TRANSFORMERS_MODELS, TextChunk, aggregateVectors, chunkText, containsPii, cosineSimilarity, createAzureEmbedding, createCohereEmbedding, createEmbeddingProvider, createHttpEmbedding, createHuggingFaceEmbedding, createNoOpCache, createOllamaEmbedding, createOpenAIEmbedding, createPiiRedactor, createTimeoutSignal, createTokenizer, createTransformersEmbedding, dotProduct, embed, embedScrapedData, estimateTokens, euclideanDistance, generateCacheKey, generateChecksum, generateEmbeddings, getChunkingStats, getDefaultCache, getDefaultModel, getDimensions, heuristicTokenCount, isEmbeddingProvider, isRetryableError, needsChunking, normalizeVector, previewInput, redactPii, resetDefaultCache, selectInput, validateCachedResult, validateInput, withResilience, withRetry, withTimeout };

package/dist/embeddings/index.d.mts

@@ -0,0 +1,3 @@
+import { A as HttpEmbeddingProvider, B as withResilience, C as RateLimitConfig, D as SafetyConfig, E as RetryConfig, F as CircuitOpenError, H as withTimeout, I as RateLimiter, L as Semaphore, O as TextChunk, P as CircuitBreaker, R as createTimeoutSignal, S as PiiRedactionConfig, T as ResilienceState, V as withRetry, _ as EmbeddingSkipped, a as EmbedRequest, b as EmbeddingSuccessSingle, c as EmbeddingCache, d as EmbeddingInputType, f as EmbeddingMetrics, g as EmbeddingResult, h as EmbeddingProviderConfig, i as CircuitState, j as createHttpEmbedding, k as HttpEmbeddingConfig, l as EmbeddingCacheConfig, m as EmbeddingProvider, n as CircuitBreakerConfig, o as EmbedResponse, p as EmbeddingOptions, r as CircuitBreakerState, s as EmbeddingAggregation, t as ChunkingConfig, u as EmbeddingInputConfig, v as EmbeddingSource, w as ResilienceConfig, x as OutputConfig, y as EmbeddingSuccessMultiple, z as isRetryableError } from "../types-BOcHQU9s.mjs";
+import { A as createNoOpCache, B as euclideanDistance, C as createTokenizer, D as needsChunking, E as heuristicTokenCount, F as validateCachedResult, H as normalizeVector, I as AggregationResult, L as aggregateVectors, M as generateChecksum, N as getDefaultCache, O as CacheStats, P as resetDefaultCache, R as cosineSimilarity, S as chunkText, T as getChunkingStats, V as getDimensions, _ as generateEmbeddings, a as createEmbeddingProvider, b as selectInput, c as createAzureEmbedding, d as createOllamaEmbedding, f as createOpenAIEmbedding, g as embedScrapedData, h as embed, i as redactPii, j as generateCacheKey, k as InMemoryEmbeddingCache, l as createCohereEmbedding, m as getDefaultModel, n as containsPii, o as isEmbeddingProvider, p as createTransformersEmbedding, r as createPiiRedactor, s as TRANSFORMERS_MODELS, t as RedactionResult, u as createHuggingFaceEmbedding, v as InputValidation, w as estimateTokens, x as validateInput, y as previewInput, z as dotProduct } from "../index-D6qfjmZQ.mjs";
+export { AggregationResult, CacheStats, ChunkingConfig, CircuitBreaker, CircuitBreakerConfig, CircuitBreakerState, CircuitOpenError, CircuitState, EmbedRequest, EmbedResponse, EmbeddingAggregation, EmbeddingCache, EmbeddingCacheConfig, EmbeddingInputConfig, EmbeddingInputType, EmbeddingMetrics, EmbeddingOptions, EmbeddingProvider, EmbeddingProviderConfig, EmbeddingResult, EmbeddingSkipped, EmbeddingSource, EmbeddingSuccessMultiple, EmbeddingSuccessSingle, HttpEmbeddingConfig, HttpEmbeddingProvider, InMemoryEmbeddingCache, InputValidation, OutputConfig, PiiRedactionConfig, RateLimitConfig, RateLimiter, RedactionResult, ResilienceConfig, ResilienceState, RetryConfig, SafetyConfig, Semaphore, TRANSFORMERS_MODELS, TextChunk, aggregateVectors, chunkText, containsPii, cosineSimilarity, createAzureEmbedding, createCohereEmbedding, createEmbeddingProvider, createHttpEmbedding, createHuggingFaceEmbedding, createNoOpCache, createOllamaEmbedding, createOpenAIEmbedding, createPiiRedactor, createTimeoutSignal, createTokenizer, createTransformersEmbedding, dotProduct, embed, embedScrapedData, estimateTokens, euclideanDistance, generateCacheKey, generateChecksum, generateEmbeddings, getChunkingStats, getDefaultCache, getDefaultModel, getDimensions, heuristicTokenCount, isEmbeddingProvider, isRetryableError, needsChunking, normalizeVector, previewInput, redactPii, resetDefaultCache, selectInput, validateCachedResult, validateInput, withResilience, withRetry, withTimeout };

package/dist/embeddings/index.mjs

@@ -0,0 +1,4 @@
+import { a as Semaphore, c as withResilience, i as RateLimiter, l as withRetry, n as CircuitBreaker, o as createTimeoutSignal, r as CircuitOpenError, s as isRetryableError, u as withTimeout } from "../http-base-DM7YNo6X.mjs";
+import { A as generateCacheKey, B as normalizeVector, C as createTokenizer, D as needsChunking, E as heuristicTokenCount, F as aggregateVectors, I as cosineSimilarity, L as dotProduct, M as getDefaultCache, N as resetDefaultCache, O as InMemoryEmbeddingCache, P as validateCachedResult, R as euclideanDistance, S as chunkText, T as getChunkingStats, _ as createHttpEmbedding, a as createPiiRedactor, b as selectInput, c as isEmbeddingProvider, d as createCohereEmbedding, f as createHuggingFaceEmbedding, g as HttpEmbeddingProvider, h as createTransformersEmbedding, i as containsPii, j as generateChecksum, k as createNoOpCache, l as TRANSFORMERS_MODELS, m as createOpenAIEmbedding, n as embedScrapedData, o as redactPii, p as createOllamaEmbedding, r as generateEmbeddings, s as createEmbeddingProvider, t as embed, u as createAzureEmbedding, v as getDefaultModel, w as estimateTokens, x as validateInput, y as previewInput, z as getDimensions } from "../embeddings-Bsymy_jA.mjs";
+
+export { CircuitBreaker, CircuitOpenError, HttpEmbeddingProvider, InMemoryEmbeddingCache, RateLimiter, Semaphore, TRANSFORMERS_MODELS, aggregateVectors, chunkText, containsPii, cosineSimilarity, createAzureEmbedding, createCohereEmbedding, createEmbeddingProvider, createHttpEmbedding, createHuggingFaceEmbedding, createNoOpCache, createOllamaEmbedding, createOpenAIEmbedding, createPiiRedactor, createTimeoutSignal, createTokenizer, createTransformersEmbedding, dotProduct, embed, embedScrapedData, estimateTokens, euclideanDistance, generateCacheKey, generateChecksum, generateEmbeddings, getChunkingStats, getDefaultCache, getDefaultModel, getDimensions, heuristicTokenCount, isEmbeddingProvider, isRetryableError, needsChunking, normalizeVector, previewInput, redactPii, resetDefaultCache, selectInput, validateCachedResult, validateInput, withResilience, withRetry, withTimeout };