@wenathlan/saddle 1.8.17 → 2.0.2

package/docs/plans/27.research.html.parsing.md

@@ -1,251 +0,0 @@
-# Pesquisa: HTML Parsing e Extração (2026)
-
-## 1. Parser Comparison
-
-| Feature | **Cheerio** | **jsdom** | **node-html-parser** | **parse5** |
-|---------|-------------|-----------|---------------------|------------|
-| Weekly downloads | ~10M | ~8M | ~300K+ | ~235M (indirect) |
-| Memory per page | ~1MB | ~50MB | ~1MB | <1MB |
-| Startup time | Instant | Fast (~100ms) | Instant | Instant |
-| API style | jQuery-like `$()` | Standard DOM `querySelector` | Simple DOM traversal | AST methods |
-| Script execution | No | Yes (limited) | No | No |
-| CSS selector support | Yes (css-select) | Yes (nwsapi) | Yes (basic) | No (AST only) |
-| HTML5 spec compliance | Good | Good | Basic | Full |
-| Malformed HTML | Good | Good | Basic | Excellent |
-| Best for | Web scraping, HTML transform | Testing, DOM APIs | High-throughput | Build tools |
-
-### Decision Tree
-
-```
-Is the content rendered by JavaScript (SPA/React/Vue)?
-  Yes → Use Playwright (not here)
-  No (static HTML):
-    Do you need DOM APIs (addEventListener, classList)?
-      Yes → jsdom
-      No → Do you need max throughput?
-        Yes → node-html-parser or parse5
-        No → Cheerio (best developer experience)
-```
-
-## 2. HTML Entities
-
-| Library | Weekly Downloads | Performance (ops/s) | Standards |
-|---------|-----------------|---------------------|-----------|
-| **entities** | ~235M | 5,838,416 (fastest) | XML, HTML4, HTML5 |
-| **html-entities** | ~31M | 2,919,637 | XML, HTML4, HTML5 |
-| **he** | ~15M | 2,318,438 | HTML5 only |
-
-```typescript
-import * as entities from "entities";
-entities.decodeHTML("asdf &amp; ÿ ü '"); // "asdf & ÿ ü '"
-```
-
-## 3. CSS Selector Engine: css-select
-
-**Version:** 7.0.0 (March 2026)
-
-```typescript
-import CSSselect from "css-select";
-const results = CSSselect.selectAll('.article > h2:first-child', elements);
-```
-
-Supports: `:scope`, `:is()`, `:where()`, `:has()`, attribute selectors, sibling combinators.
-
-## 4. Structured Data Extraction
-
-### Cheerio `$.extract()` (Declarative)
-
-```typescript
-const $ = await cheerio.fromURL('https://example.com');
-const data = $.extract({
-  title: 'title',
-  posts: [{
-    selector: '.post',
-    value: {
-      title: '.post-title',
-      link: { selector: 'a', value: 'href' },
-      date: { selector: 'time', value: 'datetime' },
-      tags: ['.tag'],
-    }
-  }]
-});
-```
-
-### Extraction Priority Order
-
-1. **JSON-LD** — `<script type="application/ld+json">`
-2. **Microdata** — `itemscope`, `itemprop` attributes
-3. **OpenGraph** — `og:` meta tags
-4. **data-* attributes** — developer hooks
-5. **CSS selectors** — last resort
-
-```typescript
-// JSON-LD extraction
-$('script[type="application/ld+json"]').each((_, el) => {
-  try {
-    const data = JSON.parse($(el).html());
-    if (data['@type'] === 'Product') {
-      console.log(data.name, data.offers?.price);
-    }
-  } catch {}
-});
-```
-
-## 5. Malformed HTML Handling
-
-### htmlparser2 (Forgiving)
-
-```typescript
-import htmlparser2 from "htmlparser2";
-const parser = new htmlparser2.Parser({
-  onopentag(name, attribs) { /* ... */ },
-  ontext(text) { /* ... */ },
-  onclosetag(tagname) { /* ... */ },
-}, {
-  decodeEntities: true,
-  xmlMode: false,  // HTML mode (forgiving)
-});
-```
-
-### tag-soup (Lightweight, 20KB gzipped)
-
-```typescript
-import { createDOMParser } from "tag-soup";
-const parser = createDOMParser({
-  areUnbalancedEndTagsIgnored: true,
-  areUnbalancedStartTagsImplicitlyClosed: true,
-});
-```
-
-## 6. HTML Sanitization
-
-### sanitize-html (Server-side, ~8.6M weekly downloads)
-
-```typescript
-import sanitizeHtml from "sanitize-html";
-const clean = sanitizeHtml(dirty, {
-  allowedTags: ['b', 'i', 'em', 'strong', 'a', 'p', 'ul', 'ol', 'li'],
-  allowedAttributes: { 'a': ['href', 'title'] },
-  transformTags: {
-    'a': sanitizeHtml.simpleTransform('a', { rel: 'noopener noreferrer' })
-  }
-});
-```
-
-### DOMPurify (Browser-side, ~40M weekly downloads)
-
-```typescript
-import DOMPurify from "dompurify";
-const clean = DOMPurify.sanitize('<b>hello</b><script>alert(1)</script>');
-// Result: '<b>hello</b>'
-```
-
-**Recommendation:** sanitize-html for server, DOMPurify for browser.
-
-## 7. Text Extraction with Formatting
-
-### Formatting-Preserving Walk
-
-```typescript
-function extractFormattedText($: cheerio.CheerioAPI, element: cheerio.Element): string {
-  let result = '';
-  $(element).contents().each((_, node) => {
-    if (node.type === 'text') {
-      result += node.data;
-    } else if (node.type === 'tag') {
-      switch (node.name) {
-        case 'br': result += '\n'; break;
-        case 'p': result += extractFormattedText($, node) + '\n\n'; break;
-        case 'strong': case 'b': result += `**${extractFormattedText($, node)}**`; break;
-        case 'em': case 'i': result += `*${extractFormattedText($, node)}*`; break;
-        case 'h1': result += `# ${extractFormattedText($, node)}\n\n`; break;
-        case 'h2': result += `## ${extractFormattedText($, node)}\n\n`; break;
-        case 'li': result += `- ${extractFormattedText($, node)}\n`; break;
-        default: result += extractFormattedText($, node);
-      }
-    }
-  });
-  return result;
-}
-```
-
-## 8. HTML-to-Markdown
-
-### Performance Comparison (166KB Wikipedia page)
-
-| Package | Time | Speed vs Turndown |
-|---------|------|-------------------|
-| **mdream** (Rust) | 0.34ms | 33x faster |
-| **mdream** (JS) | 3.26ms | 3.5x faster |
-| **Turndown** | 11.26ms | baseline |
-| **node-html-markdown** | 14.31ms | 1.3x slower |
-
-### Turndown (Industry Standard, 4.8M weekly downloads)
-
-```typescript
-import TurndownService from 'turndown';
-import turndownPluginGfm from 'turndown-plugin-gfm';
-
-const turndownService = new TurndownService({
-  headingStyle: 'atx',
-  codeBlockStyle: 'fenced',
-  bulletListMarker: '-',
-});
-turndownService.use(turndownPluginGfm.gfm);
-const markdown = turndownService.turndown(html);
-```
-
-### html-to-text (Plain Text, v10.0.0)
-
-```typescript
-import { convert } from 'html-to-text';
-const text = convert(html, {
-  wordWrap: 80,
-  tables: true,
-  links: { hideLinkUrls: true },
-});
-```
-
-## 9. Table Parsing
-
-### cheerio-tableparser (colspan/rowspan support)
-
-```typescript
-import cheerioTableparser from "cheerio-tableparser";
-cheerioTableparser($);
-const data = $("table").parsetable(true, true, true);
-// dupCols, dupRows, textMode
-```
-
-## 10. Custom Extraction Rules
-
-### Schema-Based Pattern
-
-```typescript
-const schema = {
-  title: { selector: 'h1', type: 'text' },
-  price: {
-    selector: '.price',
-    type: 'text',
-    transform: (v: string) => parseFloat(v.replace(/[^0-9.]/g, ''))
-  },
-  tags: { selector: '.tag', type: 'text', multiple: true },
-  links: { selector: 'a', type: 'attribute', attribute: 'href', multiple: true },
-};
-```
-
-### Recommended Stack (2026)
-
-| Task | Package |
-|------|---------|
-| Static HTML parsing | **Cheerio** (default) |
-| Full DOM APIs | **jsdom** |
-| High-throughput parsing | **node-html-parser** |
-| HTML entity encoding | **entities** (fastest) |
-| HTML sanitization (server) | **sanitize-html** |
-| HTML sanitization (browser) | **DOMPurify** |
-| HTML to Markdown | **Turndown** (plugins) or **mdream** (speed) |
-| HTML to plain text | **html-to-text** |
-| Article extraction | **@mozilla/readability** |
-| Malformed HTML | **htmlparser2** |
-| Structured extraction | Cheerio `$.extract()` |

package/docs/plans/28.action.plan.md

@@ -1,50 +0,0 @@
-# Action Plan
-
-## Phase: planning + research done
-- [x] Folder structure (mirrors `cli`/`debonair`): `docs/plans`, `docs/talks`, `docs/logs`, `tests/{examples,output,scripts}`, `web`, `.github/workflows`.
-- [x] Config files: `package.json` (`@wenathlan/saddle`, 65 deps), `tsconfig.json`, `biome.json`, `vitest.config.ts`, `.github/workflows/saddle-tests.yml`, `.gitignore`, `.npmrc`, `.nvmrc`, `README.md`.
-- [x] Research done (computer use, hcaptcha bypass, vercel sandbox, atlas agent browser).
-- [x] Planning docs in `docs/plans/` (01 to 12 + index).
-- [x] Applied skill arch: no `src/`, no vercel/netlify functions, mysql2, socket, no localhost, lowercase files, production = `web/`.
-
-## Next phase: implementation (waits for user approval)
-The user stated: "when i give you the go ahead to build the platform". So no code is written now (`web/` logic, db, etc).
-
-### Step 1 — Capture platform (after approval)
-- Root files in `web/`: `humancursor.js`, `trajectory.js`, `fingerprint.js`, `recorder.js`, `replay.js`, `runcapture.js`, `capture.js`, `index.html`.
-- Export to `docs/logs/<session>.json` (format in `11-movement-logs-json.md`).
-
-### Step 2 — Captcha test page
-- `tests/examples/hcaptchatest.html` (local) + real sites (cloudflare/google/hcaptcha).
-- `tests/scripts/` with detect -> solve -> record -> assert.
-- Evidence in `tests/output/`.
-
-### Step 3 — Production infra
-- `web/` with open node server (hono/express): routes `sessions`, `events`, `replays`, `captcha`.
-- Db: drizzle + mysql2 (and prisma schema). Migrations in deploy.
-- Cloudinary for assets/replays.
-
-### Step 4 — Sandbox
-- `@vercel/sandbox` for isolated agent execution.
-- Agent https post/get requests to the deployed site.
-
-### Step 5 — Tests
-- `tests/*.test.ts` (vitest) for trajectory, recorder, schema, cloudinary helper.
-- Example already at `tests/capture.test.ts`.
-
-## Architecture constraints (skill arch)
-- No `src/`. Root based in `web/`.
-- No vercel functions. No netlify functions.
-- Data: prisma, drizzle, mysql2, socket.
-- No localhost. Host parametrized. Ports randomized then locked.
-- Files lowercase. No underscore. No hyphen. English. Jsdoc.
-
-## Dependencies registered
-See `06-dependencies.md` (65 packages). Install with `npm install` when implementation starts.
-
-## Acceptance criteria (defined by user)
-- Capture movement (virtual mouse, virtual arrow, click, rotation, coordinates) in brave.
-- Export json logs to `docs/logs/`.
-- Open test page in brave and face hcaptcha/cloudflare/recaptcha.
-- Production backend deployed (vercel/netlify) with db (drizzle/prisma/mysql2) + cloudinary.
-- Agent https post/get requests to the deployed site.

package/docs/plans/29.api.reference.md

@@ -1,174 +0,0 @@
-# API Reference
-
-## Core Functions
-
-### `scrapeUrl(url, options?)`
-Scrape a URL using Playwright.
-```ts
-const result = await scrapeUrl('https://example.com', {
-  scroll: true,          // Scroll to bottom before extracting
-  extractLinks: true,    // Extract all links
-  extractImages: true,   // Extract all images
-  extractTables: true,   // Extract all tables
-  readable: false,       // Use readable mode
-  screenshot: false,     // Take a screenshot
-  headless: true,        // Run browser headless
-  timeout: 30000,        // Navigation timeout
-});
-```
-
-### `scrapeHtml(html, options?)`
-Scrape raw HTML string without a browser.
-```ts
-const result = await scrapeHtml('<html>...</html>', { extractTables: true });
-```
-
-### `scrapeWithBrowser(browser, url, options?)`
-Reuse an existing AgentBrowser instance.
-```ts
-const browser = new AgentBrowser();
-await browser.launch('chromium');
-const result = await scrapeWithBrowser(browser, 'https://example.com');
-```
-
-### `extractContent(html, options?)`
-Extract structured content from HTML.
-```ts
-const content = await extractContent(html, {
-  extractLinks: true,
-  extractImages: true,
-  extractTables: true,
-  readable: false,
-});
-```
-Returns `ExtractedContent` with: `title`, `author`, `text`, `links[]`, `images[]`, `tables[]`, `metadata`.
-
-### `serializeResult(result, options)`
-Serialize a ScrapeResult to target format.
-```ts
-const output = serializeResult(result, { format: 'markdown' });
-const json = serializeResult(result, { format: 'json', pretty: true });
-```
-Returns `SerializedOutput` with: `content`, `format`, `extension`.
-
-### `serializeHtml(html, options)`
-Serialize raw HTML directly to target format.
-```ts
-const md = serializeHtml('<h1>Hello</h1>', { format: 'markdown' });
-```
-
-### `formatForAgent(result, options?)`
-Format scraped content for AI agent consumption.
-```ts
-const agent = formatForAgent(result, {
-  maxChunkSize: 4000,     // Max tokens per chunk
-  includeSummary: true,
-  includeKeyPoints: true,
-  includeRelevantUrls: true,
-  includeMetadata: true,
-});
-```
-Returns `AgentOutput` with: `summary`, `keyPoints`, `content`, `relevantUrls`, `chunks[]`, `tokenCount`.
-
-## Classes
-
-### `AgentBrowser`
-Playwright browser automation manager.
-
-```ts
-const browser = new AgentBrowser({
-  headless: true,
-  stealth: true,        // Enable stealth mode
-  adBlock: true,        // Block ads and trackers
-  viewport: { width: 1280, height: 720 },
-  userAgent: 'custom',  // Custom UA string
-});
-
-await browser.launch('chromium');
-await browser.navigate(url, { waitUntil: 'networkidle' });
-await browser.click(selector);
-await browser.type(selector, text);
-await browser.screenshot({ fullPage: true });
-await browser.html();          // Get page HTML
-await browser.text();          // Get page text
-await browser.title();         // Get page title
-await browser.evaluate(fn);    // Run JS in page
-await browser.scrollToBottom({ step: 500, delay: 100 });
-await browser.executeCommands([  // Chain commands
-  { action: 'goto', args: { url } },
-  { action: 'extract' },
-  { action: 'screenshot' },
-]);
-await browser.close();
-```
-
-### `PygameRenderer`
-Bridge to Python/Pygame for rendering.
-
-```ts
-const renderer = new PygameRenderer({ pythonPath: 'python3' });
-await renderer.renderScreenshot(screenshotBuffer, 'output.png');
-await renderer.renderDoom(frameBuffer, 'doom-frame.png');
-renderer.close();
-```
-
-## CLI
-
-```
-Usage: webscrape <url> [options]
-
-Arguments:
-  url                   URL to scrape
-
-Options:
-  -f, --format <fmt>    Output format: markdown, json, xml, redis, text (default: markdown)
-  -o, --output <file>   Output file
-  --pretty              Pretty-print JSON/XML output
-  --scroll              Scroll to bottom before extraction
-  --readable           Use readable mode
-  --agent              Format output for AI agent consumption
-  --screenshot         Take a screenshot
-  -h, --help           Show help
-```
-
-## Types
-
-```ts
-interface ScrapeResult {
-  url: string;
-  title: string;
-  html: string;
-  text: string;
-  links: { href: string; text: string }[];
-  images: { src: string; alt: string }[];
-  tables: { headers: string[]; rows: string[][] }[];
-  metadata: Record<string, string>;
-  screenshot?: Buffer;
-  timestamp: number;
-}
-
-interface ExtractedContent {
-  title: string;
-  author: string;
-  text: string;
-  links: { href: string; text: string }[];
-  images: { src: string; alt: string }[];
-  tables: { headers: string[]; rows: string[][] }[];
-  metadata: Record<string, string>;
-}
-
-interface SerializedOutput {
-  content: string;
-  format: string;
-  extension: string;
-}
-
-interface AgentOutput {
-  summary: string;
-  keyPoints: string[];
-  content: string;
-  relevantUrls: { url: string; text: string }[];
-  chunks: { content: string; tokenCount: number }[];
-  tokenCount: number;
-}
-```

package/docs/plans/30.architecture.plan.md

@@ -1,94 +0,0 @@
-# Architecture
-
-## Module Overview
-
-```
-┌─────────────┐     ┌──────────────┐     ┌────────────────┐
-│ AgentBrowser│────▶│  scrapeUrl / │────▶│ extractContent │
-│ (browser.ts)│     │  scrapeHtml  │     │ (extract.ts)   │
-└─────────────┘     └──────────────┘     └────────────────┘
-                                                │
-                                                ▼
-┌─────────────┐     ┌──────────────────┐     ┌────────────────┐
-│ formatFor   │◀────│ serializeResult  │◀────│ ScrapeResult   │
-│ Agent       │     │ (serialize.ts)   │     │                │
-│ (agent.ts)  │     │ formats: md, xml,│     └────────────────┘
-└─────────────┘     │ json, redis, txt │
-                    └──────────────────┘
-                           │
-                           ▼
-                    ┌──────────────┐
-                    │  Pygame      │
-                    │  Renderer    │
-                    │ (renderer.ts)│
-                    └──────────────┘
-```
-
-## Module Responsibilities
-
-### `browser.ts` — AgentBrowser
-- Wraps Playwright `Browser`, `BrowserContext`, and `Page`
-- Provides chainable command execution (`executeCommands`)
-- Handles stealth mode, ad blocking, and browser lifecycle
-
-### `scrape.ts` — Scraping Engine
-- `scrapeUrl(url, options?)` — Launches browser, navigates, extracts
-- `scrapeHtml(html, options?)` — Works offline with raw HTML
-- `scrapeWithBrowser(browser, url, options?)` — Reuses existing AgentBrowser
-
-### `extract.ts` — Content Extraction
-- `extractContent(html, options?)` — Parses HTML via Cheerio
-- Extracts: title, author, text, links, images, tables, metadata
-- Returns structured `ExtractedContent`
-
-### `serialize.ts` — Multi-Format Serialization
-- `serializeResult(result, options)` — Converts `ScrapeResult` to target format
-- `serializeHtml(html, options)` — Direct HTML-to-format conversion
-- Supports: Markdown, HTML, JSON, XML, Redis, Plaintext
-
-### `formats.ts` — Format Registry
-- Maps format names to serializers
-- Provides format detection (article, list, page)
-- Extensions and MIME types
-
-### `agent.ts` — AI Agent Output
-- `formatForAgent(result, options?)` — Structures content for LLM consumption
-- Generates: summary, key points, relevant URLs, content chunks with token counts
-
-### `renderer.ts` — Pygame Bridge
-- `PygameRenderer` — Renders screenshots and DOOM game frames via Python/Pygame
-- Spawns Python subprocess for rendering
-
-### `cli.ts` — Command-Line Interface
-- Commander-based CLI for scraping from terminal
-- Accepts format, pretty-print, scroll, readable, and agent options
-
-### `utils.ts` — Utilities
-- `slugify`, `truncate`, `tokenEstimate`, `chunkText`
-- URL validation and classification (internal/external)
-
-### `dev-server.mjs` — Development Server
-- Standalone Node.js HTTP server
-- POST `/api/scrape` endpoint wrapping Playwright
-- Used by Vite dev server via proxy
-
-## Package Organization
-
-```
-webscrape/
-├── index.ts        # Public API exports
-├── browser.ts      # AgentBrowser (Playwright wrapper)
-├── scrape.ts       # Scraping engine
-├── extract.ts      # HTML content extraction
-├── serialize.ts    # Multi-format serialization
-├── formats.ts      # Format registry
-├── agent.ts        # AI agent output
-├── renderer.ts     # Pygame bridge
-├── cli.ts          # CLI entry
-├── utils.ts        # Utilities
-├── types.ts        # TypeScript interfaces
-├── web/            # Web demo UI
-├── tests/          # Test suites
-├── docs/           # Documentation
-└── dev-server.mjs  # API server for dev mode
-```