@wenathlan/saddle 1.8.2 → 1.8.5

package/docs/plans/27.research.html.parsing.md

@@ -0,0 +1,251 @@
+# Pesquisa: HTML Parsing e Extração (2026)
+
+## 1. Parser Comparison
+
+| Feature | **Cheerio** | **jsdom** | **node-html-parser** | **parse5** |
+|---------|-------------|-----------|---------------------|------------|
+| Weekly downloads | ~10M | ~8M | ~300K+ | ~235M (indirect) |
+| Memory per page | ~1MB | ~50MB | ~1MB | <1MB |
+| Startup time | Instant | Fast (~100ms) | Instant | Instant |
+| API style | jQuery-like `$()` | Standard DOM `querySelector` | Simple DOM traversal | AST methods |
+| Script execution | No | Yes (limited) | No | No |
+| CSS selector support | Yes (css-select) | Yes (nwsapi) | Yes (basic) | No (AST only) |
+| HTML5 spec compliance | Good | Good | Basic | Full |
+| Malformed HTML | Good | Good | Basic | Excellent |
+| Best for | Web scraping, HTML transform | Testing, DOM APIs | High-throughput | Build tools |
+
+### Decision Tree
+
+```
+Is the content rendered by JavaScript (SPA/React/Vue)?
+  Yes → Use Playwright (not here)
+  No (static HTML):
+    Do you need DOM APIs (addEventListener, classList)?
+      Yes → jsdom
+      No → Do you need max throughput?
+        Yes → node-html-parser or parse5
+        No → Cheerio (best developer experience)
+```
+
+## 2. HTML Entities
+
+| Library | Weekly Downloads | Performance (ops/s) | Standards |
+|---------|-----------------|---------------------|-----------|
+| **entities** | ~235M | 5,838,416 (fastest) | XML, HTML4, HTML5 |
+| **html-entities** | ~31M | 2,919,637 | XML, HTML4, HTML5 |
+| **he** | ~15M | 2,318,438 | HTML5 only |
+
+```typescript
+import * as entities from "entities";
+entities.decodeHTML("asdf &amp; ÿ ü '"); // "asdf & ÿ ü '"
+```
+
+## 3. CSS Selector Engine: css-select
+
+**Version:** 7.0.0 (March 2026)
+
+```typescript
+import CSSselect from "css-select";
+const results = CSSselect.selectAll('.article > h2:first-child', elements);
+```
+
+Supports: `:scope`, `:is()`, `:where()`, `:has()`, attribute selectors, sibling combinators.
+
+## 4. Structured Data Extraction
+
+### Cheerio `$.extract()` (Declarative)
+
+```typescript
+const $ = await cheerio.fromURL('https://example.com');
+const data = $.extract({
+  title: 'title',
+  posts: [{
+    selector: '.post',
+    value: {
+      title: '.post-title',
+      link: { selector: 'a', value: 'href' },
+      date: { selector: 'time', value: 'datetime' },
+      tags: ['.tag'],
+    }
+  }]
+});
+```
+
+### Extraction Priority Order
+
+1. **JSON-LD** — `<script type="application/ld+json">`
+2. **Microdata** — `itemscope`, `itemprop` attributes
+3. **OpenGraph** — `og:` meta tags
+4. **data-* attributes** — developer hooks
+5. **CSS selectors** — last resort
+
+```typescript
+// JSON-LD extraction
+$('script[type="application/ld+json"]').each((_, el) => {
+  try {
+    const data = JSON.parse($(el).html());
+    if (data['@type'] === 'Product') {
+      console.log(data.name, data.offers?.price);
+    }
+  } catch {}
+});
+```
+
+## 5. Malformed HTML Handling
+
+### htmlparser2 (Forgiving)
+
+```typescript
+import htmlparser2 from "htmlparser2";
+const parser = new htmlparser2.Parser({
+  onopentag(name, attribs) { /* ... */ },
+  ontext(text) { /* ... */ },
+  onclosetag(tagname) { /* ... */ },
+}, {
+  decodeEntities: true,
+  xmlMode: false,  // HTML mode (forgiving)
+});
+```
+
+### tag-soup (Lightweight, 20KB gzipped)
+
+```typescript
+import { createDOMParser } from "tag-soup";
+const parser = createDOMParser({
+  areUnbalancedEndTagsIgnored: true,
+  areUnbalancedStartTagsImplicitlyClosed: true,
+});
+```
+
+## 6. HTML Sanitization
+
+### sanitize-html (Server-side, ~8.6M weekly downloads)
+
+```typescript
+import sanitizeHtml from "sanitize-html";
+const clean = sanitizeHtml(dirty, {
+  allowedTags: ['b', 'i', 'em', 'strong', 'a', 'p', 'ul', 'ol', 'li'],
+  allowedAttributes: { 'a': ['href', 'title'] },
+  transformTags: {
+    'a': sanitizeHtml.simpleTransform('a', { rel: 'noopener noreferrer' })
+  }
+});
+```
+
+### DOMPurify (Browser-side, ~40M weekly downloads)
+
+```typescript
+import DOMPurify from "dompurify";
+const clean = DOMPurify.sanitize('<b>hello</b><script>alert(1)</script>');
+// Result: '<b>hello</b>'
+```
+
+**Recommendation:** sanitize-html for server, DOMPurify for browser.
+
+## 7. Text Extraction with Formatting
+
+### Formatting-Preserving Walk
+
+```typescript
+function extractFormattedText($: cheerio.CheerioAPI, element: cheerio.Element): string {
+  let result = '';
+  $(element).contents().each((_, node) => {
+    if (node.type === 'text') {
+      result += node.data;
+    } else if (node.type === 'tag') {
+      switch (node.name) {
+        case 'br': result += '\n'; break;
+        case 'p': result += extractFormattedText($, node) + '\n\n'; break;
+        case 'strong': case 'b': result += `**${extractFormattedText($, node)}**`; break;
+        case 'em': case 'i': result += `*${extractFormattedText($, node)}*`; break;
+        case 'h1': result += `# ${extractFormattedText($, node)}\n\n`; break;
+        case 'h2': result += `## ${extractFormattedText($, node)}\n\n`; break;
+        case 'li': result += `- ${extractFormattedText($, node)}\n`; break;
+        default: result += extractFormattedText($, node);
+      }
+    }
+  });
+  return result;
+}
+```
+
+## 8. HTML-to-Markdown
+
+### Performance Comparison (166KB Wikipedia page)
+
+| Package | Time | Speed vs Turndown |
+|---------|------|-------------------|
+| **mdream** (Rust) | 0.34ms | 33x faster |
+| **mdream** (JS) | 3.26ms | 3.5x faster |
+| **Turndown** | 11.26ms | baseline |
+| **node-html-markdown** | 14.31ms | 1.3x slower |
+
+### Turndown (Industry Standard, 4.8M weekly downloads)
+
+```typescript
+import TurndownService from 'turndown';
+import turndownPluginGfm from 'turndown-plugin-gfm';
+
+const turndownService = new TurndownService({
+  headingStyle: 'atx',
+  codeBlockStyle: 'fenced',
+  bulletListMarker: '-',
+});
+turndownService.use(turndownPluginGfm.gfm);
+const markdown = turndownService.turndown(html);
+```
+
+### html-to-text (Plain Text, v10.0.0)
+
+```typescript
+import { convert } from 'html-to-text';
+const text = convert(html, {
+  wordWrap: 80,
+  tables: true,
+  links: { hideLinkUrls: true },
+});
+```
+
+## 9. Table Parsing
+
+### cheerio-tableparser (colspan/rowspan support)
+
+```typescript
+import cheerioTableparser from "cheerio-tableparser";
+cheerioTableparser($);
+const data = $("table").parsetable(true, true, true);
+// dupCols, dupRows, textMode
+```
+
+## 10. Custom Extraction Rules
+
+### Schema-Based Pattern
+
+```typescript
+const schema = {
+  title: { selector: 'h1', type: 'text' },
+  price: {
+    selector: '.price',
+    type: 'text',
+    transform: (v: string) => parseFloat(v.replace(/[^0-9.]/g, ''))
+  },
+  tags: { selector: '.tag', type: 'text', multiple: true },
+  links: { selector: 'a', type: 'attribute', attribute: 'href', multiple: true },
+};
+```
+
+### Recommended Stack (2026)
+
+| Task | Package |
+|------|---------|
+| Static HTML parsing | **Cheerio** (default) |
+| Full DOM APIs | **jsdom** |
+| High-throughput parsing | **node-html-parser** |
+| HTML entity encoding | **entities** (fastest) |
+| HTML sanitization (server) | **sanitize-html** |
+| HTML sanitization (browser) | **DOMPurify** |
+| HTML to Markdown | **Turndown** (plugins) or **mdream** (speed) |
+| HTML to plain text | **html-to-text** |
+| Article extraction | **@mozilla/readability** |
+| Malformed HTML | **htmlparser2** |
+| Structured extraction | Cheerio `$.extract()` |

package/docs/plans/28.action.plan.md

@@ -0,0 +1,50 @@
+# Action Plan
+
+## Phase: planning + research done
+- [x] Folder structure (mirrors `cli`/`debonair`): `docs/plans`, `docs/talks`, `docs/logs`, `tests/{examples,output,scripts}`, `web`, `.github/workflows`.
+- [x] Config files: `package.json` (`@devthink/saddle`, 65 deps), `tsconfig.json`, `biome.json`, `vitest.config.ts`, `.github/workflows/saddle-tests.yml`, `.gitignore`, `.npmrc`, `.nvmrc`, `README.md`.
+- [x] Research done (computer use, hcaptcha bypass, vercel sandbox, atlas agent browser).
+- [x] Planning docs in `docs/plans/` (01 to 12 + index).
+- [x] Applied skill arch: no `src/`, no vercel/netlify functions, mysql2, socket, no localhost, lowercase files, production = `web/`.
+
+## Next phase: implementation (waits for user approval)
+The user stated: "when i give you the go ahead to build the platform". So no code is written now (`web/` logic, db, etc).
+
+### Step 1 — Capture platform (after approval)
+- Root files in `web/`: `humancursor.js`, `trajectory.js`, `fingerprint.js`, `recorder.js`, `replay.js`, `runcapture.js`, `capture.js`, `index.html`.
+- Export to `docs/logs/<session>.json` (format in `11-movement-logs-json.md`).
+
+### Step 2 — Captcha test page
+- `tests/examples/hcaptchatest.html` (local) + real sites (cloudflare/google/hcaptcha).
+- `tests/scripts/` with detect -> solve -> record -> assert.
+- Evidence in `tests/output/`.
+
+### Step 3 — Production infra
+- `web/` with open node server (hono/express): routes `sessions`, `events`, `replays`, `captcha`.
+- Db: drizzle + mysql2 (and prisma schema). Migrations in deploy.
+- Cloudinary for assets/replays.
+
+### Step 4 — Sandbox
+- `@vercel/sandbox` for isolated agent execution.
+- Agent https post/get requests to the deployed site.
+
+### Step 5 — Tests
+- `tests/*.test.ts` (vitest) for trajectory, recorder, schema, cloudinary helper.
+- Example already at `tests/capture.test.ts`.
+
+## Architecture constraints (skill arch)
+- No `src/`. Root based in `web/`.
+- No vercel functions. No netlify functions.
+- Data: prisma, drizzle, mysql2, socket.
+- No localhost. Host parametrized. Ports randomized then locked.
+- Files lowercase. No underscore. No hyphen. English. Jsdoc.
+
+## Dependencies registered
+See `06-dependencies.md` (65 packages). Install with `npm install` when implementation starts.
+
+## Acceptance criteria (defined by user)
+- Capture movement (virtual mouse, virtual arrow, click, rotation, coordinates) in brave.
+- Export json logs to `docs/logs/`.
+- Open test page in brave and face hcaptcha/cloudflare/recaptcha.
+- Production backend deployed (vercel/netlify) with db (drizzle/prisma/mysql2) + cloudinary.
+- Agent https post/get requests to the deployed site.

package/docs/plans/29.api.reference.md

@@ -0,0 +1,174 @@
+# API Reference
+
+## Core Functions
+
+### `scrapeUrl(url, options?)`
+Scrape a URL using Playwright.
+```ts
+const result = await scrapeUrl('https://example.com', {
+  scroll: true,          // Scroll to bottom before extracting
+  extractLinks: true,    // Extract all links
+  extractImages: true,   // Extract all images
+  extractTables: true,   // Extract all tables
+  readable: false,       // Use readable mode
+  screenshot: false,     // Take a screenshot
+  headless: true,        // Run browser headless
+  timeout: 30000,        // Navigation timeout
+});
+```
+
+### `scrapeHtml(html, options?)`
+Scrape raw HTML string without a browser.
+```ts
+const result = await scrapeHtml('<html>...</html>', { extractTables: true });
+```
+
+### `scrapeWithBrowser(browser, url, options?)`
+Reuse an existing AgentBrowser instance.
+```ts
+const browser = new AgentBrowser();
+await browser.launch('chromium');
+const result = await scrapeWithBrowser(browser, 'https://example.com');
+```
+
+### `extractContent(html, options?)`
+Extract structured content from HTML.
+```ts
+const content = await extractContent(html, {
+  extractLinks: true,
+  extractImages: true,
+  extractTables: true,
+  readable: false,
+});
+```
+Returns `ExtractedContent` with: `title`, `author`, `text`, `links[]`, `images[]`, `tables[]`, `metadata`.
+
+### `serializeResult(result, options)`
+Serialize a ScrapeResult to target format.
+```ts
+const output = serializeResult(result, { format: 'markdown' });
+const json = serializeResult(result, { format: 'json', pretty: true });
+```
+Returns `SerializedOutput` with: `content`, `format`, `extension`.
+
+### `serializeHtml(html, options)`
+Serialize raw HTML directly to target format.
+```ts
+const md = serializeHtml('<h1>Hello</h1>', { format: 'markdown' });
+```
+
+### `formatForAgent(result, options?)`
+Format scraped content for AI agent consumption.
+```ts
+const agent = formatForAgent(result, {
+  maxChunkSize: 4000,     // Max tokens per chunk
+  includeSummary: true,
+  includeKeyPoints: true,
+  includeRelevantUrls: true,
+  includeMetadata: true,
+});
+```
+Returns `AgentOutput` with: `summary`, `keyPoints`, `content`, `relevantUrls`, `chunks[]`, `tokenCount`.
+
+## Classes
+
+### `AgentBrowser`
+Playwright browser automation manager.
+
+```ts
+const browser = new AgentBrowser({
+  headless: true,
+  stealth: true,        // Enable stealth mode
+  adBlock: true,        // Block ads and trackers
+  viewport: { width: 1280, height: 720 },
+  userAgent: 'custom',  // Custom UA string
+});
+
+await browser.launch('chromium');
+await browser.navigate(url, { waitUntil: 'networkidle' });
+await browser.click(selector);
+await browser.type(selector, text);
+await browser.screenshot({ fullPage: true });
+await browser.html();          // Get page HTML
+await browser.text();          // Get page text
+await browser.title();         // Get page title
+await browser.evaluate(fn);    // Run JS in page
+await browser.scrollToBottom({ step: 500, delay: 100 });
+await browser.executeCommands([  // Chain commands
+  { action: 'goto', args: { url } },
+  { action: 'extract' },
+  { action: 'screenshot' },
+]);
+await browser.close();
+```
+
+### `PygameRenderer`
+Bridge to Python/Pygame for rendering.
+
+```ts
+const renderer = new PygameRenderer({ pythonPath: 'python3' });
+await renderer.renderScreenshot(screenshotBuffer, 'output.png');
+await renderer.renderDoom(frameBuffer, 'doom-frame.png');
+renderer.close();
+```
+
+## CLI
+
+```
+Usage: webscrape <url> [options]
+
+Arguments:
+  url                   URL to scrape
+
+Options:
+  -f, --format <fmt>    Output format: markdown, json, xml, redis, text (default: markdown)
+  -o, --output <file>   Output file
+  --pretty              Pretty-print JSON/XML output
+  --scroll              Scroll to bottom before extraction
+  --readable           Use readable mode
+  --agent              Format output for AI agent consumption
+  --screenshot         Take a screenshot
+  -h, --help           Show help
+```
+
+## Types
+
+```ts
+interface ScrapeResult {
+  url: string;
+  title: string;
+  html: string;
+  text: string;
+  links: { href: string; text: string }[];
+  images: { src: string; alt: string }[];
+  tables: { headers: string[]; rows: string[][] }[];
+  metadata: Record<string, string>;
+  screenshot?: Buffer;
+  timestamp: number;
+}
+
+interface ExtractedContent {
+  title: string;
+  author: string;
+  text: string;
+  links: { href: string; text: string }[];
+  images: { src: string; alt: string }[];
+  tables: { headers: string[]; rows: string[][] }[];
+  metadata: Record<string, string>;
+}
+
+interface SerializedOutput {
+  content: string;
+  format: string;
+  extension: string;
+}
+
+interface AgentOutput {
+  summary: string;
+  keyPoints: string[];
+  content: string;
+  relevantUrls: { url: string; text: string }[];
+  chunks: { content: string; tokenCount: number }[];
+  tokenCount: number;
+}
+```

package/docs/plans/30.architecture.plan.md

@@ -0,0 +1,94 @@
+# Architecture
+
+## Module Overview
+
+```
+┌─────────────┐     ┌──────────────┐     ┌────────────────┐
+│ AgentBrowser│────▶│  scrapeUrl / │────▶│ extractContent │
+│ (browser.ts)│     │  scrapeHtml  │     │ (extract.ts)   │
+└─────────────┘     └──────────────┘     └────────────────┘
+                                                │
+                                                ▼
+┌─────────────┐     ┌──────────────────┐     ┌────────────────┐
+│ formatFor   │◀────│ serializeResult  │◀────│ ScrapeResult   │
+│ Agent       │     │ (serialize.ts)   │     │                │
+│ (agent.ts)  │     │ formats: md, xml,│     └────────────────┘
+└─────────────┘     │ json, redis, txt │
+                    └──────────────────┘
+                           │
+                           ▼
+                    ┌──────────────┐
+                    │  Pygame      │
+                    │  Renderer    │
+                    │ (renderer.ts)│
+                    └──────────────┘
+```
+
+## Module Responsibilities
+
+### `browser.ts` — AgentBrowser
+- Wraps Playwright `Browser`, `BrowserContext`, and `Page`
+- Provides chainable command execution (`executeCommands`)
+- Handles stealth mode, ad blocking, and browser lifecycle
+
+### `scrape.ts` — Scraping Engine
+- `scrapeUrl(url, options?)` — Launches browser, navigates, extracts
+- `scrapeHtml(html, options?)` — Works offline with raw HTML
+- `scrapeWithBrowser(browser, url, options?)` — Reuses existing AgentBrowser
+
+### `extract.ts` — Content Extraction
+- `extractContent(html, options?)` — Parses HTML via Cheerio
+- Extracts: title, author, text, links, images, tables, metadata
+- Returns structured `ExtractedContent`
+
+### `serialize.ts` — Multi-Format Serialization
+- `serializeResult(result, options)` — Converts `ScrapeResult` to target format
+- `serializeHtml(html, options)` — Direct HTML-to-format conversion
+- Supports: Markdown, HTML, JSON, XML, Redis, Plaintext
+
+### `formats.ts` — Format Registry
+- Maps format names to serializers
+- Provides format detection (article, list, page)
+- Extensions and MIME types
+
+### `agent.ts` — AI Agent Output
+- `formatForAgent(result, options?)` — Structures content for LLM consumption
+- Generates: summary, key points, relevant URLs, content chunks with token counts
+
+### `renderer.ts` — Pygame Bridge
+- `PygameRenderer` — Renders screenshots and DOOM game frames via Python/Pygame
+- Spawns Python subprocess for rendering
+
+### `cli.ts` — Command-Line Interface
+- Commander-based CLI for scraping from terminal
+- Accepts format, pretty-print, scroll, readable, and agent options
+
+### `utils.ts` — Utilities
+- `slugify`, `truncate`, `tokenEstimate`, `chunkText`
+- URL validation and classification (internal/external)
+
+### `dev-server.mjs` — Development Server
+- Standalone Node.js HTTP server
+- POST `/api/scrape` endpoint wrapping Playwright
+- Used by Vite dev server via proxy
+
+## Package Organization
+
+```
+webscrape/
+├── index.ts        # Public API exports
+├── browser.ts      # AgentBrowser (Playwright wrapper)
+├── scrape.ts       # Scraping engine
+├── extract.ts      # HTML content extraction
+├── serialize.ts    # Multi-format serialization
+├── formats.ts      # Format registry
+├── agent.ts        # AI agent output
+├── renderer.ts     # Pygame bridge
+├── cli.ts          # CLI entry
+├── utils.ts        # Utilities
+├── types.ts        # TypeScript interfaces
+├── web/            # Web demo UI
+├── tests/          # Test suites
+├── docs/           # Documentation
+└── dev-server.mjs  # API server for dev mode
+```