@wenathlan/saddle 1.8.4 → 1.8.6

package/docs/plans/43.plan.universal.architecture.md

@@ -0,0 +1,496 @@
+# Plano: Arquitetura Universal WebScrape v2.0
+
+## 1. Visão Geral
+
+A biblioteca `@devthink/webscrape` precisa funcionar em **TODOS os modos**:
+
+| Modo | Como funciona |
+|------|--------------|
+| **Com navegador** | Playwright (Chromium/Firefox/WebKit) para páginas SPA/JS |
+| **Sem navegador** | fetch + Cheerio para HTML estático |
+| **Com Node.js** | `node:fs`, `node:child_process`, SQLite |
+| **Sem Node.js** | Bun, Deno, edge runtimes, browsers |
+| **Com web** | API HTTP (Hono), WebSocket, SSE |
+| **Sem web** | CLI, binário compilado, biblioteca |
+| **Com IA** | Token counting, chunking semântico, llms.txt, RAG |
+| **Sem IA** | Scraping puro, extração de dados |
+| **Com memória** | Redis, SQLite, IndexedDB, Workers KV |
+| **Sem memória** | In-memory LRU, efêmero |
+| **Com desktop** | CLI TUI, Electron, Tauri |
+| **Sem desktop** | Web UI, serverless |
+| **Com servidor** | API server, worker pool |
+| **Sem servidor** | Standalone, binário compilado |
+| **Com script** | npm package, ESM/CJS |
+| **Sem script** | Bun compile binary, WASM |
+
+## 2. Estrutura de Diretórios
+
+```
+WebScrape/
+├── index.ts                    # Barrel principal (lib universal)
+├── types.ts                    # Tipos + Zod schemas
+├── errors.ts                   # Hierarquia de erros
+├── events.ts                   # Typed EventEmitter (hooks)
+│
+├── # ── Core Scraping ──
+├── fetch.ts                    # HTTP fetch universal (sem navegador)
+├── browser.ts                  # AgentBrowser (Playwright)
+├── scrape.ts                   # Strategy cascade: fetch → browser
+├── extract.ts                  # Extração HTML (Cheerio + Readability)
+├── serialize.ts                # Serialização multi-formato
+├── formats.ts                  # Registros de formato
+│
+├── # ── Infraestrutura ──
+├── retry.ts                    # Retry com backoff
+├── proxy.ts                    # Proxy rotation + pool
+├── rate-limiter.ts             # Rate limiting por domínio
+├── cache.ts                    # Cache multi-camada
+├── session.ts                  # Cookie jar + sessões
+│
+├── # ── Anti-Detection ──
+├── stealth.ts                  # Anti-detection (playwright-extra)
+├── fingerprint.ts              # Geração de fingerprints
+├── headers.ts                  # Rotação de headers/UA
+│
+├── # ── Crawling ──
+├── crawler.ts                  # BFS crawling com depth limit
+├── sitemap.ts                  # Parser de sitemap XML
+├── robots.ts                   # Parser de robots.txt
+│
+├── # ── AI/LLM ──
+├── agent.ts                    # Formatação para LLMs
+├── tokens.ts                   # Token counting multi-modelo
+├── chunking.ts                 # Chunking semântico
+├── llms-txt.ts                 # Geração de llms.txt
+│
+├── # ── Batch & Concurrency ──
+├── batch.ts                    # Batch scraping concorrente
+├── pool.ts                     # Browser pool
+│
+├── # ── Middleware ──
+├── middleware.ts                # Pipeline de middleware
+│
+├── # ── Utilitários ──
+├── utils.ts                    # Funções utilitárias
+├── url.ts                      # URL normalization
+│
+├── # ── CLI ──
+├── cli.ts                      # CLI binary (commander)
+│
+├── # ── Server ──
+├── server.ts                   # API HTTP (Hono)
+│
+├── # ── Renderer ──
+├── renderer.ts                 # Pygame bridge (opcional)
+│
+├── # ── Config ──
+├── package.json
+├── tsconfig.json
+├── vite.config.ts
+├── vitest.config.ts
+├── biome.json
+├── .npmrc
+├── .gitignore
+├── .husky/
+├── .github/workflows/
+│
+├── # ── Web UI ──
+├── web/
+│   └── index.html
+│
+├── # ── Docs ──
+├── docs/
+│   ├── README.md
+│   ├── API.md
+│   ├── ARCHITECTURE.md
+│   ├── CHANGELOG.md
+│   └── RESEARCH-*.md
+│
+└── # ── Tests ──
+    └── tests/
+        ├── fetch.test.ts
+        ├── browser.test.ts
+        ├── scrape.test.ts
+        ├── extract.test.ts
+        ├── serialize.test.ts
+        ├── retry.test.ts
+        ├── proxy.test.ts
+        ├── cache.test.ts
+        ├── crawler.test.ts
+        ├── batch.test.ts
+        ├── agent.test.ts
+        ├── tokens.test.ts
+        ├── chunking.test.ts
+        ├── errors.test.ts
+        ├── events.test.ts
+        ├── middleware.test.ts
+        └── utils.test.ts
+```
+
+## 3. Strategy Cascade Architecture
+
+```
+scrape(url, options)
+    │
+    ├─── options.mode === 'fetch' ──→ fetch.ts (HTTP + Cheerio)
+    │
+    ├─── options.mode === 'browser' ──→ browser.ts (Playwright)
+    │
+    └─── options.mode === 'auto' (default) ──→ scrape.ts
+         │
+         ├── 1. Tentar fetch (rápido, barato)
+         ├── 2. Detectar se precisa de JS rendering
+         │      ├── Estático → usar resultado do fetch
+         │      └── Dinâmico → usar Playwright
+         └── 3. Return resultado mais limpo
+```
+
+## 4. Módulos Novos — Especificações
+
+### 4.1 errors.ts — Hierarquia de Erros
+
+```typescript
+WebScrapeError (base)
+├── ValidationError (400)
+├── TimeoutError (504)
+├── BlockedError (403)
+├── RateLimitError (429)
+├── ProxyError (502)
+├── ParseError (422)
+├── AuthError (401)
+└── NetworkError (503)
+```
+
+- Cada erro tem `code`, `statusCode`, `isRetryable`, `details`
+- `Object.setPrototypeOf` para `instanceof` correto
+- Recovery suggestions em cada erro
+
+### 4.2 events.ts — Hooks de Lifecycle
+
+```typescript
+interface WebScrapeEvents {
+  'request:start':    [url: string, options: RequestOptions]
+  'request:response': [response: Response, attempt: number]
+  'request:error':    [error: Error, attempt: number]
+  'request:retry':    [attempt: number, maxRetries: number]
+  'request:complete': [stats: CrawlStats]
+  'cache:hit':        [key: string]
+  'cache:miss':       [key: string]
+  'proxy:rotate':     [proxy: string, reason: string]
+  'proxy:error':      [proxy: string, error: Error]
+  'crawl:discover':   [url: string, depth: number]
+  'crawl:complete':   [stats: CrawlStats]
+}
+```
+
+- Usa `emittery` (typed, async-first, AbortSignal, Symbol.dispose)
+- Substitui EventEmitter padrão do Node
+
+### 4.3 fetch.ts — HTTP Universal
+
+- Usa `globalThis.fetch` (disponível em TODOS os runtimes 2026)
+- Suporta: cookies, headers rotation, proxy, timeout, retry
+- Detecta se a página precisa de JS rendering
+- Suporta HTTP/2 via undici (quando disponível)
+- Cookie management via tough-cookie
+
+### 4.4 proxy.ts — Proxy Rotation
+
+- Pool de proxies com health checking
+- Estratégias: round-robin, random, sticky session
+- Auto-disable em proxies mortos (3 failures consecutivas)
+- Cooldown com revive automático
+- Suporta: HTTP, HTTPS, SOCKS5
+
+### 4.5 cache.ts — Cache Multi-Camada
+
+- L1: In-memory LRU (lru-cache v11)
+- L2: Persistent (keyv com adapter: SQLite/Redis/IndexedDB)
+- TTL por tipo de conteúdo:
+  - Estático (about, policies): 24-72h
+  - Semi-estático (produtos): 1-6h
+  - Dinâmico (search): 30s-5min
+- ETag/Last-Modified conditional GET
+
+### 4.6 chunking.ts — Chunking Semântico
+
+- Split por headings Markdown (H1/H2/H3)
+- Overlap de 10-20%
+- Max 512 tokens por chunk
+- Preserva código blocks inteiros
+- Inclui heading path em cada chunk
+- Hierarchical parent-child para RAG
+
+### 4.7 tokens.ts — Token Counting Multi-Modelo
+
+- OpenAI: js-tiktoken (o200k_base)
+- Claude: API ou approximação (p50k_base ±10-30%)
+- Gemini: API ou bpe-lite (100% accuracy)
+- Fallback: estimateTokens (chars/4)
+- Edge runtime compatible
+
+### 4.8 llms-txt.ts — Geração de llms.txt
+
+- Extrai headings, links, descriptions de páginas
+- Gera formato padronizado (llmstxt.org)
+- Suporta llms-full.txt (conteúdo concatenado)
+- Validação: exato H1, blockquote summary, URLs absolutas
+
+### 4.9 middleware.ts — Pipeline
+
+- Koa-style onion model
+- Cada middleware recebe `ctx` e `next`
+- Middleware para: auth, rate-limit, retry, cache, logging
+- Extensível via plugins
+
+### 4.10 server.ts — API HTTP
+
+- Hono framework (multi-runtime)
+- Endpoints:
+  - `POST /v1/scrape` — sync scrape
+  - `POST /v1/scrape/async` — async scrape
+  - `GET /v1/scrape/:taskId` — get result
+  - `POST /v1/batch` — batch scrape
+  - `POST /v1/crawl` — start crawl
+  - `GET /health` — health check
+- SSE para progress updates
+- Rate limiting por API key
+- OpenAPI spec gerado
+
+## 5. Dependências
+
+### Production (adicionar)
+
+| Pacote | Versão | Propósito |
+|--------|--------|-----------|
+| `zod` | ^4.4.3 | Validação runtime |
+| `p-retry` | ^8.0.0 | Retry com backoff |
+| `p-limit` | ^7.3.0 | Controle concorrência |
+| `p-queue` | ^9.2.0 | Fila com prioridade |
+| `robots-parser` | ^3.0.1 | Parser robots.txt |
+| `sitemapper` | ^4.1.6 | Parser sitemap XML |
+| `turndown` | ^7.2.4 | HTML → Markdown |
+| `turndown-plugin-gfm` | ^1.0.2 | GFM tables |
+| `@mozilla/readability` | ^0.6.0 | Extração conteúdo principal |
+| `lru-cache` | ^11.0.0 | Cache LRU |
+| `keyv` | ^5.6.0 | KV store cross-runtime |
+| `hono` | ^4.0.0 | HTTP server |
+| `commander` | ^14.0.3 | CLI |
+| `emittery` | ^2.0.0 | Typed EventEmitter |
+| `normalize-url` | ^9.0.0 | URL normalization |
+
+### Production (manter)
+
+| Pacote | Nota |
+|--------|------|
+| `playwright` | ^1.60.0 |
+| `cheerio` | ^1.2.0 |
+| `jsdom` | ^29.1.1 |
+| `chalk` | ^5.6.2 |
+
+### Production (REMOVER)
+
+| Pacote | Motivo |
+|--------|--------|
+| `@inquirer/prompts` | UI interativa, não core |
+| `@inkjs/ui` | Ink components |
+| `@inquirer/core` | UI interativa |
+| `@opencode-ai/plugin` | Plugin específico |
+| `@types/blessed` | TUI blessed |
+| `better-sqlite3` | Mover para optional |
+| `blessed` | TUI framework |
+| `cfonts` | ASCII art fonts |
+| `chalk-animation` | CLI animations |
+| `cli-table3` | CLI tables |
+| `cross-spawn` | Subprocess spawning |
+| `figlet` | ASCII art |
+| `fuzzysort` | Fuzzy matching |
+| `gradient-string` | CLI gradient |
+| `hls.js` | Video streaming |
+| `ink` | React TUI |
+| `ink-big-text` | Ink component |
+| `ink-gradient` | Ink component |
+| `ink-link` | Ink component |
+| `listr2` | Task runner |
+| `marked` | MD parser (usar turndown) |
+| `ora` | Spinner |
+| `react` | UI framework |
+| `react-devtools-core` | Dev tools |
+| `react-reconciler` | React internals |
+| `scheduler` | React scheduler |
+| `sharp` | Image processing |
+
+### Optional (novo)
+
+| Pacote | Versão | Propósito |
+|--------|--------|-----------|
+| `playwright` | ^1.60.0 | Opcional para quem só quer fetch |
+| `playwright-extra` | ^1.7.9 | Stealth plugin |
+| `puppeteer-extra-plugin-stealth` | ^2.11.2 | Evasion modules |
+
+### Dev Dependencies (adicionar)
+
+| Pacote | Propósito |
+|--------|-----------|
+| `unplugin-dts` | Gerar .d.ts (sucessor vite-plugin-dts) |
+| `@arethetypeswrong/cli` | Validar tipos |
+| `publint` | Validar package.json |
+| `husky` ^9.0.0 | Git hooks |
+| `lint-staged` ^15.0.0 | Lint staged files |
+
+## 6. Configurações de Build
+
+### tsconfig.json
+
+```json
+{
+  "compilerOptions": {
+    "target": "ES2024",
+    "module": "ESNext",
+    "moduleResolution": "bundler",
+    "lib": ["ES2024", "DOM", "DOM.Iterable"],
+    "strict": true,
+    "declaration": true,
+    "declarationMap": true,
+    "sourceMap": true,
+    "outDir": "./dist",
+    "rootDir": ".",
+    "noEmit": true,
+    "isolatedModules": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true,
+    "verbatimModuleSyntax": true,
+    "types": ["node"]
+  },
+  "include": ["*.ts"],
+  "exclude": ["node_modules", "dist", "web", "tests"]
+}
+```
+
+### vite.config.ts
+
+```typescript
+import { defineConfig } from "vite";
+import dts from "vite-plugin-dts";
+import { resolve } from "path";
+
+export default defineConfig({
+  root: "web",
+  build: {
+    lib: {
+      entry: resolve(__dirname, "index.ts"),
+      name: "DevThinkWebScrape",
+      formats: ["es", "cjs"],
+      fileName: (format) => `index.${format === "es" ? "mjs" : "js"}`,
+    },
+    rollupOptions: {
+      external: [
+        "playwright", "cheerio", "jsdom",
+        "zod", "p-retry", "p-limit", "p-queue",
+        "robots-parser", "sitemapper",
+        "turndown", "turndown-plugin-gfm",
+        "@mozilla/readability", "lru-cache", "keyv",
+        "hono", "commander", "emittery", "normalize-url",
+        "node:fs", "node:path", "node:child_process", "node:http",
+      ],
+    },
+  },
+  plugins: [dts()],
+  server: { port: 5175, host: "127.0.0.1", cors: true },
+});
+```
+
+### package.json (atualizado)
+
+```json
+{
+  "name": "@devthink/webscrape",
+  "version": "2.0.0",
+  "type": "module",
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "bin": { "webscrape": "./dist/cli.js" },
+  "exports": {
+    ".":         { "types": "./dist/index.d.ts", "import": "./dist/index.js" },
+    "./fetch":   { "types": "./dist/fetch.d.ts", "import": "./dist/fetch.js" },
+    "./browser": { "types": "./dist/browser.d.ts", "import": "./dist/browser.js" },
+    "./scrape":  { "types": "./dist/scrape.d.ts", "import": "./dist/scrape.js" },
+    "./agent":   { "types": "./dist/agent.d.ts", "import": "./dist/agent.js" },
+    "./server":  { "types": "./dist/server.d.ts", "import": "./dist/server.js" },
+    "./cli":     { "import": "./dist/cli.js" },
+    "./types":   { "types": "./dist/types.d.ts", "import": "./dist/types.js" }
+  },
+  "files": ["dist", "web/", "docs/", "README.md"],
+  "sideEffects": false,
+  "scripts": {
+    "build": "tsc --noEmit && vite build",
+    "dev": "vite",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "test:coverage": "vitest run --coverage",
+    "lint": "biome check .",
+    "lint:fix": "biome check --write .",
+    "typecheck": "tsc --noEmit",
+    "check": "npm run typecheck && npm run lint && npm run test",
+    "validate": "publint && attw"
+  },
+  "engines": { "node": ">=20.0.0", "bun": ">=1.0.0" },
+  "publishConfig": { "access": "public" }
+}
+```
+
+## 7. Fases de Execução
+
+### Fase 1: Limpeza (1 subagente)
+- Remover `src/`, `webscrape.exe`, `webscrape.ts`, `web/demo.html`
+- Limpar package.json de deps desnecessárias
+- Atualizar tsconfig.json
+
+### Fase 2: Core Infrastructure (5 subagentes paralelos)
+- `errors.ts` + `events.ts` + `types.ts` (com Zod schemas)
+- `retry.ts` + `rate-limiter.ts`
+- `proxy.ts` + `session.ts` + `headers.ts`
+- `cache.ts` (multi-camada com lru-cache + keyv)
+- `middleware.ts`
+
+### Fase 3: Core Scraping (4 subagentes paralelos)
+- `fetch.ts` (HTTP universal com Cheerio)
+- `browser.ts` (refatorar AgentBrowser)
+- `scrape.ts` (strategy cascade)
+- `extract.ts` (melhorar com Readability + JSON-LD)
+
+### Fase 4: Crawling & Batch (3 subagentes paralelos)
+- `crawler.ts` (BFS + depth limit)
+- `sitemap.ts` + `robots.ts`
+- `batch.ts` + `pool.ts`
+
+### Fase 5: AI/LLM (3 subagentes paralelos)
+- `agent.ts` (melhorar formatação)
+- `tokens.ts` + `chunking.ts`
+- `llms-txt.ts`
+
+### Fase 6: Server & CLI (2 subagentes paralelos)
+- `server.ts` (Hono API)
+- `cli.ts` (refatorar com commander)
+
+### Fase 7: Build & Config (1 subagente)
+- Atualizar package.json, vite.config, biome.json
+- Criar .husky/, .github/workflows/
+- Build e validação
+
+### Fase 8: Tests (2 subagentes paralelos)
+- Testes unitários para todos os novos módulos
+- Cobertura mínima 60%
+
+### Fase 9: Docs & Preview (1 subagente)
+- Atualizar README.md, API.md, ARCHITECTURE.md
+- Atualizar web/index.html com novas features
+- Criar CHANGELOG.md
+
+**Total: ~22 subagentes de implementação**
+
+## 8. Referências
+
+- Pesquisa completa em `docs/RESEARCH-01-anti-detection.md` até `docs/RESEARCH-15-html-parsing.md`
+- 18 subagentes de pesquisa executados
+- Dados coletados de npm, GitHub, documentação oficial, benchmarks 2026

package/docs/plans/44.reference.md

@@ -0,0 +1,100 @@
+# Saddle Robot Documentation Reference
+
+> Reference index for all saddle robot documentation.
+> Auto-generated summary of planning documents.
+
+## Project Overview
+
+**saddle** -- Super bot platform combining browser automation,
+scraping, captcha bypass, and multi-platform bot capability
+with computational memory architecture.
+
+## Documentation Structure
+
+### Planning Documents (`docs/plans`)
+
+| # | Document | Purpose |
+|---|----------|---------|
+| — | `00-indice-geral.md` | Primary index (original project) |
+| — | `action-plan.md` | Action plan and next steps |
+| 1 | `saddlerobotarchitecture.md` | Super bot architecture overview |
+| 2 | `saddledeploystrategy.md` | Deploy strategies for all platforms |
+| 3 | `saddlecomputationalmemory.md` | Storage-to-compute memory system |
+| 4 | `saddlemultiplatformbot.md` | Multi-platform bot engine |
+| 5 | `saddlenpmpublish.md` | NPM package publishing guide |
+| 6 | `saddlescdnintegration.md` | SCDN integration documentation |
+| — | `bots-codigo-revisao.md` | Best AI code review bots (research) |
+| — | `bots-automacao-computacional.md` | Best computational automation bots (research) |
+| — | `bots-seguranca-cicd.md` | Bot research (security + CI/CD) |
+
+### Historical Research
+
+| Document | Purpose |
+|----------|---------|
+| `RESEARCH-01-anti-detection.md` | Anti-detection techniques |
+| `RESEARCH-02-proxy.md` | Proxy configuration |
+| `RESEARCH-03-retry-rate-limit.md` | Retry and rate limiting |
+| `RESEARCH-04-crawling.md` | Web crawling strategies |
+| `RESEARCH-05-caching.md` | Caching patterns |
+| `RESEARCH-06-content-extraction.md` | Content extraction methods |
+| `RESEARCH-07-errors-events.md` | Error handling and events |
+| `RESEARCH-08-zod-validation.md` | Schema validation |
+| `RESEARCH-09-batch-concurrency.md` | Batch processing concurrency |
+| `RESEARCH-10-universal-runtime.md` | Universal runtime configuration |
+| `RESEARCH-11-ai-integration.md` | AI integration patterns |
+| `RESEARCH-12-memory-persistence.md` | Memory persistence strategies |
+| `RESEARCH-13-server-api.md` | Server API design |
+| `RESEARCH-14-compilation.md` | Compilation strategies |
+| `RESEARCH-15-html-parsing.md` | HTML parsing techniques |
+
+### Decision Transcripts (`docs/talks`)
+
+Architecture and design decision transcripts.
+
+### Logs (`docs/logs`)
+
+JSON movement logs from Brave capture sessions.
+
+## Key Concepts
+
+### Computational Memory
+
+Transform storage objects (files, blobs in repositories)
+into virtual RAM and GPU memory for processing. This
+enables effectively unlimited memory capacity.
+
+### Multi-Platform Bot
+
+Unified bot interface operating across GitHub, GitLab,
+Forgejo, Gitea, and Discord via adapter pattern.
+
+### Deploy Strategies
+
+Six deployment strategies: Netlify, Vercel, GitHub Actions,
+GitLab CI, Forgejo Actions, Gitea Actions, and Docker/VM
+self-hosted.
+
+### SCDN Integration
+
+Static Content Delivery Network for browser assets, capture
+files, and deployment payloads with edge caching.
+
+### NPM Publishing
+
+Package `@devthink/saddle` available as both library and
+CLI binary.
+
+## Architecture Rules (from arch skill)
+
+- No `src/` folder. Root based.
+- No deep nesting. Subfolders organized by domain only.
+- lowercase code, lowercase files. No uppercase. No underscore. No hyphen.
+- English docs. JSDoc comments. No emoji.
+- Logic grouped. Up to 20 related logics per file.
+- Error catcher embedded in every script.
+- No hardcoded hostnames. No localhost. Ports randomized then locked.
+- Library works as binary, CLI, headless, and browser.
+
+## License
+
+MIT