@wenathlan/saddle 1.8.17 → 2.0.2

package/docs/plans/43.plan.universal.architecture.md

@@ -1,496 +0,0 @@
-# Plano: Arquitetura Universal WebScrape v2.0
-
-## 1. Visão Geral
-
-A biblioteca `@wenathlan/webscrape` precisa funcionar em **TODOS os modos**:
-
-| Modo | Como funciona |
-|------|--------------|
-| **Com navegador** | Playwright (Chromium/Firefox/WebKit) para páginas SPA/JS |
-| **Sem navegador** | fetch + Cheerio para HTML estático |
-| **Com Node.js** | `node:fs`, `node:child_process`, SQLite |
-| **Sem Node.js** | Bun, Deno, edge runtimes, browsers |
-| **Com web** | API HTTP (Hono), WebSocket, SSE |
-| **Sem web** | CLI, binário compilado, biblioteca |
-| **Com IA** | Token counting, chunking semântico, llms.txt, RAG |
-| **Sem IA** | Scraping puro, extração de dados |
-| **Com memória** | Redis, SQLite, IndexedDB, Workers KV |
-| **Sem memória** | In-memory LRU, efêmero |
-| **Com desktop** | CLI TUI, Electron, Tauri |
-| **Sem desktop** | Web UI, serverless |
-| **Com servidor** | API server, worker pool |
-| **Sem servidor** | Standalone, binário compilado |
-| **Com script** | npm package, ESM/CJS |
-| **Sem script** | Bun compile binary, WASM |
-
-## 2. Estrutura de Diretórios
-
-```
-WebScrape/
-├── index.ts                    # Barrel principal (lib universal)
-├── types.ts                    # Tipos + Zod schemas
-├── errors.ts                   # Hierarquia de erros
-├── events.ts                   # Typed EventEmitter (hooks)
-│
-├── # ── Core Scraping ──
-├── fetch.ts                    # HTTP fetch universal (sem navegador)
-├── browser.ts                  # AgentBrowser (Playwright)
-├── scrape.ts                   # Strategy cascade: fetch → browser
-├── extract.ts                  # Extração HTML (Cheerio + Readability)
-├── serialize.ts                # Serialização multi-formato
-├── formats.ts                  # Registros de formato
-│
-├── # ── Infraestrutura ──
-├── retry.ts                    # Retry com backoff
-├── proxy.ts                    # Proxy rotation + pool
-├── rate-limiter.ts             # Rate limiting por domínio
-├── cache.ts                    # Cache multi-camada
-├── session.ts                  # Cookie jar + sessões
-│
-├── # ── Anti-Detection ──
-├── stealth.ts                  # Anti-detection (playwright-extra)
-├── fingerprint.ts              # Geração de fingerprints
-├── headers.ts                  # Rotação de headers/UA
-│
-├── # ── Crawling ──
-├── crawler.ts                  # BFS crawling com depth limit
-├── sitemap.ts                  # Parser de sitemap XML
-├── robots.ts                   # Parser de robots.txt
-│
-├── # ── AI/LLM ──
-├── agent.ts                    # Formatação para LLMs
-├── tokens.ts                   # Token counting multi-modelo
-├── chunking.ts                 # Chunking semântico
-├── llms-txt.ts                 # Geração de llms.txt
-│
-├── # ── Batch & Concurrency ──
-├── batch.ts                    # Batch scraping concorrente
-├── pool.ts                     # Browser pool
-│
-├── # ── Middleware ──
-├── middleware.ts                # Pipeline de middleware
-│
-├── # ── Utilitários ──
-├── utils.ts                    # Funções utilitárias
-├── url.ts                      # URL normalization
-│
-├── # ── CLI ──
-├── cli.ts                      # CLI binary (commander)
-│
-├── # ── Server ──
-├── server.ts                   # API HTTP (Hono)
-│
-├── # ── Renderer ──
-├── renderer.ts                 # Pygame bridge (opcional)
-│
-├── # ── Config ──
-├── package.json
-├── tsconfig.json
-├── vite.config.ts
-├── vitest.config.ts
-├── biome.json
-├── .npmrc
-├── .gitignore
-├── .husky/
-├── .github/workflows/
-│
-├── # ── Web UI ──
-├── web/
-│   └── index.html
-│
-├── # ── Docs ──
-├── docs/
-│   ├── README.md
-│   ├── API.md
-│   ├── ARCHITECTURE.md
-│   ├── CHANGELOG.md
-│   └── RESEARCH-*.md
-│
-└── # ── Tests ──
-    └── tests/
-        ├── fetch.test.ts
-        ├── browser.test.ts
-        ├── scrape.test.ts
-        ├── extract.test.ts
-        ├── serialize.test.ts
-        ├── retry.test.ts
-        ├── proxy.test.ts
-        ├── cache.test.ts
-        ├── crawler.test.ts
-        ├── batch.test.ts
-        ├── agent.test.ts
-        ├── tokens.test.ts
-        ├── chunking.test.ts
-        ├── errors.test.ts
-        ├── events.test.ts
-        ├── middleware.test.ts
-        └── utils.test.ts
-```
-
-## 3. Strategy Cascade Architecture
-
-```
-scrape(url, options)
-    │
-    ├─── options.mode === 'fetch' ──→ fetch.ts (HTTP + Cheerio)
-    │
-    ├─── options.mode === 'browser' ──→ browser.ts (Playwright)
-    │
-    └─── options.mode === 'auto' (default) ──→ scrape.ts
-         │
-         ├── 1. Tentar fetch (rápido, barato)
-         ├── 2. Detectar se precisa de JS rendering
-         │      ├── Estático → usar resultado do fetch
-         │      └── Dinâmico → usar Playwright
-         └── 3. Return resultado mais limpo
-```
-
-## 4. Módulos Novos — Especificações
-
-### 4.1 errors.ts — Hierarquia de Erros
-
-```typescript
-WebScrapeError (base)
-├── ValidationError (400)
-├── TimeoutError (504)
-├── BlockedError (403)
-├── RateLimitError (429)
-├── ProxyError (502)
-├── ParseError (422)
-├── AuthError (401)
-└── NetworkError (503)
-```
-
-- Cada erro tem `code`, `statusCode`, `isRetryable`, `details`
-- `Object.setPrototypeOf` para `instanceof` correto
-- Recovery suggestions em cada erro
-
-### 4.2 events.ts — Hooks de Lifecycle
-
-```typescript
-interface WebScrapeEvents {
-  'request:start':    [url: string, options: RequestOptions]
-  'request:response': [response: Response, attempt: number]
-  'request:error':    [error: Error, attempt: number]
-  'request:retry':    [attempt: number, maxRetries: number]
-  'request:complete': [stats: CrawlStats]
-  'cache:hit':        [key: string]
-  'cache:miss':       [key: string]
-  'proxy:rotate':     [proxy: string, reason: string]
-  'proxy:error':      [proxy: string, error: Error]
-  'crawl:discover':   [url: string, depth: number]
-  'crawl:complete':   [stats: CrawlStats]
-}
-```
-
-- Usa `emittery` (typed, async-first, AbortSignal, Symbol.dispose)
-- Substitui EventEmitter padrão do Node
-
-### 4.3 fetch.ts — HTTP Universal
-
-- Usa `globalThis.fetch` (disponível em TODOS os runtimes 2026)
-- Suporta: cookies, headers rotation, proxy, timeout, retry
-- Detecta se a página precisa de JS rendering
-- Suporta HTTP/2 via undici (quando disponível)
-- Cookie management via tough-cookie
-
-### 4.4 proxy.ts — Proxy Rotation
-
-- Pool de proxies com health checking
-- Estratégias: round-robin, random, sticky session
-- Auto-disable em proxies mortos (3 failures consecutivas)
-- Cooldown com revive automático
-- Suporta: HTTP, HTTPS, SOCKS5
-
-### 4.5 cache.ts — Cache Multi-Camada
-
-- L1: In-memory LRU (lru-cache v11)
-- L2: Persistent (keyv com adapter: SQLite/Redis/IndexedDB)
-- TTL por tipo de conteúdo:
-  - Estático (about, policies): 24-72h
-  - Semi-estático (produtos): 1-6h
-  - Dinâmico (search): 30s-5min
-- ETag/Last-Modified conditional GET
-
-### 4.6 chunking.ts — Chunking Semântico
-
-- Split por headings Markdown (H1/H2/H3)
-- Overlap de 10-20%
-- Max 512 tokens por chunk
-- Preserva código blocks inteiros
-- Inclui heading path em cada chunk
-- Hierarchical parent-child para RAG
-
-### 4.7 tokens.ts — Token Counting Multi-Modelo
-
-- OpenAI: js-tiktoken (o200k_base)
-- Claude: API ou approximação (p50k_base ±10-30%)
-- Gemini: API ou bpe-lite (100% accuracy)
-- Fallback: estimateTokens (chars/4)
-- Edge runtime compatible
-
-### 4.8 llms-txt.ts — Geração de llms.txt
-
-- Extrai headings, links, descriptions de páginas
-- Gera formato padronizado (llmstxt.org)
-- Suporta llms-full.txt (conteúdo concatenado)
-- Validação: exato H1, blockquote summary, URLs absolutas
-
-### 4.9 middleware.ts — Pipeline
-
-- Koa-style onion model
-- Cada middleware recebe `ctx` e `next`
-- Middleware para: auth, rate-limit, retry, cache, logging
-- Extensível via plugins
-
-### 4.10 server.ts — API HTTP
-
-- Hono framework (multi-runtime)
-- Endpoints:
-  - `POST /v1/scrape` — sync scrape
-  - `POST /v1/scrape/async` — async scrape
-  - `GET /v1/scrape/:taskId` — get result
-  - `POST /v1/batch` — batch scrape
-  - `POST /v1/crawl` — start crawl
-  - `GET /health` — health check
-- SSE para progress updates
-- Rate limiting por API key
-- OpenAPI spec gerado
-
-## 5. Dependências
-
-### Production (adicionar)
-
-| Pacote | Versão | Propósito |
-|--------|--------|-----------|
-| `zod` | ^4.4.3 | Validação runtime |
-| `p-retry` | ^8.0.0 | Retry com backoff |
-| `p-limit` | ^7.3.0 | Controle concorrência |
-| `p-queue` | ^9.2.0 | Fila com prioridade |
-| `robots-parser` | ^3.0.1 | Parser robots.txt |
-| `sitemapper` | ^4.1.6 | Parser sitemap XML |
-| `turndown` | ^7.2.4 | HTML → Markdown |
-| `turndown-plugin-gfm` | ^1.0.2 | GFM tables |
-| `@mozilla/readability` | ^0.6.0 | Extração conteúdo principal |
-| `lru-cache` | ^11.0.0 | Cache LRU |
-| `keyv` | ^5.6.0 | KV store cross-runtime |
-| `hono` | ^4.0.0 | HTTP server |
-| `commander` | ^14.0.3 | CLI |
-| `emittery` | ^2.0.0 | Typed EventEmitter |
-| `normalize-url` | ^9.0.0 | URL normalization |
-
-### Production (manter)
-
-| Pacote | Nota |
-|--------|------|
-| `playwright` | ^1.60.0 |
-| `cheerio` | ^1.2.0 |
-| `jsdom` | ^29.1.1 |
-| `chalk` | ^5.6.2 |
-
-### Production (REMOVER)
-
-| Pacote | Motivo |
-|--------|--------|
-| `@inquirer/prompts` | UI interativa, não core |
-| `@inkjs/ui` | Ink components |
-| `@inquirer/core` | UI interativa |
-| `@opencode-ai/plugin` | Plugin específico |
-| `@types/blessed` | TUI blessed |
-| `better-sqlite3` | Mover para optional |
-| `blessed` | TUI framework |
-| `cfonts` | ASCII art fonts |
-| `chalk-animation` | CLI animations |
-| `cli-table3` | CLI tables |
-| `cross-spawn` | Subprocess spawning |
-| `figlet` | ASCII art |
-| `fuzzysort` | Fuzzy matching |
-| `gradient-string` | CLI gradient |
-| `hls.js` | Video streaming |
-| `ink` | React TUI |
-| `ink-big-text` | Ink component |
-| `ink-gradient` | Ink component |
-| `ink-link` | Ink component |
-| `listr2` | Task runner |
-| `marked` | MD parser (usar turndown) |
-| `ora` | Spinner |
-| `react` | UI framework |
-| `react-devtools-core` | Dev tools |
-| `react-reconciler` | React internals |
-| `scheduler` | React scheduler |
-| `sharp` | Image processing |
-
-### Optional (novo)
-
-| Pacote | Versão | Propósito |
-|--------|--------|-----------|
-| `playwright` | ^1.60.0 | Opcional para quem só quer fetch |
-| `playwright-extra` | ^1.7.9 | Stealth plugin |
-| `puppeteer-extra-plugin-stealth` | ^2.11.2 | Evasion modules |
-
-### Dev Dependencies (adicionar)
-
-| Pacote | Propósito |
-|--------|-----------|
-| `unplugin-dts` | Gerar .d.ts (sucessor vite-plugin-dts) |
-| `@arethetypeswrong/cli` | Validar tipos |
-| `publint` | Validar package.json |
-| `husky` ^9.0.0 | Git hooks |
-| `lint-staged` ^15.0.0 | Lint staged files |
-
-## 6. Configurações de Build
-
-### tsconfig.json
-
-```json
-{
-  "compilerOptions": {
-    "target": "ES2024",
-    "module": "ESNext",
-    "moduleResolution": "bundler",
-    "lib": ["ES2024", "DOM", "DOM.Iterable"],
-    "strict": true,
-    "declaration": true,
-    "declarationMap": true,
-    "sourceMap": true,
-    "outDir": "./dist",
-    "rootDir": ".",
-    "noEmit": true,
-    "isolatedModules": true,
-    "esModuleInterop": true,
-    "skipLibCheck": true,
-    "verbatimModuleSyntax": true,
-    "types": ["node"]
-  },
-  "include": ["*.ts"],
-  "exclude": ["node_modules", "dist", "web", "tests"]
-}
-```
-
-### vite.config.ts
-
-```typescript
-import { defineConfig } from "vite";
-import dts from "vite-plugin-dts";
-import { resolve } from "path";
-
-export default defineConfig({
-  root: "web",
-  build: {
-    lib: {
-      entry: resolve(__dirname, "index.ts"),
-      name: "DevThinkWebScrape",
-      formats: ["es", "cjs"],
-      fileName: (format) => `index.${format === "es" ? "mjs" : "js"}`,
-    },
-    rollupOptions: {
-      external: [
-        "playwright", "cheerio", "jsdom",
-        "zod", "p-retry", "p-limit", "p-queue",
-        "robots-parser", "sitemapper",
-        "turndown", "turndown-plugin-gfm",
-        "@mozilla/readability", "lru-cache", "keyv",
-        "hono", "commander", "emittery", "normalize-url",
-        "node:fs", "node:path", "node:child_process", "node:http",
-      ],
-    },
-  },
-  plugins: [dts()],
-  server: { port: 5175, host: "127.0.0.1", cors: true },
-});
-```
-
-### package.json (atualizado)
-
-```json
-{
-  "name": "@wenathlan/webscrape",
-  "version": "2.0.0",
-  "type": "module",
-  "main": "./dist/index.js",
-  "types": "./dist/index.d.ts",
-  "bin": { "webscrape": "./dist/cli.js" },
-  "exports": {
-    ".":         { "types": "./dist/index.d.ts", "import": "./dist/index.js" },
-    "./fetch":   { "types": "./dist/fetch.d.ts", "import": "./dist/fetch.js" },
-    "./browser": { "types": "./dist/browser.d.ts", "import": "./dist/browser.js" },
-    "./scrape":  { "types": "./dist/scrape.d.ts", "import": "./dist/scrape.js" },
-    "./agent":   { "types": "./dist/agent.d.ts", "import": "./dist/agent.js" },
-    "./server":  { "types": "./dist/server.d.ts", "import": "./dist/server.js" },
-    "./cli":     { "import": "./dist/cli.js" },
-    "./types":   { "types": "./dist/types.d.ts", "import": "./dist/types.js" }
-  },
-  "files": ["dist", "web/", "docs/", "README.md"],
-  "sideEffects": false,
-  "scripts": {
-    "build": "tsc --noEmit && vite build",
-    "dev": "vite",
-    "test": "vitest run",
-    "test:watch": "vitest",
-    "test:coverage": "vitest run --coverage",
-    "lint": "biome check .",
-    "lint:fix": "biome check --write .",
-    "typecheck": "tsc --noEmit",
-    "check": "npm run typecheck && npm run lint && npm run test",
-    "validate": "publint && attw"
-  },
-  "engines": { "node": ">=20.0.0", "bun": ">=1.0.0" },
-  "publishConfig": { "access": "public" }
-}
-```
-
-## 7. Fases de Execução
-
-### Fase 1: Limpeza (1 subagente)
-- Remover `src/`, `webscrape.exe`, `webscrape.ts`, `web/demo.html`
-- Limpar package.json de deps desnecessárias
-- Atualizar tsconfig.json
-
-### Fase 2: Core Infrastructure (5 subagentes paralelos)
-- `errors.ts` + `events.ts` + `types.ts` (com Zod schemas)
-- `retry.ts` + `rate-limiter.ts`
-- `proxy.ts` + `session.ts` + `headers.ts`
-- `cache.ts` (multi-camada com lru-cache + keyv)
-- `middleware.ts`
-
-### Fase 3: Core Scraping (4 subagentes paralelos)
-- `fetch.ts` (HTTP universal com Cheerio)
-- `browser.ts` (refatorar AgentBrowser)
-- `scrape.ts` (strategy cascade)
-- `extract.ts` (melhorar com Readability + JSON-LD)
-
-### Fase 4: Crawling & Batch (3 subagentes paralelos)
-- `crawler.ts` (BFS + depth limit)
-- `sitemap.ts` + `robots.ts`
-- `batch.ts` + `pool.ts`
-
-### Fase 5: AI/LLM (3 subagentes paralelos)
-- `agent.ts` (melhorar formatação)
-- `tokens.ts` + `chunking.ts`
-- `llms-txt.ts`
-
-### Fase 6: Server & CLI (2 subagentes paralelos)
-- `server.ts` (Hono API)
-- `cli.ts` (refatorar com commander)
-
-### Fase 7: Build & Config (1 subagente)
-- Atualizar package.json, vite.config, biome.json
-- Criar .husky/, .github/workflows/
-- Build e validação
-
-### Fase 8: Tests (2 subagentes paralelos)
-- Testes unitários para todos os novos módulos
-- Cobertura mínima 60%
-
-### Fase 9: Docs & Preview (1 subagente)
-- Atualizar README.md, API.md, ARCHITECTURE.md
-- Atualizar web/index.html com novas features
-- Criar CHANGELOG.md
-
-**Total: ~22 subagentes de implementação**
-
-## 8. Referências
-
-- Pesquisa completa em `docs/RESEARCH-01-anti-detection.md` até `docs/RESEARCH-15-html-parsing.md`
-- 18 subagentes de pesquisa executados
-- Dados coletados de npm, GitHub, documentação oficial, benchmarks 2026

package/docs/plans/44.reference.md

@@ -1,100 +0,0 @@
-# Saddle Robot Documentation Reference
-
-> Reference index for all saddle robot documentation.
-> Auto-generated summary of planning documents.
-
-## Project Overview
-
-**saddle** -- Super bot platform combining browser automation,
-scraping, captcha bypass, and multi-platform bot capability
-with computational memory architecture.
-
-## Documentation Structure
-
-### Planning Documents (`docs/plans`)
-
-| # | Document | Purpose |
-|---|----------|---------|
-| — | `00-indice-geral.md` | Primary index (original project) |
-| — | `action-plan.md` | Action plan and next steps |
-| 1 | `saddlerobotarchitecture.md` | Super bot architecture overview |
-| 2 | `saddledeploystrategy.md` | Deploy strategies for all platforms |
-| 3 | `saddlecomputationalmemory.md` | Storage-to-compute memory system |
-| 4 | `saddlemultiplatformbot.md` | Multi-platform bot engine |
-| 5 | `saddlenpmpublish.md` | NPM package publishing guide |
-| 6 | `saddlescdnintegration.md` | SCDN integration documentation |
-| — | `bots-codigo-revisao.md` | Best AI code review bots (research) |
-| — | `bots-automacao-computacional.md` | Best computational automation bots (research) |
-| — | `bots-seguranca-cicd.md` | Bot research (security + CI/CD) |
-
-### Historical Research
-
-| Document | Purpose |
-|----------|---------|
-| `RESEARCH-01-anti-detection.md` | Anti-detection techniques |
-| `RESEARCH-02-proxy.md` | Proxy configuration |
-| `RESEARCH-03-retry-rate-limit.md` | Retry and rate limiting |
-| `RESEARCH-04-crawling.md` | Web crawling strategies |
-| `RESEARCH-05-caching.md` | Caching patterns |
-| `RESEARCH-06-content-extraction.md` | Content extraction methods |
-| `RESEARCH-07-errors-events.md` | Error handling and events |
-| `RESEARCH-08-zod-validation.md` | Schema validation |
-| `RESEARCH-09-batch-concurrency.md` | Batch processing concurrency |
-| `RESEARCH-10-universal-runtime.md` | Universal runtime configuration |
-| `RESEARCH-11-ai-integration.md` | AI integration patterns |
-| `RESEARCH-12-memory-persistence.md` | Memory persistence strategies |
-| `RESEARCH-13-server-api.md` | Server API design |
-| `RESEARCH-14-compilation.md` | Compilation strategies |
-| `RESEARCH-15-html-parsing.md` | HTML parsing techniques |
-
-### Decision Transcripts (`docs/talks`)
-
-Architecture and design decision transcripts.
-
-### Logs (`docs/logs`)
-
-JSON movement logs from Brave capture sessions.
-
-## Key Concepts
-
-### Computational Memory
-
-Transform storage objects (files, blobs in repositories)
-into virtual RAM and GPU memory for processing. This
-enables effectively unlimited memory capacity.
-
-### Multi-Platform Bot
-
-Unified bot interface operating across GitHub, GitLab,
-Forgejo, Gitea, and Discord via adapter pattern.
-
-### Deploy Strategies
-
-Six deployment strategies: Netlify, Vercel, GitHub Actions,
-GitLab CI, Forgejo Actions, Gitea Actions, and Docker/VM
-self-hosted.
-
-### SCDN Integration
-
-Static Content Delivery Network for browser assets, capture
-files, and deployment payloads with edge caching.
-
-### NPM Publishing
-
-Package `@wenathlan/saddle` available as both library and
-CLI binary.
-
-## Architecture Rules (from arch skill)
-
-- No `src/` folder. Root based.
-- No deep nesting. Subfolders organized by domain only.
-- lowercase code, lowercase files. No uppercase. No underscore. No hyphen.
-- English docs. JSDoc comments. No emoji.
-- Logic grouped. Up to 20 related logics per file.
-- Error catcher embedded in every script.
-- No hardcoded hostnames. No localhost. Ports randomized then locked.
-- Library works as binary, CLI, headless, and browser.
-
-## License
-
-MIT