@wenathlan/saddle 1.8.2 → 1.8.5

package/docs/plans/25.research.server.api.md

@@ -0,0 +1,342 @@
+# Pesquisa: Server, API e Distribuição (2026)
+
+## 1. Framework Comparison
+
+### Performance Benchmarks (2026)
+
+| Framework | Runtime | req/s (Node.js) | req/s (Bun) | Bundle Size | TypeScript | GitHub Stars | npm Weekly Downloads |
+|-----------|---------|-----------------|-------------|-------------|------------|--------------|---------------------|
+| **Elysia** | Bun-native | N/A | ~184K (peak) | ~30 KB | Native (best) | ~12K | ~300K-700K |
+| **Hono** | Multi-runtime | ~45K-78K | ~55K-62K | ~12-15 KB | First-class | ~22-28K | ~1.2-9.3M |
+| **Fastify** | Node.js | ~42K-114K | ~85K | ~50 KB+ | First-class | ~33-36K | ~4-11M |
+| **Express** | Node.js | ~8K-21K | ~15-21K | ~200 KB+ | Via @types | ~65-69K | ~30-35M |
+
+### Recommendation Matrix for Scraping APIs
+
+| Use Case | Top Pick | Runner-up | Reason |
+|----------|----------|-----------|--------|
+| **Cloudflare Workers / Edge API** | Hono | — | Native support, 14 KB bundle, runs everywhere |
+| **New Node.js REST API** | Fastify / Hono | Fastify | 3-10x faster than Express, schema validation |
+| **Bun production API** | Elysia | Hono | Peak Bun throughput, end-to-end type safety |
+| **Existing Express codebase** | Express | Fastify | Lowest migration cost, largest ecosystem |
+| **Multi-runtime / portable** | Hono | — | Same code on Node, Bun, Deno, CF Workers, Lambda |
+| **Enterprise large-scale** | Fastify | Express | Plugin system, stability, 300+ plugins |
+
+### Framework-Specific Middleware
+
+| Capability | Express | Fastify | Hono | Elysia |
+|-----------|---------|---------|------|--------|
+| CORS | `cors` (npm) | `@fastify/cors` | Built-in `hono/cors` | `@elysiajs/cors` |
+| Rate limiting | `express-rate-limit` | `@fastify/rate-limit` | `hono-rate-limiter` | Custom |
+| JWT Auth | `passport`, `jsonwebtoken` | `@fastify/jwt` | `hono/jwt` | `@elysiajs/jwt` |
+| WebSocket | `ws` (standalone) | `@fastify/websocket` | `hono/ws` | Native |
+| OpenAPI/Swagger | `swagger-jsdoc` | `@fastify/swagger` | `@hono/swagger-ui` | `@elysiajs/swagger` |
+
+## 2. Rate Limiting on the API Layer
+
+### Packages by Framework
+
+| Package | Framework | Algorithm |
+|---------|-----------|-----------|
+| `express-rate-limit` 8.5.x | Express | Fixed window, pluggable stores |
+| `@fastify/rate-limit` | Fastify | Fixed window, Redis-backed |
+| `hono-rate-limiter` 0.5.x | Hono | Fixed window, Redis/CF KV |
+| `universal-rate-limit` | All | Fixed/sliding window, token bucket |
+
+### Hono Rate Limiting Example
+
+```typescript
+import { rateLimiter } from 'hono-rate-limiter';
+import { RedisStore } from '@hono-rate-limiter/redis';
+import { Redis } from '@upstash/redis';
+
+const client = new Redis({ url: REDIS_URL, token: REDIS_TOKEN });
+
+app.use('/api/*', rateLimiter({
+  windowMs: 60_000,
+  limit: 100,
+  store: new RedisStore({ client }),
+  keyGenerator: (c) => c.var.accountId,
+  standardHeaders: 'draft-7',
+}));
+```
+
+### Tiered Rate Limiting
+
+```typescript
+// Global limit
+app.use(rateLimiter({ windowMs: 60_000, limit: 1000 }));
+
+// Per-endpoint for expensive operations
+app.post('/v1/scrape', rateLimiter({
+  windowMs: 60_000,
+  limit: 10,
+  keyGenerator: (c) => c.var.userId,
+}));
+```
+
+## 3. Streaming Responses
+
+### Chunked Transfer Encoding
+
+```typescript
+app.get('/api/export', async (req, res) => {
+  res.writeHead(200, {
+    'Content-Type': 'application/json',
+    'Transfer-Encoding': 'chunked',
+  });
+  res.write('[\n');
+  let first = true;
+  for await (const batch of scraper.streamResults(url)) {
+    for (const row of batch) {
+      res.write(first ? '' : ',\n');
+      res.write(JSON.stringify(row));
+      first = false;
+    }
+    await new Promise(resolve => setImmediate(resolve));
+  }
+  res.write('\n]');
+  res.end();
+});
+```
+
+### Server-Sent Events (SSE)
+
+```typescript
+app.get('/api/scrape/stream', async (req, res) => {
+  res.setHeader('Content-Type', 'text/event-stream');
+  res.setHeader('Cache-Control', 'no-cache');
+  res.setHeader('Connection', 'keep-alive');
+  res.setHeader('X-Accel-Buffering', 'no');
+  res.flushHeaders();
+
+  const scraper = scrapeWithProgress(url);
+  let eventId = 0;
+
+  for await (const event of scraper) {
+    res.write(`id: ${++eventId}\n`);
+    res.write(`event: ${event.type}\n`);
+    res.write(`data: ${JSON.stringify(event.data)}\n\n`);
+  }
+
+  res.write('event: done\ndata: {"status":"complete"}\n\n');
+  res.end();
+  req.on('close', () => scraper.abort());
+});
+```
+
+### SSE Gotchas
+- Nginx buffers by default — add `proxy_buffering off;` and `X-Accel-Buffering: no`
+- Cloudflare free tier has 100s idle timeout — send `: keepalive\n\n` every 30s
+- Browsers limit 6 concurrent SSE connections per origin (HTTP/1.1)
+- Skip streaming for responses under 50KB
+
+## 4. WebSocket for Real-Time Progress
+
+```typescript
+import { WebSocketServer } from 'ws';
+
+const wss = new WebSocketServer({ port: 3001 });
+
+wss.on('connection', (ws) => {
+  ws.isAlive = true;
+  ws.on('pong', () => { ws.isAlive = true; });
+  ws.on('message', (data) => {
+    const msg = JSON.parse(data);
+    if (msg.type === 'subscribe') ws.jobId = msg.jobId;
+  });
+});
+
+// Heartbeat
+setInterval(() => {
+  wss.clients.forEach(ws => {
+    if (!ws.isAlive) return ws.terminate();
+    ws.isAlive = false;
+    ws.ping();
+  });
+}, 30_000);
+
+function broadcastJobUpdate(jobId: string, data: unknown) {
+  wss.clients.forEach(ws => {
+    if (ws.jobId === jobId && ws.readyState === 1) {
+      ws.send(JSON.stringify({ type: 'job_update', ...data }));
+    }
+  });
+}
+```
+
+## 5. Authentication
+
+### API Key Pattern
+
+```typescript
+async function verifyApiKey(request: any, reply: any) {
+  const apiKey = request.headers['x-api-key'];
+  if (!apiKey) return reply.code(401).send({ error: 'API key required' });
+  const user = await redis.hgetall(`apikey:${apiKey}`);
+  if (!user.user_id) return reply.code(403).send({ error: 'Invalid API key' });
+  request.user = { userId: user.user_id };
+}
+```
+
+### JWT Pattern
+
+```typescript
+import jwt from 'jsonwebtoken';
+
+function validateJwt(request: any, reply: any) {
+  const token = request.headers.authorization?.replace('Bearer ', '');
+  if (!token) return reply.code(401).send({ error: 'Token required' });
+  try {
+    request.user = jwt.verify(token, SECRET, { algorithms: ['HS256'] });
+  } catch (e) {
+    return reply.code(401).send({ error: 'Invalid token' });
+  }
+}
+```
+
+## 6. REST API Design
+
+### Standard Endpoints
+
+```
+POST   /v1/scrape                  # Sync scrape
+POST   /v1/scrape/async            # Async scrape (returns task ID)
+GET    /v1/scrape/:taskId          # Get async job result
+POST   /v1/batch                   # Batch scrape multiple URLs
+POST   /v1/crawl                   # Start recursive crawl
+GET    /v1/crawl/:crawlId          # Get crawl status/results
+GET    /health                     # Health check
+```
+
+### Request/Response Schema
+
+```json
+// POST /v1/scrape
+{
+  "url": "https://example.com",
+  "useBrowser": false,
+  "format": "markdown",
+  "waitFor": "h1",
+  "timeout": 30,
+  "proxy": "http://user:pass@proxy:port"
+}
+
+// Response
+{
+  "status": "ok",
+  "data": {
+    "url": "https://example.com",
+    "markdown": "# Content...",
+    "metadata": { "title": "...", "description": "..." },
+    "links": [...],
+    "images": [...]
+  },
+  "timing": { "duration_ms": 1234 }
+}
+```
+
+### Error Response Format
+
+```json
+{
+  "error": {
+    "code": "RATE_LIMIT_EXCEEDED",
+    "message": "Rate limit exceeded. Try again in 60 seconds.",
+    "retry_after": 60,
+    "request_id": "req_abc123"
+  }
+}
+```
+
+## 7. Docker Deployment
+
+### Multi-Stage Dockerfile
+
+```dockerfile
+# syntax=docker/dockerfile:1.4
+FROM node:22-alpine AS deps
+WORKDIR /app
+COPY package.json package-lock.json ./
+RUN npm ci --frozen-lockfile
+
+FROM node:22-alpine AS builder
+WORKDIR /app
+COPY --from=deps /app/node_modules ./node_modules
+COPY . .
+RUN npm run build
+
+FROM node:22-alpine AS runner
+WORKDIR /app
+RUN addgroup --system --gid 1001 nodejs && adduser --system --uid 1001 appuser
+ENV NODE_ENV=production
+COPY package.json package-lock.json ./
+RUN npm ci --frozen-lockfile --omit=dev && npm cache clean --force
+COPY --from=builder /app/dist ./dist
+RUN chown -R appuser:nodejs /app
+USER appuser
+EXPOSE 3000
+HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
+    CMD node -e "require('http').get('http://localhost:3000/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1)).on('error', () => process.exit(1))"
+CMD ["node", "--enable-source-maps", "dist/index.js"]
+```
+
+### Docker Compose
+
+```yaml
+services:
+  api:
+    build: .
+    ports: ["3000:3000"]
+    environment:
+      - DATABASE_URL=postgresql://user:pass@postgres:5432/scraper
+      - REDIS_URL=redis://redis:6379
+    depends_on: [postgres, redis]
+
+  postgres:
+    image: postgres:16-alpine
+    volumes: [pgdata:/var/lib/postgresql/data]
+    environment:
+      POSTGRES_DB: scraper
+      POSTGRES_USER: user
+      POSTGRES_PASSWORD: pass
+
+  redis:
+    image: redis:7-alpine
+    volumes: [redis_data:/data]
+
+volumes:
+  pgdata:
+  redis_data:
+```
+
+## 8. Serverless Deployment
+
+| Platform | Best For | Notes |
+|----------|----------|-------|
+| AWS Lambda (container) | Event-driven scraping | Up to 15 min timeout |
+| Google Cloud Run | Serverless pipeline | Separate scraper + API services |
+| Cloudflare Workers | Edge scraping API | Hono is best choice, 14 KB bundle |
+| Fly.io | Persistent containers | Good for browser-based scraping |
+| Render | Full-stack | Postgres + Redis included |
+
+## 9. Observability
+
+- Structured JSON logging (Pino for Fastify, built-in for Hono)
+- OpenTelemetry for distributed tracing
+- Metrics: request count, latency p50/p95/p99, scrape success rate
+- Per-user usage dashboards
+- Alerting on anomalous patterns
+
+## 10. Security Checklist
+
+- [ ] HTTPS everywhere
+- [ ] API key or JWT authentication
+- [ ] Rate limiting per user/API key
+- [ ] Input validation on all endpoints
+- [ ] SSRF protection (prevent scraping internal networks)
+- [ ] CORS configured properly
+- [ ] No secrets in Docker images
+- [ ] Request ID tracking for audit trails
+- [ ] Respect robots.txt
+- [ ] Data retention limits and deletion schedules

package/docs/plans/26.research.compilation.md

@@ -0,0 +1,249 @@
+# Pesquisa: Compilação e Build de Bibliotecas (2026)
+
+## 1. Bun `--compile` — Standalone Executables
+
+Bun's bundler has a mature `--compile` flag that generates standalone binaries with the Bun runtime embedded.
+
+### Basic Usage
+
+```bash
+bun build ./cli.ts --compile --outfile mycli
+./mycli
+```
+
+### Cross-Compilation Targets
+
+| Target | OS | Arch |
+|--------|-----|------|
+| `bun-linux-x64` | Linux | x64 (glibc) |
+| `bun-linux-arm64` | Linux | arm64 |
+| `bun-windows-x64` | Windows | x64 (.exe) |
+| `bun-windows-arm64` | Windows | arm64 |
+| `bun-darwin-x64` | macOS | x64 |
+| `bun-darwin-arm64` | macOS | arm64 |
+| `bun-linux-x64-musl` | Linux | x64 (Alpine) |
+| `bun-linux-arm64-musl` | Linux | arm64 (musl) |
+
+### Advanced Features
+
+```bash
+# Bytecode compilation (2x faster startup)
+bun build --compile --bytecode ./app.ts --outfile myapp
+
+# Cross-compile
+bun build --compile --target=bun-linux-x64 ./src/cli.ts --outfile myapp-linux
+
+# Embed files
+bun build --compile ./index.ts ./public/**/*.png
+
+# Build-time constants
+bun build --compile --define BUILD_VERSION='"1.2.3"' ./app.ts --outfile myapp
+
+# Minification + sourcemaps
+bun build --compile --minify --sourcemap ./app.ts --outfile myapp
+```
+
+### Key Features (2026)
+- Build-time constants via `--define`
+- Bytecode compilation for fast startup
+- Embed files/assets with `with { type: "file" }` import attributes
+- Embed SQLite databases with `with { type: "sqlite", embed: "true" }`
+- Embed N-API addons (`.node` files) directly
+- Full-stack executables (v1.2.17+): embed HTML/CSS/JS frontend
+- Windows metadata (icon, hideConsole, title)
+- macOS codesigning support
+
+## 2. tsup vs tsdown
+
+### tsup (legacy, v8.5.1 — no longer actively maintained)
+
+```typescript
+// tsup.config.ts
+import { defineConfig } from 'tsup'
+export default defineConfig({
+  entry: ['src/index.ts', 'src/cli.ts'],
+  format: ['cjs', 'esm'],
+  dts: true,
+  splitting: true,
+  clean: true,
+  minify: true,
+})
+```
+
+### tsdown (recommended successor, Rolldown-based)
+
+```bash
+npm i tsdown -D
+npx tsdown-migrate  # migrate from tsup
+```
+
+| Option | tsup | tsdown |
+|--------|------|--------|
+| Default format | `'cjs'` | `'esm'` |
+| Clean | `false` | `true` |
+| DTS | `false` | Auto-enabled |
+| Target | none | Auto-reads from `engines.node` |
+| Backend | esbuild | Rolldown (Rust) |
+
+### tsdown Unique Features
+- **`exe: true`** — bundle as Node.js SEA
+- **`exports: true`** — auto-generate `package.json` exports
+- **`publint`/`attw`** — validate package
+- **`devtools`** — Vite DevTools integration
+- **Full CSS pipeline** with preprocessors
+- **Lifecycle hooks**: `build:prepare`, `build:before`, `build:done`
+
+```typescript
+// tsdown.config.ts
+import { defineConfig } from 'tsdown'
+export default defineConfig({
+  entry: ['src/index.ts', 'src/cli.ts'],
+  format: ['cjs', 'esm'],
+  dts: true,
+  exe: false,
+  minify: true,
+  exports: true,
+  publint: true,
+})
+```
+
+## 3. Node.js SEA (Single Executable Applications)
+
+### Step-by-Step Process
+
+```bash
+# 1. Write your app
+echo 'console.log("Hello!");' > hello.js
+
+# 2. Bundle to single CJS file (esbuild/ncc/webpack)
+npx esbuild hello.ts --bundle --platform=node --outfile=hello.js --format=cjs
+
+# 3. Create sea-config.json
+echo '{"main":"hello.js","output":"sea-prep.blob","disableExperimentalSEAWarning":true}' > sea-config.json
+
+# 4. Generate the blob
+node --experimental-sea-config sea-config.json
+
+# 5. Copy node binary
+cp $(command -v node) hello
+
+# 6. Remove signature (macOS)
+codesign --remove-signature hello
+
+# 7. Inject blob with postject
+npx postject hello NODE_SEA_BLOB sea-prep.blob \
+    --sentinel-fuse NODE_SEA_FUSE_fce680ab2cc467b6e072b8b5df1996b2
+
+# 8. Re-sign (macOS)
+codesign --sign - hello
+
+# 9. Run
+./hello
+```
+
+### SEA v22+ Features
+- **Assets** — embed files via `"assets"` config field
+- **Startup snapshots** — `"useSnapshot": true`
+- **V8 code cache** — `"useCodeCache": true`
+- **Embedded `execArgv`** — `"execArgv": ["--no-warnings"]`
+
+**Key limitation:** SEA only supports CommonJS. Must bundle everything into a single JS file first.
+
+## 4. esbuild for Fast Bundling
+
+```bash
+# Browser bundle
+esbuild app.jsx --bundle --minify --sourcemap --target=chrome58 --outfile=out.js
+
+# Node.js bundle
+esbuild app.js --bundle --platform=node --target=node18 --outfile=out.js
+
+# Leave dependencies external (for libraries)
+esbuild app.jsx --bundle --platform=node --packages=external --outfile=out.js
+```
+
+### Key Features
+- Extremely fast (written in Go, parallelized)
+- TypeScript/JSX/TSX out of the box
+- CSS bundling support
+- Plugin system (`onResolve`, `onLoad`, `onStart`, `onEnd`)
+- WASM version available (`esbuild-wasm`)
+
+## 5. Creating CLI Tools Without Node.js
+
+### Options Ranked
+
+1. **Bun `--compile`** (recommended): True cross-compilation, supports Bun + Node.js APIs
+2. **Node.js SEA**: Embeds full Node.js runtime, CJS only
+3. **Deno compile**: Alternative, good for Deno-first projects
+
+## 6. Native Dependencies in Compiled Binaries
+
+### Strategies
+
+1. **Replace with pure-JS alternatives:**
+   - `better-sqlite3` → `bun:sqlite` (Bun) or `sql.js` (WASM)
+   - `bcrypt` → `bcryptjs`
+   - `sharp` → pure-JS image libs
+
+2. **Use WASM-based alternatives** — Many native libs now have WASM builds
+
+3. **Bundle with `ncc` and mark as external:**
+   ```bash
+   ncc build src/index.ts -o dist --external better-sqlite3
+   ```
+
+4. **With Bun compile**, embed `.node` file directly if you control the require path
+
+## 7. WASM Builds
+
+### Using tsdown with rolldown-plugin-wasm
+
+```typescript
+// tsdown.config.ts
+import { wasm } from 'rolldown-plugin-wasm'
+import { defineConfig } from 'tsdown'
+
+export default defineConfig({
+  entry: ['./src/index.ts'],
+  plugins: [wasm({
+    maxFileSize: 14 * 1024,
+    targetEnv: 'auto',
+  })],
+})
+```
+
+### Importing WASM Modules
+
+```typescript
+// Direct import
+import { add } from './add.wasm'
+add(1, 2)
+
+// Async init
+import init from './add.wasm?init'
+const instance = await init()
+instance.exports.add(1, 2)
+```
+
+## 8. Distribution Strategy Matrix
+
+| Target | Tool | Output | Needs Runtime? |
+|--------|------|--------|----------------|
+| npm package | tsdown | ESM + CJS + DTS | Yes (consumer's bundler) |
+| npm CLI tool | tsdown + bin field | JS + bin script | Yes (Node.js) |
+| Standalone binary | Bun compile | Native executable | No |
+| Node.js SEA | esbuild + postject | Modified node binary | Embedded Node.js |
+| Browser bundle | esbuild/tsdown | ESM/IIFE JS | No (browser) |
+| Edge/Workers | tsdown + WASM plugin | JS + .wasm | No (edge runtime) |
+| Serverless | @vercel/ncc | Single JS file | Yes (Node.js Lambda) |
+| Deno | deno compile | Native executable | No |
+
+## 9. Recommended 2026 Toolchain
+
+1. **tsdown** for library building (replaces tsup, uses Rolldown)
+2. **esbuild** for fast development bundling
+3. **Bun `--compile`** for standalone executables
+4. **Node.js SEA** when Node.js-based standalone is needed
+5. **rolldown-plugin-wasm** for WASM builds via tsdown
+6. **@vercel/ncc** for single-file Node.js deployments