@wenathlan/saddle 1.8.16 → 1.8.18

package/docs/plans/45.robotarchitecture.md

@@ -1,237 +0,0 @@
-# Saddle Robot Architecture
-
-> Super bot: agent browser, scraper, captcha bypass, GitHub/GitLab/Forgejo/Gitea automation.
-> Transform storage memory into computational memory for unlimited processing.
-
-## Overview
-
-Saddle is a self-hosted AI agent platform that combines browser automation,
-scraping, captcha bypass, and multi-platform bot functionality into a single
-runtime. It operates as a virtual machine agent that can deploy to multiple
-platforms and store unlimited memory in remote repositories.
-
-## Core Components
-
-### 1. Agent Browser Engine
-
-The browser agent captures user interactions (mouse movement, clicks, scrolling)
-and replays them. It uses Brave browser via Playwright with the real user
-profile for persistence.
-
-#### Movement Capture
-
-```
-Browser → Movement Capture → JSON Log → Storage → Replay
-```
-
-Supports:
-- Virtual mouse movement (arrow trails)
-- Click coordinates
-- Scroll events
-- Keyboard input
-- Screenshot capture
-- Session recording
-
-#### Captcha Bypass
-
-Saddle handles three captcha types:
-
-| Type | Method | Success Rate |
-|------|--------|-------------|
-| hCaptcha | Cloudflare Turnstile proxy | High |
-| Cloudflare Turnstile | Managed challenge solving | Medium-High |
-| reCAPTCHA | Token rotation + headless solve | Medium |
-
-### 2. Multi-Platform Bot Engine
-
-The bot engine operates across platforms through a unified interface:
-
-```typescript
-interface PlatformAdapter {
-  name: string
-  type: 'github' | 'gitlab' | 'forgejo' | 'gitea' | 'discord'
-  authenticate(token: string): Promise<boolean>
-  listRepos(): Promise<Repo[]>
-  createWebhook(payload: WebhookPayload): Promise<void>
-  executeBot(commands: BotCommand[]): Promise<BotResult>
-}
-```
-
-Platforms supported:
-- **GitHub** - Actions, webhooks, PR automation, issue triage
-- **GitLab** - CI/CD pipelines, merge requests, webhooks
-- **Forgejo** - Self-hosted Git, lightweight alternative
-- **Gitea** - Self-hosted Git service, minimal footprint
-- **Discord** - Bot commands, webhook reactions, automated responses
-
-### 3. Computational Memory System
-
-This is the core innovation of saddle. It transforms storage memory into
-computational memory through a memory engine.
-
-#### Memory Layers
-
-| Layer | Type | Source | Capacity |
-|-------|------|--------|----------|
-| L1 | RAM | System memory | Limited (ephemeral) |
-| L2 | GPU VRAM | Graphics memory | Limited (compute-bound) |
-| L3 | Storage RAM | Repository storage | Effectively unlimited |
-| L4 | External Buckets | HF/Kaggle/ModelScope/Terabox | Unlimited |
-
-#### Storage-to-Compute Transformation
-
-```
-Repository Storage (L3)
-    → Memory Engine reads blobs
-    → Transforms into virtual RAM
-    → Processed as computational memory
-    → Results written back to storage
-```
-
-The memory engine acts as a bridge between storage objects (files, blobs,
-assets) and computational processing. It can load file contents into
-memory, process them, and persist results back to storage.
-
-#### Memory Engine API
-
-```typescript
-interface MemoryEngine {
-  loadFromStorage(repo: string, path: string): Promise<MemoryObject>
-  process(operation: ComputeOperation): Promise<MemoryResult>
-  persist(result: MemoryResult, target: StorageTarget): Promise<void>
-  release(memoryId: string): Promise<void>
-}
-```
-
-## Architecture Diagram
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│                         Saddle Robot                          │
-│                                                                │
-│  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌───────────────┐ │
-│  │  Browser  │  │   Bot    │  │  CAPTCHA │  │ Memory Engine │ │
-│  │  Agent    │  │   Engine │  │  Bypass  │  │ (Core)        │ │
-│  └────┬─────┘  └────┬─────┘  └────┬─────┘  └──────┬────────┘ │
-│       │              │              │                │           │
-│       ▼              ▼              ▼                ▼           │
-│  ┌─────────────────────────────────────────────────────────────┐│
-│  │              Unified Processing Layer                      ││
-│  └─────────────────────────────────────────────────────────────┘│
-│       │              │              │                │           │
-│       ▼              ▼              ▼                ▼           │
-│  ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────────┐   │
-│  │  GitHub │   │  GitLab │   │  Forgejo│   │   Discord   │   │
-│  │         │   │         │   │         │   │             │   │
-│  └─────────┘   └─────────┘   └─────────┘   └─────────────┘   │
-│                                                                │
-│  ┌────────────────────────────────────────────────────────────┐│
-│  │              Storage / Memory Backends                    ││
-│  │  HuggingFace │ Kaggle │ ModelScope │ Terabox │ Repos    ││
-│  └────────────────────────────────────────────────────────────┘│
-│                                                                │
-│  ┌────────────────────────────────────────────────────────────┐│
-│  │              Deploy Targets                               ││
-│  │  Netlify │ Vercel │ Docker │ VM │ Self-Hosted │ GitHub  ││
-│  └────────────────────────────────────────────────────────────┘│
-└─────────────────────────────────────────────────────────────────┘
-```
-
-## Deploy Strategies
-
-### Strategy 1: HTTPS Communication Between Sites
-
-Two or more deployed sites communicate via HTTPS. The saddle robot
-receives commands from a web interface and executes actions.
-
-```
-Site A (web interface) ──HTTPS──▶ Site B (saddle robot backend)
-                                      │
-                                      ├──▶ GitHub API
-                                      ├──▶ GitLab API
-                                      ├──▶ Browser Engine
-                                      └──▶ Storage Backends
-```
-
-### Strategy 2: GitHub/GitLab/Forge Pipelines
-
-Use CI/CD pipelines (GitHub Actions, GitLab CI, Forgejo Actions) to
-deploy and run the saddle robot as a scheduled or triggered job.
-
-### Strategy 3: Docker/VM Deployment
-
-Package saddle as a Docker container or VM image for self-hosted
-deployment with full resource access.
-
-```dockerfile
-FROM node:20-slim
-WORKDIR /app
-COPY package*.json ./
-RUN npm ci --production
-COPY . .
-EXPOSE 3000
-CMD ["node", "server.js"]
-```
-
-## NPM Package Publishing
-
-Saddle can be published as an NPM package `@wenathlan/saddle` for
-distribution as both a library and a CLI tool.
-
-### Package Structure
-
-```
-@wenathlan/saddle/
-├── lib/
-│   ├── browser.js        (agent browser engine)
-│   ├── bot.js            (multi-platform bot engine)
-│   ├── captcha.js        (captcha bypass module)
-│   ├── memory-engine.js  (storage-to-compute transformation)
-│   └── deploy.js         (deploy strategy implementations)
-├── bin/
-│   └── saddle.js         (CLI entry point)
-├── docs/
-│   └── index.md          (package documentation)
-├── package.json
-└── README.md
-```
-
-### As Library vs Binary
-
-| Mode | Entry Point | Usage |
-|------|------------|-------|
-| Library | `require('@wenathlan/saddle')` | Import as Node module |
-| CLI | `saddle --command capture` | Run as terminal command |
-| Binary | `node_modules/.bin/saddle` | Direct execution |
-
-## SCDN Integration
-
-Saddle supports integration with Static Content Delivery Networks for
-hosting browser assets, capture files, and deployment payloads.
-
-The SCDN pipeline:
-1. Captures browser session data
-2. Uploads assets to SCDN
-3. References SCDN URLs in bot results
-4. Serves static content from edge locations
-
-## Research Status
-
-| Component | Status | Source |
-|-----------|--------|--------|
-| Agent browser | Research complete | `12-research-atlas-agent-browser.md` |
-| Captcha bypass | Research complete | `03-research-captcha-bypass.md` |
-| Sandbox/VM | Research complete | `04-research-sandbox-ai.md` |
-| Memory engine | To be implemented | New doc |
-| Multi-platform bot | To be implemented | New doc |
-| Deploy strategies | To be implemented | New doc |
-| SCDN integration | To be implemented | New doc |
-
-## Decision Log
-
-See `docs/talks/` for decision transcripts on architecture choices.
-See `docs/logs/` for json movement logs from Brave capture sessions.
-
-## License
-
-MIT

package/docs/plans/46.scdnintegration.md

@@ -1,284 +0,0 @@
-# SCDN Integration for Saddle Robot
-
-> Static Content Delivery Network integration for
-> hosting browser assets, capture files, and deployment payloads.
-
-## Overview
-
-The saddle robot integrates with Static Content Delivery
-Networks (SCDN) to serve browser assets, capture files,
-and deployment payloads from edge locations worldwide.
-
-## SCDN Use Cases
-
-1. Host browser automation assets (scripts, styles,
-   configuration files)
-2. Serve capture session data (JSON logs, video files)
-3. Deliver deployment payloads to deployed sites
-4. Cache bot responses for faster repeat execution
-5. Distribute browser profiles and extensions
-
-## SCDN Pipeline
-
-```
-Capture Pipeline
-────────────────────────────────────────────────────────
-1. Browser session captures data
-2. Assets uploaded to SCDN via HTTPS
-3. SCDN caches content at edge locations
-4. Bot workers fetch assets from nearest edge
-5. Results stored in repository/bucket storage
-```
-
-## Implementation
-
-### SCDN Client
-
-```typescript
-interface SCDNClientConfig {
-  endpoint: string
-  token: string
-  bucketId: string
-  region?: string
-  ttlSeconds: number
-}
-
-class SCDNClient {
-  private config: SCDNClientConfig
-
-  constructor(config: SCDNClientConfig) {
-    this.config = config
-  }
-
-  async upload(
-    path: string,
-    content: Buffer | string,
-    options?: { contentType?: string; ttl?: number }
-  ): Promise<{ url: string; etag: string; size: number }> {
-    try {
-      const response = await fetch(
-        `${this.config.endpoint}/${this.config.bucketId}/${path}`,
-        {
-          method: 'PUT',
-          headers: {
-            Authorization: `Bearer ${this.config.token}`,
-            'Content-Type': options?.contentType ?? 'application/octet-stream',
-            'Cache-Control': `max-age=${options?.ttl ?? this.config.ttlSeconds}`,
-          },
-          body:
-            typeof content === 'string'
-              ? new TextEncoder().encode(content)
-              : content,
-        }
-      )
-
-      if (!response.ok) {
-        throw new Error(
-          `SCDN upload failed: ${response.status} ${response.statusText}`
-        )
-      }
-
-      const etag = response.headers.get('etag') ?? ''
-      const url = `${this.config.endpoint}/${this.config.bucketId}/${path}`
-      const size =
-        typeof content === 'string'
-          ? new TextEncoder().encode(content).byteLength
-          : content.byteLength
-
-      return { url, etag, size }
-    } catch (error) {
-      throw new Error(
-        `SCDNClient.upload("${path}") failed: ${error instanceof Error ? error.message : String(error)}`
-      )
-    }
-  }
-
-  async download(path: string): Promise<Buffer> {
-    try {
-      const response = await fetch(
-        `${this.config.endpoint}/${this.config.bucketId}/${path}`,
-        {
-          headers: {
-            Authorization: `Bearer ${this.config.token}`,
-          },
-        }
-      )
-
-      if (!response.ok) {
-        throw new Error(
-          `SCDN download failed: ${response.status} ${response.statusText}`
-        )
-      }
-
-      const arrayBuffer = await response.arrayBuffer()
-      return Buffer.from(arrayBuffer)
-    } catch (error) {
-      throw new Error(
-        `SCDNClient.download("${path}") failed: ${error instanceof Error ? error.message : String(error)}`
-      )
-    }
-  }
-
-  async delete(path: string): Promise<boolean> {
-    try {
-      const response = await fetch(
-        `${this.config.endpoint}/${this.config.bucketId}/${path}`,
-        {
-          method: 'DELETE',
-          headers: {
-            Authorization: `Bearer ${this.config.token}`,
-          },
-        }
-      )
-      return response.ok
-    } catch (error) {
-      throw new Error(
-        `SCDNClient.delete("${path}") failed: ${error instanceof Error ? error.message : String(error)}`
-      )
-    }
-  }
-}
-```
-
-### Asset Upload Flow
-
-```typescript
-// In browser capture module
-import { SCDNClient } from '@wenathlan/saddle/deploy'
-
-async function uploadCaptureSession(
-  sessionData: SessionData,
-  scdnClient: SCDNClient
-): Promise<SessionResult> {
-  const sessionId = generateSessionId()
-  const sessionPath = `sessions/${sessionId}.json`
-  const screenshotPath = `screenshots/${sessionId}.png`
-
-  // Upload session data
-  const sessionResult = await scdnClient.upload(
-    sessionPath,
-    JSON.stringify(sessionData),
-    { contentType: 'application/json', ttl: 3600 }
-  )
-
-  // Upload screenshot
-  const screenshotResult = await scdnClient.upload(
-    screenshotPath,
-    sessionData.screenshot,
-    { contentType: 'image/png', ttl: 86400 }
-  )
-
-  return {
-    sessionId,
-    sessionUrl: sessionResult.url,
-    screenshotUrl: screenshotResult.url,
-  }
-}
-```
-
-## SCDN Providers
-
-| Provider | Type | Pricing | Edge Locations |
-|----------|------|---------|----------------|
-| Cloudflare Workers | Edge JS | Free tier + paid | 300+ cities |
-| Cloudflare R2 | Object storage | Free egress | 300+ cities |
-| AWS CloudFront | CDN + S3 | Pay per use | 400+ cities |
-| Bun CDN | Edge runtime | Free tier | Global |
-| Deno Deploy | Edge functions | Free tier + paid | Global |
-| Vercel Edge Config | Config storage | Includes Vercel | Global |
-| Netlify Edge | Edge functions | Free tier + paid | Global |
-
-## SCDN + Repository Integration
-
-```
-SCDN Endpoint ──HTTPS──▶ Repository Webhook
-                              │
-                              ▼
-                    Saddle Robot Deploy
-                              │
-                              ▼
-                    Browser Capture Session
-                              │
-                              ▼
-                    SCDN Asset Upload
-                              │
-                              ▼
-                    Store SCDN URLs in Repo
-```
-
-## Configuration
-
-### Environment Variables
-
-| Variable | Description | Required |
-|----------|-------------|----------|
-| `SBOT_SCDN_ENDPOINT` | SCDN API endpoint | Yes |
-| `SBOT_SCDN_TOKEN` | SCDN authentication token | Yes |
-| `SBOT_SCDN_BUCKET` | Bucket ID for storage | Yes |
-| `SBOT_SCDN_REGION` | Preferred edge region | No |
-| `SBOT_SCDN_TTL` | Default cache TTL in seconds | No |
-
-### Example Configuration
-
-```yaml
-# .saddle/scdn.yml
-scdn:
-  endpoint: "https://api.scdn.example.com/v1"
-  bucket: "saddle-assets"
-  token: "${SBOT_SCDN_TOKEN}"
-  region: "us-east-1"
-  ttl: 3600
-  retry:
-    maxAttempts: 3
-    delayMs: 1000
-```
-
-## Performance
-
-| Operation | SCDN (edge) | Direct storage |
-|-----------|------------|----------------|
-| Upload | 50-200ms | 200-2000ms |
-| Download | 10-50ms | 50-500ms |
-| Cache hit | 5-20ms | N/A |
-| Cache miss | 50-200ms | N/A |
-
-## Error Handling
-
-```typescript
-interface SCDNResult {
-  success: boolean
-  url?: string
-  error?: string
-  retryCount?: number
-}
-
-async function safeUpload(
-  client: SCDNClient,
-  path: string,
-  content: Buffer,
-  maxRetries: number = 3
-): Promise<SCDNResult> {
-  for (let attempt = 1; attempt <= maxRetries; attempt++) {
-    try {
-      const result = await client.upload(path, content)
-      return { success: true, url: result.url }
-    } catch (error) {
-      if (attempt === maxRetries) {
-        return {
-          success: false,
-          error: error instanceof Error ? error.message : String(error),
-          retryCount: attempt,
-        }
-      }
-      await new Promise((resolve) =>
-        setTimeout(resolve, 1000 * attempt)
-      )
-    }
-  }
-  return { success: false, error: 'Unexpected error' }
-}
-```
-
-## License
-
-MIT

package/docs/plans/47.multiforge.readme.md

@@ -1,129 +0,0 @@
-# OpenCode Multi-Forge - Repo como Processador Virtual
-
-## Conceito: Storage = RAM = CPU
-
-> Memória RAM e armazenamento são a mesma coisa. A diferença é uso: uma guarda, outra processa. 
-> Transformamos o lugar que guarda arquivo em lugar que processa arquivo.
-
-### Como?
-
-1. **Swap file**: `fallocate -l 16G /mnt/swapfile && mkswap && swapon` -> disco virou RAM
-2. **tmpfs**: `mount -t tmpfs -o size=8G tmpfs /mnt/ramdisk` -> RAM virou disco ultra rápido
-3. **ZRAM**: comprime RAM para caber 2-3x mais
-4. **Repo como CPU**: GitHub Actions roda código, não só guarda. Cada `workflow_dispatch` é uma chamada de função serverless.
-
-### Arquitetura Thin Client (V3)
-
-```
-[Browser - GitHub Pages 50KB] --light--> [GitHub API] --dispatch--> [Actions Farm]
-  - Preview 256 bytes                         - 4 vCPU, 16GB RAM, ilimitado público
-  - SHA256 via Worker                         - Storage transformado em RAM via swap
-  - Terminal fake                             - Resultado commitado em docs/results/
-  <--polling-- status + artifact download ------
-```
-
-### Multi-Forja = Ilimitado real
-
-| Forja | Minutos | Storage | Como usar |
-|-------|---------|---------|-----------|
-| **GitHub Public** | Ilimitado (standard) | 500MB artifacts, 10GB cache/repo | Principal |
-| **Forgejo Self-Hosted** | Ilimitado (seu hardware) | Ilimitado | Mirror do repo |
-| **Gitea Self-Hosted** | Ilimitado | Ilimitado | Mirror |
-| **Codeberg + Woodpecker** | Ilimitado FLOSS | 750MB soft, pede mais via issue | Backup open source |
-| **GitLab Free** | 400 min/mês | 10GB por projeto | Fallback Pages |
-
-**Farm:** Crie 100 repos `opencode-worker-N`, cada um com 500MB free = 50GB distribuído. Script `scripts/farm.py` dispara para o primeiro livre.
-
-### Como usar
-
-1. Deploy `index.html` no Pages
-2. Configure token com `repo` + `workflow`
-3. Arraste binário -> clica "Enviar para GitHub para processar"
-4. O binário NÃO roda no browser, só preview. O Action roda pesado.
-5. Resultado aparece em `docs/results/{job_id}/` e baixa automático.
-
-### Estrutura
-
-- `index.html` - V2 thin client GitHub only
-- `index-v3-multi-forge.html` - V3 com Forgejo/Gitea/Codeberg/GitLab
-- `.github/workflows/` - 5 workflows GitHub
-- `.forgejo/workflows/` - workflows Forgejo (swap + tmpfs)
-- `.gitea/workflows/` - idem Gitea
-- `.woodpecker/` - Codeberg Woodpecker CI
-- `.gitlab-ci.yml` - GitLab CI
-- `scripts/storage-as-ram.sh` - transforma storage em RAM
-- `scripts/farm.py` - orquestra N repos
-
-### Por que não pesa o navegador?
-
-- Browser: só File API slice + base64 chunked + fetch
-- Heavy: acontece em `/mnt/ramdisk` dentro do runner, com swap de 16GB
-- Resultado: commitado no repo, Pages serve como CDN
-
-Sensação: usuário mexe no binário normal, exporta normal, mas processamento é no repo.
-
----
-
-# V4 - Teoria Completa: Repo como Casa do OS
-
-## Fatos que você trouxe - raciocinando junto
-
-### Fato 1: No Linux tudo é arquivo
-> Everything is a file - open/read/write/close retornam file descriptor
-
-No kernel tudo é struct inode + dentry + file_operations. Processo, memória, device, socket - tudo é arquivo.
-
-```c
-struct file_operations {
-  .read, .write, .mmap, .open...
-}
-```
-
-Se implementa file_operations, É arquivo. VFS (Virtual File System Switch) abstrai ext4, tmpfs, proc, sysfs.
-
-Prova que storage e RAM são mesma coisa: ambas são files com backing diferente.
-
-### Fato 2: Formatos, mimetypes, magic bytes
-Extensão é UX. Verdade está nos primeiros bytes:
-
-```
-ELF: 7F 45 4C 46
-PE: 4D 5A (MZ) + 50 45 00 00 (PE)
-PNG: 89 50 4E 47 0D 0A 1A 0A
-ZIP: 50 4B 03 04 (por isso DOCX, APK, JAR = ZIP)
-```
-
-`file` usa libmagic para detectar. MIME type é pra web (Content-Type), magic byte é pra SO decodificar.
-
-Repo guarda esses bytes estáticos. Quando busca, decodifica via magic.
-
-### Fato 3: Encoding e Decoding
-Todo arquivo estático precisa encode/decode:
-
-- Encode: arquivo no disco (bytes estáticos)
-- Decode: loader lê magic, escolhe parser, mapeia pra RAM, executa
-
-Ex: Windows instala, arquivos ficam estáticos. Na hora de rodar, busca paginação, dump de memória, decodifica PE -> mapeia seções -> executa.
-
-Mesma coisa no nosso caso: repo guarda binário estático com magic, browser faz fetch, decodifica via WASM loader, executa.
-
-### Fato 4: Sandbox tem VFS e virtual file - Docker, K8s
-
-Docker usa OverlayFS:
-
-```
-lowerdir=/lower1:/lower2 (read-only, camadas da imagem)
-upperdir=/upper (writable)
-merged=/merged (o que processo vê)
-copy_up: primeira escrita copia arquivo inteiro do lower pro upper
-```
-
-Imagem OCI = stack de tar.gz estáticos. `docker run` = monta overlay + cria namespace.
-
-Kubernetes ephemeral storage = tmpfs + emptyDir que é RAM.
-
-Ou seja: arquivos estáticos viram sistema rodando via overlay.
-
-### Fato 5: Virtual file e virtual RAM, VMs
-
-VM = arquivo estático (qc