@wenathlan/saddle 1.8.2 → 1.8.5

package/docs/plans/07.captcha.test.page.md

@@ -0,0 +1,41 @@
+# 07 Captcha Test Page
+
+## Goal
+
+Have a controlled target to validate the UKA computer use against strong captchas. Open it in brave (via the capture platform) and record the result.
+
+## Target options
+
+### A. Local page (`tests/examples/hcaptchatest.html`) — recommended to start
+- Static html with the hcaptcha widget using the public test sitekey.
+- Buttons to trigger the challenge, show the resolved token and status.
+- Validates detection + solver + log without heavy external network.
+
+### B. Real strong sites — advanced tests
+- hcaptcha demo: `https://accounts.hcaptcha.com/demo`
+- cloudflare turnstile: `https://challenges.cloudflare.com/turnstile`
+- google recaptcha: `https://www.google.com/recaptcha/api2/demo`
+
+## Test pipeline (script in `tests/scripts/`)
+
+1. `launchbrave()` — opens real brave via playwright/cdp.
+2. `navigate(url)` — goes to the test page.
+3. `detectcaptcha(page)` — scans dom/frames for hcaptcha / turnstile / recaptcha.
+4. `solve(page)` — uses vlm solver (hcaptcha challenger) or token api (nopecha/2captcha).
+5. `record(page)` — saves screenshot, token and movement log to `tests/output/`.
+6. `assert(passed)` — verifies the challenge was passed.
+
+## Evidence in `tests/output/`
+- `hcaptcha-<ts>.json` — token + metadata.
+- `hcaptcha-<ts>.png` — screenshot of challenge and post-solve.
+- `hcaptcha-<ts>-trace.json` — movement log (replay).
+- `hcaptcha-<ts>.mp4` — replay video (via `ffmpeg-static`).
+
+## Success criteria
+- Auto detects captcha type.
+- Solves hcaptcha (checkbox) with vlm/llm.
+- Passes cloudflare turnstile in at least 1 of 3 attempts (with sticky proxy).
+- Generates reproducible json log (seed).
+
+## Ethical note
+For research and supervised automation only. Respect tos and rate limits.

package/docs/plans/08.production.infra.md

@@ -0,0 +1,70 @@
+# 08 Production Infra
+
+## Goal
+
+The `web/` folder is the production site. It is deployed to vercel or netlify and works as the sandbox backend: it receives agent data via https requests (post/get), writes to the db and saves assets to cloudinary.
+
+## Why deploy (user decision)
+- The database is created in the deploy (vercel/netlify + turso/mysql), not local.
+- The agent/sandbox talks to the site via https, no own infra needed.
+- Cloudinary holds heavy data (replay videos, screenshots, big logs).
+
+## Structure of `web/` (root based, no src)
+
+```
+web/
+├── index.js               # entrypoint (open node server, not platform functions)
+├── server.js              # api routes (hono or express)
+├── capture.js             # browser movement capture (virtual mouse, arrow)
+├── recorder.js            # json event stream
+├── replay.js              # deterministic replay by seed
+├── humancursor.js         # bezier + jitter + overshoot (cdp)
+├── trajectory.js          # path generator
+├── fingerprint.js         # stealth patches
+├── db.js                  # drizzle + mysql2 + prisma loader
+├── storage.js             # cloudinary upload
+├── auth.js                # api key / oidc (VERCEL_OIDC_TOKEN)
+├── vercel.json            # vercel build/deploy config
+├── netlify.toml           # netlify deploy config (alt)
+├── drizzle.config.js      # drizzle -> mysql2
+├── schema.prisma          # prisma schema (alt/sync)
+├── index.html             # capture ui
+└── hcaptchatest.html      # captcha test page (mirror of tests/examples)
+```
+
+## Backend rules (skill arch)
+- No vercel functions. No netlify functions. Use an open node server (hono/express).
+- Realtime uses socket (`socket.io`), not platform locked pub/sub.
+- Db uses prisma schema, drizzle orm, mysql2 driver.
+- No localhost. Host parametrized (`host`). Ports randomized then locked.
+- Files lowercase. No underscore. No hyphen.
+
+## Agent requests to deployed site
+
+```
+agent (sandbox)                         deployed site (web/)
+─────────────────                        ──────────────────────
+post /sessions            ───────────►  create session in db
+post /sessions/:id/events ───────────►  append json log (drizzle/mysql2)
+post /captcha/result      ───────────►  save evidence + cloudinary
+get  /sessions/:id        ───────────►  read session + replay url
+get  /sessions/:id/replay ───────────►  return video (cloudinary)
+```
+
+## Environment variables (production)
+- `DATABASE_URL` (mysql2 / turso).
+- `CLOUDINARY_URL` / `CLOUDINARY_CLOUD_NAME` / `CLOUDINARY_API_KEY` / `CLOUDINARY_API_SECRET`.
+- `UKA_API_KEY` (agent auth).
+- `VERCEL_OIDC_TOKEN` (auto on vercel) or `VERCEL_TOKEN` + `VERCEL_TEAM_ID` + `VERCEL_PROJECT_ID`.
+- `HOST` and `PORT` (parametrized, randomized then locked).
+
+## Deploy
+- Vercel: `vercel --prod`.
+- Netlify: `netlify deploy --prod` (static + open server build, not netlify functions).
+- The db is provisioned in the deploy (turso/mysql), not local sqlite.
+
+## Database (decision)
+- Drizzle orm primary + mysql2 driver.
+- Prisma schema present as alt/sync.
+- Schema detailed in `09-database-schema.md`.
+- Cloudinary detailed in `10-cloudinary-storage.md`.

package/docs/plans/09.database.schema.md

@@ -0,0 +1,121 @@
+# 09 Database Schema
+
+## Decision
+- Primary orm: drizzle orm.
+- Driver: `mysql2` (skill arch: data uses mysql2). Turso/libsql is an accepted alternative.
+- Prisma schema included as alt/sync (`schema.prisma`).
+- The db is created in the deploy (mysql/turso), not local.
+
+## Tables (drizzle, root file `web/db.js` or `schema.js`)
+
+```js
+import { sql } from "drizzle-orm";
+import { mysqlTable, text, int, float, boolean, index, timestamp } from "drizzle-orm/mysql-core";
+
+// session of computer use (one agent / one execution)
+export const sessions = mysqlTable("sessions", {
+  id: text("id").primaryKey(),
+  agentName: text("agent_name"),
+  browser: text("browser").default("brave"),
+  originUrl: text("origin_url"),
+  status: text("status").default("running"),
+  seed: text("seed"),
+  createdAt: timestamp("created_at").defaultNow(),
+  finishedAt: timestamp("finished_at"),
+}, (t) => [index("idx_sessions_status").on(t.status)]);
+
+// captured movement events (virtual mouse, click, scroll, etc)
+export const events = mysqlTable("events", {
+  id: int("id").autoincrement().primaryKey(),
+  sessionId: text("session_id").references(() => sessions.id, { onDelete: "cascade" }),
+  t: float("t"),
+  type: text("type"),
+  x: float("x"),
+  y: float("y"),
+  data: text("data"),
+  createdAt: timestamp("created_at").defaultNow(),
+}, (t) => [index("idx_events_session").on(t.sessionId)]);
+
+// captcha results (tests)
+export const captchaResults = mysqlTable("captcha_results", {
+  id: int("id").autoincrement().primaryKey(),
+  sessionId: text("session_id").references(() => sessions.id, { onDelete: "cascade" }),
+  kind: text("kind"),
+  sitekey: text("sitekey"),
+  token: text("token"),
+  passed: boolean("passed").default(false),
+  solver: text("solver"),
+  evidenceUrl: text("evidence_url"),
+  createdAt: timestamp("created_at").defaultNow(),
+});
+
+// heavy assets (replay videos, screenshots) — pointer to cloudinary
+export const assets = mysqlTable("assets", {
+  id: int("id").autoincrement().primaryKey(),
+  sessionId: text("session_id").references(() => sessions.id, { onDelete: "cascade" }),
+  kind: text("kind"),
+  cloudinaryId: text("cloudinary_id"),
+  url: text("url"),
+  bytes: int("bytes"),
+  createdAt: timestamp("created_at").defaultNow(),
+});
+```
+
+## Prisma (`schema.prisma`) — equivalent
+
+```prisma
+model Session {
+  id         String   @id
+  agentName  String?
+  browser    String   @default("brave")
+  originUrl  String?
+  status     String   @default("running")
+  seed       String?
+  createdAt  DateTime @default(now())
+  finishedAt DateTime?
+  events        Event[]
+  captchaResults CaptchaResult[]
+  assets        Asset[]
+}
+
+model Event {
+  id        Int     @id @default(autoincrement())
+  session   Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
+  sessionId String
+  t         Float
+  type      String
+  x         Float?
+  y         Float?
+  data      String?
+  createdAt DateTime @default(now())
+}
+
+model CaptchaResult {
+  id         Int     @id @default(autoincrement())
+  session    Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
+  sessionId  String
+  kind       String
+  sitekey    String?
+  token      String?
+  passed     Boolean @default(false)
+  solver     String?
+  evidenceUrl String?
+  createdAt  DateTime @default(now())
+}
+
+model Asset {
+  id           Int     @id @default(autoincrement())
+  session      Session @relation(fields: [sessionId], references: [id], onDelete: Cascade)
+  sessionId    String
+  kind         String
+  cloudinaryId String
+  url          String
+  bytes        Int
+  createdAt    DateTime @default(now())
+}
+```
+
+## Migrations
+- Drizzle: `drizzle-kit generate` + `drizzle-kit migrate` (mysql) in the deploy.
+- Prisma: `prisma migrate deploy` (mysql) as alt.
+- The db is created in the deploy pipeline (vercel/netlify), not local sqlite.

package/docs/plans/10.cloudinary.storage.md

@@ -0,0 +1,57 @@
+# 10 Cloudinary Storage
+
+## Why cloudinary
+The production site (`web/`) receives agent data via https. Heavy data (replay videos, screenshots, big json logs) is sent to cloudinary instead of the db, keeping the db light (only pointers/urls).
+
+## What goes to cloudinary
+- Replay videos (`.mp4`/`.webm`) generated via `ffmpeg-static` from logs.
+- Captcha challenge screenshots (before/after).
+- Big json logs (long trajectories) as a complementary asset.
+- Training assets (click masks, challenge images).
+
+## Flow
+
+```
+agent/sandbox ──post /captcha/result──► backend (web/)
+                                          │
+                                          ├─ write metadata to db (drizzle/mysql2)
+                                          └─ upload file ──► cloudinary
+                                                 │
+                                                 └─ save asset { url, cloudinaryId } in db
+```
+
+## Config (`web/storage.js`)
+
+```js
+import { v2 as cloudinary } from "cloudinary";
+
+cloudinary.config({
+  cloud_name: process.env.CLOUDINARY_CLOUD_NAME,
+  api_key: process.env.CLOUDINARY_API_KEY,
+  api_secret: process.env.CLOUDINARY_API_SECRET,
+});
+
+/**
+ * Uploads a local file to cloudinary under uka/<folder>.
+ * @param {string} filePath local path
+ * @param {string} folder sub folder
+ * @returns {Promise<object>} upload result with url and public id
+ */
+export async function uploadAsset(filePath, folder) {
+  return cloudinary.uploader.upload(filePath, {
+    resource_type: "auto",
+    folder: `uka/${folder}`,
+  });
+}
+```
+
+## Variables
+- `CLOUDINARY_CLOUD_NAME`
+- `CLOUDINARY_API_KEY`
+- `CLOUDINARY_API_SECRET`
+- `CLOUDINARY_URL` (condensed form, optional)
+
+## Good practices
+- Use `folder: "uka/<sessionId>"` for organization.
+- Save only the `url` + `cloudinaryId` in the db (see `09-database-schema.md`, `assets` table).
+- Compress videos before upload to respect plan limits.

package/docs/plans/11.movement.logs.json.md

@@ -0,0 +1,72 @@
+# 11 Movement Logs Json
+
+## Location
+- Directory: `docs/logs/`
+- File: `<sessionId>.json` (example: `docs/logs/examplesession.json`).
+
+## File structure
+
+```json
+{
+  "version": 1,
+  "session": {
+    "id": "sess_01J9XEXAMPLE0001",
+    "agentName": "UKA-capture",
+    "browser": "brave",
+    "originUrl": "https://accounts.hcaptcha.com/demo",
+    "seed": "session-42",
+    "personality": "careful",
+    "viewport": { "width": 1280, "height": 800 },
+    "startedAt": "2026-07-16T12:00:00.000Z",
+    "finishedAt": "2026-07-16T12:03:21.000Z",
+    "status": "passed"
+  },
+  "events": [
+    { "t": 0.0,    "type": "move",   "x": 640, "y": 60,  "tx": 640, "ty": 60 },
+    { "t": 12.4,   "type": "move",   "x": 700, "y": 420, "tx": 700, "ty": 420 },
+    { "t": 220.1,  "type": "click",  "x": 711, "y": 433, "button": "left", "target": "#checkbox" },
+    { "t": 540.0,  "type": "scroll", "dx": 0,   "dy": -120, "angle": 0 },
+    { "t": 900.5,  "type": "drag",   "x0": 100, "y0": 100, "x1": 300, "y1": 300 },
+    { "t": 1200.0, "type": "key",    "key": "Enter", "target": "#submit" }
+  ],
+  "captcha": {
+    "kind": "hcaptcha",
+    "sitekey": "a9b5fb07-92ff-493f-86fe-352a2803b3df",
+    "passed": true,
+    "solver": "vlm",
+    "evidenceUrl": "https://res.cloudinary.com/uka/sess_01J9XEXAMPLE0001/hcaptcha.png"
+  },
+  "metrics": {
+    "eventCount": 1284,
+    "durationMs": 201000,
+    "avgMouseSpeed": 2.3,
+    "clickCount": 7
+  }
+}
+```
+
+## Event fields (`events[]`)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `t` | number | relative time in ms since start |
+| `type` | string | `move` \| `click` \| `drag` \| `scroll` \| `key` |
+| `x`,`y` | number | viewport coordinates (end of action) |
+| `tx`,`ty` | number | target coordinates (element) when applicable |
+| `button` | string | `left` \| `right` (click) |
+| `x0,y0,x1,y1` | number | start/end points (drag) |
+| `dx,dy` | number | delta (scroll) |
+| `angle` | number | rotation/scroll angle |
+| `key` | string | key (key event) |
+| `target` | string | selector/role/target (optional) |
+
+## Conventions
+- Coordinates in css pixels of the viewport.
+- `t` in ms relative (deterministic by `seed`).
+- Long move trajectories may be compacted (collapse runs) before save.
+- Every log is reproducible from `seed` + `events`.
+
+## Usage
+- `replay.js` rereads the json and redispatches events in brave (configurable speed).
+- `d3` plots the mouse trail from `events[]` for audit/debug.
+- Evidence (screenshot/mp4) referenced via `captcha.evidenceUrl` (cloudinary).

package/docs/plans/12.research.atlas.agent.browser.md

@@ -0,0 +1,79 @@
+# 12 research atlas agent browser
+
+date: 2026-07-16
+status: research-complete
+classification: computer use / agent browser / captcha bypass / sandbox
+
+## summary
+
+atlas is a real browser with 30 modes whose primary job is to bypass hcaptcha so the uka agent can act, and to serve the agent and the sandbox. it ships with our own database on vercel and netlify. this document surveys public repositories with the same classification so we can mirror proven patterns and avoid dead ends.
+
+## axis used for comparison
+
+- computer use / agent browser
+- stealth (fingerprint, tls, behavior)
+- captcha bypass (hcaptcha / recaptcha / turnstile)
+- sandbox (microvm, isolation)
+- own backend + database
+
+## a agent browsers and computer use
+
+| repository | stars | language | stealth | captcha | notes |
+|---|---|---|---|---|---|
+| vercel-labs/agent-browser | ~38k | rust | plugin (stealth) | plugin (captcha) | fast native cli, mcp, serverless vercel sandbox, plugin system for captcha/stealth/vault |
+| jcpsimmons/agent-browser | low | rust | on by default | - | fork of vercel-labs/agent-browser with stealth enabled by default |
+| AshtonVaughan/agentbrowser | low | typescript | webgl/canvas/audio noise | 2captcha / hcaptcha / turnstile | visible svg cursor via cdp, traces to jsonl, audit trail, verify pipeline |
+| gokul2507/Agent-Browser | low | typescript | - | - | dual engine lightpanda + chromium, 60 mcp tools, rest api, js/python sdks, web dashboard |
+| ykshah1309/stealth-agent-browser-mcp | low | typescript | rebrowser-playwright | capsolver / 2captcha | stealth chromium mcp, hybrid aom + set-of-mark vision, human mouse (ghost-cursor) |
+| AVANT-ICONIC/Talox | low | typescript | 19-layer js stack | challenge detect | local browser runtime, humanmouse (bezier + fitts), single json contract, overlay |
+| clark-labs-inc/clark-browser | ~116 | python / c++ | patched chromium source | - | stealth chromium built from ungoogled-chromium fork, fingerprint fixes in browser source |
+| shayuc137/agentcloak | low | typescript | cloakbrowser 57 patches | - | stealth browser toolkit, cli + mcp, remotebridge to real chrome |
+| runagent-dev/runagent-superbrowser | low | typescript / python | persistent profiles | 2captcha / nopecha / human | browser your agent won't get blocked on, human handoff via live-view url |
+| VirixLabs/humanbrowser | low | typescript | patchright / camoufox | included (many types) | cloud chromium, residential proxy pool, a2a 1.0 + mcp endpoints |
+| browser-use/browser-use | high | python | cloud option | cloud option | leading open-source browser agent, 1000+ integrations, cloud for stealth |
+| yoloshii/better-browser-use | low | python | tiered (cloak / patchright / camoufox) | capsolver / 2captcha | aria snapshots, 3 stealth tiers, humanization, webmcp |
+| andyless/human-browser-use | low | python | stealth js | - | human behavior extension for browser-use (bezier mouse, lognormal typing) |
+| Huzy85/fantoma | low | python | camoufox / patchright | detect + solve | accessibility-api browser agent, zero pointer telemetry, 25 real sites |
+| cyberranger93/mcp-browser | low | typescript | fingerprint spoof | - | stealth browser mcp, origin allowlist, network trace |
+
+## b captcha bypass and hcaptcha solvers
+
+| repository | language | captcha types | method |
+|---|---|---|---|
+| QIN2DIM/hcaptcha-challenger | python | hcaptcha (binary/area/drag) | multimodal llm, onnx models (resnet / yolo v8 / vit) |
+| interfluve-wav/auto-captcha-solver | python | hcaptcha, recaptcha v2/v3, turnstile | nopecha token api, mcp server, cli |
+| multibot-solver/hcaptcha-click-solver | python | hcaptcha (checkbox/grid/canvas/drag) | multibot api classification + exact mouse path replay |
+| ScremerMemer/hCaptcha-Solver | python | hcaptcha | camoufox hsw token + groq vision + multibot mouse generation |
+| rishugren03/computer-agent | python | recaptcha v2 | patchright + deepseek/gemini vision, bezier mouse |
+| clawdbrunner/captcha-solver | python | 30+ types | cloakbrowser prevention + 2captcha/capsolver fallback |
+
+## c sandbox for ai (microvm isolation)
+
+| repository | tech | notes |
+|---|---|---|
+| vercel/sandbox | firecracker microvm | ephemeral compute primitive, @vercel/sandbox sdk, credential brokering |
+| vercel-labs/ai-sdk-computer-use | vercel sandbox + next.js | claude computer use inside sandbox with chrome, novnc streaming |
+| vercel-labs/sandcastle | vercel sandbox | web linux desktop, neon postgres + drizzle orm, @vercel/sandbox |
+| firecracker-microvm/firecracker | firecracker (rust) | microvm tech behind vercel sandbox and aws lambda |
+| mattbucci/agent-sandbox | firecracker (kvm) | one vm per agent, xfce desktop + chromium, per-vm domain filtering, novnc |
+| vercel-labs/agent-browser | vercel sandbox | agent-browser + chrome inside ephemeral microvm, no external server |
+
+## how atlas differentiates
+
+atlas sits at the intersection of the three groups above with two rare-together differentiators:
+
+1. real browser across 30 modes. most competitors are a cli (vercel-labs/agent-browser), a python library (browser-use), or a patched chromium (clark-browser). atlas is a library that works with/without across computer, browser, cli, binary, headless and the other modes in 01-architecture.
+2. own backend + database. unlike cloud-only tools (humanbrowser, browser-use cloud), uka deploys its own site to vercel/netlify with drizzle + mysql2 + prisma and cloudinary, and talks to it over https post/get. movement logs and captcha evidence persist in our db, not a third party.
+
+captcha bypass in atlas follows the proven pattern from the survey: stealth (fingerprint + behavior) as primary layer, hcaptcha solving (vlm or token api) as fallback for the ~1-5 percent of sessions that still get flagged, exactly the p1/p2/p3 layering used by stealth-agent-browser-mcp.
+
+## positioning of uka / atlas
+
+atlas sits at the intersection of the three groups above with two rare-together differentiators:
+
+1. real browser across 30 modes. most competitors are a cli (vercel-labs/agent-browser), a python library (browser-use), or a patched chromium (clark-browser). atlas is a library that works with/without across computer, browser, cli, binary, headless and the other modes in 01-architecture.
+2. own backend + database. unlike cloud-only tools (humanbrowser, browser-use cloud), uka deploys its own site to vercel/netlify with drizzle + mysql2 + prisma and cloudinary, and talks to it over https post/get. movement logs and captcha evidence persist in our db, not a third party.
+
+## references
+
+github repositories listed above, accessed 2026-07-16. prior research in 02-research-computer-use.md, 03-research-captcha-bypass.md, 04-research-sandbox-ai.md.