camofox-browser 2.1.0 → 2.4.1

package/CHANGELOG.md

@@ -1,5 +1,131 @@
 # Changelog
 
+## [Unreleased]
+
+## Release Audit: v2.3.0 -> v2.4.1
+
+### What shipped in this line
+- **Security hardening** tightened exposed deployment defaults with loopback-only bind, non-loopback API key enforcement, private-network navigation blocking, and fail-fast proxy deployment validation.
+- **Proxy and geo session identity** moved from user-only scoping to `userId + sessionKey`, allowing parallel sessions with distinct proxy and geo profiles without unsafe reuse or eviction collisions.
+- **Lifecycle control** added staged idle cleanup plus daemon exit policy, with activity-aware timer disarming so live sessions are not collected accidentally.
+- **Fingerprint environment controls** added deployment-level defaults for OS, WebGL, screen dimensions, and humanization, with strict parsing and clear generation-time versus launch-time behavior.
+- **Structured extraction** introduced schema-driven extraction across core API, CLI, and OpenClaw, including validation-time 400s and runtime 422s with stable field-path reporting.
+- **OpenAPI and interactive docs** added `/openapi.json` and `/api/docs`, then hardened the spec and origin handling to match real server behavior.
+- **Release-lane hardening** shipped in `v2.4.1`, ensuring Docker/GHCR publication no longer fails solely because optional GeoLite download during `camoufox-js fetch` is temporarily unavailable.
+
+### Reading guide
+- **`2.4.0`** is the main Wave 2 delivery release.
+- **`2.4.1`** is the follow-up patch that fixes the release-distribution lane while inheriting the full `2.4.0` surface.
+
+## [2.4.1] - 2026-05-05
+
+### Upgrade Notes
+- **Patch scope**: `2.4.1` keeps the full Wave 2 surface from `2.4.0` and only changes release-distribution behavior.
+- **Operator impact**: Docker/GHCR publication no longer fails solely because `camoufox-js fetch` cannot download the optional GeoLite database during image build.
+
+### Fixed
+- **Docker release builds** now tolerate transient `camoufox-js fetch` / GeoLite MMDB download failures during image creation.
+- **Release-lane consistency** now matches the existing best-effort `postinstall` fetch contract already used by package installation.
+
+## [2.4.0] - 2026-05-05
+
+### Upgrade Notes
+- **Wave 2 delivery** adds OpenAPI documentation, deployment-level fingerprint controls, staged idle lifecycle management, session-level proxy/geo overrides, and structured extraction without removing previous route aliases.
+- **Operational posture** is more defensive than in `2.3.0`: exposed deployments now default to loopback-only binding, require an API key on non-loopback binds, reject unsafe private-network navigation by default, and fail fast on unsupported proxy deployment assumptions.
+
+### Added
+- **OpenAPI 3.1.0 specification** at `/openapi.json` with request/response schemas, auth requirements, and representative route coverage.
+- **Interactive Swagger UI** at `/api/docs` for live inspection and request testing.
+- **Fingerprint environment controls** for `CAMOFOX_OS`, `CAMOFOX_ALLOW_WEBGL`, `CAMOFOX_SCREEN_WIDTH`, `CAMOFOX_SCREEN_HEIGHT`, and `CAMOFOX_HUMANIZE`.
+- **Idle lifecycle policy** with staged cleanup (`CAMOFOX_IDLE_TIMEOUT_MS`) and daemon exit (`CAMOFOX_IDLE_EXIT_TIMEOUT_MS`).
+- **Session-level proxy/geo overrides** through `proxyProfile`, raw `proxy` fields, and `geoMode`.
+- **OpenClaw proxy/geo parity** for `/tabs/open`.
+- **Structured extraction** across core API, CLI, and OpenClaw with schema validation and deterministic JSON output.
+
+### Changed
+- **Session identity and reuse** now key proxy/geo behavior on `userId + sessionKey` instead of `userId` alone.
+- **Context pool eviction** now uses `profileKey`, preventing sibling sessions from evicting each other incorrectly.
+- **OpenAPI docs behavior** now derives server origin from the incoming request and safe defaults instead of assuming a single static external origin.
+
+### Fixed
+- **Proxy profile validation** now rejects malformed configuration and preserves conflict behavior when an existing session profile disagrees with new proxy/geo input.
+- **Lifecycle cleanup correctness** now avoids cleanup reentry, preserves reused/live contexts, and only arms daemon exit under valid idle conditions.
+- **Fingerprint env application** now routes screen constraints into fingerprint generation rather than launch-only options, preserving the intended sidecar semantics.
+- **Structured extraction contracts** now reject invalid root schemas/selectors and align API, CLI, and OpenClaw error semantics.
+- **OpenAPI request contracts** now mark required fields correctly and remove mismatched schema claims such as unsupported `/act` coverage.
+
+### Security
+- Default server bind is `127.0.0.1` via `CAMOFOX_HOST`, and non-loopback binds require `CAMOFOX_API_KEY`.
+- Navigation target validation blocks loopback/private/link-local/metadata hosts by default on exposed deployments unless `CAMOFOX_ALLOW_PRIVATE_NETWORK=true`.
+- Proxy-enabled exposed deployments fail fast unless the operator explicitly opts into private-network allowance.
+
+### Docs
+- README, skills, and agent-facing references were updated to document the shipped Wave 2 surfaces.
+- OpenAPI discovery wording, subset-scope wording, request contracts, and origin handling were corrected to match actual shipped behavior.
+
+### Tests
+- Added E2E coverage for security hardening, proxy/geo overrides, OpenClaw proxy/geo support, fingerprint env controls, lifecycle cleanup/exit, OpenAPI docs, and structured extraction.
+- Added unit coverage for profile-key eviction, lifecycle state handling, proxy profile parsing, structured extractor schema/runtime contracts, and URL security validation.
+
+## [2.3.0] - 2026-05-03
+
+### Upgrade Notes
+- **New Wave 1 surfaces** add trace artifact retrieval and image-only extraction on top of the existing tracing/resource services. These are additive endpoints and do not remove any previous route or alias.
+- **Conditional auth coverage** now includes the image listing route when `CAMOFOX_API_KEY` is set, aligning it with the surrounding extraction/tracing surfaces.
+
+### Added
+- **Trace artifact management** — `GET /sessions/:userId/traces`, `GET /sessions/:userId/traces/:filename`, and `DELETE /sessions/:userId/traces/:filename`
+- **Image listing route** — `GET /tabs/:tabId/images` for image-only extraction with selector, extension, blob-resolution, and lazy-load options
+- **Wave 1 regression coverage** for trace ownership/path handling, timeout cleanup, chunk-stop coordination, and image-route auth/behavior
+
+### Fixed
+- Trace artifact ownership now uses collision-safe owner tokens rather than lossy userId sanitization
+- Trace artifact handling now rejects spoofed paths, keeps managed files inside the traces root, and tolerates vanished files during list operations
+- Trace timeout cleanup now stays coordinated with both manual stop and in-flight chunk-stop operations
+- `extractImages()` no longer requires a fake `userId` shim in its shared extractor contract
+
+### Changed
+- README and release metadata now reflect shipped Wave 1 trace/image capabilities
+- Package and OpenClaw plugin versions now advance together to `2.3.0`
+
+## [2.2.1] - 2026-04-09
+
+### Changed
+- Version bump for release-prep (v2.2.0 tag exists; this patch carries final release framing)
+
+## [2.2.0] - 2026-04-09
+
+### Upgrade Notes
+- **Local-state sidecar versioning** introduces fail-closed compatibility checks. If local state files are incompatible with the running version, the server will refuse to start the affected session and log an error with the specific path to delete. For sidecar metadata files, only the indicated file needs removal. For profile-level incompatibilities (e.g., Camoufox engine version mismatch), the error may indicate deleting the entire profile directory — follow the error message guidance.
+- **API key guard** is now conditionally applied to core and OpenClaw protected endpoints (tab creation, navigation, interaction, session management, downloads, tracing, console) when `CAMOFOX_API_KEY` is set. The `POST /stop` route requires `CAMOFOX_ADMIN_KEY` unconditionally. Unset deployments are unaffected.
+
+### Added
+- **Conditional API-key guard** (`CAMOFOX_API_KEY`) on core and OpenClaw protected endpoints — tab creation, navigation, interaction, session management, downloads, tracing, console. Guard enforced only when env var is set; unset deployments are unaffected. `POST /stop` uses a separate unconditional `CAMOFOX_ADMIN_KEY` guard
+- **Canonical profile invariants** — staged first-use, rollback-on-failure, cookie race guard
+- **Local-state sidecar versioning** with fail-closed compatibility checks and migration support
+- **Snapshot pagination** with offset-based windowing for large page snapshots
+- **OpenClaw parity** — snapshot, navigate, scroll endpoints aligned with plugin contract
+- **Macro navigate** and scroll parity with initial-download capture
+- **Plugin surface cleanup** — publish/install/plugin artifact contract validation
+
+### Fixed
+- Server env whitelist: added `DISPLAY`, `HANDLER_TIMEOUT_MS`, `MAX_CONCURRENT_PER_USER`
+- Unified CLI port and idle-timeout defaults with canonical config
+- Session lifecycle: staged first-use + rollback, cookie race guard, dist rebuild, tab-cap test
+
+### Changed
+- README, skills, and governance docs synced to shipped behavior
+
+## [2.1.1] - 2026-03-08
+
+### Fixed
+- Unknown element ref now returns HTTP 400 with guidance message instead of ambiguous error
+
+## [2.1.0] - 2026-03-08
+
+### Fixed
+- **Ref system improvements** — strict ref parsing, expanded element roles in snapshot, stale ref detection
+
 ## [2.0.5] - 2026-03-08
 
 ### Fixed

package/README.md

@@ -11,6 +11,7 @@
 
 - [Why CamoFox?](#why-camofox)
 - [Features](#features)
+- [Preview Status](#preview-status)
 - [Quick Start](#quick-start)
 - [CLI](#cli)
 - [Console Capture](#console-capture)
@@ -19,6 +20,7 @@
 - [Usage with AI Agents](#usage-with-ai-agents)
 - [Architecture](#architecture)
 - [API Reference](#api-reference)
+- [Structured Extract](#structured-extract)
 - [Search Macros](#search-macros)
 - [Geo Presets](#geo-presets)
 - [Environment Variables](#environment-variables)
@@ -50,19 +52,53 @@
 - **Multi-Session** — concurrent isolated browser contexts per `userId` (defaults: max 50 sessions, max 10 tabs/session)
 - **Persistent Browser Profiles** — Each user gets a dedicated Firefox profile. Cookies, localStorage, IndexedDB, and all browser storage persist across sessions automatically.
 - **Geo Presets** — 8 built-in region presets (locale/timezone/geolocation) + custom presets file
+- **Session-Level Proxy/Geo Overrides** — per-session proxy configuration via named profiles or raw credentials, with hybrid geo modes (`explicit-wins` or `proxy-locked`)
 - **14 Search Macros** — Google, YouTube, Amazon, Reddit (search + subreddit JSON), Wikipedia, Twitter, Yelp, Spotify, Netflix, LinkedIn, Instagram, TikTok, Twitch
 - **Element Refs** — accessibility snapshots annotated with stable `eN` element references for precise interaction
-- **Cookie Persistence** — import Netscape/Playwright-style cookies into a session (optional, gated by API key)
+- **Cookie Persistence** — import Netscape/Playwright-style cookies into a session (bearer auth required only when `CAMOFOX_API_KEY` is set)
 - **OpenClaw Plugin** — OpenClaw-compatible endpoints (`/start`, `/tabs/open`, `/act`, etc.)
 - **TypeScript** — strict mode, typed request shapes, modular Express routes
-- **YouTube Transcript Extraction** — yt-dlp primary pipeline with browser fallback
+- **YouTube Transcript Extraction** — yt-dlp + browser fallback (service-level; no public API route currently exposed)
 - **Snapshot Pagination** — offset-based windowing for large page snapshots
+- **Image Listing Route** — image-only extraction over the shared resource extractor with selector, extension, lazy-load, and blob-resolution controls
+- **Structured Extract** — deterministic schema-driven JSON extraction across core API, CLI, and OpenClaw without arbitrary JavaScript
 - **Browser Health Monitoring** — health probe with recovery/degraded state tracking
 - 🖥️ **CLI Mode** — 50+ commands for terminal-based browser automation
 - 🔐 **Auth Vault** — AES-256-GCM encrypted credential storage (LLM-safe)
 - 📜 **Pipeline Scripting** — Execute command scripts from files
 - 🔍 **Console Capture** — capture and filter browser console messages and uncaught errors
 - 📼 **Playwright Tracing** — record and export Playwright traces for debugging
+- 🗂️ **Trace Artifact Management** — list, download, and delete managed trace ZIPs per user session
+
+## Preview Status
+
+CamoFox Browser Server is in **Preview** (Phase 1). Preview releases are functional for browser automation and agent integration, but carry specific compatibility commitments and explicit non-goals.
+
+### What Preview Means
+- The REST API and CLI are usable for agent workflows today; [CamoFox MCP](https://github.com/redf0x1/camofox-mcp) is available as an external companion integration
+- New features may be added between minor versions
+- Backward-compatible aliases are maintained for renamed or moved endpoints (see [Compatibility Policy](#compatibility-policy))
+- Local state (profiles, registries, sessions) uses versioned formats with fail-closed integrity checks
+
+### What Preview Does NOT Guarantee
+- **Frozen API surface** — endpoint behavior, request shapes, or response formats may change between minor versions
+- **Automatic local-state migration** — browser profiles, download registries, and session files use versioned sidecar formats; incompatible upgrades require manual reset (see [Local State Recovery](#local-state-recovery))
+- **Downgrade safety** — rolling back to an older version may require clearing local state
+- **Fixed GA timeline** — promotion to GA requires meeting evidence-based exit criteria, not a calendar date
+
+### Compatibility Policy
+During Preview, CamoFox follows an **additive-only deprecation model**:
+- **Legacy aliases** (e.g., `listItemId` accepted alongside `sessionKey`, OpenClaw `/act` routing to core endpoints) continue to work alongside their replacements
+- **Deprecated fields** are accepted silently; no removal until GA or a documented migration window with advance notice in CHANGELOG
+- **No existing endpoint** is removed in a minor version — removals happen only in major versions with prior CHANGELOG notice
+
+### Local State Recovery
+Browser profiles, download registries, and CLI session files use versioned sidecar formats. When upgrading CamoFox:
+- **Compatible versions**: State loads normally
+- **Incompatible or corrupt state**: The server refuses to load incompatible profiles and download registries; the CLI rejects incompatible saved-session files. Both log an actionable error with the specific recovery path.
+- **Recovery**: Delete the affected profile directory, session file, or download registry as indicated in the error message. Clean state is recreated on next use.
+
+Supported sidecars include limited forward-migration paths (e.g., fingerprint v0 → v1); when no migration path exists for a given version, the server refuses to load the file and logs an actionable recovery message. There is no silent repair or downgrade path — this fail-closed default prevents data corruption at the cost of manual intervention on unsupported version jumps.
 
 ## Quick Start
 
@@ -102,6 +138,8 @@ docker run -d \
   --name camofox-browser \
   -p 9377:9377 \
   -p 6080:6080 \
+  -e CAMOFOX_HOST=0.0.0.0 \
+  -e CAMOFOX_API_KEY=change-me \
   -v ~/.camofox:/home/node/.camofox \
   camofox-browser
 ```
@@ -117,9 +155,10 @@ services:
     ports:
       - "9377:9377"
     environment:
+      CAMOFOX_HOST: "0.0.0.0"
       CAMOFOX_PORT: "9377"
-      # Optional auth gates
-      # CAMOFOX_API_KEY: "change-me"
+      # Required when CAMOFOX_HOST is non-loopback
+      CAMOFOX_API_KEY: "change-me"
       # CAMOFOX_ADMIN_KEY: "change-me"
       # Optional: proxy routing (also enables Camoufox geoip mode)
       # PROXY_HOST: ""
@@ -146,7 +185,7 @@ CamoFox Browser includes a powerful CLI for browser automation directly from the
 npm install -g camofox-browser
 
 # Or use npx (no install needed)
-npx camofox open https://example.com
+npx camofox-browser open https://example.com
 ```
 
 ### Quick Start
@@ -176,6 +215,7 @@ camofox get-url                        # Get current page URL
 camofox get-text                       # Get page text content
 camofox get-links                      # Get all links on page
 camofox get-tabs                       # List open tabs
+camofox extract-structured @schema.json # Extract deterministic JSON from a schema
 
 # Interaction
 camofox click <ref>                    # Click element by ref
@@ -426,57 +466,75 @@ AI Agent (MCP / OpenClaw / REST Client)
 
 Base URL: `http://localhost:9377`
 
+> **Security defaults:** `CAMOFOX_HOST` now defaults to `127.0.0.1`. If you bind beyond loopback (for example `0.0.0.0` in Docker or PaaS), `CAMOFOX_API_KEY` becomes required at startup. On non-loopback binds, navigation targets on loopback/private/link-local/metadata hosts are blocked by default unless you explicitly set `CAMOFOX_ALLOW_PRIVATE_NETWORK=true`. If you also configure `PROXY_HOST`/`PROXY_PORT`, exposed deployments must opt into `CAMOFOX_ALLOW_PRIVATE_NETWORK=true` until proxy-side private-target validation exists.
+
+### API Documentation
+
+The Camofox Browser API includes OpenAPI 3.1.0 docs for a representative subset of the shipped route surface:
+
+- **Interactive API Explorer**: [http://localhost:9377/api/docs](http://localhost:9377/api/docs) — Swagger UI with live request testing
+- **OpenAPI Specification**: [http://localhost:9377/openapi.json](http://localhost:9377/openapi.json) — Machine-readable OpenAPI 3.1.0 spec
+
+The OpenAPI spec covers a representative subset of core and OpenClaw endpoints, including request schemas, response shapes, authentication requirements, and validation rules.
+
 ### Core Endpoints
 
 Note: For any endpoint that targets an existing tab (`/tabs/:tabId/...`), the server resolves `tabId` **within a `userId` scope**. If you omit `userId`, you will typically get `404 Tab not found`.
 
 | Method | Endpoint | Description | Required | Auth |
 |--------|----------|-------------|----------|------|
-| POST | `/sessions/:userId/cookies` | Import cookies into a user session (Playwright cookie objects) | Path: `userId`; Body: `{ "cookies": Cookie[] }` | `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/sessions/:userId/cookies` | Import cookies into a user session (Playwright cookie objects) | Path: `userId`; Body: `{ "cookies": Cookie[] }` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/health` | Health check (also pre-launches the browser) | None | None |
 | GET | `/presets` | List available geo presets (built-in + custom) | None | None |
-| POST | `/tabs` | Create a new tab (supports `preset` + per-field overrides) | Body: `userId` + (`sessionKey` or `listItemId`) | None |
+| POST | `/tabs` | Create a new tab (supports `preset` + per-field overrides) | Body: `userId` + (`sessionKey` or `listItemId`) | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/tabs?userId=...` | List all tabs for a user (OpenClaw-compatible response shape) | Query: `userId` | None |
-| POST | `/tabs/:tabId/navigate` | Navigate to a URL, or expand a search `macro` + `query` | Body: `userId` + (`url` or `macro`) | None |
+| POST | `/tabs/:tabId/navigate` | Navigate to a URL, or expand a search `macro` + `query` | Body: `userId` + (`url` or `macro`) | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/tabs/:tabId/snapshot?userId=...` | Accessibility snapshot annotated with `eN` element refs | Query: `userId` | None |
-| POST | `/tabs/:tabId/wait` | Wait for page readiness (DOM + optional network idle) | Body: `userId` | None |
-| POST | `/tabs/:tabId/click` | Click by `ref` (e.g. `e12`) or CSS `selector` | Body: `userId` + (`ref` or `selector`) | None |
-| POST | `/tabs/:tabId/type` | Type into an element by `ref` or CSS `selector` | Body: `userId` + (`ref` or `selector`) + `text` | None |
-| POST | `/tabs/:tabId/press` | Press a key (e.g. `Enter`, `Escape`) | Body: `userId` + `key` | None |
-| POST | `/tabs/:tabId/scroll` | Scroll up/down by pixels | Body: `userId` | None |
-| POST | `/tabs/:tabId/scroll-element` | Scroll specific element into view | Body: userId, ref/selector | None |
-| POST | `/tabs/:tabId/back` | Go back | Body: `userId` | None |
-| POST | `/tabs/:tabId/forward` | Go forward | Body: `userId` | None |
-| POST | `/tabs/:tabId/refresh` | Refresh | Body: `userId` | None |
+| POST | `/tabs/:tabId/wait` | Wait for page readiness (DOM + optional network idle) | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/click` | Click by `ref` (e.g. `e12`) or CSS `selector` | Body: `userId` + (`ref` or `selector`) | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/type` | Type into an element by `ref` or CSS `selector` | Body: `userId` + (`ref` or `selector`) + `text` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/press` | Press a key (e.g. `Enter`, `Escape`) | Body: `userId` + `key` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/scroll` | Scroll up/down/left/right by pixels | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/scroll-element` | Scroll specific element into view | Body: userId, ref/selector | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/back` | Go back | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/forward` | Go forward | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/refresh` | Refresh | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/tabs/:tabId/links?userId=...&limit=50&offset=0` | Extract links (paginated) | Query: `userId` | None |
+| GET | `/tabs/:tabId/images?userId=...` | List extracted images | Query: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/tabs/:tabId/screenshot?userId=...&fullPage=true` | Screenshot (PNG bytes) | Query: `userId` | None |
 | GET | `/tabs/:tabId/stats?userId=...` | Tab stats + visited URLs | Query: `userId` | None |
-| DELETE | `/tabs/:tabId` | Close a tab (expects JSON body: `{ "userId": "..." }`) | Body: `userId` | None |
-| DELETE | `/tabs/group/:listItemId` | Close a tab group (expects JSON body: `{ "userId": "..." }`) | Body: `userId` | None |
-| DELETE | `/sessions/:userId` | Close all sessions for a user | Path: `userId` | None |
+| DELETE | `/tabs/:tabId` | Close a tab (expects JSON body: `{ "userId": "..." }`) | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| DELETE | `/tabs/group/:listItemId` | Close a tab group (expects JSON body: `{ "userId": "..." }`) | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| DELETE | `/sessions/:userId` | Close all sessions for a user | Path: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/sessions/:userId/toggle-display` | Toggle display mode (headless/headed/virtual) | Path: `userId`; Body: `{ "headless": true\|false\|"virtual" }` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/tabs/:tabId/cookies` | Export tab cookies | Query: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/tabs/:tabId/downloads` | List tab downloads | Query: `userId` | None |
 | GET | `/users/:userId/downloads` | List user downloads | Path: `userId` | None |
 | GET | `/downloads/:downloadId` | Download metadata | Query: `userId` | None |
 | GET | `/downloads/:downloadId/content` | Stream download content | Query: `userId` | None |
-| DELETE | `/downloads/:downloadId` | Delete tracked download | Body or Query: `userId` | None |
-| POST | `/tabs/:tabId/extract-resources` | Extract downloadable resources | Body: `userId` | None |
-| POST | `/tabs/:tabId/batch-download` | Batch download resources | Body: `userId` | None |
+| DELETE | `/downloads/:downloadId` | Delete tracked download | Body or Query: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/extract-resources` | Extract downloadable resources | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/batch-download` | Batch download resources | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | POST | `/tabs/:tabId/resolve-blobs` | Resolve blob URLs to base64 | Body: `userId` + `urls[]` | None |
 | POST | `/tabs/:tabId/trace/start` | Start trace recording | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | POST | `/tabs/:tabId/trace/stop` | Stop and save trace ZIP | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | POST | `/tabs/:tabId/trace/chunk/start` | Start trace chunk | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | POST | `/tabs/:tabId/trace/chunk/stop` | Stop chunk and save ZIP | Body: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/tabs/:tabId/trace/status` | Check trace status | Query: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| GET | `/sessions/:userId/traces` | List managed trace ZIPs for a user | Path: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| GET | `/sessions/:userId/traces/:filename` | Download a managed trace ZIP | Path: `userId`, `filename` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| DELETE | `/sessions/:userId/traces/:filename` | Delete a managed trace ZIP | Path: `userId`, `filename` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/tabs/:tabId/console` | Get console messages | Query: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/tabs/:tabId/errors` | Get uncaught JS errors | Query: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | POST | `/tabs/:tabId/console/clear` | Clear console + errors | Body or Query: `userId` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+| POST | `/tabs/:tabId/extract-structured` | Extract deterministic JSON from a structured schema | Body: `userId` + `schema` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 
 ### Toggle Display Mode
 ```bash
 POST /sessions/:userId/toggle-display
 {"headless": "virtual"}
 ```
+**Auth:** Conditional — requires `Authorization: Bearer $CAMOFOX_API_KEY` when `CAMOFOX_API_KEY` is set.
 Switch browser between headless and headed mode. When encountering CAPTCHAs or issues requiring visual interaction, switch to headed mode to show the browser window.
 
 Returns:
@@ -538,12 +596,75 @@ OpenClaw-compatible aliases (used by the OpenClaw plugin).
 | Method | Endpoint | Description | Required | Auth |
 |--------|----------|-------------|----------|------|
 | GET | `/` | Status (alias of `/health`) | None | None |
-| POST | `/tabs/open` | Open tab (OpenClaw request/response shape) | Body: `userId` + `url` | None |
+| POST | `/tabs/open` | Open tab (OpenClaw request/response shape) | Body: `userId` + `url` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | POST | `/start` | Start browser engine | None | None |
 | POST | `/stop` | Stop browser engine | None | `x-admin-key: $CAMOFOX_ADMIN_KEY` |
-| POST | `/navigate` | Navigate (OpenClaw request shape: `targetId` in body) | Body: `userId` + `targetId` + `url` | None |
+| POST | `/navigate` | Navigate (OpenClaw request shape: `targetId` in body) | Body: `userId` + `targetId` + `url` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
 | GET | `/snapshot?userId=...&targetId=...` | Snapshot (OpenClaw response shape) | Query: `userId` + `targetId` | None |
-| POST | `/act` | Combined actions (`click`, `type`, `press`, `scroll`, `scrollIntoView`, `hover`, `wait`, `close`) | Body: `userId` + `targetId` + `kind` | None |
+| POST | `/act` | Combined actions (`click`, `type`, `press`, `scroll`, `scrollIntoView`, `hover`, `wait`, `close`, `extractStructured`) | Body: `userId` + `targetId` + `kind` | Conditional: `Authorization: Bearer $CAMOFOX_API_KEY` |
+
+### Structured Extract
+
+Structured extract returns deterministic JSON from a DOM schema without arbitrary JavaScript. Use it when you want stable data contracts instead of ad-hoc `evaluate()` calls.
+
+Core API:
+
+```bash
+curl -X POST "$CAMOFOX_URL/tabs/$TAB_ID/extract-structured" \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "userId": "agent1",
+    "schema": {
+      "kind": "object",
+      "fields": {
+        "title": { "kind": "text", "selector": "h1", "required": true, "trim": true },
+        "products": {
+          "kind": "list",
+          "selector": ".product",
+          "item": {
+            "kind": "object",
+            "fields": {
+              "name": { "kind": "text", "selector": ".name", "required": true, "trim": true },
+              "href": { "kind": "url", "selector": "a.product-link", "attr": "href", "required": true }
+            }
+          }
+        }
+      }
+    }
+  }'
+```
+
+CLI:
+
+```bash
+camofox extract-structured @schema.json <tabId> --user <userId> --format json
+```
+
+OpenClaw:
+
+```bash
+curl -X POST "$CAMOFOX_URL/act" \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "kind": "extractStructured",
+    "targetId": "tab-123",
+    "userId": "agent1",
+    "schema": {
+      "kind": "object",
+      "fields": {
+        "title": { "kind": "text", "selector": "h1", "required": true }
+      }
+    }
+  }'
+```
+
+Notes:
+
+- invalid schemas fail with HTTP 400
+- required runtime misses fail the whole request with HTTP 422 and `fieldPath`
+- optional scalar/object/list nodes normalize to `null` / `null` / `[]`
+- selectors must be CSS; no XPath, arbitrary JavaScript, or AI extraction
+- raw resource extraction and structured extraction stay separate on purpose
 
 ## Search Macros
 
@@ -591,6 +712,119 @@ curl -X POST http://localhost:9377/tabs \
 
 Custom presets: set `CAMOFOX_PRESETS_FILE=/path/to/presets.json` (JSON object; keys become preset names).
 
+## Session-Level Proxy and Geo Overrides
+
+CamoFox supports session-level proxy and geolocation configuration with a hybrid model that combines server defaults with per-session overrides.
+
+### Proxy Configuration Model
+
+**Server-level baseline** (via environment variables):
+- `PROXY_HOST`, `PROXY_PORT`, `PROXY_USERNAME`, `PROXY_PASSWORD` — applied as the default for all sessions
+- Enables Camoufox geoip mode when configured
+
+**Session-level overrides** (via `POST /tabs` or CLI):
+- `proxyProfile` — select a named proxy profile from `CAMOFOX_PROXY_PROFILES_FILE`
+- `proxy` — provide raw proxy fields (`host`, `port`, `username`, `password`) directly
+- Session-level proxy overrides the server baseline for that specific `userId + sessionKey` combination
+
+**Session identity rules**:
+- The same `userId` may run different `sessionKey` profiles in parallel with different proxy/geo configurations
+- The same `userId + sessionKey` combination maintains a stable proxy/geo identity — requests with conflicting proxy/geo fields are rejected
+- Session reuse and cleanup scope proxy/geo identity by `userId + sessionKey`, not just `userId`
+
+### Geo Mode Behavior
+
+CamoFox offers two geo modes that control how explicit geo fields (locale, timezone, geolocation) interact with proxy-derived geo:
+
+**`geoMode=explicit-wins`** (default):
+- Explicit geo fields (locale, timezone, geolocation) remain authoritative
+- Proxy-derived geo suggestions are ignored
+- Use this mode when you want precise geo control regardless of proxy location
+
+**`geoMode=proxy-locked`**:
+- Explicit geo fields that conflict with proxy-derived geo are rejected
+- Proxy-derived geo is authoritative
+- Use this mode to ensure geo consistency with proxy exit location
+
+### CLI Examples
+
+Session-level proxy with named profile:
+```bash
+camofox open https://example.com --proxy-profile tokyo-exit --user agent1
+```
+
+Session-level proxy with raw fields:
+```bash
+camofox open https://example.com \
+  --proxy-host proxy.example.com \
+  --proxy-port 8080 \
+  --proxy-username user \
+  --proxy-password pass \
+  --user agent1
+```
+
+Combine proxy with geo mode:
+```bash
+camofox open https://example.com \
+  --proxy-profile london-exit \
+  --geo uk \
+  --geo-mode proxy-locked \
+  --user agent1
+```
+
+### API Examples
+
+Using a named proxy profile with explicit geo:
+```bash
+curl -X POST http://localhost:9377/tabs \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "userId": "agent1",
+    "sessionKey": "task1",
+    "url": "https://example.com",
+    "proxyProfile": "tokyo-exit",
+    "preset": "japan",
+    "geoMode": "explicit-wins"
+  }'
+```
+
+Using raw proxy fields with proxy-locked geo:
+```bash
+curl -X POST http://localhost:9377/tabs \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "userId": "agent1",
+    "sessionKey": "task2",
+    "url": "https://example.com",
+    "proxy": {
+      "host": "proxy.example.com",
+      "port": 8080,
+      "username": "user",
+      "password": "pass"
+    },
+    "geoMode": "proxy-locked"
+  }'
+```
+
+### Named Proxy Profiles
+
+Define proxy profiles in a JSON file and point `CAMOFOX_PROXY_PROFILES_FILE` to it:
+
+```json
+{
+  "tokyo-exit": {
+    "server": "http://tokyo.proxy.example.com:8080",
+    "username": "user",
+    "password": "pass"
+  },
+  "london-exit": {
+    "server": "http://london.proxy.example.com:8080"
+  }
+}
+```
+
+Then use profiles by name in API requests or CLI commands.
+
 ## Environment Variables
 
 | Variable | Default | Description |
@@ -598,8 +832,10 @@ Custom presets: set `CAMOFOX_PRESETS_FILE=/path/to/presets.json` (JSON object; k
 | `CAMOFOX_PORT` | `9377` | Server port |
 | `PORT` | (optional) | Alternative port env var (common in PaaS) |
 | `NODE_ENV` | `development` | Node environment |
+| `CAMOFOX_HOST` | `127.0.0.1` | Server bind host. Set `0.0.0.0` for Docker/PaaS/network exposure. Non-loopback binds require `CAMOFOX_API_KEY`. |
 | `CAMOFOX_ADMIN_KEY` | (empty) | Required for `POST /stop` (sent via `x-admin-key`) |
-| `CAMOFOX_API_KEY` | (empty) | Enables cookie import endpoint; sent via `Authorization: Bearer ...` |
+| `CAMOFOX_API_KEY` | (empty) | Guards protected endpoints (tab creation, navigation, interaction, session management, downloads, image extraction, tracing, console) via `Authorization: Bearer` header when set. Required whenever `CAMOFOX_HOST` exposes the server beyond loopback. |
+| `CAMOFOX_ALLOW_PRIVATE_NETWORK` | `true` on loopback binds, `false` otherwise | Allows navigation to loopback/private/link-local/metadata targets. Leave unset for the safe default; enable only for trusted deployments that intentionally need internal-network reachability. |
 | `CAMOFOX_HEADLESS` | `true` | Display mode: `true` (headless), `false` (headed), `virtual` (Xvfb) |
 | `CAMOFOX_VNC_RESOLUTION` | `1920x1080x24` | Virtual Xvfb display resolution (`WIDTHxHEIGHTxDEPTH`) |
 | `CAMOFOX_VNC_TIMEOUT_MS` | `120000` | Max VNC session duration in ms before auto-stop |
@@ -618,15 +854,17 @@ Custom presets: set `CAMOFOX_PRESETS_FILE=/path/to/presets.json` (JSON object; k
 | `CAMOFOX_VNC_BASE_PORT` | `6080` | noVNC/websockify base port |
 | `CAMOFOX_VNC_HOST` | `localhost` | noVNC host in returned URL |
 | `CAMOFOX_CLI_USER` | `cli-default` | Default CLI user id |
-| `CAMOFOX_IDLE_TIMEOUT_MS` | `1800000` | CLI server idle timeout |
+| `CAMOFOX_IDLE_TIMEOUT_MS` | `1800000` | Stage 1 idle cleanup threshold (ms) |
+| `CAMOFOX_IDLE_EXIT_TIMEOUT_MS` | `1800000` | Stage 2 daemon exit quiet window (ms, defaults to match Stage 1) |
 | `CAMOFOX_PRESETS_FILE` | (unset) | Optional JSON file defining/overriding geo presets |
+| `CAMOFOX_PROXY_PROFILES_FILE` | (unset) | Optional JSON file defining named proxy profiles for session-level overrides |
 | `CAMOFOX_SESSION_TIMEOUT` | `1800000` | Session idle timeout in ms (min `60000`) |
 | `CAMOFOX_MAX_SESSIONS` | `50` | Maximum concurrent sessions |
 | `CAMOFOX_MAX_TABS` | `10` | Maximum tabs per session |
-| `PROXY_HOST` | (empty) | Proxy host (enables proxy routing) |
-| `PROXY_PORT` | (empty) | Proxy port |
-| `PROXY_USERNAME` | (empty) | Proxy username |
-| `PROXY_PASSWORD` | (empty) | Proxy password |
+| `PROXY_HOST` | (empty) | Proxy host (server-level default; enables proxy routing) |
+| `PROXY_PORT` | (empty) | Proxy port (server-level default) |
+| `PROXY_USERNAME` | (empty) | Proxy username (server-level default) |
+| `PROXY_PASSWORD` | (empty) | Proxy password (server-level default) |
 | `CAMOFOX_MAX_SNAPSHOT_CHARS` | `80000` | Max characters in snapshot before truncation |
 | `CAMOFOX_SNAPSHOT_TAIL_CHARS` | `5000` | Characters preserved at end of truncated snapshot |
 | `CAMOFOX_BUILDREFS_TIMEOUT_MS` | `12000` | Timeout for building element refs |
@@ -635,6 +873,34 @@ Custom presets: set `CAMOFOX_PRESETS_FILE=/path/to/presets.json` (JSON object; k
 | `CAMOFOX_FAILURE_THRESHOLD` | `3` | Consecutive failures before health degradation |
 | `CAMOFOX_YT_DLP_TIMEOUT_MS` | `30000` | Timeout for yt-dlp subtitle extraction |
 | `CAMOFOX_YT_BROWSER_TIMEOUT_MS` | `25000` | Timeout for browser transcript fallback |
+| `CAMOFOX_OS` | (unset) | Optional server-wide Camoufox OS override (`windows`, `macos`, `linux`, or comma-separated list for randomization) |
+| `CAMOFOX_ALLOW_WEBGL` | (unset) | Optional server-wide WebGL override; malformed values fail startup |
+| `CAMOFOX_SCREEN_WIDTH` | (unset) | Optional screen width override; applied only with `CAMOFOX_SCREEN_HEIGHT` |
+| `CAMOFOX_SCREEN_HEIGHT` | (unset) | Optional screen height override; applied only with `CAMOFOX_SCREEN_WIDTH` |
+| `CAMOFOX_HUMANIZE` | (unset) | Optional server-wide humanization override |
+
+> `CAMOFOX_OS` and `CAMOFOX_SCREEN_*` are **generation-time** controls: they affect only newly generated fingerprints and have no effect while an existing `fingerprint.json` sidecar is in use. Reset the profile or delete `fingerprint.json` to force regeneration under the new defaults. `CAMOFOX_ALLOW_WEBGL` and `CAMOFOX_HUMANIZE` are **launch-time** overrides and apply on every browser launch regardless of whether a sidecar is reused.
+
+### Idle Lifecycle Policy
+
+CamoFox implements a two-stage idle lifecycle for graceful cleanup and daemon exit:
+
+**Stage 1 — Idle Cleanup**
+- After `CAMOFOX_IDLE_TIMEOUT_MS` of idle time (default: 30 minutes), the server runs cleanup to close idle sessions and tabs
+- Cleanup is delayed if browser contexts are launching or sessions are being created
+- New interactive activity (tab creation, navigation, interaction) cancels any pending cleanup
+
+**Stage 2 — Daemon Exit**
+- After Stage 1 cleanup completes, the server waits for `CAMOFOX_IDLE_EXIT_TIMEOUT_MS` (default: matches Stage 1 timeout)
+- If no new activity occurs during this quiet window, the daemon process exits gracefully
+- Any new request activity cancels the pending exit timer
+
+**Activity detection**:
+- Live tabs, launching browser contexts, or staged session creation count as active work and prevent cleanup
+- Empty sessions (sessions with no tabs) do not block cleanup but do disarm pending daemon exit
+- New interactive activity resets both cleanup and exit timers
+
+This two-stage model ensures cleanup runs before daemon exit, preventing resource leaks while allowing the server to shut down cleanly when fully idle.
 
 ## Deployment
 
@@ -644,7 +910,9 @@ Custom presets: set `CAMOFOX_PRESETS_FILE=/path/to/presets.json` (JSON object; k
 docker build -t camofox-browser .
 docker run -p 9377:9377 -p 6080:6080 \
   -v ~/.camofox:/home/node/.camofox \
+  -e CAMOFOX_HOST=0.0.0.0 \
   -e CAMOFOX_PORT=9377 \
+  -e CAMOFOX_API_KEY=change-me \
   camofox-browser
 ```
 
@@ -660,6 +928,8 @@ fly deploy
 ### Railway
 
 - Create a new project → deploy from this GitHub repo
+- Set `CAMOFOX_HOST=0.0.0.0`
+- Set `CAMOFOX_API_KEY` to a strong secret
 - Set `CAMOFOX_PORT=9377` (Railway will also provide `PORT`, which is supported)
 - Ensure the service exposes port `9377`
 
@@ -667,6 +937,8 @@ fly deploy
 
 - Create a new Web Service → deploy from this GitHub repo
 - Use Docker (recommended) and expose port `9377`
+- Set `CAMOFOX_HOST=0.0.0.0`
+- Set `CAMOFOX_API_KEY` to a strong secret
 - Set `CAMOFOX_PORT=9377` (or rely on Render `PORT`)
 
 ### System Requirements
@@ -747,4 +1019,3 @@ This project is based on [camofox-browser](https://github.com/jo-inc/camofox-bro
 ## Crypto Scam Warning
 
 Sketchy people are doing sketchy things with crypto tokens named "Camofox" now that this project is getting attention. **Camofox is not a crypto project and will never be one.** Any token, coin, or NFT using the Camofox name has nothing to do with us.
-

package/dist/src/cli/commands/content.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"content.d.ts","sourceRoot":"","sources":["../../../../src/cli/commands/content.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAKpC,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,UAAU,CAAC;AAgE3C,wBAAgB,uBAAuB,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,UAAU,GAAG,IAAI,CA0RnF"}
+{"version":3,"file":"content.d.ts","sourceRoot":"","sources":["../../../../src/cli/commands/content.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAKpC,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,UAAU,CAAC;AAkF3C,wBAAgB,uBAAuB,CAAC,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,UAAU,GAAG,IAAI,CAoTnF"}