@wenathlan/saddle 1.8.6 → 1.8.8

package/README.md

@@ -1,22 +1,20 @@
-# Saddle
-
 <p align="center">
   <img src="docs/assets/saddlemark.svg" alt="Saddle" width="720" />
 </p>
 
 <p align="center">
   <strong>Storage-backed jobs, scraping contracts and portable runners for Node.js.</strong><br/>
-  <strong>Binary computing agent, agent browser, computer-use, scraper and packager.</strong><br/>
+  <strong>Binary computing engine, agent browser, scraper and packager.</strong><br/>
   <a href="https://github.com/wenathlan/saddle/actions/workflows/ci.yml"><img src="https://github.com/wenathlan/saddle/actions/workflows/ci.yml/badge.svg" alt="CI" /></a>
-  <a href="https://github.com/wenathlan/saddle/releases/tag/v1.8.6"><img src="https://img.shields.io/badge/release-v1.8.6-d35d3d" alt="Release 1.8.6" /></a>
+  <a href="https://github.com/wenathlan/saddle/releases/tag/v1.8.8"><img src="https://img.shields.io/badge/release-v1.8.8-d35d3d" alt="Release 1.8.8" /></a>
   <a href="https://github.com/wenathlan/saddle/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-GPL--3.0-202a2f" alt="GPL 3.0 license" /></a>
 </p>
 
-> **Core idea:** storage is the durable side of the working set; the runner is replaceable; the artifact is the boundary. **Storage == Compute** — RAM and disk are the same construct, differing only by usage flag.
+> **Core idea:** storage is the durable side of the working set; the runner is replaceable; the artifact is the boundary. **Storage == Compute** means that the same bytes can be retained or processed according to an explicit usage flag.
 
-Saddle is a **JavaScript ESM engine** for jobs that move data between storage, a working set, an injected runner and durable artifacts. It is also a **virtual machine you publish as a package** that runs on other people's computers (GitHub Actions, Forgejo, Gitea, GitLab, Codeberg, free Docker containers) and turns unlimited third-party storage buckets into virtual RAM/GPU/CPU. Nothing runs on the operator's local machine.
+Saddle is a **JavaScript ESM engine** for jobs that move data between storage, a bounded working set, a caller-injected runner and durable artifacts. It is also a virtual machine published as a package: the caller can run it on GitHub Actions, Forgejo, Gitea, GitLab, Codeberg, Docker or another third-party compute surface. The engine does not require the operator's local machine, does not embed credentials and does not choose a mandatory cloud provider.
 
-Ships as a library, CLI, binary, n8n node, CRX extension, Android/iOS and Tauri desktop app. The canonical JavaScript package is `@wenathlan/saddle`; GitHub Packages npm, Maven and GHCR use the transferred `wenathlan` owner namespace, while NuGet and RubyGems retain their unscoped ecosystem package names.
+The canonical JavaScript package is `@wenathlan/saddle`. GitHub Packages npm, Maven and GHCR use the `wenathlan` owner namespace; NuGet and RubyGems retain their ecosystem package names. Older `@devthink`, `@iakadion` and `io.devthink` references in archived documents are historical records, not current package identities.
 
 ## Start here
 
@@ -35,160 +33,173 @@ const context = formatforagent(result, { maxchunksize: 2000, keypoints: 4 });
 console.log(context.summary);
 ```
 
-Deterministic example with no network:
+The deterministic examples and tests do not require network access or real credentials:
 
 ```bash
 node examples/publicapi.js
+npm test
 ```
 
-## What is included
+## Progressive architecture
+
+The project documentation follows a progressive arc. The foundation describes the storage and runner model; the engine describes the contracts that make the model executable; productization describes the package, extension, workflow and web surfaces.
+
+### Foundation: storage, runners and working sets
+
+Saddle treats a repository, bucket or object store as durable state and a third-party runner as a replaceable processor. GitHub Actions is one adapter, not the core. Forgejo, Gitea, GitLab, Codeberg, Docker and caller-owned runners can implement the same runner contracts.
+
+The physical limit remains explicit: remote storage is not VRAM. A storage-to-RAM bridge can stage a bounded working set through a local filesystem, tmpfs, mmap, cache or caller-owned storage adapter, but it cannot remove network latency or create the bandwidth of a GPU bus. The engine exposes that distinction instead of hiding it behind marketing language.
+
+The execution model is:
+
+```text
+repository or bucket -> runner working set -> process -> durable artifact
+        persistent state       virtual processor       published boundary
+```
+
+The repository may act as a disk, a CI workflow may act as a function call, Pages may act as a static bus and a release artifact may act as the durable boundary. `workflow_dispatch`, `repository_dispatch` and HTTP adapters remain caller-configured interfaces.
+
+### Engine: contracts instead of vendor lock-in
 
-| Area | Contract | Result |
+| Area | Contracts shipped | Result |
 | --- | --- | --- |
-| Jobs | `engine`, `scheduler`, `inprocess` | `prepare → process → sync → cleanup` |
-| Storage | local, chunked, content-addressed, S3-compatible, GitHub Contents, file hosting | durable objects, ranges, dedupe and sync |
-| Working set | memory bridge, modes, objects, transforms | storage-to-compute and compute-to-storage |
-| Scraping | robots, cache, extraction, semantic facts, schema, scraper | text, metadata, links, controls and structured output |
-| Crawl | normalization, priority frontier, BFS crawler, per-domain budgets and persistent frontier | bounded domain-aware crawling |
-| Browser | snapshots, tabs, frames, actions, fingerprint, session, replay and injected agent | browser actions without vendor lock-in |
+| Jobs | `engine`, `scheduler`, `inprocess` | `prepare -> process -> sync -> cleanup` |
+| Storage | local, chunked, content-addressed, S3-compatible, GitHub Contents and file-hosting adapters | durable objects, ranges, dedupe and sync |
+| Working set | memory bridge, modes, objects and transforms | storage-to-compute and compute-to-storage flows |
+| Scraping | robots, cache, extraction, semantic facts, schema and normalization | bounded text, metadata, links, controls and structured output |
+| Crawl | normalization, priority frontier, BFS crawler and persistent frontier contracts | domain-aware bounded crawling |
+| Browser | snapshots, tabs, frames, actions, fingerprint, session and replay contracts | caller-owned browser automation without a mandatory provider |
 | Operations | queues, idempotency, saga, retry, circuit breaker, health and heartbeat | controlled execution and recovery |
 | Protocols | JSON, NDJSON, SSE, blocks, API envelopes and MCP | transport-neutral messages |
-| Delivery | manifests, workflow registry, binary/container plans | package and runner surfaces |
+| Delivery | manifests, workflow registry, extension packaging and release assets | repeatable package and runner surfaces |
 | Integrations | GitHub, GitLab, Forgejo, app lifecycle, command scopes and delivery adapters | caller-owned provider connectivity |
-| Agent Browser | capture & replay, stealth, fingerprint | Brave capture, movement replay, session recording |
-| Compute Backends | github-actions, huggingface, gitlab-ci, kaggle, oracle-cloud | free runners chain |
-| Storage Backends | HF, Kaggle, Terabox, R2, Telegram, Discord via rclone | unlimited disk as RAM |
-| Extension | Manifest V3 bridge, snapshot protocol, popup and service worker | user initiated browser control |
+
+The root entry point is transport-neutral. Node filesystem, HTTP server, persistent sessions and Playwright are explicit subpaths or optional adapters. The library accepts caller-provided fetchers, browser transports, storage adapters, persistence, proxy pools, captcha evidence handlers, webhook secrets and remote credentials.
+
+### Productization: one engine, many shells
+
+The same contracts can be surfaced as an npm library, CLI, binary, Manifest V3 browser extension, webhook server, MCP transport, workflow action, container image, Maven package, NuGet package or RubyGem. These surfaces are adapters around the engine; they are not separate sources of truth.
 
 ## Public API
 
 | Export | Purpose |
 | --- | --- |
-| `saddleurl` | choose fetch or injected browser path |
-| `scrapeurl` | fetch one URL and extract |
-| `scrapehtml` | extract from HTML without network |
+| `saddleurl` | choose a fetch or caller-injected browser path |
+| `scrapeurl` | fetch one URL and extract bounded content |
+| `scrapehtml` | extract from HTML without network access |
 | `extractcontent` | structured extraction |
-| `serializeresult` | serialize as JSON, Markdown, XML |
-| `formatforagent` | summary, chunks, token count |
+| `serializeresult` | serialize JSON, Markdown or XML results |
+| `formatforagent` | summary, chunks and token count |
 | `batchscrape` | bounded URL groups |
-| `crawlurl` | crawl contract |
-| `browseragent` | navigation, click, type, screenshot |
-| `mcpserver` / `mcptransport` | MCP tools over JSONL/HTTP |
+| `crawlurl` | crawl contract with domain and budget controls |
+| `browseragent` | caller-owned navigation, click, type and screenshot actions |
+| `mcpserver` / `mcptransport` | MCP tools over JSONL or HTTP |
 | `nodeserver` | Web Request/Response handler |
+| `engine` / `scheduler` | job lifecycle and runner dispatch |
+| `release-assets` | SHA256SUMS, SBOM and provenance metadata for caller-selected artifacts |
 
-Complete API: `docs/libraryapi.md`. Surface overview: [`docs/productindex.md`](docs/productindex.md). Usage examples: [`docs/usage.md`](docs/usage.md).
-
-## The execution model
-
-Saddle coordinates contracts instead of hiding providers. A repo + CI runner is a virtual processor:
-
-- Repo = Disk (persistent state)
-- CI = CPU (workflow_dispatch = function call)
-- Pages = Bus + CDN
-- Static site = BIOS
-- repository_dispatch = IPC
-
-```js
-import { engine, eventbus, inprocess, scheduler } from "@wenathlan/saddle";
-import { localmemory } from "@wenathlan/saddle/memory-node";
-import { localstorage } from "@wenathlan/saddle/storage-node";
-const events = eventbus();
-const run = engine({
-  storage: localstorage("./.saddle-data"),
-  memory: localmemory(),
-  scheduler: scheduler([inprocess()]),
-  events
-});
-const result = await run.run(
-  { name: "example", input: { value: 42 } },
-  ({ job }) => ({ jobid: job.id, ok: true })
-);
-```
-
-The caller still chooses how to provide `fetcher`, browser transport, persistence, proxy pool, captcha solver, webhook secret and remote credentials. The root entry is transport-neutral; Node filesystem and HTTP adapters are explicit subpaths such as `@wenathlan/saddle/storage-node`, `@wenathlan/saddle/memory-node`, `@wenathlan/saddle/server-node`, `@wenathlan/saddle/sessions-file` and `@wenathlan/saddle/queue-persistent`. Saddle does not embed secrets, fixed hosts or a mandatory cloud vendor.
+The complete export map is documented in [`docs/libraryapi.md`](docs/libraryapi.md). The product index is in [`docs/productindex.md`](docs/productindex.md), and runnable examples are in [`docs/usage.md`](docs/usage.md).
 
 ## Browser extension
 
-Version 1.8.6 includes a pure JavaScript Manifest V3 reference surface in [`extension/`](extension/), a read-only page-world `pagefacts` boundary, snapshot diffs and persisted browser context metadata. It is deliberately narrow: the user invokes the action, the popup sends a versioned command, the service worker routes it, and an isolated content bridge returns bounded page metadata, visible text or a user initiated action result. The exported `permissionpolicy` keeps base permissions minimal and makes optional escalation caller-owned.
+The extension is a pure JavaScript Manifest V3 reference surface in [`extension/`](extension/). It contains a popup, service worker, isolated content bridge, read-only page-world `pagefacts` boundary, snapshot diffs and persisted window/tab/frame context for explicit resume.
 
 ```bash
 # load the unpacked extension from chrome://extensions
 ls extension/manifest.json extension/worker.js extension/content.js extension/popup.html
 
-# build an isolated unpacked artifact using the package version
+# build an isolated artifact using the version supplied by the caller or release tag
 npm run extension:build -- --output build/extension
 ```
 
-The extension requests `activeTab`, `scripting` and `storage`; it does not request broad host permissions, cookies, `webRequest`, debugger access or arbitrary page code execution. Its public contracts are available from `@wenathlan/saddle/extension`. Published releases attach `saddle-extension-<version>.zip`; the source manifest remains a stable unpacked reference. See [`extension/README.md`](extension/README.md) for the unpacked and release artifact flows.
+The base permission set is `activeTab`, `scripting` and `storage`. It does not request broad host permissions, cookies, `webRequest`, debugger access or arbitrary page code execution. Optional host escalation remains caller-owned. Releases attach `saddle-extension-<version>.zip`; cross-browser profiles remain adapter work.
 
-## CLI
+## Security boundaries
+
+| Boundary | Policy |
+| --- | --- |
+| Credentials | injected by the caller or repository secret; never committed or printed |
+| Network | HTTP/HTTPS targets are validated; private-target access remains caller policy |
+| Crawling | robots rules, crawl delay, limits and budgets are explicit |
+| Storage | adapters are replaceable; the core does not own a provider account |
+| Runtime | Node-only filesystem, HTTP, Playwright and release metadata stay outside the transport-neutral root |
+| Extension | page-world reads are bounded, token-correlated and read-only |
+| Failure | retry, circuit breaker, idempotency and resume are configurable |
+| Releases | version comes from the `vX.Y.Z` tag and must match `package.json` |
+
+Version 1.8.8 preserves the dependency-free JavaScript scrape contracts and groups the related crawl context in `scrape/crawl.js`. The retry and circuit context now lives in `runtime/retry.js`, while scraper-specific error classification is part of `core/errors.js`. The root lockfile and security gates remain authoritative; see [`docs/reorganization-1.8.8.md`](docs/reorganization-1.8.8.md) for the ownership decisions and [`docs/securityaudit-1.8.7.md`](docs/securityaudit-1.8.7.md) for the preceding security baseline.
+
+## Package surfaces and release automation
+
+Workflows use the release tag and the local `releaseversion` action. They do not contain a manually edited version number. The action fetches the tag, checks out its commit and rejects a release when the tag version does not match the root `package.json`.
+
+| Registry | Artifact | Workflow |
+| --- | --- | --- |
+| GitHub Packages npm | `@wenathlan/saddle@<version>` | `publishgithubnpm.yml` |
+| Public npmjs | `@wenathlan/saddle@<version>` | `publishnpmjs.yml` |
+| GHCR | `ghcr.io/wenathlan/saddle:<version>` | `publishghcr.yml` |
+| Maven | `io.wenathlan:saddle:<version>` | `publishmaven.yml` |
+| NuGet | `Saddle.<version>.nupkg` | `publishnuget.yml` |
+| RubyGems | `saddle <version>` | `publishrubygems.yml` |
+
+Release assets are caller-selected and deterministic: `SHA256SUMS`, `sbom.cdx.json` in CycloneDX 1.5 shape and `provenance.intoto.jsonl` in an in-toto statement shape. The adapter does not publish, authenticate or choose a registry. The npm token previously sent in chat is compromised and must never be used; public npmjs publication uses only the owner-managed `NPM_TOKEN` repository secret.
+
+## GitHub Pages web surface
+
+The marketing site lives under [`web/`](web/) with a root-based TypeScript/React layout. It has no `client/` or `src/` subdirectory. Vite normalizes the base path and all visual assets resolve through a shared helper, so the same build works at `/` and `/saddle/`.
 
 ```bash
-saddle help
-saddle modes
-saddle runexample
-saddle mcp
+npm run web:check
+VITE_BASE_PATH=/saddle npm run web:build:pages
 ```
 
-## Security boundaries
+Small public configuration and visual assets live under `web/public/`. The development collector is `web/public/debugcollector.js` and uses `/debuglogs`; it is not part of the production build. The obsolete `web/public/__manus__` directory is intentionally absent.
 
-| Boundary | Policy |
-| --- | --- |
-| Credentials | injected at runtime; never committed |
-| Network | http/https validated; private targets blocked |
-| Crawling | robots rules and crawl delay explicit |
-| Storage | adapters replaceable |
-| Runtime | Node HTTP isolated |
-| Failure | retry, circuit breaker, idempotency configurable |
-
-## Package surfaces
-
-| Registry | Artifact | Workflow | Status |
-| --- | --- | --- | --- |
-| GitHub npm | `@wenathlan/saddle@1.8.6` | publishgithubnpm.yml | pending release |
-| GHCR | `ghcr.io/wenathlan/saddle:1.8.6` and `latest` | publishghcr.yml | pending release |
-| Maven | `io.wenathlan:saddle:1.8.6` | publishmaven.yml | pending release |
-| NuGet | `Saddle.1.8.6.nupkg` | publishnuget.yml | pending release |
-| RubyGems | `saddle 1.8.6` | publishrubygems.yml | pending release |
-| npmjs | `@wenathlan/saddle@1.8.6` | publishnpmjs.yml | pending release |
-
-## Development
+## Development and release gates
 
 ```bash
 npm ci
-npm test
 npm run check
 npm run formatcheck
+npm test
 npm run pack:check
+npm audit --audit-level=high
+npm run web:check
+VITE_BASE_PATH=/saddle npm run web:build:pages
 ```
 
-Test suite deterministic, no network or real credentials required.
+The engine test suite is deterministic and does not require real credentials or network access. The release path is: update `package.json` and the manifest files, update `changelog.md`, run all gates, create `v<package-version>`, push the tag and create the GitHub release. Registry workflows then derive the same version from that release tag.
 
 ## Repository map
 
-```
-core/          errors, events and identifiers
+```text
+core/          engine errors, scrape error taxonomy, events, identifiers and hashing
 domain/        jobs, artifacts, sessions and providers
 memory/        working-set bridge, modes, objects and transforms
 storage/       local, chunked, remote and file-hosting adapters
-scrape/        robots, cache, extraction, schema and scraper
-crawl/         URL normalization, crawler and persistent frontier
+scrape/        robots, cache, extraction, schema, normalization and grouped crawl contracts
 queue/         queue, idempotency, saga and recovery
-browser/       fingerprint, session and agent contracts
-browser/       snapshots, tabs, frames, actions and recorder contracts
-mcp/           optional server and JSONL/HTTP transport
+browser/       fingerprint, session, agent and Playwright adapter contracts
+extension/     Manifest V3 reference surface and packager
 protocol/      JSON, NDJSON, SSE and block serializers
-workflow/      manifests, templates and registry
-tests/         deterministic engine coverage
-docs/          architecture, API, release and registry notes
-surfaces/      browser, extension, desktop, mobile and n8n contracts
+workflow/      manifests, templates and registry contracts
+packager/      package and publication plans
+release/       checksums, SBOM and provenance metadata
+runtime/       engine orchestration, capability detection, worker and grouped retry context
+web/           root-based static marketing site
+tests/         deterministic engine and extension coverage
+docs/          architecture, API, security, release and registry notes
 ```
 
-Root-based JavaScript ESM layout, no src/ directory, no TypeScript build required.
+The engine remains pure JavaScript ESM with JSDoc comments in English. The web surface is TypeScript/React, while the published library has no TypeScript build requirement and no hardcoded host, port or credential.
+
+## Historical documentation
+
+Earlier README snapshots remain in `docs/plans/README.md`, `docs/talks9/README.md` and `docs/talks9/README (2).md` as archival evidence. Their useful architecture ideas were consolidated here, while stale `@devthink`, `@iakadion`, `io.devthink`, Node 20/22, `client/src` and speculative provider quotas were not copied into the canonical contract.
 
 ## Current scope
 
-Version 1.8.6 extends the 1.8.5 engine contracts with isolated page facts, extension snapshot diffs, persisted browser context metadata, deterministic release assets and root-based Pages deployment. Native runtimes, browser binaries, n8n host registration, provider credentials, persistent databases and production deployment remain caller-selected adapters. The next improvements should extend these contracts without coupling the core to one forge, registry, browser or storage vendor.
+Version 1.8.8 extends the 1.8.7 engine with conservative context regrouping. Browser binaries, provider credentials, n8n host registration, persistent databases, captcha solvers and production deployment remain caller-selected adapters. Future work should extend contracts without coupling the core to one forge, registry, browser or storage vendor.
 
 ## License
 

package/api/service.js

@@ -1,7 +1,7 @@
 /**
  * saddle service exposes universal routes without choosing hono fastify express or another server.
  */
-import { crawl } from "../crawl/crawler.js";
+import { crawl } from "../scrape/crawl.js";
 import { ratelimiter } from "./rate.js";
 import { errorresponse, jsonresponse, sseresponse } from "./http.js";
 import { assertpublicurl } from "./security.js";

package/core/errors.js

@@ -35,3 +35,23 @@ export function aserror(error, jobid) {
   if (error?.name === "saddleerror") return error;
   return saddleerror(`job ${jobid} failed`, { code: errorcodes.jobfailed, cause: error, details: { jobid } });
 }
+
+/** Scrape error presets keep status, recovery and retry decisions explicit. */
+export const errorcatalog = Object.freeze({
+  timeout: { code: "E1001", statuscode: 504, retryable: true, recovery: "WAIT_AND_RETRY" },
+  connectionrefused: { code: "E1002", statuscode: 503, retryable: true, recovery: "WAIT_AND_RETRY" },
+  dns: { code: "E1003", statuscode: 503, retryable: true, recovery: "ROTATE_PROXY" },
+  ratelimited: { code: "E2001", statuscode: 429, retryable: true, recovery: "WAIT_AND_RETRY" },
+  forbidden: { code: "E2002", statuscode: 403, retryable: false, recovery: "REVIEW_ROBOTS_TXT" },
+  notfound: { code: "E2003", statuscode: 404, retryable: false, recovery: "STOP_CRAWLING" },
+  parse: { code: "E4002", statuscode: 422, retryable: false, recovery: "STOP_CRAWLING" },
+  captcha: { code: "E4003", statuscode: 403, retryable: false, recovery: "REVIEW_ROBOTS_TXT" },
+  session: { code: "E5001", statuscode: 401, retryable: true, recovery: "ROTATE_USER_AGENT" },
+  config: { code: "E6001", statuscode: 400, retryable: false, recovery: "STOP_CRAWLING" }
+});
+
+/** Creates a stable scrape error with recovery metadata. */
+export function webscrapeerror(kind, message, options = {}) { const preset = errorcatalog[kind] ?? errorcatalog.config; const error = new Error(message, { cause: options.cause }); error.name = "webscrapeerror"; error.code = options.code ?? preset.code; error.statuscode = options.statuscode ?? preset.statuscode; error.retryable = options.retryable ?? preset.retryable; error.recovery = options.recovery ?? preset.recovery; error.severity = options.severity ?? (error.statuscode >= 500 ? "high" : "medium"); error.details = options.details ?? {}; return error; }
+
+/** Converts unknown failures into the stable scrape taxonomy. */
+export function classifyerror(error) { if (error?.name === "webscrapeerror") return error; const message = String(error?.message ?? error); if (/timeout|aborted/i.test(message)) return webscrapeerror("timeout", message, { cause: error }); if (/dns|enotfound/i.test(message)) return webscrapeerror("dns", message, { cause: error }); return webscrapeerror("config", message, { cause: error }); }

package/docs/actionsincident.md

@@ -27,3 +27,7 @@ The repository metadata audit found `main` as the default branch and one open De
 The repository homepage About editor was opened in the authenticated browser and saved with the canonical Saddle description. The repository API should now report that description instead of the placeholder `saddle`.
 
 The six non-main Dependabot tips were preserved as `archive-dependabot-*` tags, their PRs were closed, and their branch refs were removed. The cleanup leaves `main` as the only active branch without deleting the archived commit objects.
+
+The first `v1.8.6` release fan-out passed release validation, GitHub Packages npm, public npmjs, Maven, NuGet, RubyGems and extension packaging. GHCR alone failed because the Docker image ran `npm ci --omit=dev` against the root manifest, whose dev-only Vite peer graph is rejected by npm's strict peer resolver. The corrective Dockerfile now sets `NPM_CONFIG_LEGACY_PEER_DEPS=true` and passes `--legacy-peer-deps`; the GHCR workflow also checks out the release tag for both release and manual dispatch paths.
+
+The corrected manual GHCR run [31706464064](https://github.com/wenathlan/saddle/actions/runs/31706464064) completed successfully, including the Docker build and push. The six registry outcomes for `v1.8.6` are therefore green: GitHub Packages npm, public npmjs, Maven, NuGet, RubyGems and GHCR. The Pages deployment remains green in [run 31705301175](https://github.com/wenathlan/saddle/actions/runs/31705301175).

package/docs/enginearchitecture.md

@@ -12,17 +12,18 @@ The project has no `src` folder. The root is the map of the engine.
 
 | folder | responsibility |
 |---|---|
-| `core` | errors, ids, clock, events, and tracing primitives |
+| `core` | engine errors, scrape error taxonomy, ids, events, and hashing primitives |
 | `domain` | jobs, sessions, artifacts, providers, and runtime records |
 | `storage` | storage adapter contract, local backend, and checksums |
 | `memory` | working set preparation, sync, and cleanup |
 | `runners` | provider factories and deterministic scheduling |
-| `runtime` | engine orchestration and output encoding |
+| `runtime` | engine orchestration, capability detection, worker boundary, retry and circuit context |
+| `scrape` | single page extraction, response normalization, robots policy and grouped crawl context |
 | `cli` | explicit command surface with local error handling |
 | `tests` | local deterministic tests without credentials |
 | `examples` | small runnable integration examples |
 
-All internal file names are lowercase and contain no underscore or hyphen. Related logic stays grouped and each module remains small enough to reason about in isolation.
+All internal file names are lowercase and contain no underscore or hyphen. Related logic stays grouped and each module remains small enough to reason about in isolation. Version 1.8.8 groups crawl URL normalization, traversal, frontier budgets and persistence in `scrape/crawl.js`; it does not merge distinct storage cache, job queue or browser session contracts merely because they use similar words.
 
 ## public contracts
 
@@ -64,7 +65,7 @@ The same contracts support paired modes. A mode can exist without its pair, and
 | browser | headless job definition | capture and replay adapter |
 | network | local deterministic job | remote provider and storage adapter |
 
-The first cut implements library, cli, binary entry point, internal memory, and internal file. The other modes are extension points, not hardcoded promises.
+The first cut implements library, cli, binary entry point, internal memory, and internal file. The other modes are extension points, not hardcoded promises. Version 1.8.8 preserves this boundary while reducing duplicated active context files.
 
 ## infrastructure rules
 

package/docs/libraryapi.md

@@ -28,7 +28,7 @@ The public API is designed around injected transports. Consumers can use the sam
 | `workflowtriggers` / `triggermatch` | normalize and match manual, event, schedule and retry starts |
 | `resumablerun` / `transitionrun` | recover remote run state through legal transitions |
 | `extractsemantic` | expose bounded headings, landmarks, controls and links |
-| `crawlfrontier` | prioritize URLs and enforce page and domain budgets |
+| `crawlfrontier` / `normalizeurl` / `persistentqueue` | grouped crawl context for URL normalization, traversal budgets and durable frontier state |
 | `provenance` / `mergeprovenance` | link context chunks to source and retrieval evidence |
 | `metricstore` | collect bounded counters and duration summaries |
 | `authorize` | verify caller credentials through an injected verifier |
@@ -58,8 +58,8 @@ The `fetcher`, browser adapter, persistence adapter, proxy pool, captcha solver,
 
 The package exposes explicit subpaths for `./browser`, `./bot`, `./captcha`, `./memory-engine`, and `./deploy`. Desktop, mobile, and n8n contracts are exported from the root entry; the root entry remains the complete JavaScript API for consumers that prefer one import.
 
-Version 1.1 adds `./extension`, which exposes browser-neutral message, snapshot and service-worker routing contracts. The concrete Manifest V3 files live in `extension/`; they are not imported by the core at runtime and do not require Chrome when the library is used as a Node package. The browser surface also exposes snapshots, tab/frame context, action results, bounded action batches and recording through `./browser`.
+The `./extension` subpath exposes browser-neutral message, snapshot and service-worker routing contracts. The concrete Manifest V3 files live in `extension/`; they are not imported by the core at runtime and do not require Chrome when the library is used as a Node package. The browser surface also exposes snapshots, tab/frame context, action results, bounded action batches and recording through `./browser`.
 
-The current main branch adds `desktopmanifest`, `mobilemanifest`, `desktopadapter`, `mobileadapter`, `n8nnode`, `n8nmatch`, `n8nexecute`, `controlsurface`, `controlservice`, and `workerbridge`. These factories describe surface boundaries and invoke caller-owned handlers; they do not install a native toolkit, start an n8n server, create a dashboard, or store credentials. They will be included in the next versioned release after the compatibility gates are complete.
+The current release exposes `desktopmanifest`, `mobilemanifest`, `desktopadapter`, `mobileadapter`, `n8nnode`, `n8nmatch`, `n8nexecute`, `controlsurface`, `controlservice`, and `workerbridge`. These factories describe surface boundaries and invoke caller-owned handlers; they do not install a native toolkit, start an n8n server, create a dashboard, or store credentials.
 
 The root entry is cross-runtime safe for the tested core contract. Filesystem, Node HTTP, persistent queue, file sessions, local memory and captcha evidence adapters remain available through explicit Node-only files or subpaths. `runtimecontract` reports the capabilities of the current global scope, while `memorystorage` provides a process-local backend for browser workers, Deno, Bun and deterministic tests.

package/docs/release.md

@@ -6,10 +6,10 @@ The release path is intentionally split into source validation, package validati
 
 | step | owner | condition |
 |---|---|---|
-| package version | repository | `package.json` is `1.8.5` |
+| package version | repository | `package.json` matches the release tag, for example `1.8.8` |
 | quality gate | GitHub Actions | `npm run pack:check` passes |
-| tag | repository owner | tag `v1.8.5` points to the validated release commit |
-| GitHub release | repository owner | release `v1.8.5` is created from the validated tag |
+| tag | repository owner | tag `v1.8.8` points to the validated release commit |
+| GitHub release | repository owner | release `v1.8.8` is created from the validated tag |
 | GitHub Packages | GitHub Actions | `publishgithubnpm.yml`, `publishghcr.yml`, `publishmaven.yml`, `publishnuget.yml`, and `publishrubygems.yml` use `GITHUB_TOKEN` |
 | public npmjs | owner-managed GitHub Actions secret | `publishnpmjs.yml` uses `NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}` and derives the version from the release tag or latest release in manual maintenance runs |
 | extension zip | GitHub Actions | `buildextension.yml` derives the version from the release tag, validates the unpacked artifact and attaches `saddle-extension-<version>.zip` |
@@ -22,8 +22,8 @@ The npm token previously sent in chat is compromised and must not be used. GitHu
 
 ```text
 npm run pack:check
-git tag v1.8.2
-git push origin v1.8.2
+git tag v1.8.8
+git push origin v1.8.8
 ```
 
 The release-created event is the publication and extension-asset trigger for the release workflows. A dry-run verifies package shape and local tests, but cannot verify registry ownership, Trusted Publisher configuration, package scope authorization, package visibility or browser-store submission; those remain settings controlled by the owner.

package/docs/releaseassets.md

@@ -4,7 +4,7 @@ The Node-only release adapter creates deterministic metadata for caller-selected
 
 ```bash
 npm run release:assets -- \
-  --version 1.8.6 \
+  --version 1.8.8 \
   --output build/release \
   --artifact build/saddle.tgz \
   --build-type caller-build \

package/docs/reorganization-1.8.8.md

@@ -0,0 +1,31 @@
+# Reorganization 1.8.8
+
+## Purpose
+
+Version 1.8.8 applies the architecture skill's grouping rule to active JavaScript contracts. The objective is not to erase historical features or flatten unrelated domains. The objective is to give each correlated responsibility one canonical owner, reduce redundant folder boundaries, and preserve public names and deterministic behavior.
+
+## Audit result
+
+The active package contains a JavaScript runtime and a legacy TypeScript scrape tree. The TypeScript tree is not imported by the active root entry or package export map. It remains in place because it documents a broader historical surface and contains features that would require a separate compatibility decision before conversion to the current JavaScript library boundary.
+
+| Context | Previous ownership | Version 1.8.8 owner | Decision |
+| --- | --- | --- | --- |
+| Crawl acquisition | `crawl/normalize.js`, `crawl/frontier.js`, `crawl/crawler.js`, `crawl/persistent.js` | `scrape/crawl.js` | Consolidated. URL normalization, traversal, frontier budgets and durable crawl state share one context and preserve all public function names. |
+| Retry protection | `retry/policy.js`, `retry/circuit.js` | `runtime/retry.js` | Consolidated. Both modules protect caller-owned runners, storage and network handlers from transient failure storms. |
+| Scrape error classification | `errors/taxonomy.js` | `core/errors.js` | Consolidated. Generic engine errors and scrape recovery metadata share one stable error boundary. |
+| Page cache | `scrape/cache.js` | `scrape/cache.js` | Kept separate. It caches single-page scrape results and robots policy. |
+| Storage cache | `storage/cache.js` | `storage/cache.js` | Kept separate. It manages hot and cold byte-backed tiers, encoding, eviction and revalidation. |
+| Job persistence | `queue/persistent.js` | `queue/persistent.js` | Kept separate. It persists generic job lifecycle records rather than crawl URLs. |
+| Browser sessions | `browser/session.js`, `domain/sessions.js`, `sessions/*` | existing owners | Kept separate. These contracts represent browser identity, domain validation and file or replay persistence. |
+
+## Compatibility rule
+
+The root export barrel continues to expose `normalizeurl`, `sameorigin`, `crawlfrontier`, `persistentqueue`, `crawl`, `retrypolicy`, `circuitbreaker`, `webscrapeerror` and `classifyerror`. Internal imports, tests and package payload declarations now point to the canonical owners. No public export was renamed for the regrouping.
+
+## Verification rule
+
+The reorganization is accepted only when syntax checks, deterministic tests, format checks, package dry-run, dependency audit, web typecheck and Pages build pass. A deleted file is considered redundant only after its imports, tests, package payload and documented public names have been checked.
+
+## Deferred historical surface
+
+The legacy TypeScript modules under `scrape/` remain a separate historical surface. They include richer browser, renderer, cache, middleware, server, session, pool, sitemap and format contracts. They are not silently merged into the active JavaScript modules because doing so would either discard features or introduce an undeclared TypeScript and external dependency build into the published core.

package/docs/securityaudit-1.8.7.md

@@ -0,0 +1,21 @@
+# Security and web audit for 1.8.7
+
+This audit records the baseline collected before remediation. The repository owner can cross-check the live GitHub view at [Saddle Security](https://github.com/wenathlan/saddle/security) and the [Dependabot alerts API](https://docs.github.com/en/rest/dependabot/alerts).
+
+## Baseline
+
+GitHub reported **42 open Dependabot alerts**: 2 critical, 14 high, 23 medium and 3 low. Five alerts were associated with the root `package-lock.json`; the remaining alerts were associated with the tracked `scrape/package-lock.json`. The largest groups were `undici`, `hono`, `vite`, `shell-quote`, `brace-expansion`, `postcss`, `ip-address` and `sharp` in the nested scrape manifest.
+
+The local root `npm audit` reported 4 vulnerabilities in the current root installation: 2 moderate, 1 high and 1 critical. The root audit is a separate view from GitHub's repository-wide alert count and does not include the stale nested scrape dependency graph unless that directory is audited independently.
+
+Code scanning returned no analysis found, and secret scanning returned that the feature is disabled. These are coverage gaps, not evidence that the repository has no code or secret findings. The remediation therefore includes enabling or documenting the appropriate GitHub security controls without fabricating a clean result.
+
+## Web inventory
+
+The requested directory `web/public/manos` does not exist. The actual platform directory is `web/public/__manus__`, containing `debug-collector.js`; it is a small runtime support directory rather than a product asset directory. The public visual assets are tracked under `web/public/assets/` as four WebP files.
+
+The first path audit found root-absolute application entry and route paths, while the asset files themselves are in the correct public directory. The Pages site is served below `/saddle/`, so every asset and internal route must be resolved through the Vite base path rather than assuming `/`.
+
+## Remediation policy
+
+The 1.8.7 work will prioritize Node.js built-ins for new logic, update direct and transitive dependencies through the root lockfile, isolate or remove the obsolete nested scrape dependency graph when it is not part of the published engine, preserve the small `__manus__` support directory unless its script is proven unnecessary, and make web assets base-aware. A final audit will distinguish resolved advisories from external or unfixable advisories rather than hiding them.

package/extension/manifest.json

@@ -1,7 +1,7 @@
 {
   "manifest_version": 3,
   "name": "Saddle browser bridge",
-  "version": "1.8.6",
+  "version": "1.8.8",
   "description": "User initiated page snapshots through the Saddle browser contract.",
   "minimum_chrome_version": "110",
   "permissions": ["activeTab", "scripting", "storage"],

package/index.js

@@ -28,7 +28,7 @@ export * from "./runners/heartbeat.js";
 export * from "./workflow/triggers.js";
 export * from "./dispatch/resumable.js";
 export * from "./scrape/semantic.js";
-export * from "./crawl/frontier.js";
+export * from "./scrape/crawl.js";
 export * from "./ai/provenance.js";
 export * from "./observability/metrics.js";
 export * from "./api/auth.js";
@@ -73,9 +73,6 @@ export * from "./protocol/blocks.js";
 export * from "./workflow/manifest.js";
 export * from "./workflow/templates.js";
 export * from "./workflow/registry.js";
-export * from "./crawl/normalize.js";
-export * from "./crawl/crawler.js";
-export * from "./crawl/persistent.js";
 export * from "./scrape/schema.js";
 export * from "./scrape/normalize.js";
 export * from "./mcp/server.js";
@@ -102,9 +99,7 @@ export * from "./surfaces/adapters.js";
 export * from "./surfaces/controls.js";
 export * from "./surfaces/operations.js";
 export * from "./library/public.js";
-export * from "./errors/taxonomy.js";
-export * from "./retry/policy.js";
-export * from "./retry/circuit.js";
+export * from "./runtime/retry.js";
 export * from "./storage/githubcontents.js";
 export * from "./storage/filehosting.js";
 export * from "./persistence/migrations.js";

package/library/public.js

@@ -1,7 +1,7 @@
 /**
  * public library helpers compose fetch extraction serialization chunking and crawl contracts.
  */
-import { crawl } from "../crawl/crawler.js";
+import { crawl } from "../scrape/crawl.js";
 import { chunkmarkdown } from "../ai/chunk.js";
 import { estimatetokens } from "../ai/tokens.js";
 import { extracthtml } from "../scrape/extract.js";

package/mcp/server.js

@@ -1,7 +1,7 @@
 /**
  * mcp server implements a small JSON RPC tool surface and remains transport agnostic.
  */
-import { crawl } from "../crawl/crawler.js";
+import { crawl } from "../scrape/crawl.js";
 import { extractwithschema } from "../scrape/schema.js";
 import { jsonencode } from "../protocol/json.js";
 import { browsertools } from "./browser.js";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@wenathlan/saddle",
-  "version": "1.8.6",
+  "version": "1.8.8",
   "description": "binary computing engine that turns distributed storage into a publishable working set",
   "type": "module",
   "private": false,
@@ -84,7 +84,6 @@
     "queue",
     "dispatch",
     "scrape",
-    "crawl",
     "api",
     "mcp",
     "browser",
@@ -94,8 +93,6 @@
     "webhook",
     "surfaces",
     "library",
-    "errors",
-    "retry",
     "server",
     "binary",
     "format",
@@ -113,7 +110,7 @@
     "LICENSE"
   ],
   "scripts": {
-    "check": "node --check index.js && node --check runtime/engine.js && node --check cli/main.js && node --check extension/protocol.js && node --check extension/serviceworker.js && node --check extension/worker.js && node --check extension/content.js && node --check extension/pagebridge.js && node --check extension/popup.js && node --check extension/permissions.js && node --check extension/build.js && node --check release/assets.js && node --check sessions/replay.js && node --check browser/recorder.js && node --check scrape/normalize.js",
+    "check": "node --check index.js && node --check runtime/engine.js && node --check runtime/retry.js && node --check cli/main.js && node --check extension/protocol.js && node --check extension/serviceworker.js && node --check extension/worker.js && node --check extension/content.js && node --check extension/pagebridge.js && node --check extension/popup.js && node --check extension/permissions.js && node --check extension/build.js && node --check release/assets.js && node --check sessions/replay.js && node --check browser/recorder.js && node --check scrape/normalize.js && node --check scrape/crawl.js",
     "formatcheck": "node format/check.js",
     "test": "node --test tests/*.test.js",
     "run": "node cli/main.js",
@@ -178,7 +175,6 @@
     "vaul": "^1.1.2",
     "wouter": "^3.3.5",
     "zod": "^4.1.12",
-    "@builder.io/vite-plugin-jsx-loc": "^0.1.1",
     "@tailwindcss/typography": "^0.5.15",
     "@tailwindcss/vite": "^4.1.3",
     "@types/express": "4.17.21",
@@ -198,6 +194,6 @@
     "typescript": "5.6.3",
     "vite": "^7.1.7",
     "vite-plugin-manus-runtime": "^0.0.58",
-    "vitest": "^2.1.4"
+    "vitest": "^4.1.10"
   }
 }