@elaraai/east-claude-plugin 1.0.99 → 1.0.100

package/skills/e3/reference/browser.md

@@ -0,0 +1,93 @@
+# Running e3 in a browser — `@elaraai/e3-web`
+
+Part of the `e3` skill.
+
+e3-web runs e3 entirely in a page, with no server: a repository kept in the
+browser's own storage (records in IndexedDB, objects in OPFS, locks through
+Web Locks), every East program on a pool of Web Workers, and e3's whole API
+answered in the page by e3-api-server's own routes. An app is three scripts:
+
+```typescript
+// unit.worker.ts — where East programs run: east-web-std answers for east-node-std's
+// platform functions, and e3's own (e3-api-client's Platform) reach the e3 worker
+import { serveUnits } from '@elaraai/e3-web/units';
+serveUnits();
+```
+
+```typescript
+// e3.worker.ts — e3 itself: its storage, orchestrator, runner and API
+import { serveE3 } from '@elaraai/e3-web/worker';
+serveE3({
+  units: () => new Worker(new URL('./unit.worker.ts', import.meta.url), { type: 'module' }),
+});
+```
+
+```typescript
+// main.ts — the page: e3-api-client's calls, every one answered by the e3 worker
+import { createWebE3 } from '@elaraai/e3-web';
+import {
+  dataflowExecute, packageImport, repoCreate, repoList, workspaceCreate, workspaceDeploy,
+} from '@elaraai/e3-api-client';
+
+const e3 = await createWebE3(new Worker(new URL('./e3.worker.ts', import.meta.url), { type: 'module' }));
+const opts = { token: null, fetch: e3.fetch };
+
+// The first visit makes the repository; later visits find it where they left it.
+if (!(await repoList(e3.apiUrl, opts)).includes('default')) {
+  await repoCreate(e3.apiUrl, 'default', opts);
+  // The package zip e3.export wrote at build time, served beside the page
+  const zip = new Uint8Array(await (await fetch('/sales-planning-1.0.0.zip')).arrayBuffer());
+  const { name, version } = await packageImport(e3.apiUrl, 'default', zip, opts);
+  await workspaceCreate(e3.apiUrl, 'default', 'main', opts);
+  await workspaceDeploy(e3.apiUrl, 'default', 'main', `${name}@${version}`, opts);
+  await dataflowExecute(e3.apiUrl, 'default', 'main', {}, opts);
+}
+```
+
+`e3.apiUrl` is `https://e3-web.invalid`, an origin reserved so that a request
+sent without `e3.fetch` fails rather than reaching the network, and
+`e3.fetch`, the global `fetch`'s signature, is answered by the e3 worker. Both
+go to e3-api-client (`RequestOptions.fetch`) and to e3-ui-components'
+`<E3Provider>` (`E3Config.fetch`: the **e3-ui** skill). An app's own platform
+package is served beside the standard ones:
+`serveUnits({ platforms: { '@acme/pricing': PricingPlatform } })`.
+
+| `serveE3` option | Default | Effect |
+|---|---|---|
+| `units` | — | starts a unit worker: a Web Worker whose script calls `serveUnits()` |
+| `width` | the cores | the most units that run at once, and so the most unit workers, each with a heap of its own |
+| `persist` | `true` | `false` keeps repositories in memory, gone with the page |
+| `name` | `'e3'` | the storage's name: e3 workers of one name share their repositories, in one tab or several |
+| `identify` | none | each request's caller, `{ sub, email?, roles }`, from the request; a request to a repository whose caller it does not identify is answered 401. None: the page's one caller may do everything |
+| `access` | `oneShotAccessByRoles()` with `identify`, else `any` | the grant a caller holds for a one-shot, a split call, or a function call naming its own runner |
+| `wholeIntakeLimit` | 256 MiB | the largest delivery one intake unit takes in whole |
+
+What works is the API, as the local server answers it: every API suite of
+e3-api-tests passes against it in Chromium, as an admin and, in its reader
+cases, as a reader. Paged reads and key search, function calls, one-shot and
+split calls, record mutations, deploys that migrate records or build indexes,
+uploads, package import and export, gc, and dataflow runs — tasks in parallel
+up to the pool's width (the cores), split tasks in pieces, re-run reactively.
+A repository outlives a reload, and two tabs share an origin's repositories as
+two processes share a directory. A closed tab's locks are free; a run it left
+is taken as one whose process died, the next run served what finished from
+the cache; and a job or an upload's commit it left is recorded failed, never
+run again.
+
+What a browser cannot do — each fails naming it, never a partial answer:
+
+- **Commands.** A custom task, or a task on the `custom` runtime, is recorded
+  `error`: "a browser runs no commands".
+- **Platform functions with no browser meaning.** east-web-std has no
+  FileSystem, Env or large-JSON reader, and east-node-io and the python
+  packages have no browser equivalent: a unit calling one fails naming the
+  function, and one listing a package its worker does not serve fails naming
+  the package.
+- **Segments on demand.** A unit's inputs are staged whole into its worker:
+  waiting for a segment needs a cross-origin isolated page, which GitHub Pages
+  cannot give. Split a large input into pieces with `e3.partition`, which
+  bounds a unit's memory anywhere; a delivery that cannot be cut into pieces
+  is refused above `wholeIntakeLimit`, naming the fix.
+- **A memory budget.** Memory is not measured: the pool is as wide as the
+  cores, there is no `--memory`, and no record names a peak.
+- **The CLI.** `e3` drives e3 over HTTP, which an in-page e3 does not serve.

package/skills/e3/reference/cli.md

@@ -0,0 +1,120 @@
+# The e3 CLI
+
+Part of the `e3` skill.
+
+Every `<repo>` is a local path or a server URL, `http(s)://<host>/repos/<name>`;
+where `[repo]` is optional it defaults to `$E3_REPO`, then `.`. Dataset paths are
+flat, `<ws>.<name>` — an input or a task output, with "did you mean" suggestions;
+a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
+
+| Command | Does |
+|---|---|
+| **Repositories** | |
+| `e3 repo create [repo] [--exist-ok]` | Create one (`--exist-ok`: succeed if it exists). |
+| `e3 repo status [repo]` · `e3 repo remove [repo] [-r]` | Show objects, packages, workspaces · remove (`-r`: its workspaces first). |
+| `e3 repo gc [repo] [--dry-run] [--keep-runs <n>] [--keep-days <d>] [--min-age <ms>]` | Drop old run history, then what nothing names (below). |
+| `e3 repo list <server>` | The repositories a server holds. |
+| **Packages** | |
+| `e3 package import [repo] <zip>` · `export [repo] <pkg[@ver]> <zip>` | Import · export (`--quiet`: errors only). |
+| `e3 package list [repo]` · `remove [repo] <pkg[@ver]>` | |
+| **Workspaces** | |
+| `e3 workspace create [repo] <ws>` · `list [repo]` · `status [repo] <ws>` · `remove [repo] <ws>` | `status` shows tasks, datasets and locks. |
+| `e3 workspace copy [repo] <from> <to>` | Make `<to>` what `<from>` is now — its package and every input, record and task output, so nothing runs again — made, or replaced whole with its runs. It writes refs only, so its cost follows the number of datasets, never their size; a write to either afterwards leaves the other as it was. Refused while a dataflow runs in `<to>`. |
+| `e3 workspace deploy [repo] <ws> <pkg[@ver]>` | Deploy an imported package. |
+| `… --from-zip <zip>` · `… --from-source <src.ts> [--functions <manifest…>]` | Import the zip (or bundle the source) and deploy, creating the workspace. |
+| `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on, and with `--from-source` the workspace packages exporting their functions at once; `--memory`: what those may reserve (local). |
+| `e3 workspace export [repo] <ws> <zip> [--name <n>] [--version <v>]` | The workspace's state as a package. |
+| **Datasets** | |
+| `e3 dataset get [repo] <ws.name> [-f east\|json\|beast2]` | Print a value. |
+| `e3 dataset set [repo] <ws.name> <file> [--type <spec> \| --type-file <path>]` | Write from `.east`, `.beast2`, `.json` or `.csv` (JSON and CSV need the type). |
+| `e3 dataset set [repo] <ws.name> --from-file <path.beast2> [-j <n>] [--memory <size>]` | Take a beast2 file in as the value, as a deploy takes a `file` source in, saying how far it has got; `-j`, `--memory`: the budget its intake units run under (local). |
+| `e3 dataset list [repo] <ws> [-l]` · `status [repo] <ws.name>` · `find [repo] <ws> <pattern>` | Paths (`-l`: kind, type, status, size) · one dataset · by substring or glob. |
+| **Running** | |
+| `e3 dataflow run [repo] <ws> [--filter <task>] [--force \| --force-task <task>…] [-j <n>] [--memory <size>] [-v]` | Run what is stale, then print the outputs' paths. `--filter`: one task, by its exact name, and the tasks it depends on. `--force`: re-run every task the run runs (under `--filter`, its task); `--force-task`, repeated for each: re-run the tasks it names, whose dependents re-run when an output changes. A task the graph lacks, or one `--filter` leaves out, refuses the run. A task whose input nothing has set — an input never set, or a record declared with no initial value and not yet seeded — is left waiting, with its dependents (`[WAIT]`, `Waiting:`), and the run completes without them; they run once it is set. |
+| `e3 task list [repo] <ws>` | Tasks with the status of each one's latest run. |
+| `e3 task logs [repo] <ws.task> [-n <lines>] [--all] [--follow] [--execution <task>/<inputs>/<id>]` | The last 200 lines by default, of the latest attempt over the task's current inputs, or its latest run; `--follow` stays on that execution; `--execution`: one unit's log, as a split task's log names it (local). |
+| `e3 run <repo> <pkg[@ver].task> [inputs.beast2…] -o <out> [--force] [-v] [-j <n>] [--memory <size>]` | Run one task ad hoc. |
+| `e3 call <repo> <pkg[@ver].fn> [args…] [-o <out.beast2>] [-j <n>] [--memory <size>] [-v]` · `e3 call <repo> -w <ws> <fn> [args…]` | Call a function; each argument is an `.east` literal or a `.beast2`/`.json`/`.east` file, and with `-w` one written `@<keypath>` (`@.inputs.sales`) is that dataset of the workspace, pinned at its hash when the call starts. |
+| **Records** (`-w <ws>` required) | |
+| `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
+| `e3 seed <repo> <record> <file.beast2> -w <ws> [-j <n>] [--memory <size>]` | Commit a record's first state from a beast2 file, as its `$init`: for a record declared with no initial value, unassigned until then (local). The file is taken in as a `file` source is, never read whole; a record that holds a state is refused. |
+| `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target — a summary's totals added to as `±n`; local). |
+| `e3 reindex <repo> <record> -w <ws> [--index <name>] [-j <n>] [--memory <size>]` · `e3 compact <repo> <record> -w <ws>` | Rebuild indexes (local) · collapse the history to a `$compact` root, the state kept. |
+| **Development** | |
+| `e3 watch <src.ts> <repo> <ws> [--start] [--schema <p>] [--inputs <p>] [--abort-on-change] [--functions <manifest…>] [-j <n>] [--memory <size>]` | Redeploy on each change (and run, with `--start`). |
+| `e3 convert [input] [--from <f>] [--to <f>] [--type <spec>] [-o <out>]` | Convert between `.east`, `.json` and `.beast2`. |
+| `e3 completion install [--shell <s>]` · `uninstall` · `bash` \| `zsh` \| `fish` | Shell completion. |
+| **Servers** | |
+| `e3 auth login <server> [--no-browser]` · `logout <server>` · `status` · `token <server>` · `whoami [server]` | OAuth2 device flow; credentials per server (`token` prints one for curl). |
+
+**The budget.** `-j`/`--jobs <n>` is the cores — runner processes in flight, a
+task or a unit each — and `--memory <size>` (`8G`, `512M`) what they may reserve
+between them. The defaults are `E3_JOBS` and `E3_MEMORY`, else what e3 may use:
+its CPU affinity capped by a cgroup quota, and the cgroup's `memory.max` or
+physical memory, less a reserve. A unit of a split task reserves the largest
+peak a unit of its stage has reached in the run, so a stage runs its first unit
+alone and then fans out; on Linux and macOS a guard stops the newest unit when
+the runners together pass the budget, and runs it again once it fits. A deploy
+takes its `file` sources in `-j` at a time, their pieces' intake units running
+under the same budget. A source deploy, or a `watch`, exports the workspace
+packages its imported functions come from `-j` at a time, a process each, on
+this machine; against a server, as many at once as this machine has CPUs. The
+commands that run East — `workspace deploy`,
+`dataset set --from-file`, `dataflow run`, `run`, `call`, `mutate`, `seed`,
+`reindex` and `watch` — take the flags for a local repository; against a server the CLI
+refuses them, since the server runs the work under its own budget
+(`e3-api-server -j`, `--memory`).
+
+**A deploy says how far it has got**, on stderr. Each `file` source prints a
+line once it is in: its size, time and rate, and how it was taken in
+(`unchanged, already in the store`, `taken in by east-c`, or `carried` for a
+value that is not a collection), with why the first time a runner fell back to
+another. A terminal also keeps a live line for the files in flight, with their
+pieces, the rate and the time left across them. A source deploy prints a line
+for each workspace package whose functions it links: `exported <n> function(s)
+of <pkg> (east-py export-functions)`, or `reused <n> function(s) of <pkg>,
+unchanged since its last export` when the export cache held them. Against a
+server, the deploy line
+says what the job is doing, such as migrating a record or building an index;
+the deliveries then upload a few at a time, each printing a line once it is in,
+and the live line says what the server's commit is doing with each in flight.
+While a deploy runs, its lock carries the same progress for `e3-ui` or any
+client to read (`workspaceLockStatus`). A workspace deployed for the first time
+has no status until its deploy ends, so the lock is the only place to see it.
+
+**`-v`** passes `-v` to the runners, which print to the task's logs, identically
+on every runtime, how each input was read — opened lazily, and what reading it
+came to, or decoded whole, and the resident memory that added — and a timing
+and peak-memory block. It never changes hashes or caching (add `--force` to see
+it for a cached task), and works against a server.
+**Where a task's time goes.** `EAST_PROFILE=1` in the environment of a local
+`e3 dataflow run` (or `run`, `call`, `mutate`) has the east-c and east-py
+runners profile each unit — the runners inherit e3's environment — and the
+task's log (`e3 task logs`) then ends with every East function and platform
+function it called, by self time, with its calls and where it was written and
+called: the python lines of a function `East.importFunction` linked in
+included. Each input read lazily is an entry too, named as `-v` names it
+(`input 1`): the time its reads spent decoding segments, a call each — keyed
+reads at random beyond the segments the runner keeps, or a scan repeated,
+which `-v`'s account counts — and not the reading function's own time.
+`EAST_PROFILE_INTERVAL=300` adds the profile so far every five minutes while a
+long task runs. east-node does not profile. Like `-v` it changes no hash: add
+`--force` for a cached task.
+**`E3_SCRATCH_DIR`** moves a local run's per-execution scratch directories
+(default `<repo>/tmp/scratch`, on the object store's disk; on tmpfs, outputs sit
+in memory until stored).
+
+**`repo gc`** keeps each workspace's last 10 runs (`--keep-runs`), every run
+from the last 7 days (`--keep-days`) and the run its current state came from,
+with every execution those runs used; every execution a workspace's current
+state is served from, so a re-run stays cached; every execution from the last 7
+days; and whatever is still running. An attempt recorded running that the local
+runner says cannot finish is recorded interrupted, and goes as any that ended.
+It then removes the objects nothing names, and staging files older than
+`--min-age` (60 s). `--dry-run` reports what would go.
+
+A package zip names the release of e3 that exported it, and an import refuses a
+zip a newer release exported, naming that release: import it with an e3 at least
+as new as the SDK that exported it. A dataset upload names the transfer protocol
+its e3 speaks, and a server refuses one that speaks another, naming both
+releases and which of them to upgrade.

package/skills/e3/reference/from-code.md

@@ -0,0 +1,48 @@
+# Driving e3 from code
+
+`@elaraai/e3-api-client` against a server, `@elaraai/e3-core` on a local repository, and `e3-api-server`. Part of the `e3` skill.
+
+**`@elaraai/e3-api-client`** talks to a server over HTTP. Every call is
+`(url, repo, …, options)`, `options` being
+`{ token: string | null, retry?, verbose?, fetch? }` (`e3 auth token <server>`
+prints a token); `fetch` is the `fetch` every request of the call goes
+through, the global one unless given — an in-page e3's `e3.fetch` (below).
+Dataset paths are `TreePath`s — an input definition's `.path`, for one. A
+request a workspace's lock refuses throws `ApiError` `workspace_locked`,
+naming the holder: its pid, boot id, command and when it took the lock.
+
+| Area | Functions |
+|---|---|
+| Repositories | `repoList(url, opts)`, `repoCreate(url, name, opts)`, `repoRemove(url, name, opts)`, `repoStatus(url, repo, opts)`, `repoRecord` (its release and upgrades), `repoGc(url, repo, gcRequest, opts)` — a job it polls; `repoGcStart` and `repoGcStatus` apart |
+| Packages | `packageList(url, repo, opts)`, `packageGet(url, repo, name, version, opts)`, `packageImport(url, repo, zipBytes, opts)`, `packageExport(url, repo, name, version, opts)` → zip bytes, `packageRemove` |
+| Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceBundle(url, repo, ws, opts)` → the workspace as it is, as a bundle's bytes — its package, every dataset's ref, the executions its tasks' outputs come from, and every object those reach — and `workspaceBundleImport(url, repo, ws, bundle, opts)` → the workspace made from one under a name no workspace has, in another repository or another e3 (`workspace_exists`, `package_exists`, `bundle_invalid`), its tasks up to date with no run; a host mounts these routes where a bundle fits in a request: the local server, and e3-web, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
+| Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, and several reads in one request (at most `DATASET_BATCH_MAX`, 10,000, each at one content hash, in a body of at most 16 MiB: a larger one is refused 413, `payload_too_large`): `datasetFindKeys(…, { queries, index?, hash? }, opts)` — key searches, `datasetGetRows(…, { keys, hash? }, opts)` — a Dict's rows by `.east` key, each with its position — or `datasetGetRows(…, { keysOf: { path, index, runs, hash }, hash? }, opts)`, by the keys the entries of some runs of another record's index name, which the server reads from the index so none is sent (a drawing's details for its window's cells), a stale `keysOf.hash` thrown as `RunsHashMismatchError` naming the record's content — and `datasetGetRuns(…, { runs, hash, index?, join? }, opts)` — the entries of several runs as one window, a rows or runs answer holding the first `count` the page budget reaches (by runs, of `runEntries`); and where a narrowing's rows sit in a record, from its index alone: `datasetPlace(…, { index, runs, offset?, limit?, hash }, opts)` → `{ segments, placed, runEntries, totalElements, hash }` — how many of the runs' entries each record segment holds (`{ segment, offset, count, rows }`), for the first `placed` of a window from `offset` (the next window from `offset + placed`), and `datasetWithin(…, { index, runs, keys, hash }, opts)` → `{ within, hash }` — whether each index entry key (`(ik=…, k=…)`) lies in one of the runs, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
+| Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
+| Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts)` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress? })` — a program over a dataset's pieces, as a job it polls, under the same rule (or `splitCallLaunch`, `splitCallStatus` and `splitCallExplain`) |
+| Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
+| From East | `Platform` and the `platform_*` functions (`platform_dataset_get`, `platform_dataflow_execute`, …): the same calls as platform functions, for an East program that drives a server |
+
+**`@elaraai/e3-core`** does the same on a local repository, given a storage
+backend (`new LocalStorage()`) and the repository's path; work that runs East
+takes a runner, `new LocalTaskRunner(repo)`.
+
+| Area | Functions |
+|---|---|
+| Repositories | `repoInit(path)`, `repoFind(startPath?)`, `repositoryOpen(storage, repo)` (checks the repository and applies the upgrades it owes), `repositoryUpgradeStep(storage, repo, { budgetMs, waitMs? })` → `{ owed }` — a host's job applies them so, a part per run under its time limit, each part taking a step up where the last stopped, `repoGc(storage, repo, { dryRun?, minAge?, keepRuns?, keepDays?, retention?, executionAlive? })` — holding the repository still, or, with `retention: { windowMs }`, beside running work: an object goes once unreachable for the window and not written or re-referenced since; given a runner's `executionAlive`, an attempt recorded running that cannot finish is recorded interrupted and pruned; `repoGcStep(storage, repo, step, { windowMs, pruneMs?, markMs?, concurrency?, executionAlive?, … })` runs that one step at a time, each returning the next (`GcStepType`), `pruneMs` and `markMs` bounding a prune step and a mark step for a host with a time limit |
+| Packages | `packageImport(storage, repo, zipPath)`, `packageExport(storage, repo, name, version, zipPath)`, `packageList`, `packageRemove` |
+| Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceBundleExport(storage, repo, ws, writable, { have? })` — the workspace as a bundle to a WHATWG stream, leaving out the objects `have` says the receiving repository holds — and `workspaceBundleImport(storage, repo, ws, source)` — the workspace made from a bundle (a stream or chunks) under a name no workspace has, its objects stored as they come and its refs and record written once the repository holds everything they name, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
+| Datasets | `workspaceGetDataset(storage, repo, ws, treePath)`, `workspaceSetDataset(storage, repo, ws, treePath, value, type)`, `datasetAdoptFile(storage, repo, ws, treePath, file, { runner, onProgress? })` → `{ hash, size, segments, rows, taken, runners? }`, `taken` being `known`, `carried` or `taken` (by the `runners` named); `deliveryAdoptStored(storage, repo, ws, { address, type }, sha256, { runner?, onProgress?, signal? })` — a delivery the store holds taken in as a stored value of the type, naming no dataset, for a record's first state |
+| Runs | `dataflowExecute(storage, repo, ws, options?)`; `LocalOrchestrator` to start, poll and cancel a run |
+| Records | `recordMutate(storage, runner, repo, ws, record, mutation, args, { actor })`, `recordHistory`, `recordDescribe`, `recordCompact`, `recordReindex`; `recordSystemCommit(storage, runner, repo, ws, record, { name, target, actor, args? })` — `$rollback` to a commit, `$restore` of a state at the record's migrations, and `$init` on a record declared with no initial value: its first state, from a stored value, with the whole chain applied |
+
+`@elaraai/e3-core/test` exports the contract suites another storage backend runs
+over itself.
+
+**`e3-api-server`** serves repositories: `--repos <dir>` (each subdirectory a
+repository) or `--repo <path>` (one, served as `default`), `-p`/`--port` (3000),
+`-H`/`--host` (localhost), `--cors`, `-j`/`--memory` (the server's budget, for
+every run and call it serves and every upload it takes in), and auth — `--oidc`, a built-in provider for `e3
+auth login` (`--token-expiry`, `--refresh-token-expiry`), or an external JWT
+issuer (`--auth-key`, `--auth-issuer`, `--auth-audience`). In code,
+`createServer({ reposDir | singleRepoPath, port, host, … })` starts one, and its
+route factories mount on another host.

package/skills/e3/reference/rack.md

@@ -0,0 +1,139 @@
+# Rack delegation from a local repository
+
+Part of the `e3` skill.
+
+**Setup and routing**
+
+| Signature | Description | Example |
+| --- | --- | --- |
+| `e3 rack enroll --listen <host:port> [--allow <ip,...>] [--advertise-url <origin>] [--label <label>] [--no-wait]` | Starts the user-wide hub and prints the source origin, the hub's certificate SHA-256, the matching release and a single-use token for the companion's hidden prompts. Waits for a new registration, not activation. Behind an SSH reverse tunnel, listen on the loopback and advertise the relay's origin. | `e3 rack enroll --listen 127.0.0.1:7331 --allow 127.0.0.1 --advertise-url https://relay.example:7401` |
+| `e3 rack policy [repo] set <glob> rack\|local [--size <size>] [--timeout <min>] [--allow-host-access]` | First whole-task-name match wins; `*` and `?` are wildcards. Size: serverless, small, medium, large, xlarge. | `e3 rack policy . set 'train_*' rack --size large` |
+| `e3 rack policy [repo] enable\|disable\|mode <opt-in\|auto>` | Enable/disable saved opt-in; `mode auto` selects eligible unmatched tasks. Defaults: disabled, opt-in. | `e3 rack policy . enable` |
+| `e3 rack policy [repo] defaults [--no-spill] [--claim-timeout <sec>] [--retry-task-failures-locally]` | Defaults: large, 1440-minute timeout, spill on, 30-second claim recheck, no task-failure retry. | `e3 rack policy . defaults --no-spill` |
+| `e3 rack policy [repo] show\|unset <glob>\|reset` | Inspect rules, remove an exact pattern, or delete the saved policy. | `e3 rack policy . show` |
+| `e3 dataflow run <repo> <ws> --rack\|--no-rack\|--rack-only` | Select rack routing for this run; rack-only disables capacity spill but keeps infrastructure fallback. ❗ Refused for remote URLs. | `e3 dataflow run . dev --rack` |
+| `e3 watch <source> <repo> <ws> --start --rack` | Reuses a rack session across watch runs and refreshes capacity; a second Ctrl-C stops the watch at once. ❗ `--rack` and `--rack-only` require `--start`. | `e3 watch src/main.ts . dev --start --rack` |
+| `e3 rack status\|list\|remove <id-or-label>\|approve <id-or-label>\|stop` | Inspect fleet, revoke a rack, admit the boot a quarantined rack last reported, or stop the shared hub. Status never starts an absent hub. | `e3 rack status` |
+| `e3 rack config [--listen <host:port>] [--allow <ip,...>] [--idle-exit <min>] [--advertise-url <origin>] [--tls\|--no-tls] [--require-approval\|--no-require-approval]` | Configure the shared listener (off by default; HTTPS with the hub's own certificate unless `--no-tls`); every rack and data URL is built on `--advertise-url` when set; zero idle-exit disables automatic shutdown; `--require-approval` quarantines a rack after each reboot until `e3 rack approve` admits it (enrollment admits the first boot). ❗ Plain HTTP only on the loopback or a private address, with `--allow`. | `e3 rack config --listen 10.0.0.5:7331` |
+
+The local process owns graph planning, repository and cache. After a cache
+miss, eligible plain task, map and merge bodies can execute on the rack; input
+closures and every output object transfer by hash. The session verifies the
+whole output graph before success. Placement never changes task/cache hashes.
+Local bodies and fallback share the original cores/memory budget; rack bodies
+use remote capacity. Calls, mutations and intake stay local. Stock runner
+configurations qualify; commands, custom environments/platforms and unscannable
+IR stay local. Host file/environment/network access stays local unless a rule
+deliberately grants `--allow-host-access` and its resources exist on the rack.
+
+The listener serves HTTPS with the hub's own self-signed certificate, made on
+first use and kept in `hub-tls.beast2`; a rack pins it by its SHA-256, which
+enrollment prints, so TLS runs end to end through any relay or tunnel. The
+usual path is an SSH reverse tunnel from the developer machine to a relay the
+rack reaches, with the hub on the loopback (`--allow 127.0.0.1`) and the relay's
+origin advertised. Removing the certificate file, with the hub stopped, makes a
+new one, which every worker must then pin again. `--no-tls` is for a TLS
+terminator in front of the hub; the hub refuses plain HTTP on a wildcard or
+public address, or without `--allow`. A listener on every address needs
+`--advertise-url` to enroll.
+
+The handoff targets a prepared independent worker VM with the matching verified
+`e3-rack` release. Its `configure --allocation <host-provisioned-path>` prompts
+for the source origin, the certificate SHA-256 for an HTTPS origin the hub
+serves, the worker label and settings; `enroll --allocation <path>` prompts for
+the one-time token. Origins, pins and credentials are encrypted together. A
+plain-HTTP origin needs `--allow-vpn-http` on both commands and host
+authorization for that exact endpoint, and belongs only on a path encrypted end
+to end, such as an SSH tunnel, never across a VPN hub that decrypts traffic.
+Never put a source address or token in worker command arguments or plaintext
+files. The local mint intentionally displays its token; do not log that output.
+
+Enrollment does not start services or authorize task claims. The local CLI
+observes a new registration (optionally filtered by label); compare it with the
+worker's result. Before a first heartbeat, status is `enrolled-not-started`.
+After managed resume, a ready heartbeat advertises free plus occupied slots;
+busy workers stay eligible. Fenced, draining or stopping workers withdraw their
+capacity and show `unavailable`. After 45 seconds without a heartbeat, readiness
+expires; lease polling cannot prolong it. Source boot approval, when required,
+remains an additional gate. This advertisement does not authorize host resource
+admission or removal of service/network quarantine.
+Resources and mutable whole-GPU assignments belong to the host lifecycle, not
+local enrollment flags. The companion accepts up to 32 mutually trusted sources
+in one worker. After draining and stopping its services and sockets, use
+`configure --allocation <path> --add-source`, confirm the shared trust boundary,
+and enter the additional origin privately. It prints the new source UUID for
+`enroll --allocation <path> --source <UUID>`; add `--allow-vpn-http` to both for a
+plain-HTTP origin. All sources must be enrolled before startup. Each has its own identity,
+client, cache and completion journal; actual execution shares one worker-wide
+task/GPU limit. Per-source capacity reports are not additive reservations.
+A worker is the trust boundary: a shared agent process can access all its
+sources' credentials, so separate clients, and a cloud deployment and developer
+hubs, need separate worker VMs. The hub keeps no task data; what a worker keeps
+between tasks is its own policy, and an agent that keeps nothing works unchanged. Worker-local `source list --allocation <path>`
+prints UUIDs/enrollment status; `--show-origins` adds origins only on a terminal.
+After drain/stop and source-side revocation, `source remove` or `source reset`
+with `--allocation <path> --source <UUID> --acknowledge-revocation` asks for UUID
+confirmation. Reset prints a new UUID for fresh enrollment; old pending results
+and caches stay in the retired namespace. Other sources and retained data are
+preserved, and host endpoint policy must be reconciled before activation. The
+flag acknowledges revocation; it does not revoke remotely.
+
+Worker-local `registry status --allocation <path>` reports only whether encrypted
+registry credentials are present; it can run while active and does not test access.
+After drain/stop, `registry configure --allocation <path>` privately prompts for
+username/token without re-entering settings. `registry clear --allocation <path>`
+asks for the installation UUID and removes only the stored access. Both require
+the same release, the writer lock and no unresolved enrollment/service transition;
+they preserve current identities, rotations, settings, journals and cached data.
+Credentials stay in the existing encrypted bundle, with no Docker login file.
+A later activation receives them; services remain stopped and the next boot fenced.
+Repeating configure with the same values or clear when absent does not write.
+Clearing/replacing does not revoke a token remotely or purge cached images.
+
+The companion's
+`reconcile-services` applies a host-provided next allocation and service budgets
+after a completed drain, with stopped units, the previous plan hash and exact
+operation/boot binding. It retains ciphertext/journals and network quarantine;
+it does not move a GPU, start a worker or change its executable release. For a
+compatible release change, invoke the target verified executable's `switch-release`
+with the same drain/plan bindings and unchanged resource budgets at the next
+allocation generation. It checks both managed-state contracts, updates only the
+runtime version in the current encrypted configuration, and preserves identities
+and journals through interruption or downgrade. Both four-file releases must stay
+available; a downgrade is a new operation, never a saved-state restore. Services
+stay stopped and quarantined. If the selected release never resumed, the exact
+recorded previous executable's `recover-release` binds the completed release
+receipt, current plan and new operation at another generation increase. It
+requires the original drain or a valid fenced journal with no operation at the
+failed generation, verifies both releases' recorded hashes, and preserves current
+credentials and journals. A prior resume attempt, even across reboot, refuses
+recovery. General journal repair remains unfinished. Named host management, source aliases, network
+activation, incompatible-state migrations and hardware acceptance remain
+companion cloud #253 work. This and durable completion retry are implemented in
+companion PR #254, separately from the original main protocol pin.
+
+Policy is `<repo>/rack/policy.beast2`; hub state is `~/.e3/rack` (or
+`E3_RACK_HOME`). A hub `dataflow run` starts exits after 15 idle minutes, and a
+run waits up to 10 seconds for an enrolled rack to report healthy (with none
+enrolled, or a policy that selects no task for a rack, it starts at once): run
+`e3 rack hub` as a service, or set `--idle-exit 0`, for a hub racks can always
+reach. A data URL lasts 15 minutes, which a transfer may use whole; an object is
+at most 5 GiB. The control schema currently requires protocol 2 exactly.
+Agents must report bundled e3 at least the project's release. Infrastructure
+failure falls back locally with a separate execution attempt, as does an
+attempt the rack stopped on its own; the run's own cancellation and the task's
+timeout never do. One cancelled subscriber does not stop work shared by other
+sessions of the same canonical repository and e3 release.
+
+**Programmatic attachment** (`@elaraai/e3-rack`)
+
+| Signature | Description | Example |
+| --- | --- | --- |
+| `openRackRunner({ repoPath, workspace, storage, label, budget?, settings?, policy?, rackOnly? })` | Opens a runner/session; preserve the caller's existing budget and process settings, use `rackSlots` to extend orchestrator width, and close in `finally`. | `const rack = await openRackRunner({ repoPath, workspace: 'dev', storage, label: 'run', budget });` |
+| `createDataflowRunnerFactory(options?)` | Pass to `createServer({ dataflowRunner })`; one session per API dataflow, using the server's shared budget. Explicit server opt-in overrides policy.enabled; rules still apply. | `const server = await createServer({ reposDir: './repos', dataflowRunner: createDataflowRunnerFactory() });` |
+
+The API server's optional rack package is needed only for
+`e3-api-server --repos ./repos --rack`. Portable handlers receive a runner
+factory and do not import hub code. Custom environments, remote function
+delegation, automatic reconnect after hub restart and embedded UI integration
+are deferred.