@elaraai/east-claude-plugin 1.0.98 → 1.0.100

package/skills/e3/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: e3
-description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex, e3.recordIndex.interval / e3.recordIndex.area for reads by a time window or a 2D window, and e3.recordIndex.series / e3.recordIndex.summary for a block of keys over a span of time tiles and their totals) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack enroll/policy, dataflow run --rack, watch --start --rack, optional e3-api-server --rack, (7) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui()."
+description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations, declared whole in one e3.record(name, type, { initial, indexes, mutations, migrations }) call: mutation.reduce/edit/patch; index.on/key/keys, index.interval / index.area for reads by a time window or a 2D window, index.series / index.summary for a block of keys over a span of time tiles and their totals, and index.nested for what a record's rows nest — jobs, a value a day, child rows, trees — filed item by item, with totals and parents' subtotals; and migrated between versions (migration.value/rows/rekey, in order), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack enroll/policy, dataflow run --rack, watch --start --rack, optional e3-api-server --rack, (7) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui()."
 ---
 
 # East Execution Engine (e3)
@@ -29,6 +29,10 @@ writing or changing East code:
    programs, exact and far cheaper, and the signatures omit the runtime rules
    that make East code correct.
 
+Tested e3 packages (inputs, tasks, stream tasks and partitions, custom tasks,
+functions, records with their mutations and migrations) are in the index too:
+search with `package: "e3"`.
+
 Nothing is injected for you; the search is the step.
 
 ## Quick Start
@@ -66,160 +70,26 @@ What do you need?
 │   │   └─ Split over an input, in pieces → wrap that input: e3.partition(dataset, { by? })
 │   ├─ A shell command           → e3.customTask(name, [inputs], outputType, command)
 │   ├─ A named function (RPC)    → e3.function(name, fn, config?) — called by name, persists nothing
-│   ├─ Audited state             → e3.record + e3.mutation.reduce | edit | patch, e3.recordIndex
-│   │   ├─ Read by a time window or a 2D window → e3.recordIndex.interval | e3.recordIndex.area
-│   │   └─ Read a block of keys over time tiles, or their totals → e3.recordIndex.series | e3.recordIndex.summary
-│   ├─ A record's type changes   → e3.migration.value | rows | rekey(name, record, fn, { after? })
+│   ├─ Audited state             → e3.record(name, type, { initial?, indexes?, mutations?, migrations? }), declared whole
+│   │   ├─ Its writes            → mutations: (mutation) => ({ name: mutation.reduce | edit | patch })
+│   │   ├─ An order to page in   → indexes: (index) => ({ name: index.on([...fields]) | index.key(fn) | index.keys(fn) }) (reference/record-indexes.md)
+│   │   ├─ Read by a time window or a 2D window → index.interval | index.area
+│   │   ├─ Read a block of keys over time tiles, or their totals → index.series | index.summary
+│   │   └─ Read what rows nest (jobs, a value a day, child rows, trees) a tile at a time, with totals and subtotals
+│   │       → index.nested({ collections, tile, summary? })
+│   ├─ A record's type changes   → migrations: (migration) => [migration.value | rows | rekey(name, fn), …], in order (reference/record-migrations.md)
 │   ├─ A UI task                 → ui(name, [inputs], fn) from @elaraai/e3-ui (the e3-ui skill)
 │   ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
 │   └─ Bundle and export         → e3.package(name, version, ...items); e3.export(pkg, zipPath)
-├─ Operate it from the shell     → the e3 CLI (below): deploy, run, read and write datasets, mutate records, gc
-├─ Delegate local task bodies to a rack → e3 rack enroll + policy; dataflow run --rack, watch --start --rack
+├─ Operate it from the shell     → the e3 CLI (reference/cli.md): deploy, run, read and write datasets, mutate records, gc
+├─ Delegate local task bodies to a rack → e3 rack enroll + policy; dataflow run --rack, watch --start --rack (reference/rack.md)
 └─ Drive it from code
-    ├─ A server, over HTTP       → @elaraai/e3-api-client
+    ├─ A server, over HTTP       → @elaraai/e3-api-client (reference/from-code.md)
     ├─ A local repository        → @elaraai/e3-core
     ├─ Serve repositories        → e3-api-server --repos <dir>, or createServer()
-    └─ e3 in a browser, no server → @elaraai/e3-web: serveUnits, serveE3, createWebE3 ("Running e3 in a browser")
+    └─ e3 in a browser, no server → @elaraai/e3-web: serveUnits, serveE3, createWebE3 (reference/browser.md)
 ```
 
-## Rack delegation from a local repository
-
-**Setup and routing**
-
-| Signature | Description | Example |
-| --- | --- | --- |
-| `e3 rack enroll --listen <host:port> [--allow <ip,...>] [--advertise-url <origin>] [--label <label>] [--no-wait]` | Starts the user-wide hub and prints the source origin, the hub's certificate SHA-256, the matching release and a single-use token for the companion's hidden prompts. Waits for a new registration, not activation. Behind an SSH reverse tunnel, listen on the loopback and advertise the relay's origin. | `e3 rack enroll --listen 127.0.0.1:7331 --allow 127.0.0.1 --advertise-url https://relay.example:7401` |
-| `e3 rack policy [repo] set <glob> rack\|local [--size <size>] [--timeout <min>] [--allow-host-access]` | First whole-task-name match wins; `*` and `?` are wildcards. Size: serverless, small, medium, large, xlarge. | `e3 rack policy . set 'train_*' rack --size large` |
-| `e3 rack policy [repo] enable\|disable\|mode <opt-in\|auto>` | Enable/disable saved opt-in; `mode auto` selects eligible unmatched tasks. Defaults: disabled, opt-in. | `e3 rack policy . enable` |
-| `e3 rack policy [repo] defaults [--no-spill] [--claim-timeout <sec>] [--retry-task-failures-locally]` | Defaults: large, 1440-minute timeout, spill on, 30-second claim recheck, no task-failure retry. | `e3 rack policy . defaults --no-spill` |
-| `e3 rack policy [repo] show\|unset <glob>\|reset` | Inspect rules, remove an exact pattern, or delete the saved policy. | `e3 rack policy . show` |
-| `e3 dataflow run <repo> <ws> --rack\|--no-rack\|--rack-only` | Select rack routing for this run; rack-only disables capacity spill but keeps infrastructure fallback. ❗ Refused for remote URLs. | `e3 dataflow run . dev --rack` |
-| `e3 watch <source> <repo> <ws> --start --rack` | Reuses a rack session across watch runs and refreshes capacity; a second Ctrl-C stops the watch at once. ❗ `--rack` and `--rack-only` require `--start`. | `e3 watch src/main.ts . dev --start --rack` |
-| `e3 rack status\|list\|remove <id-or-label>\|approve <id-or-label>\|stop` | Inspect fleet, revoke a rack, admit the boot a quarantined rack last reported, or stop the shared hub. Status never starts an absent hub. | `e3 rack status` |
-| `e3 rack config [--listen <host:port>] [--allow <ip,...>] [--idle-exit <min>] [--advertise-url <origin>] [--tls\|--no-tls] [--require-approval\|--no-require-approval]` | Configure the shared listener (off by default; HTTPS with the hub's own certificate unless `--no-tls`); every rack and data URL is built on `--advertise-url` when set; zero idle-exit disables automatic shutdown; `--require-approval` quarantines a rack after each reboot until `e3 rack approve` admits it (enrollment admits the first boot). ❗ Plain HTTP only on the loopback or a private address, with `--allow`. | `e3 rack config --listen 10.0.0.5:7331` |
-
-The local process owns graph planning, repository and cache. After a cache
-miss, eligible plain task, map and merge bodies can execute on the rack; input
-closures and every output object transfer by hash. The session verifies the
-whole output graph before success. Placement never changes task/cache hashes.
-Local bodies and fallback share the original cores/memory budget; rack bodies
-use remote capacity. Calls, mutations and intake stay local. Stock runner
-configurations qualify; commands, custom environments/platforms and unscannable
-IR stay local. Host file/environment/network access stays local unless a rule
-deliberately grants `--allow-host-access` and its resources exist on the rack.
-
-The listener serves HTTPS with the hub's own self-signed certificate, made on
-first use and kept in `hub-tls.beast2`; a rack pins it by its SHA-256, which
-enrollment prints, so TLS runs end to end through any relay or tunnel. The
-usual path is an SSH reverse tunnel from the developer machine to a relay the
-rack reaches, with the hub on the loopback (`--allow 127.0.0.1`) and the relay's
-origin advertised. Removing the certificate file, with the hub stopped, makes a
-new one, which every worker must then pin again. `--no-tls` is for a TLS
-terminator in front of the hub; the hub refuses plain HTTP on a wildcard or
-public address, or without `--allow`. A listener on every address needs
-`--advertise-url` to enroll.
-
-The handoff targets a prepared independent worker VM with the matching verified
-`e3-rack` release. Its `configure --allocation <host-provisioned-path>` prompts
-for the source origin, the certificate SHA-256 for an HTTPS origin the hub
-serves, the worker label and settings; `enroll --allocation <path>` prompts for
-the one-time token. Origins, pins and credentials are encrypted together. A
-plain-HTTP origin needs `--allow-vpn-http` on both commands and host
-authorization for that exact endpoint, and belongs only on a path encrypted end
-to end, such as an SSH tunnel, never across a VPN hub that decrypts traffic.
-Never put a source address or token in worker command arguments or plaintext
-files. The local mint intentionally displays its token; do not log that output.
-
-Enrollment does not start services or authorize task claims. The local CLI
-observes a new registration (optionally filtered by label); compare it with the
-worker's result. Before a first heartbeat, status is `enrolled-not-started`.
-After managed resume, a ready heartbeat advertises free plus occupied slots;
-busy workers stay eligible. Fenced, draining or stopping workers withdraw their
-capacity and show `unavailable`. After 45 seconds without a heartbeat, readiness
-expires; lease polling cannot prolong it. Source boot approval, when required,
-remains an additional gate. This advertisement does not authorize host resource
-admission or removal of service/network quarantine.
-Resources and mutable whole-GPU assignments belong to the host lifecycle, not
-local enrollment flags. The companion accepts up to 32 mutually trusted sources
-in one worker. After draining and stopping its services and sockets, use
-`configure --allocation <path> --add-source`, confirm the shared trust boundary,
-and enter the additional origin privately. It prints the new source UUID for
-`enroll --allocation <path> --source <UUID>`; add `--allow-vpn-http` to both for a
-plain-HTTP origin. All sources must be enrolled before startup. Each has its own identity,
-client, cache and completion journal; actual execution shares one worker-wide
-task/GPU limit. Per-source capacity reports are not additive reservations.
-A worker is the trust boundary: a shared agent process can access all its
-sources' credentials, so separate clients, and a cloud deployment and developer
-hubs, need separate worker VMs. The hub keeps no task data; what a worker keeps
-between tasks is its own policy, and an agent that keeps nothing works unchanged. Worker-local `source list --allocation <path>`
-prints UUIDs/enrollment status; `--show-origins` adds origins only on a terminal.
-After drain/stop and source-side revocation, `source remove` or `source reset`
-with `--allocation <path> --source <UUID> --acknowledge-revocation` asks for UUID
-confirmation. Reset prints a new UUID for fresh enrollment; old pending results
-and caches stay in the retired namespace. Other sources and retained data are
-preserved, and host endpoint policy must be reconciled before activation. The
-flag acknowledges revocation; it does not revoke remotely.
-
-Worker-local `registry status --allocation <path>` reports only whether encrypted
-registry credentials are present; it can run while active and does not test access.
-After drain/stop, `registry configure --allocation <path>` privately prompts for
-username/token without re-entering settings. `registry clear --allocation <path>`
-asks for the installation UUID and removes only the stored access. Both require
-the same release, the writer lock and no unresolved enrollment/service transition;
-they preserve current identities, rotations, settings, journals and cached data.
-Credentials stay in the existing encrypted bundle, with no Docker login file.
-A later activation receives them; services remain stopped and the next boot fenced.
-Repeating configure with the same values or clear when absent does not write.
-Clearing/replacing does not revoke a token remotely or purge cached images.
-
-The companion's
-`reconcile-services` applies a host-provided next allocation and service budgets
-after a completed drain, with stopped units, the previous plan hash and exact
-operation/boot binding. It retains ciphertext/journals and network quarantine;
-it does not move a GPU, start a worker or change its executable release. For a
-compatible release change, invoke the target verified executable's `switch-release`
-with the same drain/plan bindings and unchanged resource budgets at the next
-allocation generation. It checks both managed-state contracts, updates only the
-runtime version in the current encrypted configuration, and preserves identities
-and journals through interruption or downgrade. Both four-file releases must stay
-available; a downgrade is a new operation, never a saved-state restore. Services
-stay stopped and quarantined. If the selected release never resumed, the exact
-recorded previous executable's `recover-release` binds the completed release
-receipt, current plan and new operation at another generation increase. It
-requires the original drain or a valid fenced journal with no operation at the
-failed generation, verifies both releases' recorded hashes, and preserves current
-credentials and journals. A prior resume attempt, even across reboot, refuses
-recovery. General journal repair remains unfinished. Named host management, source aliases, network
-activation, incompatible-state migrations and hardware acceptance remain
-companion cloud #253 work. This and durable completion retry are implemented in
-companion PR #254, separately from the original main protocol pin.
-
-Policy is `<repo>/rack/policy.beast2`; hub state is `~/.e3/rack` (or
-`E3_RACK_HOME`). A hub `dataflow run` starts exits after 15 idle minutes, and a
-run waits up to 10 seconds for an enrolled rack to report healthy (with none
-enrolled, or a policy that selects no task for a rack, it starts at once): run
-`e3 rack hub` as a service, or set `--idle-exit 0`, for a hub racks can always
-reach. A data URL lasts 15 minutes, which a transfer may use whole; an object is
-at most 5 GiB. The control schema currently requires protocol 2 exactly.
-Agents must report bundled e3 at least the project's release. Infrastructure
-failure falls back locally with a separate execution attempt, as does an
-attempt the rack stopped on its own; the run's own cancellation and the task's
-timeout never do. One cancelled subscriber does not stop work shared by other
-sessions of the same canonical repository and e3 release.
-
-**Programmatic attachment** (`@elaraai/e3-rack`)
-
-| Signature | Description | Example |
-| --- | --- | --- |
-| `openRackRunner({ repoPath, workspace, storage, label, budget?, settings?, policy?, rackOnly? })` | Opens a runner/session; preserve the caller's existing budget and process settings, use `rackSlots` to extend orchestrator width, and close in `finally`. | `const rack = await openRackRunner({ repoPath, workspace: 'dev', storage, label: 'run', budget });` |
-| `createDataflowRunnerFactory(options?)` | Pass to `createServer({ dataflowRunner })`; one session per API dataflow, using the server's shared budget. Explicit server opt-in overrides policy.enabled; rules still apply. | `const server = await createServer({ reposDir: './repos', dataflowRunner: createDataflowRunnerFactory() });` |
-
-The API server's optional rack package is needed only for
-`e3-api-server --repos ./repos --rack`. Portable handlers receive a runner
-factory and do not import hub code. Custom environments, remote function
-delegation, automatic reconnect after hub restart and embedded UI integration
-are deferred.
-
 ## Authoring a package
 
 ### Inputs — `e3.input(name, type, source?)`
@@ -335,17 +205,26 @@ const shout = e3.task('shout', [greet.output],
 - **An East function written in python, or in another node package**:
   `East.importFunction(pkg, name, FunctionType(...))` in the body. `e3.export`
   resolves it and embeds its IR, so the task is pure IR on any runner, its
-  locations kept so an error or a profile names the function's own lines. A uv
+  locations kept so an error or a profile names the function's own lines (so
+  moving them re-runs the task: **Caching**, below). A uv
   workspace member is found by name in `uv.lock`, and its root module's
-  `east_functions` dict is exported with `east-py export-functions` (from the
-  project's `.venv`, else `EAST_PY` or PATH); an npm member is found by its
-  `package.json` name, and its built `./functions` export's `eastFunctions` is
-  exported with `east-node export-functions` (`EAST_NODE`). A package built
-  elsewhere is passed as its manifest — `{ functions: [...] }` / `--functions` —
-  which wins. The declared type must equal the exported one exactly, and each
-  platform function it calls must be provided by the task's runner, by name or
-  by family (`east-py-std` ≡ `@elaraai/east-node-std` ≡ `east-c-std`;
-  `east-py-io` ≡ `@elaraai/east-node-io`).
+  `east_functions` dict is exported with `east-py export-functions` (`EAST_PY`
+  when set, else the project's `.venv`, else PATH); an npm member is found by
+  its `package.json` name, and its built `./functions` export's `eastFunctions`
+  is exported with `east-node export-functions` (`EAST_NODE`). Each member is
+  exported in one process however many tasks import from it, the members side
+  by side, as many at once as the export has cores (a deploy's `-j`,
+  `e3.export`'s `cores`). A python member's manifests are kept in
+  `node_modules/.cache/e3/functions` under the directory the export runs in,
+  keyed on its sources and those of the local packages it depends on, the
+  workspace's `uv.lock` and `pyproject.toml`, the east-py exporting it and the
+  functions and providers each manifest is for, so a redeploy whose python has
+  not changed runs no export; deleting the directory clears it, and a node
+  member is exported every time. A package built elsewhere is passed as its
+  manifest — `{ functions: [...] }` / `--functions` — which wins. The declared
+  type must equal the exported one exactly, and each platform function it calls
+  must be provided by the task's runner, by name or by family (`east-py-std` ≡
+  `@elaraai/east-node-std` ≡ `east-c-std`; `east-py-io` ≡ `@elaraai/east-node-io`).
 - **Environments** are derived at export from each `{ custom }` platform: the
   providing workspace package's closure (`pyproject.toml` + `uv.lock` + sdists,
   or `package.json` + lockfile + `npm pack`) is captured into the package and
@@ -501,7 +380,7 @@ const change = e3.streamTask('change', {
 | `e3.customTask(name, inputs, outputType, command, { environment? })` | A task that runs a bash script: `command($, inputPaths, outputPath)` builds it from the staged beast2 input files, and it writes the output as a beast2 file. |
 | `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API) and its result returned inline: not in the dataflow, and a call persists nothing. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A server bounds a call by a timeout and a 1 MiB result, so long work belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
 | `e3.package(name, version, ...items)` | Bundles the items and everything they depend on — pass the last tasks, records and functions; a package passed in contributes its contents. The name and version must be valid file names. |
-| `e3.export(pkg, zipPath, { functions? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself; `functions` lists manifests built elsewhere). |
+| `e3.export(pkg, zipPath, { functions?, cores? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself, `cores` packages at once, the CPUs by default; `functions` lists manifests built elsewhere). |
 
 ```typescript
 const process = e3.customTask('process', [rawData], StringType,
@@ -513,15 +392,19 @@ await e3.export(e3.package('tools', '1.0.0', process, add), '/tmp/tools.zip');
 
 ### Records — audited state, written only through mutations
 
-`e3.record(name, type, initialValue?)` is a dataset at `.records.<name>` that
-tasks read and react to like any input, but that only its **mutations** write:
+`e3.record(name, type, { initial?, indexes?, mutations?, migrations? })` is a
+dataset at `.records.<name>` that tasks read and react to like any input, but
+that only its **mutations** write:
 a raw `dataset set` is refused. A mutation is pure, synchronous East — no async
 body and no platform call, since the compare-and-swap retry runs it again
 against fresher state — named by an identifier. Each commit records its parent,
 the state, the mutation, its arguments, the actor and the time. Deploy mints a
 `$init` commit from the initial value, and a redeploy keeps state and history.
-The record's version vector carries the commit hash, so even a mutation that
-leaves the state unchanged triggers downstream tasks.
+A large initial value is given as a function that makes it —
+`initial: () => East.compile(generate, [])(50_000n)` — called once, as the
+package is exported, never where the declaration is only read (a page binding
+the record). The record's version vector carries the commit hash, so even a
+mutation that leaves the state unchanged triggers downstream tasks.
 
 Declared with no initial value, a record is **unassigned** until its first
 state is committed — `e3 seed <repo> <record> <file.beast2> -w <ws>` from a
@@ -532,38 +415,54 @@ and a mutation on it is refused until then. So a record whose first state is
 data that arrives after the deploy — a customer list, a reference table —
 never runs its dependents on an empty value first.
 
+A record is **declared whole**: its type, its initial value and its parts, in
+one call, never added to after. Each part comes from a builder the call hands
+its declaration, typed by the record's type: `indexes` gets `index` and returns
+the indexes by name, `mutations` gets `mutation` and returns the mutations by
+name, and `migrations` gets `migration` and returns its steps in order. Each is
+checked where the record is declared, every refusal naming the record and the
+part, and the record carries each by its name — `plans.mutations.add_plan`,
+`plans.indexes.by_due` — which a package, a binding (`Record.bind(plans)`,
+`Data.bindPaged(plans, { indexes: ['by_due'] })`) and TypeScript all read.
+
 | Form | Body | Use it when |
 |---|---|---|
-| `e3.mutation.reduce(name, rec, fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
-| `e3.mutation.edit(name, rec, fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
-| `e3.mutation.patch(rec, name = 'patch')` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
+| `mutation.reduce(fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
+| `mutation.edit(fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
+| `mutation.patch()` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
 
 Every form commits a delta addressed by key, rewriting only the segments it
 touches (a `reduce` of a record that is not a Dict or Set writes it whole), and
 takes `{ runner? }`. In an `edit`, repeated edits of one key fold, a `set` of the
 value a row already holds changes nothing, and a `delete` or `update` of a key
 the record does not hold is a conflict naming the key: the caller's view of the
-record is stale. `e3.mutation.editType(recordType)` is the type of `edit`.
+record is stale. `e3.mutation.editType(recordType)` is the type of `edit`. A
+mutation is named by its key, an identifier; a part's functions are typed by the
+record's type, so name the type (`PlansType`) to write them, in the declaration
+or beside it.
 
 ```typescript
 const PlanType = StructType({
   title: StringType, owner: StringType, due: DateTimeType, resources: SetType(StringType),
 });
-const plans = e3.record('plans', DictType(StringType, PlanType), new Map());
-
-const addPlan = e3.mutation.reduce('add_plan', plans,
-  East.function([plans.type, StringType, PlanType], plans.type, ($, state, id, plan) => {
-    const next = $.let(state.copy());          // the state is frozen: copy first
-    $(next.insert(id, plan));
-    return next;
-  }));
-
-const reschedule = e3.mutation.edit('reschedule', plans,
-  East.function([plans.type, StringType, DateTimeType, e3.mutation.editType(plans.type)], NullType,
-    ($, state, id, due, edit) => {
-      const plan = $.let(state.get(id));       // decodes one segment, not the record
-      $(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
-    }));
+const PlansType = DictType(StringType, PlanType);
+
+const plans = e3.record('plans', PlansType, {
+  initial: new Map(),
+  mutations: (mutation) => ({
+    add_plan: mutation.reduce(East.function([PlansType, StringType, PlanType], PlansType, ($, state, id, plan) => {
+      const next = $.let(state.copy());          // the state is frozen: copy first
+      $(next.insert(id, plan));
+      return next;
+    })),
+    reschedule: mutation.edit(East.function([PlansType, StringType, DateTimeType, e3.mutation.editType(PlansType)], NullType,
+      ($, state, id, due, edit) => {
+        const plan = $.let(state.get(id));       // decodes one segment, not the record
+        $(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
+      })),
+    patch: mutation.patch(),                     // a view's save
+  }),
+});
 ```
 
 Costs count in segments: a Dict or Set record is cut by key and by each entry's
@@ -571,511 +470,43 @@ encoded size, so wide rows get segments of a few rows and a one-row edit stays
 cheap. A view reads every field of the rows it shows, so keep a bulky payload it
 does not display in a second record keyed the same way.
 
-#### Secondary indexes — `e3.recordIndex(name, record, { on | key | keys, value? })`
-
-A second collection over a Dict record, in another sort order, maintained in the
-same commit, so a view by an attribute is a page rather than a scan. `on`
-names row fields in index order: one field uses its value as the key, several
-use a Struct in that order, and a Set field gives an entry per element. It
-also carries field metadata for `Data.bindPaged(record, { indexes })` and
-`Slice.bindRecord` (see **e3-ui**). `key` gives one computed entry per row;
-`keys` returns a Set, one entry per element (a row
-naming five resources appears five times, an empty set not at all); `value` is a
-projection a view renders without touching the record. The functions are pure
-and synchronous, since an index maintained commit by commit must equal its
-rebuild to the byte. The name is an identifier, never `primary` (the record's own
-order). Read through one with `index=<name>` on a page or key search; `e3
-reindex` rebuilds. A deploy builds an index the package declares and the state
-lacks, drops one it no longer declares, and runs nothing for the rest; a build
-is a split task over the record.
-
-```typescript
-const byDue = e3.recordIndex('by_due', plans, { on: ['due'] });
-const byOwnerDue = e3.recordIndex('by_owner_due', plans, { on: ['owner', 'due'] });
-const byOwner = e3.recordIndex('by_owner', plans, {
-  key:   East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner),
-  value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
-});
-const byResource = e3.recordIndex('by_resource', plans, {
-  keys: East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources),
-});
-const pkg = e3.package('planning', '1.0.0',
-  plans, addPlan, reschedule, e3.mutation.patch(plans), byDue, byOwnerDue, byOwner, byResource);
-```
-
-#### Interval and area indexes — `e3.recordIndex.interval`, `e3.recordIndex.area`
-
-These indexes answer more than an order — as do the series and summary indexes
-below — each with a constructor of its own whose keys the SDK generates;
-`e3.recordIndex` refuses an `interval` option, naming its constructor. Underneath each is an ordinary index — the same build,
-mutation, storage, paging and `e3 reindex` — and the record Slice reads a
-window through either (see **e3-ui**). Their generated keys are query
-machinery: edits and sessions always target the record's primary key.
-
-**`e3.recordIndex.interval(name, record, { on?, from, to })`** keeps events
-visible while they overlap a selected time window. `from` and `to` are the
-row's DateTime or Option<DateTime> fields; optional `on` fields are exact
-leading choices:
-
-```typescript
-const byTime = e3.recordIndex.interval('by_time', events, { from: 'start', to: 'end' });
-const byStatusTime = e3.recordIndex.interval('by_status_time', events, {
-  on: ['status'], from: 'start', to: 'end',
-});
-```
-
-It stores at most three entries per event per leading-field combination,
-independent of duration. The record Slice adapter resolves a half-open
-`[from, to)` window with at most 55 disjoint bounded queries per exact prefix;
-each matching original record occurs once, including events that started
-earlier or span the whole window. It never scans the record or guesses a
-maximum duration. An event ending at `from` is outside; an instant at `from`
-is inside and one at `to` is outside. A missing or non-positive end is an
-instant. A missing start remains selectable as Unset/backlog.
-
-Use `on: ['start']` only when the intended query is **starts within**, since it
-cannot preserve carry-in events. When both declarations are available, a Slice
-range on the start field uses the interval declaration. The generated interval
-keys are query machinery; record edits and sessions always target the original
-primary key. Calendar's `Calendar.events(binding, { slice, entries, … })`
-requires a matching interval declaration and the primary `entries` binding to
-confirm saves even after an edit leaves the selected window.
-
-**`e3.recordIndex.area(name, record, { on?, x, y, width?, height?, at?, cell })`**
-reads the rows whose box meets a 2D window — what a floor plan's viewport
-shows — the interval index's twin:
-
-```typescript
-// racks: Dict<String, { hall: String, x: Option<Float>, y: Option<Float>, w: Float, h: Float }>
-const racksByArea = e3.recordIndex.area('racks_by_area', racks, {
-  x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01,
-});
-const racksByHallArea = e3.recordIndex.area('racks_by_hall_area', racks, {
-  on: ['hall'], x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01,
-});
-```
-
-| Option | The box's |
-|---|---|
-| `x`, `y` | place: two Float fields, or two Option<Float> fields whose `none` is a row not yet placed |
-| `width`, `height` | size: two Float fields, or neither for a point |
-| `at` | `'corner'`, the default — `x`, `y` is the box's minimum corner, as a zone's is — or `'centre'`, as an item's is |
-| `cell` | finest cell: its side in world units, about the smallest thing drawn — 0.01 for a floor in metres, 1 for one in millimetres. It is rounded down to a power of two, so the keys' arithmetic is exact on every runtime, and has no default: a wrong one costs a silent tenfold read |
-| `on` | exact leading fields, as the interval index's; a Set field fans out |
-
-Each option takes only the row's fields of its type, so a wrong one does not
-compile, and the definition checks them again, naming the option, the field and
-its type; a row field named `__area` is refused, the name being the key's. The
-domain is the square ±`cell`·2²⁷ world units around zero (±1,048 km at a 0.01
-cell, which rounds to 2⁻⁷), in 29 levels. Each row is filed once, under the
-deepest level whose cell is at least twice its box's larger side, in the cell
-holding its box's centre: the key `{ …on, __area: Integer }`. A row not yet
-placed files under `__area` −1, which the record Slice reads as the unplaced
-clause; one outside the domain, not finite, or larger than half of it files
-under the root, which every window reads. The index **carries its rows**: each
-entry's projection is the whole row, so a window reads the index alone and no
-record segment, and it takes no `value`. The record Slice reads a viewport,
-grown by an eighth of its larger side on each side, as at most 266 disjoint
-searches per exact prefix, sent in one request, which together hold every row
-whose box meets it, each once — beside rows of the cells searched that it
-misses, which the reader draws or not.
-
-#### Series and summary indexes — `e3.recordIndex.series`, `e3.recordIndex.summary`
-
-What a planning board reads: the events on a block of keys — a Plan's
-resources — over a span of time, and, zoomed out past what a screen can draw,
-how busy each key is. Both file a row's event by the key it is on and by each
-time tile it touches — `day`, `week` (ISO, from Monday), `month`, `quarter` or
-`year`, in UTC, half-open — their key `{ …on, __tile, <key> }`. The tile leads
-the key, so a block of keys in one tile is one contiguous range:
-
-```typescript
-// jobs: Dict<String, { press: String, start: Option<DateTime>, end: Option<DateTime>, … }>
-const jobsByPressWeek = e3.recordIndex.series('jobs_by_press_week', jobs, {
-  key: 'press', from: 'start', to: 'end', tile: 'week',
-});
-const jobsByPressMonth = e3.recordIndex.summary('jobs_by_press_month', jobs, {
-  key: 'press', from: 'start', to: 'end', tile: 'month',
-});
-```
+A record's indexes (`index.on` / `key` / `keys`, `interval`, `area`,
+`series`, `summary`, `nested`) are in
+[reference/record-indexes.md](reference/record-indexes.md), and its migrations
+in [reference/record-migrations.md](reference/record-migrations.md).
 
-| Option | |
-|---|---|
-| `key` | the field naming what an event is on: any field but a Set |
-| `from`, `to` | the event's start and end: DateTime or Option<DateTime> fields. A row with no start is in no tile; no end, or one at or before the start, is an instant, in the one tile holding it |
-| `tile` | `day`, `week`, `month`, `quarter` or `year` |
-| `on` | exact leading fields, before the tile; a Set field fans out |
-
-**A series index** is an ordinary index whose keys the SDK generates, an entry
-per tile an event touches, and it **carries its rows**, as the area index does:
-a block of keys over a span of tiles reads the index alone, a search a tile, in
-one request. **A summary index** holds, under each key, `{ count, ms }`: how
-many rows touch its tile, and the milliseconds of it they cover — a row's shares
-add up to its length, and an instant's is none. Each entry stands for every row
-of its key and names none, so its window's `key` is Null, a read of it joins
-nothing, and a read that needs its rows — a joined read, a placement
-(`datasetPlace`) or a membership read (`datasetWithin`) — is refused, naming the
-index; the record's describe says it is a summary. A write
-changes its totals by what its rows took away and gave — its program reads the
-record, never the summary — and the apply adds that to each total, dropping one
-whose count reaches zero; the totals are Integers, so a maintained summary is the
-one `e3 reindex` sums, byte for byte, and a build split into pieces adds the
-pieces' totals of a key. A change that would leave a total no rebuild sums — a
-count below zero, or time with no row — fails the write, naming the total and
-`e3 reindex`, which rebuilds it; a total past the largest Integer fails the
-write or the build, naming a finer tile.
-
-An event touching more than 4,096 tiles — eleven years of days — is refused
-where it is written, naming it, its span and the tile: declare a coarser tile.
-Each option takes only the row's fields of its type, as the area index's do,
-and the definition checks them again; a row field named `__tile` is refused, the
-name being the key's, and so are a `value` and an `on` field that is the key,
-the start or the end: the key follows the tile, and the tiles are cut by the
-start and the end.
-
-#### Migrations — how a record's type changes
-
-| Form | Function | Runs as |
-|---|---|---|
-| `e3.migration.value(name, rec, fn, { after? })` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
-| `e3.migration.rows(name, rec, fn, { after? })` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
-| `e3.migration.rekey(name, rec, fn, { after? })` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
-
-A record's steps are one chain: each names the step before it with `after` (a
-step of the same record, leaving it as this step's input type), passing the last
-step passes the chain, and the last step leaves the record as its declared type.
-A step's name is an identifier, unique on the record, and a workspace records the
-steps it has applied by name — so moving a step's code, or re-exporting it with a
-newer e3, does not run it again, and an edit to an applied step's body is not
-detected. The functions are pure, synchronous East.
-
-```typescript
-const RowV1Type = StructType({ title: StringType });
-const RowV2Type = StructType({ title: StringType, owner: StringType });
-const plans = e3.record('plans', DictType(StringType, RowV2Type), new Map());
-
-const addOwner = e3.migration.rows('add_owner', plans,
-  East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' })));
-// The next version adds its step after this one: e3.migration.value('…', plans, fn, { after: addOwner })
-const pkg = e3.package('planning', '2.0.0', plans, addOwner);
-```
-
-What a deploy does with each record, decided before it writes anything:
-
-| The workspace holds | Against the package's chain | The deploy |
-|---|---|---|
-| no state | — | mints `$init` from the initial value, the whole chain counted applied; declared with no initial value, leaves it unassigned (`unassigned`) for `e3 seed` |
-| state | every step applied, the type unchanged | keeps it, with a `$deploy` commit when the package changed |
-| state | a proper prefix applied | runs the rest, a `$migrate:<name>` commit each |
-| state | every step applied, but the type changed | refused: a type change needs a migration, or `--schema reset` |
-| state | applied steps the chain does not start with | refused, naming the steps applied and those declared |
-| a record the package no longer declares | — | refused unless `--allow-drop-records` |
-
-`--schema <policy>` on `e3 workspace deploy` and `e3 watch`: `migrate` (the
-default), `fail` (run no migration; refuse) or `reset` (reset the record to its
-initial value, with a `$reset` commit (`reset`); one declared with no initial
-value goes back to unassigned (`unassign`), for `e3 seed` again). `--plan` prints what the deploy would do
-to each record, index and input and writes nothing — from a zip or a source it
-imports nothing, and a server plans only a package it holds. The steps run
-before the deploy writes: a failed step leaves the workspace as it was, and a
-deploy run again is served the steps that finished from the cache.
-
-What a deploy does with each input — a dataset an `e3.input` declares, which
-people set:
-
-| The input now | Its type | The deploy |
-|---|---|---|
-| holds the value the deployed package gave it, or is new | — | the new package's value (`package`) |
-| someone set it | — | the new package's value under `--inputs reset`, the default, and says it reset it (`reset`) |
-| someone set it | the new package's | kept, under `--inputs keep-edited` (`keep`) |
-| someone set it | changed | the new package's value, and says why (`reset`) |
-| the deployed package took it from a file, and it holds a value | — | the new package's value under either policy, since a value set since cannot be told from the file's, and says so (`reset`) |
-| the new package takes it from a file | — | its file, as every deploy does (`file`); a deploy that reads no file, a server's, leaves it unset for its client to send |
-| the new package no longer declares it | — | gone, with its value (`drop`) |
-
-## The e3 CLI
-
-Every `<repo>` is a local path or a server URL, `http(s)://<host>/repos/<name>`;
-where `[repo]` is optional it defaults to `$E3_REPO`, then `.`. Dataset paths are
-flat, `<ws>.<name>` — an input or a task output, with "did you mean" suggestions;
-a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
-
-| Command | Does |
-|---|---|
-| **Repositories** | |
-| `e3 repo create [repo] [--exist-ok]` | Create one (`--exist-ok`: succeed if it exists). |
-| `e3 repo status [repo]` · `e3 repo remove [repo] [-r]` | Show objects, packages, workspaces · remove (`-r`: its workspaces first). |
-| `e3 repo gc [repo] [--dry-run] [--keep-runs <n>] [--keep-days <d>] [--min-age <ms>]` | Drop old run history, then what nothing names (below). |
-| `e3 repo list <server>` | The repositories a server holds. |
-| **Packages** | |
-| `e3 package import [repo] <zip>` · `export [repo] <pkg[@ver]> <zip>` | Import · export (`--quiet`: errors only). |
-| `e3 package list [repo]` · `remove [repo] <pkg[@ver]>` | |
-| **Workspaces** | |
-| `e3 workspace create [repo] <ws>` · `list [repo]` · `status [repo] <ws>` · `remove [repo] <ws>` | `status` shows tasks, datasets and locks. |
-| `e3 workspace copy [repo] <from> <to>` | Make `<to>` what `<from>` is now — its package and every input, record and task output, so nothing runs again — made, or replaced whole with its runs. It writes refs only, so its cost follows the number of datasets, never their size; a write to either afterwards leaves the other as it was. Refused while a dataflow runs in `<to>`. |
-| `e3 workspace deploy [repo] <ws> <pkg[@ver]>` | Deploy an imported package. |
-| `… --from-zip <zip>` · `… --from-source <src.ts> [--functions <manifest…>]` | Import the zip (or bundle the source) and deploy, creating the workspace. |
-| `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on; `--memory`: what those may reserve (local). |
-| `e3 workspace export [repo] <ws> <zip> [--name <n>] [--version <v>]` | The workspace's state as a package. |
-| **Datasets** | |
-| `e3 dataset get [repo] <ws.name> [-f east\|json\|beast2]` | Print a value. |
-| `e3 dataset set [repo] <ws.name> <file> [--type <spec> \| --type-file <path>]` | Write from `.east`, `.beast2`, `.json` or `.csv` (JSON and CSV need the type). |
-| `e3 dataset set [repo] <ws.name> --from-file <path.beast2> [-j <n>] [--memory <size>]` | Take a beast2 file in as the value, as a deploy takes a `file` source in, saying how far it has got; `-j`, `--memory`: the budget its intake units run under (local). |
-| `e3 dataset list [repo] <ws> [-l]` · `status [repo] <ws.name>` · `find [repo] <ws> <pattern>` | Paths (`-l`: kind, type, status, size) · one dataset · by substring or glob. |
-| **Running** | |
-| `e3 dataflow run [repo] <ws> [--filter <task>] [--force \| --force-task <task>…] [-j <n>] [--memory <size>] [-v]` | Run what is stale, then print the outputs' paths. `--filter`: one task, by its exact name, and the tasks it depends on. `--force`: re-run every task the run runs (under `--filter`, its task); `--force-task`, repeated for each: re-run the tasks it names, whose dependents re-run when an output changes. A task the graph lacks, or one `--filter` leaves out, refuses the run. A task whose input nothing has set — an input never set, or a record declared with no initial value and not yet seeded — is left waiting, with its dependents (`[WAIT]`, `Waiting:`), and the run completes without them; they run once it is set. |
-| `e3 task list [repo] <ws>` | Tasks with the status of each one's latest run. |
-| `e3 task logs [repo] <ws.task> [-n <lines>] [--all] [--follow] [--execution <task>/<inputs>/<id>]` | The last 200 lines by default, of the latest attempt over the task's current inputs, or its latest run; `--follow` stays on that execution; `--execution`: one unit's log, as a split task's log names it (local). |
-| `e3 run <repo> <pkg[@ver].task> [inputs.beast2…] -o <out> [--force] [-v] [-j <n>] [--memory <size>]` | Run one task ad hoc. |
-| `e3 call <repo> <pkg[@ver].fn> [args…] [-o <out.beast2>] [-j <n>] [--memory <size>] [-v]` · `e3 call <repo> -w <ws> <fn> [args…]` | Call a function; each argument is an `.east` literal or a `.beast2`/`.json`/`.east` file, and with `-w` one written `@<keypath>` (`@.inputs.sales`) is that dataset of the workspace, pinned at its hash when the call starts. |
-| **Records** (`-w <ws>` required) | |
-| `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
-| `e3 seed <repo> <record> <file.beast2> -w <ws> [-j <n>] [--memory <size>]` | Commit a record's first state from a beast2 file, as its `$init`: for a record declared with no initial value, unassigned until then (local). The file is taken in as a `file` source is, never read whole; a record that holds a state is refused. |
-| `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target — a summary's totals added to as `±n`; local). |
-| `e3 reindex <repo> <record> -w <ws> [--index <name>] [-j <n>] [--memory <size>]` · `e3 compact <repo> <record> -w <ws>` | Rebuild indexes (local) · collapse the history to a `$compact` root, the state kept. |
-| **Development** | |
-| `e3 watch <src.ts> <repo> <ws> [--start] [--schema <p>] [--inputs <p>] [--abort-on-change] [--functions <manifest…>] [-j <n>] [--memory <size>]` | Redeploy on each change (and run, with `--start`). |
-| `e3 convert [input] [--from <f>] [--to <f>] [--type <spec>] [-o <out>]` | Convert between `.east`, `.json` and `.beast2`. |
-| `e3 completion install [--shell <s>]` · `uninstall` · `bash` \| `zsh` \| `fish` | Shell completion. |
-| **Servers** | |
-| `e3 auth login <server> [--no-browser]` · `logout <server>` · `status` · `token <server>` · `whoami [server]` | OAuth2 device flow; credentials per server (`token` prints one for curl). |
-
-**The budget.** `-j`/`--jobs <n>` is the cores — runner processes in flight, a
-task or a unit each — and `--memory <size>` (`8G`, `512M`) what they may reserve
-between them. The defaults are `E3_JOBS` and `E3_MEMORY`, else what e3 may use:
-its CPU affinity capped by a cgroup quota, and the cgroup's `memory.max` or
-physical memory, less a reserve. A unit of a split task reserves the largest
-peak a unit of its stage has reached in the run, so a stage runs its first unit
-alone and then fans out; on Linux and macOS a guard stops the newest unit when
-the runners together pass the budget, and runs it again once it fits. A deploy
-takes its `file` sources in `-j` at a time, their pieces' intake units running
-under the same budget. The commands that run East — `workspace deploy`,
-`dataset set --from-file`, `dataflow run`, `run`, `call`, `mutate`, `seed`,
-`reindex` and `watch` — take the flags for a local repository; against a server the CLI
-refuses them, since the server runs the work under its own budget
-(`e3-api-server -j`, `--memory`).
-
-**A deploy says how far it has got**, on stderr. Each `file` source prints a
-line once it is in: its size, time and rate, and how it was taken in
-(`unchanged, already in the store`, `taken in by east-c`, or `carried` for a
-value that is not a collection), with why the first time a runner fell back to
-another. A terminal also keeps a live line for the files in flight, with their
-pieces, the rate and the time left across them. Against a server, the deploy line
-says what the job is doing, such as migrating a record or building an index;
-the deliveries then upload a few at a time, each printing a line once it is in,
-and the live line says what the server's commit is doing with each in flight.
-While a deploy runs, its lock carries the same progress for `e3-ui` or any
-client to read (`workspaceLockStatus`). A workspace deployed for the first time
-has no status until its deploy ends, so the lock is the only place to see it.
-
-**`-v`** passes `-v` to the runners, which print to the task's logs, identically
-on every runtime, how each input was read — opened lazily, and what reading it
-came to, or decoded whole, and the resident memory that added — and a timing
-and peak-memory block. It never changes hashes or caching (add `--force` to see
-it for a cached task), and works against a server.
-**Where a task's time goes.** `EAST_PROFILE=1` in the environment of a local
-`e3 dataflow run` (or `run`, `call`, `mutate`) has the east-c and east-py
-runners profile each unit — the runners inherit e3's environment — and the
-task's log (`e3 task logs`) then ends with every East function and platform
-function it called, by self time, with its calls and where it was written and
-called: the python lines of a function `East.importFunction` linked in
-included. Each input read lazily is an entry too, named as `-v` names it
-(`input 1`): the time its reads spent decoding segments, a call each — keyed
-reads at random beyond the segments the runner keeps, or a scan repeated,
-which `-v`'s account counts — and not the reading function's own time.
-`EAST_PROFILE_INTERVAL=300` adds the profile so far every five minutes while a
-long task runs. east-node does not profile. Like `-v` it changes no hash: add
-`--force` for a cached task.
-**`E3_SCRATCH_DIR`** moves a local run's per-execution scratch directories
-(default `<repo>/tmp/scratch`, on the object store's disk; on tmpfs, outputs sit
-in memory until stored).
-
-**`repo gc`** keeps each workspace's last 10 runs (`--keep-runs`), every run
-from the last 7 days (`--keep-days`) and the run its current state came from,
-with every execution those runs used; every execution a workspace's current
-state is served from, so a re-run stays cached; every execution from the last 7
-days; and whatever is still running. An attempt recorded running that the local
-runner says cannot finish is recorded interrupted, and goes as any that ended.
-It then removes the objects nothing names, and staging files older than
-`--min-age` (60 s). `--dry-run` reports what would go.
-
-A package zip names the release of e3 that exported it, and an import refuses a
-zip a newer release exported, naming that release: import it with an e3 at least
-as new as the SDK that exported it. A dataset upload names the transfer protocol
-its e3 speaks, and a server refuses one that speaks another, naming both
-releases and which of them to upgrade.
-
-## Driving e3 from code
-
-**`@elaraai/e3-api-client`** talks to a server over HTTP. Every call is
-`(url, repo, …, options)`, `options` being
-`{ token: string | null, retry?, verbose?, fetch? }` (`e3 auth token <server>`
-prints a token); `fetch` is the `fetch` every request of the call goes
-through, the global one unless given — an in-page e3's `e3.fetch` (below).
-Dataset paths are `TreePath`s — an input definition's `.path`, for one. A
-request a workspace's lock refuses throws `ApiError` `workspace_locked`,
-naming the holder: its pid, boot id, command and when it took the lock.
-
-| Area | Functions |
-|---|---|
-| Repositories | `repoList(url, opts)`, `repoCreate(url, name, opts)`, `repoRemove(url, name, opts)`, `repoStatus(url, repo, opts)`, `repoRecord` (its release and upgrades), `repoGc(url, repo, gcRequest, opts)` — a job it polls; `repoGcStart` and `repoGcStatus` apart |
-| Packages | `packageList(url, repo, opts)`, `packageGet(url, repo, name, version, opts)`, `packageImport(url, repo, zipBytes, opts)`, `packageExport(url, repo, name, version, opts)` → zip bytes, `packageRemove` |
-| Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
-| Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, and several reads in one request (at most `DATASET_BATCH_MAX`, 10,000, each at one content hash, in a body of at most 16 MiB: a larger one is refused 413, `payload_too_large`): `datasetFindKeys(…, { queries, index?, hash? }, opts)` — key searches, `datasetGetRows(…, { keys, hash? }, opts)` — a Dict's rows by `.east` key, each with its position — or `datasetGetRows(…, { keysOf: { path, index, runs, hash }, hash? }, opts)`, by the keys the entries of some runs of another record's index name, which the server reads from the index so none is sent (a drawing's details for its window's cells), a stale `keysOf.hash` thrown as `RunsHashMismatchError` naming the record's content — and `datasetGetRuns(…, { runs, hash, index?, join? }, opts)` — the entries of several runs as one window, a rows or runs answer holding the first `count` the page budget reaches (by runs, of `runEntries`); and where a narrowing's rows sit in a record, from its index alone: `datasetPlace(…, { index, runs, offset?, limit?, hash }, opts)` → `{ segments, placed, runEntries, totalElements, hash }` — how many of the runs' entries each record segment holds (`{ segment, offset, count, rows }`), for the first `placed` of a window from `offset` (the next window from `offset + placed`), and `datasetWithin(…, { index, runs, keys, hash }, opts)` → `{ within, hash }` — whether each index entry key (`(ik=…, k=…)`) lies in one of the runs, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
-| Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
-| Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts)` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress? })` — a program over a dataset's pieces, as a job it polls, under the same rule (or `splitCallLaunch`, `splitCallStatus` and `splitCallExplain`) |
-| Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
-| From East | `Platform` and the `platform_*` functions (`platform_dataset_get`, `platform_dataflow_execute`, …): the same calls as platform functions, for an East program that drives a server |
-
-**`@elaraai/e3-core`** does the same on a local repository, given a storage
-backend (`new LocalStorage()`) and the repository's path; work that runs East
-takes a runner, `new LocalTaskRunner(repo)`.
-
-| Area | Functions |
-|---|---|
-| Repositories | `repoInit(path)`, `repoFind(startPath?)`, `repositoryOpen(storage, repo)` (checks the repository and applies the upgrades it owes), `repositoryUpgradeStep(storage, repo, { budgetMs, waitMs? })` → `{ owed }` — a host's job applies them so, a part per run under its time limit, each part taking a step up where the last stopped, `repoGc(storage, repo, { dryRun?, minAge?, keepRuns?, keepDays?, retention?, executionAlive? })` — holding the repository still, or, with `retention: { windowMs }`, beside running work: an object goes once unreachable for the window and not written or re-referenced since; given a runner's `executionAlive`, an attempt recorded running that cannot finish is recorded interrupted and pruned; `repoGcStep(storage, repo, step, { windowMs, pruneMs?, markMs?, concurrency?, executionAlive?, … })` runs that one step at a time, each returning the next (`GcStepType`), `pruneMs` and `markMs` bounding a prune step and a mark step for a host with a time limit |
-| Packages | `packageImport(storage, repo, zipPath)`, `packageExport(storage, repo, name, version, zipPath)`, `packageList`, `packageRemove` |
-| Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
-| Datasets | `workspaceGetDataset(storage, repo, ws, treePath)`, `workspaceSetDataset(storage, repo, ws, treePath, value, type)`, `datasetAdoptFile(storage, repo, ws, treePath, file, { runner, onProgress? })` → `{ hash, size, segments, rows, taken, runners? }`, `taken` being `known`, `carried` or `taken` (by the `runners` named); `deliveryAdoptStored(storage, repo, ws, { address, type }, sha256, { runner?, onProgress?, signal? })` — a delivery the store holds taken in as a stored value of the type, naming no dataset, for a record's first state |
-| Runs | `dataflowExecute(storage, repo, ws, options?)`; `LocalOrchestrator` to start, poll and cancel a run |
-| Records | `recordMutate(storage, runner, repo, ws, record, mutation, args, { actor })`, `recordHistory`, `recordDescribe`, `recordCompact`, `recordReindex`; `recordSystemCommit(storage, runner, repo, ws, record, { name, target, actor, args? })` — `$rollback` to a commit, `$restore` of a state at the record's migrations, and `$init` on a record declared with no initial value: its first state, from a stored value, with the whole chain applied |
-
-`@elaraai/e3-core/test` exports the contract suites another storage backend runs
-over itself.
-
-**`e3-api-server`** serves repositories: `--repos <dir>` (each subdirectory a
-repository) or `--repo <path>` (one, served as `default`), `-p`/`--port` (3000),
-`-H`/`--host` (localhost), `--cors`, `-j`/`--memory` (the server's budget, for
-every run and call it serves and every upload it takes in), and auth — `--oidc`, a built-in provider for `e3
-auth login` (`--token-expiry`, `--refresh-token-expiry`), or an external JWT
-issuer (`--auth-key`, `--auth-issuer`, `--auth-audience`). In code,
-`createServer({ reposDir | singleRepoPath, port, host, … })` starts one, and its
-route factories mount on another host.
-
-## Running e3 in a browser — `@elaraai/e3-web`
-
-e3-web runs e3 entirely in a page, with no server: a repository kept in the
-browser's own storage (records in IndexedDB, objects in OPFS, locks through
-Web Locks), every East program on a pool of Web Workers, and e3's whole API
-answered in the page by e3-api-server's own routes. An app is three scripts:
-
-```typescript
-// unit.worker.ts — where East programs run: east-web-std answers for east-node-std's
-// platform functions, and e3's own (e3-api-client's Platform) reach the e3 worker
-import { serveUnits } from '@elaraai/e3-web/units';
-serveUnits();
-```
-
-```typescript
-// e3.worker.ts — e3 itself: its storage, orchestrator, runner and API
-import { serveE3 } from '@elaraai/e3-web/worker';
-serveE3({
-  units: () => new Worker(new URL('./unit.worker.ts', import.meta.url), { type: 'module' }),
-});
-```
-
-```typescript
-// main.ts — the page: e3-api-client's calls, every one answered by the e3 worker
-import { createWebE3 } from '@elaraai/e3-web';
-import {
-  dataflowExecute, packageImport, repoCreate, repoList, workspaceCreate, workspaceDeploy,
-} from '@elaraai/e3-api-client';
-
-const e3 = await createWebE3(new Worker(new URL('./e3.worker.ts', import.meta.url), { type: 'module' }));
-const opts = { token: null, fetch: e3.fetch };
-
-// The first visit makes the repository; later visits find it where they left it.
-if (!(await repoList(e3.apiUrl, opts)).includes('default')) {
-  await repoCreate(e3.apiUrl, 'default', opts);
-  // The package zip e3.export wrote at build time, served beside the page
-  const zip = new Uint8Array(await (await fetch('/sales-planning-1.0.0.zip')).arrayBuffer());
-  const { name, version } = await packageImport(e3.apiUrl, 'default', zip, opts);
-  await workspaceCreate(e3.apiUrl, 'default', 'main', opts);
-  await workspaceDeploy(e3.apiUrl, 'default', 'main', `${name}@${version}`, opts);
-  await dataflowExecute(e3.apiUrl, 'default', 'main', {}, opts);
-}
-```
-
-`e3.apiUrl` is `https://e3-web.invalid`, an origin reserved so that a request
-sent without `e3.fetch` fails rather than reaching the network, and
-`e3.fetch`, the global `fetch`'s signature, is answered by the e3 worker. Both
-go to e3-api-client (`RequestOptions.fetch`) and to e3-ui-components'
-`<E3Provider>` (`E3Config.fetch`: the **e3-ui** skill). An app's own platform
-package is served beside the standard ones:
-`serveUnits({ platforms: { '@acme/pricing': PricingPlatform } })`.
-
-| `serveE3` option | Default | Effect |
-|---|---|---|
-| `units` | — | starts a unit worker: a Web Worker whose script calls `serveUnits()` |
-| `persist` | `true` | `false` keeps repositories in memory, gone with the page |
-| `name` | `'e3'` | the storage's name: e3 workers of one name share their repositories, in one tab or several |
-| `identify` | none | each request's caller, `{ sub, email?, roles }`, from the request; a request to a repository whose caller it does not identify is answered 401. None: the page's one caller may do everything |
-| `access` | `oneShotAccessByRoles()` with `identify`, else `any` | the grant a caller holds for a one-shot, a split call, or a function call naming its own runner |
-| `wholeIntakeLimit` | 256 MiB | the largest delivery one intake unit takes in whole |
-
-What works is the API, as the local server answers it: every API suite of
-e3-api-tests passes against it in Chromium, as an admin and, in its reader
-cases, as a reader. Paged reads and key search, function calls, one-shot and
-split calls, record mutations, deploys that migrate records or build indexes,
-uploads, package import and export, gc, and dataflow runs — tasks in parallel
-up to the pool's width (the cores), split tasks in pieces, re-run reactively.
-A repository outlives a reload, and two tabs share an origin's repositories as
-two processes share a directory. A closed tab's locks are free; a run it left
-is taken as one whose process died, the next run served what finished from
-the cache; and a job or an upload's commit it left is recorded failed, never
-run again.
-
-What a browser cannot do — each fails naming it, never a partial answer:
-
-- **Commands.** A custom task, or a task on the `custom` runtime, is recorded
-  `error`: "a browser runs no commands".
-- **Platform functions with no browser meaning.** east-web-std has no
-  FileSystem, Env or large-JSON reader, and east-node-io and the python
-  packages have no browser equivalent: a unit calling one fails naming the
-  function, and one listing a package its worker does not serve fails naming
-  the package.
-- **Segments on demand.** A unit's inputs are staged whole into its worker:
-  waiting for a segment needs a cross-origin isolated page, which GitHub Pages
-  cannot give. Split a large input into pieces with `e3.partition`, which
-  bounds a unit's memory anywhere; a delivery that cannot be cut into pieces
-  is refused above `wholeIntakeLimit`, naming the fix.
-- **A memory budget.** Memory is not measured: the pool is as wide as the
-  cores, there is no `--memory`, and no record names a peak.
-- **The CLI.** `e3` drives e3 over HTTP, which an in-page e3 does not serve.
-
-## A local repository
-
-```
-repo/
-├── repository.beast2  # the release that last wrote it, and the upgrades it has had
-├── repository-upgrade.beast2  # while an upgrade is under way: where its last part stopped
-├── metadata.beast2    # its name and status
-├── objects/           # content-addressed: values, segments, manifests, programs
-├── packages/          # package refs
-├── workspaces/        # each workspace's state, dataset refs, and its latest run's state and events
-├── dataflows/         # run records
-├── executions/        # execution attempts: status, owner, logs
-├── running/           # an index of the attempts recorded running, by task: what a status reads
-├── runs/              # an index of each task's runs, its units left out: what its history and logs read
-├── adoptions/         # the manifest each delivered file, or piece of one, became
-├── locks/             # locks, their holders, and how far each says it has got
-├── gc/                # gc beside running work: unreachable notes, a stepped run's parts
-├── envs/              # built execution environments
-└── tmp/               # scratch and staged uploads
-```
-
-An e3 opening a repository an older release wrote applies the upgrades it has
-not had, in place; one a newer e3 upgraded is refused, naming that release.
+## Caching
 
 **Caching.** An execution is keyed by its task — program, runner and
 environment — and its inputs' hashes, so a task re-runs only when one of those
 changes (`--force` bypasses the cache, and `--force-task <task>` for the tasks
 it names: a task whose code e3 cannot see changed, such as a platform
-function's, or one reading data from outside e3). A change to an input re-runs its
+function's, or one reading data from outside e3). A task's program holds the
+source locations of its body and of every function it embeds, an imported
+python or node function's too, since its errors, a caught error's `stack` and
+the closures it returns name them: an edit that moves one of those lines — a
+comment or an import added above a function — re-runs the task. One that leaves
+them where they were — another function, the end of a module, another package —
+re-runs only the tasks whose environment it is in, their runner's `{ custom }`
+packages. A change to an input re-runs its
 consumers, even during a run. A split task is cached a unit at a time: each piece
 and each merge is its own execution, and pieces are cut by content, so an append
 or an insertion re-runs only the pieces it reaches and the merges above them.
 
+## Reference files
+
+Read the one a task needs; each holds its subject whole.
+
+| File | Holds |
+|---|---|
+| [reference/record-indexes.md](reference/record-indexes.md) | Secondary indexes, interval and area indexes, series and summary indexes, nested series indexes |
+| [reference/record-migrations.md](reference/record-migrations.md) | `migration.value` / `rows` / `rekey`, and how a record's type changes |
+| [reference/cli.md](reference/cli.md) | Every `e3` command: repositories, packages, workspaces, datasets, tasks, runs, calls, records, watch, convert, auth, gc |
+| [reference/from-code.md](reference/from-code.md) | `@elaraai/e3-api-client`, `@elaraai/e3-core`, `e3-api-server` |
+| [reference/browser.md](reference/browser.md) | `@elaraai/e3-web`: serveUnits, serveE3, createWebE3, and what a browser cannot do |
+| [reference/rack.md](reference/rack.md) | `e3 rack enroll` / `policy`, routing a run to a rack, `@elaraai/e3-rack` |
+| [reference/repository.md](reference/repository.md) | A local repository's layout, and upgrades |
+
 ## Related skills
 
 - **east** — the language task bodies are written in.