@elaraai/east-codex-plugin 1.0.107 → 1.0.109

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@elaraai/east-codex-plugin",
3
- "version": "1.0.107",
3
+ "version": "1.0.109",
4
4
  "description": "East programming language ecosystem - a statically typed, expression-based language for business logic, data processing, UI definitions, and durable execution",
5
5
  "license": "AGPL-3.0-or-later",
6
6
  "type": "module",
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: e3
3
- description: "East Execution Engine (e3): durable, content-addressed dataflow for East programs. Use when (1) authoring a package with @elaraai/e3 — e3.input, e3.task, e3.streamTask with e3.output and e3.partition (bounded memory, work split over an input), e3.customTask, e3.function, e3.package, e3.export, East.importFunction; (2) records — audited state written only through mutations (e3.record with mutation.reduce/edit/patch), their indexes (index.on/key/keys, interval, area, series, summary, nested, entries) and migrations (migration.value/rows/rekey); (3) the e3 CLI — repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, watch, convert, auth, and the cores and memory a run may use; (4) driving e3 from code (@elaraai/e3-api-client, @elaraai/e3-core, e3-api-server); (5) caching, reactive re-runs and garbage collection; (6) delegating task bodies to a rack (e3 rack); (7) e3 in a browser with @elaraai/e3-web. UI tasks are e3-ui's ui()."
3
+ description: "East Execution Engine (e3): durable, content-addressed dataflow for East programs. Use when (1) authoring a package with @elaraai/e3 — e3.input, e3.task, e3.streamTask with e3.output and e3.partition (bounded memory, work split over an input), e3.customTask, e3.function, e3.package, e3.export, East.importFunction; (2) records — audited state written only through mutations (e3.record with mutation.reduce/edit/patch), their indexes (index.on/key/keys, interval, area, series, summary, nested, entries; e3.index reads one in a task or function) and migrations (migration.value/rows/rekey); (3) the e3 CLI — repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, watch, convert, auth, and the cores and memory a run may use; (4) driving e3 from code (@elaraai/e3-api-client, @elaraai/e3-core, e3-api-server); (5) caching, reactive re-runs and garbage collection; (6) delegating task bodies to a rack (e3 rack); (7) e3 in a browser with @elaraai/e3-web. UI tasks are e3-ui's ui()."
4
4
  ---
5
5
 
6
6
  # East Execution Engine (e3)
@@ -70,7 +70,8 @@ What do you need?
70
70
  │ │ └─ Split over an input, in pieces → wrap that input: e3.partition(dataset, { by? })
71
71
  │ ├─ A shell command → e3.customTask(name, [inputs], outputType, command)
72
72
  │ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name; a call through a workspace is kept for its lifetime
73
- │ │ └─ It reads workspace data too → e3.function(name, { inputs, args, output, runner? }, fn) — inputs read at each call
73
+ │ │ ├─ It reads workspace data too → e3.function(name, { inputs, args, output, runner? }, fn) — inputs read at each call
74
+ │ │ └─ Over a page's unsaved edits → add drafts: [record, …] — a call lays mutation calls over those records, and saves nothing
74
75
  │ ├─ Audited state → e3.record(name, type, { initial?, indexes?, mutations?, migrations? }), declared whole
75
76
  │ │ ├─ Its writes → mutations: (mutation) => ({ name: mutation.reduce | edit | patch })
76
77
  │ │ ├─ An order to page in → indexes: (index) => ({ name: index.on([...fields]) | index.key(fn) | index.keys(fn) }) (reference/record-indexes.md)
@@ -78,13 +79,16 @@ What do you need?
78
79
  │ │ ├─ Read a block of keys over time tiles, or their totals → index.series | index.summary
79
80
  │ │ ├─ Read what rows nest (jobs, a value a day, child rows, trees) a tile at a time, with totals and subtotals
80
81
  │ │ │ → index.nested({ collections, tile, summary? })
81
- │ │ └─ Read a few keys of a Dict field across every row (each row's figures by category) → index.entries(field)
82
+ │ │ ├─ Read a few keys of a Dict field across every row (each row's figures by category) → index.entries(field)
83
+ │ │ └─ Read an index inside a task or a function → e3.index(record.indexes.<name>) among its inputs: the entries, with its kind's readers — of, between, overlapping, events, totals, within, a nested index's items, subtree, subtotals — each a read: each($, body) | collect($)
82
84
  │ ├─ A record's type changes → migrations: (migration) => [migration.value | rows | rekey(name, fn), …], in order (reference/record-migrations.md)
83
85
  │ ├─ A UI task → ui(name, [inputs], fn) from @elaraai/e3-ui (the e3-ui skill)
84
86
  │ ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
85
87
  │ └─ Bundle and export → e3.package(name, version, ...items); e3.export(pkg, zipPath)
86
88
  ├─ Operate it from the shell → the e3 CLI (reference/cli.md): deploy, run, read and write datasets, mutate records, gc
87
89
  ├─ Delegate local task bodies to a rack → e3 rack enroll + policy; dataflow run --rack, watch --start --rack (reference/rack.md)
90
+ │ ├─ A split task's pieces fill the rack → booked by what each reserves, capped half as much again; --memory-limit bounds the cap, --gpu asks for the GPU
91
+ │ └─ A task with its own python/node environment → nothing new in the package: its layers are built for the rack's platform
88
92
  └─ Drive it from code
89
93
  ├─ A server, over HTTP → @elaraai/e3-api-client (reference/from-code.md)
90
94
  ├─ A local repository → @elaraai/e3-core
@@ -236,6 +240,10 @@ const shout = e3.task('shout', [greet.output],
236
240
  derivation: `{ python: { project } }`, `{ node: { project } }`,
237
241
  `{ tools: { files: [...] } }` (prebuilt binaries on PATH, such as a C runner)
238
242
  or `{ image: { digest: 'repo@sha256:<64 hex>' } }` (cloud; a tag is refused).
243
+ A task with a `python` or `node` environment also runs on a rack
244
+ (`e3 dataflow run --rack`): the launching machine installs the environment
245
+ for the rack's platform and sends it as layers, with no container runtime;
246
+ `tools` and `image` environments stay local there (reference/rack.md).
239
247
 
240
248
  ```python
241
249
  # packages/pricing/src/pricing/__init__.py — the member's root module
@@ -385,8 +393,8 @@ const change = e3.streamTask('change', {
385
393
  | Definition | What it is |
386
394
  |---|---|
387
395
  | `e3.customTask(name, inputs, outputType, command, { environment? })` | A task that runs a bash script: `command($, inputPaths, outputPath)` builds it from the staged beast2 input files, and it writes the output as a beast2 file. |
388
- | `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API, `Func.bind`) and its result returned inline: not in the dataflow. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A call through a workspace is the server's: it outlives the request that starts it — the client starts it and polls it to its end — and runs until its timeout, the server's ceiling (ten minutes locally) unless it asks for less; it can be cancelled; and the server keeps it for its lifetime, an hour for a success or a cancel and a week for any other end: who made it, its arguments, the versions of the inputs it read, its value and its logs (reference/from-code.md). A call through the package runs within one request and keeps nothing. A result is at most 1 MiB, so a large one belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
389
- | `e3.function(name, { inputs, args, output, runner?, environment?, async? }, fn)` | A function that reads the workspace as well: `inputs` (an input, a task's output or a record, as a task takes them) are read as stored when each call launches, pinned at their hashes and named in the result's `inputs`; the caller passes `args` alone. The body takes the inputs, then the arguments. `e3.package` collects the inputs and what they depend on; nothing reruns when one changes. The inputs are read one at a time and the call waits for no run, so a call during a run may read one input before the run rewrites it and another after: the hashes in `inputs` say which. Called through a workspace only (`e3 call -w`); an unassigned input is `invalid`, naming it and why. ❗ Refused: an `e3.partition` input, and the `custom` runtime with inputs. `FunctionSpec` names the spec's type. |
396
+ | `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API, `Func.bind`) and its result returned inline: not in the dataflow. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A call through a workspace is the server's: it outlives the request that starts it — the client starts it and polls it to its end — and runs until its timeout, the server's ceiling (ten minutes locally) unless it asks for less; it can be cancelled; and the server keeps it for its lifetime, an hour for a success or a cancel and a week for any other end: who made it, its arguments and drafts, the versions of the inputs it read, its value and its logs (reference/from-code.md). A call through the package runs within one request and keeps nothing. A result is at most 1 MiB, so a large one belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
397
+ | `e3.function(name, { inputs, drafts?, args, output, runner?, environment?, async? }, fn)` | A function that reads the workspace as well: `inputs` (an input, a task's output, a record or an index of one, `e3.index(record.indexes.<name>)`, as a task takes them) are read as stored when each call launches, pinned at their hashes and named in the result's `inputs`; the caller passes `args` alone. The body takes the inputs, then the arguments. `e3.package` collects the inputs and what they depend on; nothing reruns when one changes. The inputs are read one at a time and the call waits for no run, so a call during a run may read one input before the run rewrites it and another after: the hashes in `inputs` say which. A record and its indexes are read at one state of it. Called through a workspace only (`e3 call -w`); an unassigned input is `invalid`, naming it and why. `drafts` lists the records, each read by one of `inputs` (itself, or an index of it), that a call may lay **unsaved edits** over: a draft is a mutation call that is not committed, `{ record, mutation, args }` (`e3 call -w <ws> <fn> [args…] --draft <record>.<mutation> [args…]`, reference/cli.md). A call's drafts run in the order passed, as a Save runs them, each over the state the one before it left, and the body reads each drafted record, and its indexes, at the state that leaves; a dataset argument naming a drafted record is read as saved, at the state the drafts were laid over. Nothing is saved: no commit, no record moved, no task re-run — the record and what reads it stay as they were for every reader. The call keeps its drafts as it keeps its arguments, for whoever may read the call, and each draft's mutation runs as a Save's does, an execution recorded and cached. The result's `inputs` name the drafted state's hash, and its `drafts` the commit each drafted record stood at. A draft of a record the function does not list, or one the record does not take (an unknown mutation, arguments of the wrong number or type), is `invalid`, naming the draft; one the record refuses ends the call as its Save would end, the body not run: `failed` or `timed_out` (the first line of stderr says which draft, or that the drafts were all laid as the timeout ended), or `conflict` when the record changed under the edit. Listing a record is the author's word that the body is fit to run over edits nobody saved, so a function that sends a message or writes elsewhere lists none. ❗ Refused: an `e3.partition` input, the `custom` runtime with inputs, and in `drafts` what is no record or a record no input reads. `FunctionSpec` names the spec's type. |
390
398
  | `e3.package(name, version, ...items)` | Bundles the items and everything they depend on — pass the last tasks, records and functions; a package passed in contributes its contents. The name and version must be valid file names. |
391
399
  | `e3.export(pkg, zipPath, { functions?, cores? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself, `cores` packages at once, the CPUs by default; `functions` lists manifests built elsewhere). |
392
400
 
@@ -485,7 +493,7 @@ cheap. A view reads every field of the rows it shows, so keep a bulky payload it
485
493
  does not display in a second record keyed the same way.
486
494
 
487
495
  A record's indexes (`index.on` / `key` / `keys`, `interval`, `area`,
488
- `series`, `summary`, `nested`, `entries`) are in
496
+ `series`, `summary`, `nested`, `entries`, and `e3.index` to read one) are in
489
497
  [reference/record-indexes.md](reference/record-indexes.md), and its migrations
490
498
  in [reference/record-migrations.md](reference/record-migrations.md).
491
499
 
@@ -513,12 +521,12 @@ Read the one a task needs; each holds its subject whole.
513
521
 
514
522
  | File | Holds |
515
523
  |---|---|
516
- | [reference/record-indexes.md](reference/record-indexes.md) | Secondary indexes, interval and area indexes, series and summary indexes, nested series indexes, entries indexes |
524
+ | [reference/record-indexes.md](reference/record-indexes.md) | Secondary indexes, interval and area indexes, series and summary indexes, nested series indexes, entries indexes, and reading an index inside a task or a function (`e3.index`) |
517
525
  | [reference/record-migrations.md](reference/record-migrations.md) | `migration.value` / `rows` / `rekey`, and how a record's type changes |
518
526
  | [reference/cli.md](reference/cli.md) | Every `e3` command: repositories, packages, workspaces, datasets, tasks, runs, calls, records, watch, convert, auth, gc |
519
527
  | [reference/from-code.md](reference/from-code.md) | `@elaraai/e3-api-client`, `@elaraai/e3-core`, `e3-api-server` |
520
528
  | [reference/browser.md](reference/browser.md) | `@elaraai/e3-web`: serveUnits, serveE3, createWebE3, and what a browser cannot do |
521
- | [reference/rack.md](reference/rack.md) | `e3 rack enroll` / `policy`, routing a run to a rack, `@elaraai/e3-rack` |
529
+ | [reference/rack.md](reference/rack.md) | `e3 rack enroll` / `policy`, routing a run to a rack, how a rack is filled and what a run prints, tasks with an environment on a rack, `@elaraai/e3-rack` |
522
530
  | [reference/repository.md](reference/repository.md) | A local repository's layout, and upgrades |
523
531
 
524
532
  ## Related skills
@@ -35,6 +35,7 @@ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
35
35
  | `e3 task logs [repo] <ws.task> [-n <lines>] [--all] [--follow] [--execution <task>/<inputs>/<id>]` | The last 200 lines by default, of the latest attempt over the task's current inputs, or its latest run; `--follow` stays on that execution; `--execution`: one unit's log, as a split task's log names it (local). |
36
36
  | `e3 run <repo> <pkg[@ver].task> [inputs.beast2…] -o <out> [--force] [-v] [-j <n>] [--memory <size>]` | Run one task ad hoc. |
37
37
  | `e3 call <repo> <pkg[@ver].fn> [args…] [-o <out.beast2>] [-j <n>] [--memory <size>] [-v]` · `e3 call <repo> -w <ws> <fn> [args…]` | Call a function; each argument is an `.east` literal or a `.beast2`/`.json`/`.east` file, and with `-w` one written `@<keypath>` (`@.inputs.sales`) is that dataset of the workspace, pinned at its hash when the call starts. A function that declares `inputs` reads them from the workspace itself, so it takes its arguments alone and is called with `-w`; without it, it is refused, naming its inputs. The value goes to stdout, and the function's stdout and stderr to stderr. Through a server's workspace the call is the server's, kept for its lifetime: it runs as long as its timeout allows, a call longer than a second is named (`Call <id> runs on`), and Ctrl-C asks the server to stop it (exit 130). |
38
+ | `… -w <ws> <fn> [args…] --draft <record>.<mutation> [args…]` | Lay an unsaved edit over a record the function lists in `drafts`, and save nothing: the mutation runs as `e3 mutate` would run it, its arguments read the same way, and the function reads the record at the state that leaves. Repeatable, laid in the order written, each over the state the one before it left. `--draft` reads every word up to the next option, so it is written after the call's own arguments, or the drafts are ended with `--`; a draft ends where its mutation's arguments do, and a word that starts with a dash is written `--draft=<word>`. Each draft is read against its mutation before anything runs: one that names no record or mutation of the workspace, or has too few arguments or one of the wrong type, is refused, named by its place (`Draft 2 (jobs.move): …`). A draft the record refuses ends the call before the function runs, and exits 1: `Draft jobs.patch (2 of 3) failed with exit code: 1; the function did not run`, `… had not been laid when the call timed out after <n>ms; …`, `The call's drafts were laid, and its timeout of <n>ms ended before the function ran: …` when laying them took all of the call's time, or, when the record changed under the edit, `Draft jobs.patch (2 of 3) conflicted: … — the edit no longer matches the record; re-read it and call again`. |
38
39
  | **Records** (`-w <ws>` required) | |
39
40
  | `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
40
41
  | `e3 seed <repo> <record> <file.beast2> -w <ws> [-j <n>] [--memory <size>]` | Commit a record's first state from a beast2 file, as its `$init`: for a record declared with no initial value, unassigned until then (local). The file is taken in as a `file` source is, never read whole; a record that holds a state is refused. |
@@ -18,8 +18,8 @@ naming the holder: its pid, boot id, command and when it took the lock.
18
18
  | Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceBundle(url, repo, ws, opts)` → the workspace as it is, as a bundle's bytes — its package, every dataset's ref, the executions its tasks' outputs come from, and every object those reach — and `workspaceBundleImport(url, repo, ws, bundle, opts)` → the workspace made from one under a name no workspace has, in another repository or another e3 (`workspace_exists`, `package_exists`, `bundle_invalid`), its tasks up to date with no run; a host mounts these routes where a bundle fits in a request: the local server, and e3-web, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
19
19
  | Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, and several reads in one request (at most `DATASET_BATCH_MAX`, 10,000, each at one content hash, in a body of at most 16 MiB: a larger one is refused 413, `payload_too_large`): `datasetFindKeys(…, { queries, index?, hash? }, opts)` — key searches, `datasetGetRows(…, { keys, hash? }, opts)` — a Dict's rows by `.east` key, each with its position — or `datasetGetRows(…, { keysOf: { path, index, runs, hash }, hash? }, opts)`, by the keys the entries of some runs of another record's index name, which the server reads from the index so none is sent (a drawing's details for its window's cells), a stale `keysOf.hash` thrown as `RunsHashMismatchError` naming the record's content — and `datasetGetRuns(…, { runs, hash, index?, join? }, opts)` — the entries of several runs as one window, a rows or runs answer holding the first `count` the page budget reaches (by runs, of `runEntries`); and where a narrowing's rows sit in a record, from its index alone: `datasetPlace(…, { index, runs, offset?, limit?, hash }, opts)` → `{ segments, placed, runEntries, totalElements, hash }` — how many of the runs' entries each record segment holds (`{ segment, offset, count, rows }`), for the first `placed` of a window from `offset` (the next window from `offset + placed`), and `datasetWithin(…, { index, runs, keys, hash }, opts)` → `{ within, hash }` — whether each index entry key (`(ik=…, k=…)`) lies in one of the runs, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
20
20
  | Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
21
- | Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; within one request, keeping nothing; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts, { signal?, onStarted?, startWaitMs? })` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`), within one request, keeping nothing; `oneShotCall(url, repo, ws, request, opts, { signal?, onStarted?, startWaitMs? })` — the same, under the same rule, started as a kept call through the workspace (below), and run within one request on a server that keeps none (`oneShotStart` its start apart); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress?, signal? })` → `{ result, output, report }` — a program over a dataset's pieces, as a job it polls, under the same rule; `output` is `array`, `set`, `dict { merge }`, `fold { combine, zero }`, or `first { limit }` — an `array`'s first `limit` elements in input order, its pieces run in order and stopped once they hold them; `then`, the final function, takes the assembled output and then the arguments not partitioned, in order, and one of another arity is `invalid`, and how it ran: its pieces, the units it ran and the cache served, and its peak; a value over the call's result limit is `too_large`, naming its stored `output`, which `collectionOpen(url, repo, output, opts)` → `{ type, totalElements, page({ offset, limit }) }` reads a window at a time, fetching only the segments each window overlaps; aborting `signal` cancels the job, whose units the server stops (or `splitCallLaunch`, `splitCallStatus`, `splitCallCancel` — `cancelled`, or the status the job had already ended with — and `splitCallExplain`) |
22
- | Calls through a workspace | A call through a workspace is the server's, and outlives the request that starts it: `workspaceFunctionCall` starts it, waits up to the server's window (20 s; `startWaitMs` less), and polls a call still running to its end, each poll held until it ends or a while passes, however long its timeout lets it run — the server's ceiling, ten minutes locally, unless its `limits` ask for less; aborting `signal` asks the server to stop it, and `onStarted(id)` is told the id of one that runs on; a server that keeps no calls is called through `workspaceFunctionCallSync`, the sync route, which a caller may name itself. Apart: `workspaceFunctionStart(…, opts, { waitMs? })` → `{ id, status }` (`?wait=0` answers `running` at once), `workspaceFunctionCallStatus(url, repo, ws, id, opts, { waitMs?, signal? })` → `running`, `completed` with the result, `cancelled` naming who asked, or `lost`, `workspaceFunctionCallCancel(url, repo, ws, id, opts)`. The server keeps each call for its lifetime, an hour for a success or a cancel and a week for any other end: `workspaceCallList(url, repo, ws, opts, { function?, state?, before?, limit? })` — a page of them, the latest first, its `next` the next page's `before` (`FUNCTION_CALLS_PAGE_DEFAULT` 50, `…_MAX` 200), and `workspaceCallGet(url, repo, ws, id, opts)` — one whole: its function and package (`none` for a one-shot), its arguments (values by the objects they were stored as, datasets by the hashes they were pinned at), the inputs it read, its limits, its value's object, its log tails and when it is forgotten (`expiresAt`); `objectGet` reads a value by its hash. A one-shot started as a call (`oneShotCall`) is kept, polled, cancelled and listed as a function's call is, its program the caller's, under its one-shot grant. A workspace keeps no call that another made: `internal`, "keeps no call" |
21
+ | Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits, drafts }, opts)` — each argument `variant('value', beast2Bytes)`; within one request, keeping nothing, and reading no workspace, so its `drafts` are `[]` (one passed is `invalid`); the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts, { signal?, onStarted?, startWaitMs? })` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`, each `{ path, index, hash }`: `index` names the index of the record at `path` that an input read, `none` for a dataset itself, and every input of one record names the one hash it was pinned at), and the request's `drafts` unsaved edits laid over the records the function reads (below); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`), within one request, keeping nothing; `oneShotCall(url, repo, ws, request, opts, { signal?, onStarted?, startWaitMs? })` — the same, under the same rule, started as a kept call through the workspace (below), and run within one request on a server that keeps none (`oneShotStart` its start apart); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress?, signal? })` → `{ result, output, report }` — a program over a dataset's pieces, as a job it polls, under the same rule; `output` is `array`, `set`, `dict { merge }`, `fold { combine, zero }`, or `first { limit }` — an `array`'s first `limit` elements in input order, its pieces run in order and stopped once they hold them; `then`, the final function, takes the assembled output and then the arguments not partitioned, in order, and one of another arity is `invalid`, and how it ran: its pieces, the units it ran and the cache served, and its peak; a value over the call's result limit is `too_large`, naming its stored `output`, which `collectionOpen(url, repo, output, opts)` → `{ type, totalElements, page({ offset, limit }) }` reads a window at a time, fetching only the segments each window overlaps; aborting `signal` cancels the job, whose units the server stops (or `splitCallLaunch`, `splitCallStatus`, `splitCallCancel` — `cancelled`, or the status the job had already ended with — and `splitCallExplain`) |
22
+ | Calls through a workspace | A call through a workspace is the server's, and outlives the request that starts it: `workspaceFunctionCall` starts it, waits up to the server's window (20 s; `startWaitMs` less), and polls a call still running to its end, each poll held until it ends or a while passes, however long its timeout lets it run — the server's ceiling, ten minutes locally, unless its `limits` ask for less; aborting `signal` asks the server to stop it, and `onStarted(id)` is told the id of one that runs on; a server that keeps no calls is called through `workspaceFunctionCallSync`, the sync route, which a caller may name itself. Apart: `workspaceFunctionStart(…, opts, { waitMs? })` → `{ id, status }` (`?wait=0` answers `running` at once), `workspaceFunctionCallStatus(url, repo, ws, id, opts, { waitMs?, signal? })` → `running`, `completed` with the result, `cancelled` naming who asked, or `lost`, `workspaceFunctionCallCancel(url, repo, ws, id, opts)`. The server keeps each call for its lifetime, an hour for a success or a cancel and a week for any other end: `workspaceCallList(url, repo, ws, opts, { function?, state?, before?, limit? })` — a page of them, the latest first, its `next` the next page's `before` (`FUNCTION_CALLS_PAGE_DEFAULT` 50, `…_MAX` 200), and `workspaceCallGet(url, repo, ws, id, opts)` — one whole: its function and package (`none` for a one-shot), its arguments (values by the objects they were stored as, datasets by the hashes they were pinned at), its drafts (each `{ record, mutation, args }`: the mutation's arguments by the object they were stored as, `none` for a mutation that takes none; the commit each drafted record stood at is its result's `drafts`), the inputs it read (a drafted record's, and an index of it, at the drafted state, once its drafts are laid), its limits, its value's object, its log tails and when it is forgotten (`expiresAt`); `objectGet` reads a value by its hash. A one-shot started as a call (`oneShotCall`) is kept, polled, cancelled and listed as a function's call is, its program the caller's, under its one-shot grant. A workspace keeps no call that another made: `internal`, "keeps no call". **Drafts**: a function's call may pass `drafts`, unsaved edits of records the function lists (its signature's `drafts`), each `{ record, mutation, args }` — a mutation call's beast2 arguments, as `workspaceRecordMutate` takes them — and `[]` for none. The server lays them in the order given, each by the mutation's own program over the state the one before it left, and the function reads each drafted record, and its indexes, at the state that leaves, and a dataset argument naming one as saved, at the state the drafts were laid over; nothing is saved — the record and what reads it stay as they were for every reader — while the call keeps its drafts, for whoever may read it, and each draft's mutation runs as a Save's does, an execution recorded and cached. The result's `inputs` name the drafted state's hash for such a record, and its `drafts` (`{ record, commit }`) the commit each stood at, so a caller that holds another commit knows its edits were laid over newer saves. A draft found wrong — a record the function does not list, an unknown mutation, arguments of the wrong number or type — is an `invalid` result naming the draft, and nothing runs; one whose mutation fails or outlasts the call's time ends the call `failed` or `timed_out` before the function runs, the result's `draftEnd` naming it, which `functionDraftEnded(result)` reads as `{ record, mutation, place, of, how }` (`place` and `of` Integers, `how` `.failed` or `.timed_out`), or `null`, and the first line of `stderr` saying so for people (`draft jobs.patch (2 of 3) failed`); a call whose timeout ends once its drafts are all laid ends `timed_out`, its `draftEnd` saying they were laid, which `functionDraftsOutlasted(result)` reads, its first line `drafts laid (3 of 3), and the call's timeout ended before its function ran`. Read the end from `draftEnd` through these, never from the line, which a function's own stderr may begin the same way. One that no longer fits the record, which changed under the edit, is the outcome `conflict` (`{ record, mutation, place, of, message }`, the draft named by its place among the call's), the kept call's state `conflict`. A draft is passed only where its mutation may be committed: one the server refuses throws `ApiError` `permission_denied`, its `path` `draft <record>.<mutation>`, and nothing is started |
23
23
  | Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
24
24
  | From East | `Platform` and the `platform_*` functions (`platform_dataset_get`, `platform_dataflow_execute`, …): the same calls as platform functions, for an East program that drives a server |
25
25
 
@@ -7,13 +7,13 @@ Part of the `e3` skill.
7
7
  | Signature | Description | Example |
8
8
  | --- | --- | --- |
9
9
  | `e3 rack enroll --listen <host:port> [--allow <ip,...>] [--advertise-url <origin>] [--label <label>] [--no-wait]` | Starts the user-wide hub and prints the source origin, the hub's certificate SHA-256, the matching release and a single-use token for the companion's hidden prompts. Waits for a new registration, not activation. Behind an SSH reverse tunnel, listen on the loopback and advertise the relay's origin. | `e3 rack enroll --listen 127.0.0.1:7331 --allow 127.0.0.1 --advertise-url https://relay.example:7401` |
10
- | `e3 rack policy [repo] set <glob> rack\|local [--size <size>] [--timeout <min>] [--allow-host-access]` | First whole-task-name match wins; `*` and `?` are wildcards. Size: serverless, small, medium, large, xlarge. | `e3 rack policy . set 'train_*' rack --size large` |
10
+ | `e3 rack policy [repo] set <glob> rack\|local [--size <size>] [--memory-limit <size>] [--gpu] [--timeout <min>] [--allow-host-access]` | First whole-task-name match wins; `*` and `?` are wildcards; setting a rule again replaces it whole. Size: serverless (2 cores, 2 GiB), small (2, 4 GiB), medium (4, 8 GiB), large (8, 16 GiB), xlarge (16, 32 GiB). `--memory-limit 8G` is a ceiling on one piece's cap, never a request; `--gpu` asks for a rack's GPU. | `e3 rack policy . set 'train_*' rack --size large` |
11
11
  | `e3 rack policy [repo] enable\|disable\|mode <opt-in\|auto>` | Enable/disable saved opt-in; `mode auto` selects eligible unmatched tasks. Defaults: disabled, opt-in. | `e3 rack policy . enable` |
12
12
  | `e3 rack policy [repo] defaults [--no-spill] [--claim-timeout <sec>] [--retry-task-failures-locally]` | Defaults: large, 1440-minute timeout, spill on, 30-second claim recheck, no task-failure retry. | `e3 rack policy . defaults --no-spill` |
13
13
  | `e3 rack policy [repo] show\|unset <glob>\|reset` | Inspect rules, remove an exact pattern, or delete the saved policy. | `e3 rack policy . show` |
14
- | `e3 dataflow run <repo> <ws> --rack\|--no-rack\|--rack-only` | Select rack routing for this run; rack-only disables capacity spill but keeps infrastructure fallback. ❗ Refused for remote URLs. | `e3 dataflow run . dev --rack` |
14
+ | `e3 dataflow run <repo> <ws> --rack\|--no-rack\|--rack-only` | Select rack routing for this run; rack-only disables capacity spill but keeps infrastructure fallback: work waits two minutes for an unhealthy rack to come back, and is leased once more after a failed transfer, before it runs locally. ❗ Refused for remote URLs. | `e3 dataflow run . dev --rack` |
15
15
  | `e3 watch <source> <repo> <ws> --start --rack` | Reuses a rack session across watch runs and refreshes capacity; a second Ctrl-C stops the watch at once. ❗ `--rack` and `--rack-only` require `--start`. | `e3 watch src/main.ts . dev --start --rack` |
16
- | `e3 rack status\|list\|remove <id-or-label>\|approve <id-or-label>\|stop` | Inspect fleet, revoke a rack, admit the boot a quarantined rack last reported, or stop the shared hub. Status never starts an absent hub. | `e3 rack status` |
16
+ | `e3 rack status\|list\|remove <id-or-label>\|approve <id-or-label>\|stop` | Inspect fleet, revoke a rack, admit the boot a quarantined rack last reported, or stop the shared hub. Status never starts an absent hub, and lists how its last leases ended, each with its reason. | `e3 rack status` |
17
17
  | `e3 rack config [--listen <host:port>] [--allow <ip,...>] [--idle-exit <min>] [--advertise-url <origin>] [--tls\|--no-tls] [--require-approval\|--no-require-approval]` | Configure the shared listener (off by default; HTTPS with the hub's own certificate unless `--no-tls`); every rack and data URL is built on `--advertise-url` when set; zero idle-exit disables automatic shutdown; `--require-approval` quarantines a rack after each reboot until `e3 rack approve` admits it (enrollment admits the first boot). ❗ Plain HTTP only on the loopback or a private address, with `--allow`. | `e3 rack config --listen 10.0.0.5:7331` |
18
18
 
19
19
  The local process owns graph planning, repository and cache. After a cache
@@ -21,10 +21,60 @@ miss, eligible plain task, map and merge bodies can execute on the rack; input
21
21
  closures and every output object transfer by hash. The session verifies the
22
22
  whole output graph before success. Placement never changes task/cache hashes.
23
23
  Local bodies and fallback share the original cores/memory budget; rack bodies
24
- use remote capacity. Calls, mutations and intake stay local. Stock runner
25
- configurations qualify; commands, custom environments/platforms and unscannable
26
- IR stay local. Host file/environment/network access stays local unless a rule
27
- deliberately grants `--allow-host-access` and its resources exist on the rack.
24
+ use remote capacity: each lease asks its cores, what it reserves and its cap
25
+ (a split task's piece one core, reserving what it was measured to need, capped
26
+ half as much again; a task its size's cores, at most the most a rack has,
27
+ reserving its cap, its `--memory-limit`, else its size's memory), a rack is given only what its free
28
+ resources hold by what each lease reserves, and only work whose cap all its
29
+ memory holds, and a run is as much wider as the racks have cores. `e3 rack list` shows each rack's resources and what is free. A run prints `[RACK] <task> → <rack>` when a rack claims a
30
+ task, `[RACK] <task> input transfer retried (2 of 5): <cause>` for a transfer
31
+ given another try, `[LOCAL] <task> (<reason>)` for a rack's work this machine
32
+ runs after all (`spill`, `no-rack-for-tier:<tier>`, or the rack's reason, such
33
+ as `input-transfer-failed: <cause>` or `guest-crashed: <detail>`, or
34
+ `memory-limit-exceeded: needs 3 GB, limit 2 GB` for work whose cap passes its
35
+ rule's limit or every rack's memory),
36
+ `[RACK] <task> waiting: needs 1 core and 4 GB, <rack> has 0 cores and 1 GB
37
+ free` for work queued with no rack having room for what it reserves,
38
+ `[RACK] <task> outgrew its 3 GB cap on <rack>: runs again under 4.5 GB` for a
39
+ split task's piece its rack stopped at its cap (a whole task fails instead:
40
+ `memory-limit-exceeded (cap 16 GB); raise --memory-limit or --size`),
41
+ `[RACK] <task> stopped on <rack> at 2.6 GB, its host short of memory: runs
42
+ again reserving 2.6 GB` for a piece a rack stopped to keep its host from
43
+ running out, as a local run's guard stops one, and ends
44
+ with `Rack summary: 1 task and 12 units on <rack> · 0 waited for capacity · 0
45
+ ran again · 1 transfer retried · 0 ran locally` (each task or unit once,
46
+ though it was leased again). On a rack that admits by
47
+ its resources a split task's pieces are queued together, booked by what each
48
+ reserves, and the rack's guard keeps them from running its host out of memory;
49
+ a first run's piece reserves e3's estimate, as a local run does, so it is as
50
+ wide as a later run and measures no piece first. Calls, mutations and intake stay local. Stock runner
51
+ configurations qualify, and so does a task that declares a `python` or `node`
52
+ environment, with the project platform packages its runner names; commands, a
53
+ platform package with no environment (`non-stock-platform:<name>`) and
54
+ unscannable IR stay local. Host file/environment/network access stays local
55
+ unless a rule deliberately grants `--allow-host-access` and its resources exist
56
+ on the rack.
57
+
58
+ **A task that declares an environment** needs nothing new in its package. On a
59
+ rack whose agent composes environments (`environments` among its features in
60
+ `e3 rack list`; `Rack: <rack> (…, environments for x86_64-manylinux_2_28)` as
61
+ a run starts), the launching machine installs the environment a second time
62
+ for the rack's platform, with the same frozen installs, packs it as layers the
63
+ rack fetches through the hub, and prints `[ENV] 3f9a2b1c0d4e for
64
+ x86_64-manylinux_2_28: building` then `[ENV] 3f9a2b1c0d4e built in 48 s: 4
65
+ layers, 2.1 GB (1 new)`. No container runtime on either machine; nothing is
66
+ compiled for the rack; a second run builds nothing, and a change to the
67
+ project's code makes one small new layer. The layers are kept in the
68
+ repository's `rack/env-layers`, which `e3 repo gc` sweeps. ❗ What cannot go
69
+ stays local, with its reason on the task's `[LOCAL]` line: `environment:tools`
70
+ / `environment:image` (its kind), `environment:windows-host` (a Windows
71
+ launching machine), `no-rack-for-environment:<tier>` (no rack composes them),
72
+ `environment:no-wheel:<package>` and `environment:native-package:<package>`
73
+ (nothing is compiled for another platform), `environment:pnpm` (a pnpm before
74
+ 11, for another platform), `environment:build-failed` (the tool's message is
75
+ in `e3 task logs`; a watch's next run builds it again). A task's tier is its runner's; `--gpu` puts an `east-py`
76
+ task on `py-datascience`, and is refused for another runner
77
+ (`gpu-not-east-py`).
28
78
 
29
79
  The listener serves HTTPS with the hub's own self-signed certificate, made on
30
80
  first use and kept in `hub-tls.beast2`; a rack pins it by its SHA-256, which
@@ -118,7 +168,14 @@ run waits up to 10 seconds for an enrolled rack to report healthy (with none
118
168
  enrolled, or a policy that selects no task for a rack, it starts at once): run
119
169
  `e3 rack hub` as a service, or set `--idle-exit 0`, for a hub racks can always
120
170
  reach. A data URL lasts 15 minutes, which a transfer may use whole; an object is
121
- at most 5 GiB. The control schema currently requires protocol 2 exactly.
171
+ at most 5 GiB. The hub serves one rack eight transfers at once (one more is told
172
+ to ask again), and reads an object from a byte on (`Range`), so a transfer a slow
173
+ link cut short resumes; it keeps a rack's idle connection for 75 seconds.
174
+ A hub and the e3 that calls it speak control protocol 3 exactly:
175
+ a hub an older e3 started is asked to exit once idle, and `e3 rack stop` with
176
+ that e3 replaces it now. A rack's agent and its source speak rack protocol 3:
177
+ an agent of another protocol is refused by name each time it asks, and
178
+ `e3 rack list` shows its rack as `incompatible: …; update the rack` until it is.
122
179
  Agents must report bundled e3 at least the project's release. Infrastructure
123
180
  failure falls back locally with a separate execution attempt, as does an
124
181
  attempt the rack stopped on its own; the run's own cancellation and the task's
@@ -129,11 +186,11 @@ sessions of the same canonical repository and e3 release.
129
186
 
130
187
  | Signature | Description | Example |
131
188
  | --- | --- | --- |
132
- | `openRackRunner({ repoPath, workspace, storage, label, budget?, settings?, policy?, rackOnly? })` | Opens a runner/session; preserve the caller's existing budget and process settings, use `rackSlots` to extend orchestrator width, and close in `finally`. | `const rack = await openRackRunner({ repoPath, workspace: 'dev', storage, label: 'run', budget });` |
189
+ | `openRackRunner({ repoPath, workspace, storage, label, budget?, settings?, policy?, rackOnly?, unreachableWaitMs?, onPlacement?, onEnvironment? })` | Opens a runner/session; preserve the caller's existing budget and process settings, use `rackSlots` to extend orchestrator width, and close in `finally`. Under `rackOnly`, work waits `unreachableWaitMs` (two minutes) for an unhealthy rack before it runs locally. `onPlacement` is told where each task or unit runs; `onEnvironment` that an environment's layers are being built for a rack, and what the build came to. | `const rack = await openRackRunner({ repoPath, workspace: 'dev', storage, label: 'run', budget });` |
133
190
  | `createDataflowRunnerFactory(options?)` | Pass to `createServer({ dataflowRunner })`; one session per API dataflow, using the server's shared budget. Explicit server opt-in overrides policy.enabled; rules still apply. | `const server = await createServer({ reposDir: './repos', dataflowRunner: createDataflowRunnerFactory() });` |
134
191
 
135
192
  The API server's optional rack package is needed only for
136
193
  `e3-api-server --repos ./repos --rack`. Portable handlers receive a runner
137
- factory and do not import hub code. Custom environments, remote function
138
- delegation, automatic reconnect after hub restart and embedded UI integration
139
- are deferred.
194
+ factory and do not import hub code. `tools` and `image` environments on a
195
+ rack, remote function delegation, automatic reconnect after hub restart and
196
+ embedded UI integration are deferred.
@@ -9,6 +9,7 @@ Secondary, interval and area, series and summary, nested series, and entries ind
9
9
  - Series and summary indexes — `index.series`, `index.summary`
10
10
  - Nested series indexes — `index.nested`
11
11
  - Entries indexes — `index.entries(field)`
12
+ - Reading an index inside a task or a function — `e3.index(record.indexes.<name>)`, and a reader for each kind: `of`, `between`, `overlapping`, `events`, `totals`, `within`
12
13
 
13
14
  ## Secondary indexes — `index.on([...fields])`, `index.key(fn)`, `index.keys(fn)`
14
15
 
@@ -259,10 +260,10 @@ const assemblies = e3.record('assemblies', AssembliesType, {
259
260
 
260
261
  | Builder (`index.nested.…`) | A field holding |
261
262
  |---|---|
262
- | `items({ from, to })`, `items({ at })` | `Dict<String, Struct>`: things people drag, by id; their times are DateTime or Option<DateTime> fields — no start files an item nowhere, no end (or one at or before its start) makes it an instant. `{ rollup: true }` files each under every ancestor too |
263
+ | `items({ from, to })`, `items({ at })` | `Dict<K, Struct>`: things people drag, by id, of any key type but a DateTime; their times are DateTime or Option<DateTime> fields — no start files an item nowhere, no end (or one at or before its start) makes it an instant. `{ rollup: true }` files each under every ancestor too |
263
264
  | `values({ rollup? })` | `Dict<DateTime, V>`: a value per instant. `rollup` keeps every parent's subtotals |
264
- | `rows({ … })` | `Dict<String, Struct>`: child rows, their own collections declared the same way |
265
- | `recursive()` | `Dict<String, Self>`: more rows like the one listing it, to any depth |
265
+ | `rows({ … })` | `Dict<K, Struct>`: child rows, keyed by any type but a DateTime (a role, a number, a String), their own collections declared the same way. `rows({})` declares child rows that nest nothing in time: the index names them among its ids and files nothing of them |
266
+ | `recursive()` | `Dict<K, Self>`: more rows like the one listing it, to any depth, keyed as the rows listing it are |
266
267
 
267
268
  TypeScript types `collections` field by field from the record's type alone: a
268
269
  misnamed time, or a builder on the wrong kind of field, is an error where it is
@@ -272,11 +273,22 @@ series index, which takes no `on` and no `value`.
272
273
 
273
274
  **What it files.** Every entry is keyed `{ collection, row, path, run, level,
274
275
  tile, item }` (e3-types' `NestedEntryKeyType`): the collection's place, the
275
- field names from the row down (`members` then `load`); the row's key; the Dict
276
- keys from the row down to the node (`[]` for the row, `m-3` for a member); the
277
- run; the level and tile; and the item. A place and a path are held as a key
276
+ field names from the row down (`members` then `load`); the row's key; the child
277
+ rows' keys from the row down to the node (none for the row, `m-3` for a member);
278
+ the run; the level and tile; and the item. A place and a path are held as a key
278
279
  holds a list, folded from the last: `.step (key="members", rest=.step
279
- (key="load", rest=.end))`. Each run of a node is one contiguous range a level:
280
+ (key="load", rest=.end))`.
281
+
282
+ A child row's key and an item's id are held in their own types, never as text.
283
+ They are the index's **ids**: one variant with a case for each collection of
284
+ child rows and of items, named by its place's field names joined with a dot
285
+ (`members`, `members.tasks`), each holding that collection's key type. So a
286
+ member's path is `.step (key=.members "m-3", rest=.end)`, a job's item
287
+ `.id .jobs "j-0042"`, and a crew's hours by role, a variant, sit under
288
+ `.step (key=.hours .operator, rest=.end)`. Keys sort in their types' own order.
289
+ A declared field's name holds no dot.
290
+
291
+ Each run of a node is one contiguous range a level:
280
292
 
281
293
  | Run | Holds |
282
294
  |---|---|
@@ -374,3 +386,182 @@ an insert, one whose value changed an update by the value's patch — and the
374
386
  lean rows where it changed the row's other fields. After a patch, an edit or a
375
387
  reduce alike, a maintained entries index, its lean rows and the indexes beside
376
388
  them are what `e3 reindex` rebuilds, byte for byte.
389
+
390
+ ## Reading an index inside a task or a function — `e3.index(record.indexes.<name>)`
391
+
392
+ A page reads an index through the API (`index=<name>`, `Data.bindPaged`). A
393
+ task, a stream task or a function reads one by taking it as an input:
394
+ `e3.index(record.indexes.<name>)` in its `inputs`. The body is given the
395
+ index's entries — the collection the record's state holds, `Dict<{ ik, k }, P>`,
396
+ opened lazily and frozen as any input is — where a body given the record gets
397
+ its rows in the record's own order, and reads every one to find a few.
398
+
399
+ ```typescript
400
+ const JobType = StructType({ title: StringType, press: StringType });
401
+ const jobs = e3.record('jobs', DictType(StringType, JobType), {
402
+ initial: new Map(),
403
+ indexes: (index) => ({ jobs_by_press: index.on(['press']) }),
404
+ mutations: (mutation) => ({ patch: mutation.patch() }),
405
+ });
406
+ const byPress = e3.index(jobs.indexes.jobs_by_press);
407
+
408
+ // A function: how many jobs a press holds, read from the index alone.
409
+ const jobsOnPress = e3.function('jobs_on_press', { inputs: [byPress], args: [StringType], output: IntegerType },
410
+ ($, jobsByPress, press) => jobsByPress.prefix({ ik: press }).size());
411
+
412
+ // More of a row than the index carries: the record beside it, read by the entry's `k`.
413
+ const titlesOnPress = e3.function('titles_on_press', { inputs: [byPress, jobs], args: [StringType], output: DictType(StringType, StringType) },
414
+ ($, jobsByPress, jobs, press) => {
415
+ const titles = $.let(new Map(), DictType(StringType, StringType));
416
+ $.for(jobsByPress.prefix({ ik: press }), ($, _carried, key) => { $(titles.insert(key.k, jobs.get(key.k).title)); });
417
+ return titles;
418
+ });
419
+
420
+ // A task takes it the same way. Its body's parameter is the input's type.
421
+ const watched = e3.input('watched', StringType, variant('value', 'P1'));
422
+ const watchedJobs = e3.task('watched_jobs', [byPress, watched],
423
+ East.function([byPress.type, StringType], IntegerType, ($, jobsByPress, press) => jobsByPress.prefix({ ik: press }).size()));
424
+ ```
425
+
426
+ **The entries.** `ik` is the index's key and `k` the row's, `ik` first, so
427
+ every entry of one index key is one contiguous run, in the record's key order,
428
+ and a range of index keys is one run too. `P` is what the index carries. The
429
+ input's `type` is the type the state holds, so a task's body declares its
430
+ parameter by it:
431
+
432
+ | Index | `ik` | `k` | Carries |
433
+ |---|---|---|---|
434
+ | `index.on(['press'])` | the field's value | the row's key | nothing (Null), or its `value` |
435
+ | `index.on(['press', 'due'])` | `{ press, due }` | the row's key | nothing, or its `value` |
436
+ | `index.key(fn)`, `index.keys(fn)` | what `fn` returns, or an element of its Set | the row's key | nothing, or its `value` |
437
+ | `index.entries('byService')` | the field's key | the row's key | that key's value |
438
+ | `index.interval`, `index.area`, `index.series` | the generated key above | the row's key | nothing for an interval; the row for an area or a series |
439
+ | `index.summary` | `{ on, key, tile }` | Null: an entry stands for every row of its key | `{ count, ms }` |
440
+ | `index.nested` | `{ collection, row, path, run, level, tile, item }` | the row's key | the item, the totals or the subtotal |
441
+
442
+ It takes the indexes e3 keeps beside one as well: a nested series index's span
443
+ (`machines.indexes.machines_in_time.nested.span`, keyed `{ collection, tile }`,
444
+ its `k` Null) and an entries index's lean rows
445
+ (`stations.indexes.stations_by_service.entries.rows`). Beside a nested series
446
+ index or an entries index, an index that carries nothing or the row carries the
447
+ lean row. Its `type` says so, and so does TypeScript: the body's parameter, and
448
+ what a read of it gives, are typed as the row.
449
+
450
+ **A reader for each kind.** A plain index's rows under a key are one run, which
451
+ `prefix` reads. The indexes a planning board reads are not one run: a series
452
+ index files each event once, at the level its length fits, an interval index
453
+ answers a window with a run of starts and one a tree node above the window's
454
+ first instant, an area index with a cell a level, and each files more under
455
+ those runs than a window holds. So an index input arrives, in an `e3.function`
456
+ with a spec and an `e3.streamTask`, with its kind's readers on it — it is still
457
+ the entries, and `prefix`, `range`, `size` and a keyed read work as before — and
458
+ `e3.index(def).read(entries)` gives the same over any other expression of the
459
+ index's type, a task's own function's parameter among them.
460
+
461
+ ```typescript
462
+ // The hours a press is booked for in a window, from its series index alone.
463
+ const hoursOnPress = e3.function('hours_on_press', {
464
+ inputs: [e3.index(jobs.indexes.jobs_by_press_week)],
465
+ args: [StringType, DateTimeType, DateTimeType],
466
+ output: FloatType,
467
+ }, ($, jobsByPress, press, from, to) => {
468
+ const sheets = $.let(0.0);
469
+ jobsByPress.events({ key: press, from, to }).each($, ($, job) => { $.assign(sheets, sheets.add(job.sheets)); });
470
+ return sheets.divide(8000.0);
471
+ });
472
+
473
+ // The same rows as a Dict by the job's own key.
474
+ const held = jobsByPress.events({ key: press, from, to }).collect($);
475
+
476
+ // In a task, whose body is its own East function.
477
+ const byPress = e3.index(jobs.indexes.jobs_by_press_week);
478
+ East.function([byPress.type, StringType], IntegerType, ($, entries, press) =>
479
+ byPress.read(entries).events({ key: press, from, to }).collect($).size());
480
+ ```
481
+
482
+ A reader answers a **read**: the rows its words say and no others, each once,
483
+ in the index's own order.
484
+
485
+ | Form | Gives | Holds in memory |
486
+ |---|---|---|
487
+ | `read.each($, ($, carried, key) => { … })` | the body run once a row: what the index carries for it, and the row's key. It takes no label and runs to its end: skip a row with `$.if`, leave the function with `$.return` | a segment of the index at a time |
488
+ | `read.collect($)` | a new Dict by the row's key of what `each` gives | the rows it collects |
489
+
490
+ | Declared by | Reader | Gives, each row once | Runs read |
491
+ |---|---|---|---|
492
+ | `index.on`, `index.key`, `index.keys` | `of(key)`: the rows under an index key, or under its first fields (`{ owner }` of `['owner', 'due']`). `between({ from?, to? })`: the rows from one index key up to another | what the index carries, and the row's key | one |
493
+ | `index.interval` | `overlapping({ on?, from, to })`: the rows that overlap `[from, to)`. `unset({ on? })`: the rows with no start | what the index carries, and the row's key | at most 55, in the index's order: the starts in the window, then one a tree node above its first instant |
494
+ | `index.series` | `events({ on?, key, from, to })`: the events on `key` that overlap `[from, to)` | the row, and its key | one a level, from the index's tile to the year, in the index's order of its levels |
495
+ | `index.summary` | `totals({ on?, key, from, to })`: each tile of `key` that overlaps `[from, to)` | `{ count, ms }`, and the tile's start in place of a row's key | one |
496
+ | `index.area` | `within({ on?, x, y, width, height })`: the rows whose box meets the window. `unplaced({ on? })`: the rows with no place | the row, and its key | at most 266, as a page's read of a viewport, in the index's order: a cell before the cells inside it |
497
+ | `index.entries` | `of(key)`: that key's entry on every row that holds one | the entry's value, and the row's key | one |
498
+ | `index.nested` | of one node of a row's tree, below | below | one, or one a level |
499
+
500
+ A nested series index's span and an entries index's lean rows are read as
501
+ plain indexes are, with `of` and `between`. A span's entry stands for every
502
+ row of its key and names none, so its read gives the index key,
503
+ `{ collection, tile }`, in a row's key's place.
504
+
505
+ **A nested series index's readers.** A nested series index files a row's tree
506
+ a node at a time, each node's entries in runs, so each reader reads one node:
507
+ `{ collection, row, path?, from, to }` — the collection's place (its field
508
+ names from the row down, `['units', 'jobs']`, a recursive field adding none),
509
+ the row's key, and the Dict keys from the row down to the node, the row itself
510
+ where `path` is left out. TypeScript offers the places the index declares, and
511
+ types what a read gives by the place.
512
+
513
+ ```typescript
514
+ const inTime = e3.index(lines.indexes.lines_in_time);
515
+ const sheetsOnUnit = e3.function('sheets_on_unit', {
516
+ inputs: [inTime], args: [StringType, StringType, DateTimeType, DateTimeType], output: FloatType,
517
+ }, ($, linesInTime, line, unit, from, to) => {
518
+ const sheets = $.let(0.0);
519
+ linesInTime.items({ collection: ['units', 'jobs'], row: line, path: [variant('units', unit)], from, to })
520
+ .each($, ($, job) => { $.assign(sheets, sheets.add(job.sheets)); });
521
+ return sheets;
522
+ });
523
+ // The line's mean of its units' load a day, which the index keeps.
524
+ linesInTime.subtotals({ collection: ['units', 'load'], row: line, rollup: 'mean', from, to }).collect($);
525
+ ```
526
+
527
+ | Reader | Gives, each once | Needs | Runs read |
528
+ |---|---|---|---|
529
+ | `items(node)` | the node's own items that overlap `[from, to)`, each with its id, in its collection's own key type; or its values in the window, each with its instant | | one a level for items timed `{ from, to }`; one for values and items timed `{ at }` |
530
+ | `subtree(node)` | the items of every node beneath the node that overlap the window, each with `{ path, id }`: the keys from the node down to the one holding it — a path, `end` or a `step` and the rest — and its id, each an id of the index's (`NestedUnderType(index)` is the type) | `items(…, { rollup: true })` | one a level |
531
+ | `totals(node)` | the node's totals of the collection in each summary tile that overlaps the window — `{ count, ms, values }`, `values` the sum, least, greatest and latest of numbers — with the tile's start | `summary` | one |
532
+ | `subtreeTotals(node)` | the same, of the items beneath the node | `rollup: true` and `summary` | one |
533
+ | `subtotals({ …node, rollup, level? })` | a parent's subtotal of its children's values in each tile of `level` that overlaps the window, with the tile's start. `rollup` is written as the collection declares it; `level` is the index's tile where left out, or a coarser one | `values({ rollup })` | one |
534
+
535
+ A node that holds values of its own draws them and keeps no subtotal, so its
536
+ `subtotals` is empty; a row or a node that is not there reads as empty too. A
537
+ place the index does not file, a collection not rolled up, a rollup it does not
538
+ keep, a level finer than its tile and totals of an index with no `summary` are
539
+ each refused where the reader is called, naming what the index does keep.
540
+
541
+ | Rule | |
542
+ |---|---|
543
+ | **Exact** | A series, a nested series or an area index files more under the runs a window reads than the window holds, and an event past a year under each year it touches: the reader tests each entry, and gives it once. A plain index whose rows fan out — `index.keys`, a Set field — gives a row once across the keys of a read, and holds the keys of the rows it gave to do so |
544
+ | **The window's edges** | `[from, to)`, for every reader with a span of time: an event that ends at `from` is outside; an instant at `from` is inside and one at `to` outside; an event with no end, or one at or before its start, is an instant. A window that ends at or before its start holds no row. A row with no start is in no window: a series index files it nowhere, and an interval index gives it by `unset`. An area's window holds its edges: a box that touches it is in it, and one of a negative size holds none. A zero is one edge whatever its sign: a width of `-0.0` is a width of none, as it is to a page |
545
+ | **Order** | A read gives its rows in the index's own order: one of several runs reads them in the order the index holds them, so the rows come as a walk of the index meets them. Across a series index's levels that is not time order — the index orders a level by its name: day, month, quarter, week, year — so sort what is collected where order matters |
546
+ | **Leading fields** | An index declared with `on` takes every leading field, exactly, as `on: { status: "open" }`, in every reader: TypeScript requires it, `unset` and `unplaced` included. One left out, one that leads nothing, or a value of another type is refused where the reader is called, naming the fields |
547
+ | **Its kind's readers** | TypeScript offers an index the readers of its kind. `e3.index(def).read` refuses another index's entries, naming both: by their type, and, where two indexes' entries have one type, by the readers the expression already carries — a body's parameter for the other index, or what the other's `read` was given. An expression that carries none is known by its type alone, so in a task's own function two parameters of one type are the author's to keep apart. `unset` and `unplaced` are refused for an index whose start, or place, is no Option field: every row then has one |
548
+ | **More of the row** | Where an index carries less than the row, take the record as an input beside it and read the row by its key, at the same state. Those keyed reads come in the index's order, not the record's, so an index that carries the row is the one to read many rows through |
549
+ | **Cost** | A read costs its runs, each a range read: its two ends through the pager's cache and what lies between. It never costs the index. Over 24,000 jobs: a key's rows decode 1 of the index's 20 segments, a week of an interval 12 of 61, a week of a press's series 2 of 20, a room of an area 4 of 19; over forty lines of four units, each read of one node of a nested series index 1 of 31 — the same on east-node, east-c and east-py |
550
+ | **The page's runs** | Each reader reads the runs a page's reader of its kind searches, less the tiles and the margin a page reads beyond a window to pan into |
551
+
552
+ **What a read costs.** A body reads the entries with `prefix` and `range`, by
553
+ key, by size and by a loop, as it reads any lazily opened Dict: a run costs the
554
+ segment each of its ends lands in and the segments between them, never the
555
+ index, and the size of a run costs its two ends alone. A body that uses the
556
+ input any other way decodes the index whole, as it would any input; a runner's
557
+ `-v` says which, an input at a time.
558
+
559
+ | Rule | |
560
+ |---|---|
561
+ | **One state** | The inputs of one record — the record, and any of its indexes — are read at one state of it, so a body never meets an index entry for a row it reads otherwise |
562
+ | **A task** | is decided again whenever its record changes, and runs again when the index it reads did: its execution is keyed by the index, so a write that leaves the index as it was is served by the cache |
563
+ | **A function** | reads the index as the record is stored when the call starts. Its result's `inputs` name the record, the index and the hash the record was pinned at, which its other inputs of that record name too |
564
+ | **Drafts** | A function may list the record in `drafts` whether it reads the record or an index of it: a call's drafts are laid over the record, and its indexes hold them as they would after the Save |
565
+ | **No state yet** | A record with no state makes a task wait and a call `invalid`, as it does for the record as an input. So does a state that holds no such index — one written outside the doors that build every index its record declares — the task's status naming the record and the index, and the run going on without it |
566
+ | **The package** | `e3.package` collects the index's record |
567
+ | **Refused where declared** | Anything but an index its record carries by name, naming what was given; a task's parameter that is not the input's `type`; an index inside `e3.partition` (every piece reads it whole: pass the `e3.index` itself); an index as an input of `e3.customTask`, whose command reads files; a function on the `custom` runtime, which takes no inputs |
@@ -18,6 +18,7 @@ repo/
18
18
  ├── locks/ # locks, their holders, and how far each says it has got
19
19
  ├── gc/ # gc beside running work: unreachable notes, a stepped run's parts
20
20
  ├── envs/ # built execution environments
21
+ ├── rack/ # rack delegation: its policy, and the layers kept of environments built for a rack
21
22
  └── tmp/ # scratch and staged uploads
22
23
  ```
23
24