@elaraai/east-codex-plugin 1.0.108 → 1.0.110
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.build/daemon/lsp.js +55 -7
- package/.build/daemon/server.js +53 -5
- package/.codex-plugin/plugin.json +1 -1
- package/index.json +1781 -199
- package/package.json +1 -1
- package/skills/e3/SKILL.md +20 -14
- package/skills/e3/reference/cli.md +4 -2
- package/skills/e3/reference/from-code.md +2 -2
- package/skills/e3/reference/rack.md +20 -6
- package/skills/e3/reference/record-indexes.md +198 -7
- package/skills/e3-create/SKILL.md +7 -2
- package/skills/e3-ui/SKILL.md +80 -68
- package/skills/east/SKILL.md +9 -4
- package/skills/east/reference/beast2.md +10 -1
- package/skills/east-py/SKILL.md +8 -2
- package/skills/east-py/reference/beast2.md +1 -0
- package/skills/east-py-datascience/SKILL.md +8 -0
- package/skills/east-ui/SKILL.md +1 -0
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@elaraai/east-codex-plugin",
|
|
3
|
-
"version": "1.0.
|
|
3
|
+
"version": "1.0.110",
|
|
4
4
|
"description": "East programming language ecosystem - a statically typed, expression-based language for business logic, data processing, UI definitions, and durable execution",
|
|
5
5
|
"license": "AGPL-3.0-or-later",
|
|
6
6
|
"type": "module",
|
package/skills/e3/SKILL.md
CHANGED
|
@@ -1,8 +1,12 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: e3
|
|
3
|
-
description: "East Execution Engine (e3): durable, content-addressed dataflow for East programs. Use when (1) authoring a package with @elaraai/e3 — e3.input, e3.task, e3.streamTask with e3.output and e3.partition (bounded memory, work split over an input), e3.customTask, e3.function, e3.package, e3.export, East.importFunction; (2) records — audited state written only through mutations (e3.record with mutation.reduce/edit/patch), their indexes (index.on/key/keys, interval, area, series, summary, nested, entries) and migrations (migration.value/rows/rekey); (3) the e3 CLI — repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, watch, convert, auth, and the cores and memory a run may use; (4) driving e3 from code (@elaraai/e3-api-client, @elaraai/e3-core, e3-api-server); (5) caching, reactive re-runs and garbage collection; (6) delegating task bodies
|
|
3
|
+
description: "East Execution Engine (e3): durable, content-addressed dataflow for East programs. Use when (1) authoring a package with @elaraai/e3 — e3.input, e3.task, e3.streamTask with e3.output and e3.partition (bounded memory, work split over an input), e3.customTask, e3.function, e3.package, e3.export, East.importFunction; (2) records — audited state written only through mutations (e3.record with mutation.reduce/edit/patch), their indexes (index.on/key/keys, interval, area, series, summary, nested, entries; e3.index reads one in a task or function) and migrations (migration.value/rows/rekey); (3) the e3 CLI — repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, watch, convert, auth, and the cores and memory a run may use; (4) driving e3 from code (@elaraai/e3-api-client, @elaraai/e3-core, e3-api-server); (5) caching, reactive re-runs and garbage collection; (6) delegating task bodies, ad-hoc runs… See the detailed scope below."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
+
## Detailed skill scope
|
|
7
|
+
|
|
8
|
+
East Execution Engine (e3): durable, content-addressed dataflow for East programs. Use when (1) authoring a package with @elaraai/e3 — e3.input, e3.task, e3.streamTask with e3.output and e3.partition (bounded memory, work split over an input), e3.customTask, e3.function, e3.package, e3.export, East.importFunction; (2) records — audited state written only through mutations (e3.record with mutation.reduce/edit/patch), their indexes (index.on/key/keys, interval, area, series, summary, nested, entries; e3.index reads one in a task or function) and migrations (migration.value/rows/rekey); (3) the e3 CLI — repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, watch, convert, auth, and the cores and memory a run may use; (4) driving e3 from code (@elaraai/e3-api-client, @elaraai/e3-core, e3-api-server); (5) caching, reactive re-runs and garbage collection; (6) delegating task bodies, ad-hoc runs and function calls to a rack (e3 rack); (7) e3 in a browser with @elaraai/e3-web. UI tasks are e3-ui's ui().
|
|
9
|
+
|
|
6
10
|
# East Execution Engine (e3)
|
|
7
11
|
|
|
8
12
|
e3 runs East programs as a durable dataflow over content-addressed data: every
|
|
@@ -70,7 +74,8 @@ What do you need?
|
|
|
70
74
|
│ │ └─ Split over an input, in pieces → wrap that input: e3.partition(dataset, { by? })
|
|
71
75
|
│ ├─ A shell command → e3.customTask(name, [inputs], outputType, command)
|
|
72
76
|
│ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name; a call through a workspace is kept for its lifetime
|
|
73
|
-
│ │
|
|
77
|
+
│ │ ├─ It reads workspace data too → e3.function(name, { inputs, args, output, runner? }, fn) — inputs read at each call
|
|
78
|
+
│ │ └─ Over a page's unsaved edits → add drafts: [record, …] — a call lays mutation calls over those records, and saves nothing
|
|
74
79
|
│ ├─ Audited state → e3.record(name, type, { initial?, indexes?, mutations?, migrations? }), declared whole
|
|
75
80
|
│ │ ├─ Its writes → mutations: (mutation) => ({ name: mutation.reduce | edit | patch })
|
|
76
81
|
│ │ ├─ An order to page in → indexes: (index) => ({ name: index.on([...fields]) | index.key(fn) | index.keys(fn) }) (reference/record-indexes.md)
|
|
@@ -78,13 +83,14 @@ What do you need?
|
|
|
78
83
|
│ │ ├─ Read a block of keys over time tiles, or their totals → index.series | index.summary
|
|
79
84
|
│ │ ├─ Read what rows nest (jobs, a value a day, child rows, trees) a tile at a time, with totals and subtotals
|
|
80
85
|
│ │ │ → index.nested({ collections, tile, summary? })
|
|
81
|
-
│ │
|
|
86
|
+
│ │ ├─ Read a few keys of a Dict field across every row (each row's figures by category) → index.entries(field)
|
|
87
|
+
│ │ └─ Read an index inside a task or a function → e3.index(record.indexes.<name>) among its inputs: the entries, with its kind's readers — of, between, overlapping, events, totals, within, a nested index's items, subtree, subtotals — each a read: each($, body) | collect($)
|
|
82
88
|
│ ├─ A record's type changes → migrations: (migration) => [migration.value | rows | rekey(name, fn), …], in order (reference/record-migrations.md)
|
|
83
89
|
│ ├─ A UI task → ui(name, [inputs], fn) from @elaraai/e3-ui (the e3-ui skill)
|
|
84
90
|
│ ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
|
|
85
91
|
│ └─ Bundle and export → e3.package(name, version, ...items); e3.export(pkg, zipPath)
|
|
86
|
-
├─ Operate it from the shell → the e3 CLI (reference/cli.md): deploy, run, read and write datasets, mutate records, gc
|
|
87
|
-
├─ Delegate local task bodies to a rack → e3 rack enroll + policy; dataflow run --rack, watch --start --rack (reference/rack.md)
|
|
92
|
+
├─ Operate it from the shell → the e3 CLI (reference/cli.md): deploy, run, read and write datasets, call functions (named, or `call --ir` a function's IR file), mutate records, gc
|
|
93
|
+
├─ Delegate local task bodies to a rack → e3 rack enroll + policy; dataflow run --rack, watch --start --rack; one task or one call: run --rack, call --rack (reference/rack.md)
|
|
88
94
|
│ ├─ A split task's pieces fill the rack → booked by what each reserves, capped half as much again; --memory-limit bounds the cap, --gpu asks for the GPU
|
|
89
95
|
│ └─ A task with its own python/node environment → nothing new in the package: its layers are built for the rack's platform
|
|
90
96
|
└─ Drive it from code
|
|
@@ -125,15 +131,15 @@ A `file` source:
|
|
|
125
131
|
naming the input and the first differing field) and again at deploy, before
|
|
126
132
|
the workspace is touched;
|
|
127
133
|
- is hashed first, so a delivery the store already holds costs that one read;
|
|
128
|
-
- is taken in without being read whole or modified
|
|
129
|
-
a new file, never edit it in place. A collection is taken in on the runners —
|
|
134
|
+
- is taken in without being read whole or modified. A collection is taken in on the runners —
|
|
130
135
|
east-c, or east-node where there is no east-c — in pieces of its segments, as
|
|
131
136
|
many at once as the budget allows: each row is walked, the bytes the Writer
|
|
132
137
|
writes for it kept as they stand and any other row written again, so the
|
|
133
138
|
delivery is stored as the Writer's value whichever writer wrote it, and a new
|
|
134
139
|
delivery stores only the segments that changed. A deploy or an upload
|
|
135
140
|
stopped part way takes up again from the pieces it finished. Any other value
|
|
136
|
-
is taken in
|
|
141
|
+
is taken in as a copy, unless the store holds it already, so the file stays
|
|
142
|
+
its user's to write again;
|
|
137
143
|
- may hold no segment of more than 64 MiB, the most a collection is read in at
|
|
138
144
|
once. A value encoded whole is one such segment, and so can be a segment of
|
|
139
145
|
wide rows from an older Writer, which bounded a segment by its element count
|
|
@@ -391,8 +397,8 @@ const change = e3.streamTask('change', {
|
|
|
391
397
|
| Definition | What it is |
|
|
392
398
|
|---|---|
|
|
393
399
|
| `e3.customTask(name, inputs, outputType, command, { environment? })` | A task that runs a bash script: `command($, inputPaths, outputPath)` builds it from the staged beast2 input files, and it writes the output as a beast2 file. |
|
|
394
|
-
| `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API, `Func.bind`) and its result returned inline: not in the dataflow. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A call through a workspace is the server's: it outlives the request that starts it — the client starts it and polls it to its end — and runs until its timeout, the server's ceiling (ten minutes locally) unless it asks for less; it can be cancelled; and the server keeps it for its lifetime, an hour for a success or a cancel and a week for any other end: who made it, its arguments, the versions of the inputs it read, its value and its logs (reference/from-code.md). A call through the package runs within one request and keeps nothing. A result is at most 1 MiB, so a large one belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
|
|
395
|
-
| `e3.function(name, { inputs, args, output, runner?, environment?, async? }, fn)` | A function that reads the workspace as well: `inputs` (an input, a task's output or
|
|
400
|
+
| `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API, `Func.bind`) and its result returned inline: not in the dataflow. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A call through a workspace is the server's: it outlives the request that starts it — the client starts it and polls it to its end — and runs until its timeout, the server's ceiling (ten minutes locally) unless it asks for less; it can be cancelled; and the server keeps it for its lifetime, an hour for a success or a cancel and a week for any other end: who made it, its arguments and drafts, the versions of the inputs it read, its value and its logs (reference/from-code.md). A call through the package runs within one request and keeps nothing. A result is at most 1 MiB, so a large one belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). `e3 call --rack` runs a local repository's call on an enrolled rack (reference/rack.md), and `e3 call --ir <file>` calls a function from its IR file, with no package (reference/cli.md). |
|
|
401
|
+
| `e3.function(name, { inputs, drafts?, args, output, runner?, environment?, async? }, fn)` | A function that reads the workspace as well: `inputs` (an input, a task's output, a record or an index of one, `e3.index(record.indexes.<name>)`, as a task takes them) are read as stored when each call launches, pinned at their hashes and named in the result's `inputs`; the caller passes `args` alone. The body takes the inputs, then the arguments. `e3.package` collects the inputs and what they depend on; nothing reruns when one changes. The inputs are read one at a time and the call waits for no run, so a call during a run may read one input before the run rewrites it and another after: the hashes in `inputs` say which. A record and its indexes are read at one state of it. Called through a workspace only (`e3 call -w`); an unassigned input is `invalid`, naming it and why. `drafts` lists the records, each read by one of `inputs` (itself, or an index of it), that a call may lay **unsaved edits** over: a draft is a mutation call that is not committed, `{ record, mutation, args }` (`e3 call -w <ws> <fn> [args…] --draft <record>.<mutation> [args…]`, reference/cli.md). A call's drafts run in the order passed, as a Save runs them, each over the state the one before it left, and the body reads each drafted record, and its indexes, at the state that leaves; a dataset argument naming a drafted record is read as saved, at the state the drafts were laid over. Nothing is saved: no commit, no record moved, no task re-run — the record and what reads it stay as they were for every reader. The call keeps its drafts as it keeps its arguments, for whoever may read the call, and each draft's mutation runs as a Save's does, an execution recorded and cached. The result's `inputs` name the drafted state's hash, and its `drafts` the commit each drafted record stood at. A draft of a record the function does not list, or one the record does not take (an unknown mutation, arguments of the wrong number or type), is `invalid`, naming the draft; one the record refuses ends the call as its Save would end, the body not run: `failed` or `timed_out` (the first line of stderr says which draft, or that the drafts were all laid as the timeout ended), or `conflict` when the record changed under the edit. Listing a record is the author's word that the body is fit to run over edits nobody saved, so a function that sends a message or writes elsewhere lists none. ❗ Refused: an `e3.partition` input, the `custom` runtime with inputs, and in `drafts` what is no record or a record no input reads. `FunctionSpec` names the spec's type. |
|
|
396
402
|
| `e3.package(name, version, ...items)` | Bundles the items and everything they depend on — pass the last tasks, records and functions; a package passed in contributes its contents. The name and version must be valid file names. |
|
|
397
403
|
| `e3.export(pkg, zipPath, { functions?, cores? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself, `cores` packages at once, the CPUs by default; `functions` lists manifests built elsewhere). |
|
|
398
404
|
|
|
@@ -491,7 +497,7 @@ cheap. A view reads every field of the rows it shows, so keep a bulky payload it
|
|
|
491
497
|
does not display in a second record keyed the same way.
|
|
492
498
|
|
|
493
499
|
A record's indexes (`index.on` / `key` / `keys`, `interval`, `area`,
|
|
494
|
-
`series`, `summary`, `nested`, `entries`) are in
|
|
500
|
+
`series`, `summary`, `nested`, `entries`, and `e3.index` to read one) are in
|
|
495
501
|
[reference/record-indexes.md](reference/record-indexes.md), and its migrations
|
|
496
502
|
in [reference/record-migrations.md](reference/record-migrations.md).
|
|
497
503
|
|
|
@@ -519,12 +525,12 @@ Read the one a task needs; each holds its subject whole.
|
|
|
519
525
|
|
|
520
526
|
| File | Holds |
|
|
521
527
|
|---|---|
|
|
522
|
-
| [reference/record-indexes.md](reference/record-indexes.md) | Secondary indexes, interval and area indexes, series and summary indexes, nested series indexes, entries indexes |
|
|
528
|
+
| [reference/record-indexes.md](reference/record-indexes.md) | Secondary indexes, interval and area indexes, series and summary indexes, nested series indexes, entries indexes, and reading an index inside a task or a function (`e3.index`) |
|
|
523
529
|
| [reference/record-migrations.md](reference/record-migrations.md) | `migration.value` / `rows` / `rekey`, and how a record's type changes |
|
|
524
|
-
| [reference/cli.md](reference/cli.md) | Every `e3` command: repositories, packages, workspaces, datasets, tasks, runs, calls, records, watch, convert, auth, gc |
|
|
530
|
+
| [reference/cli.md](reference/cli.md) | Every `e3` command: repositories, packages, workspaces, datasets, tasks, runs, calls (named, or from an IR file), records, watch, convert, auth, gc |
|
|
525
531
|
| [reference/from-code.md](reference/from-code.md) | `@elaraai/e3-api-client`, `@elaraai/e3-core`, `e3-api-server` |
|
|
526
532
|
| [reference/browser.md](reference/browser.md) | `@elaraai/e3-web`: serveUnits, serveE3, createWebE3, and what a browser cannot do |
|
|
527
|
-
| [reference/rack.md](reference/rack.md) | `e3 rack enroll` / `policy`, routing a run to a rack, how a rack is filled and what a run prints, tasks with an environment on a rack, `@elaraai/e3-rack` |
|
|
533
|
+
| [reference/rack.md](reference/rack.md) | `e3 rack enroll` / `policy`, routing a run to a rack, one task or one function call on a rack (`run --rack`, `call --rack`), how a rack is filled and what a run prints, tasks with an environment on a rack, `@elaraai/e3-rack` |
|
|
528
534
|
| [reference/repository.md](reference/repository.md) | A local repository's layout, and upgrades |
|
|
529
535
|
|
|
530
536
|
## Related skills
|
|
@@ -33,8 +33,10 @@ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
|
|
|
33
33
|
| `e3 dataflow run [repo] <ws> [--filter <task>] [--force \| --force-task <task>…] [-j <n>] [--memory <size>] [-v]` | Run what is stale, then print the outputs' paths. `--filter`: one task, by its exact name, and the tasks it depends on. `--force`: re-run every task the run runs (under `--filter`, its task); `--force-task`, repeated for each: re-run the tasks it names, whose dependents re-run when an output changes. A task the graph lacks, or one `--filter` leaves out, refuses the run. A task whose input nothing has set — an input never set, or a record declared with no initial value and not yet seeded — is left waiting, with its dependents (`[WAIT]`, `Waiting:`), and the run completes without them; they run once it is set. |
|
|
34
34
|
| `e3 task list [repo] <ws>` | Tasks with the status of each one's latest run. |
|
|
35
35
|
| `e3 task logs [repo] <ws.task> [-n <lines>] [--all] [--follow] [--execution <task>/<inputs>/<id>]` | The last 200 lines by default, of the latest attempt over the task's current inputs, or its latest run; `--follow` stays on that execution; `--execution`: one unit's log, as a split task's log names it (local). |
|
|
36
|
-
| `e3 run <repo> <pkg[@ver].task> [inputs.beast2…] -o <out> [--force] [-v] [-j <n>] [--memory <size>]` | Run one task ad hoc. |
|
|
37
|
-
| `e3 call <repo> <pkg[@ver].fn> [args…] [-o <out.beast2>] [-j <n>] [--memory <size>] [-v]` · `e3 call <repo> -w <ws> <fn> [args…]` | Call a function; each argument is an `.east` literal or a `.beast2`/`.json`/`.east` file, and with `-w` one written `@<keypath>` (`@.inputs.sales`) is that dataset of the workspace, pinned at its hash when the call starts. A function that declares `inputs` reads them from the workspace itself, so it takes its arguments alone and is called with `-w`; without it, it is refused, naming its inputs. The value goes to stdout, and the function's stdout and stderr to stderr. Through a server's workspace the call is the server's, kept for its lifetime: it runs as long as its timeout allows, a call longer than a second is named (`Call <id> runs on`), and Ctrl-C asks the server to stop it (exit 130). |
|
|
36
|
+
| `e3 run <repo> <pkg[@ver].task> [inputs.beast2…] -o <out> [--force] [-v] [-j <n>] [--memory <size>] [--rack\|--rack-only]` | Run one task ad hoc (a bare package name runs its greatest version). Each input file is taken in as `--from-file` takes one, a collection never read whole, so a task split over it runs in pieces, and any other value as a copy, so the file stays yours to write again; the output is written as the value, a segment at a time, the file at `-o` replaced and never written into (a link there is written where it leads; a file you may not write is refused; a pipe or a terminal, as `/dev/stdout`, is written into; in a directory you may not write, a file nothing else links to is written into, and one another name links to is refused). `--rack`: on an enrolled rack (local; reference/rack.md). |
|
|
37
|
+
| `e3 call <repo> <pkg[@ver].fn> [args…] [-o <out.beast2>] [-j <n>] [--memory <size>] [-v]` · `e3 call <repo> -w <ws> <fn> [args…]` | Call a function; each argument is an `.east` literal or a `.beast2`/`.json`/`.east` file, and with `-w` one written `@<keypath>` (`@.inputs.sales`) is that dataset of the workspace, pinned at its hash when the call starts. A function that declares `inputs` reads them from the workspace itself, so it takes its arguments alone and is called with `-w`; without it, it is refused, naming its inputs. The value goes to stdout, and the function's stdout and stderr to stderr. Through a server's workspace the call is the server's, kept for its lifetime: it runs as long as its timeout allows, a call longer than a second is named (`Call <id> runs on`), and Ctrl-C asks the server to stop it (exit 130). `--rack`/`--rack-only`: on an enrolled rack (local; reference/rack.md), its drafts laid here. |
|
|
38
|
+
| `… -w <ws> <fn> [args…] --draft <record>.<mutation> [args…]` | Lay an unsaved edit over a record the function lists in `drafts`, and save nothing: the mutation runs as `e3 mutate` would run it, its arguments read the same way, and the function reads the record at the state that leaves. Repeatable, laid in the order written, each over the state the one before it left. `--draft` reads every word up to the next option, so it is written after the call's own arguments, or the drafts are ended with `--`; a draft ends where its mutation's arguments do, and a word that starts with a dash is written `--draft=<word>`. Each draft is read against its mutation before anything runs: one that names no record or mutation of the workspace, or has too few arguments or one of the wrong type, is refused, named by its place (`Draft 2 (jobs.move): …`). A draft the record refuses ends the call before the function runs, and exits 1: `Draft jobs.patch (2 of 3) failed with exit code: 1; the function did not run`, `… had not been laid when the call timed out after <n>ms; …`, `The call's drafts were laid, and its timeout of <n>ms ended before the function ran: …` when laying them took all of the call's time, or, when the record changed under the edit, `Draft jobs.patch (2 of 3) conflicted: … — the edit no longer matches the record; re-read it and call again`. |
|
|
39
|
+
| `e3 call <repo> [-w <ws>] --ir <fn.beast2> [args…] [--runner east-node\|east-c\|east-py] [--platform <pkg>…\|none]` | Call the function an IR file holds (`.beast2`, `.json` or `.east`), its signature read from the IR, named by the file's name. It runs on `--runner`'s runtime (east-node by default) with that runtime's standard package, or the packages `--platform` names (`none`: none, a platform-free call). Through a server it needs `-w` and is one of the workspace's one-shot calls, under the caller's grant: a reader runs a platform-free one alone. It reads no record, so it takes no `--draft`. |
|
|
38
40
|
| **Records** (`-w <ws>` required) | |
|
|
39
41
|
| `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
|
|
40
42
|
| `e3 seed <repo> <record> <file.beast2> -w <ws> [-j <n>] [--memory <size>]` | Commit a record's first state from a beast2 file, as its `$init`: for a record declared with no initial value, unassigned until then (local). The file is taken in as a `file` source is, never read whole; a record that holds a state is refused. |
|
|
@@ -18,8 +18,8 @@ naming the holder: its pid, boot id, command and when it took the lock.
|
|
|
18
18
|
| Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceBundle(url, repo, ws, opts)` → the workspace as it is, as a bundle's bytes — its package, every dataset's ref, the executions its tasks' outputs come from, and every object those reach — and `workspaceBundleImport(url, repo, ws, bundle, opts)` → the workspace made from one under a name no workspace has, in another repository or another e3 (`workspace_exists`, `package_exists`, `bundle_invalid`), its tasks up to date with no run; a host mounts these routes where a bundle fits in a request: the local server, and e3-web, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
|
|
19
19
|
| Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, and several reads in one request (at most `DATASET_BATCH_MAX`, 10,000, each at one content hash, in a body of at most 16 MiB: a larger one is refused 413, `payload_too_large`): `datasetFindKeys(…, { queries, index?, hash? }, opts)` — key searches, `datasetGetRows(…, { keys, hash? }, opts)` — a Dict's rows by `.east` key, each with its position — or `datasetGetRows(…, { keysOf: { path, index, runs, hash }, hash? }, opts)`, by the keys the entries of some runs of another record's index name, which the server reads from the index so none is sent (a drawing's details for its window's cells), a stale `keysOf.hash` thrown as `RunsHashMismatchError` naming the record's content — and `datasetGetRuns(…, { runs, hash, index?, join? }, opts)` — the entries of several runs as one window, a rows or runs answer holding the first `count` the page budget reaches (by runs, of `runEntries`); and where a narrowing's rows sit in a record, from its index alone: `datasetPlace(…, { index, runs, offset?, limit?, hash }, opts)` → `{ segments, placed, runEntries, totalElements, hash }` — how many of the runs' entries each record segment holds (`{ segment, offset, count, rows }`), for the first `placed` of a window from `offset` (the next window from `offset + placed`), and `datasetWithin(…, { index, runs, keys, hash }, opts)` → `{ within, hash }` — whether each index entry key (`(ik=…, k=…)`) lies in one of the runs, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
|
|
20
20
|
| Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
|
|
21
|
-
| Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; within one request, keeping nothing; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts, { signal?, onStarted?, startWaitMs? })` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`), within one request, keeping nothing; `oneShotCall(url, repo, ws, request, opts, { signal?, onStarted?, startWaitMs? })` — the same, under the same rule, started as a kept call through the workspace (below), and run within one request on a server that keeps none (`oneShotStart` its start apart); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress?, signal? })` → `{ result, output, report }` — a program over a dataset's pieces, as a job it polls, under the same rule; `output` is `array`, `set`, `dict { merge }`, `fold { combine, zero }`, or `first { limit }` — an `array`'s first `limit` elements in input order, its pieces run in order and stopped once they hold them; `then`, the final function, takes the assembled output and then the arguments not partitioned, in order, and one of another arity is `invalid`, and how it ran: its pieces, the units it ran and the cache served, and its peak; a value over the call's result limit is `too_large`, naming its stored `output`, which `collectionOpen(url, repo, output, opts)` → `{ type, totalElements, page({ offset, limit }) }` reads a window at a time, fetching only the segments each window overlaps; aborting `signal` cancels the job, whose units the server stops (or `splitCallLaunch`, `splitCallStatus`, `splitCallCancel` — `cancelled`, or the status the job had already ended with — and `splitCallExplain`) |
|
|
22
|
-
| Calls through a workspace | A call through a workspace is the server's, and outlives the request that starts it: `workspaceFunctionCall` starts it, waits up to the server's window (20 s; `startWaitMs` less), and polls a call still running to its end, each poll held until it ends or a while passes, however long its timeout lets it run — the server's ceiling, ten minutes locally, unless its `limits` ask for less; aborting `signal` asks the server to stop it, and `onStarted(id)` is told the id of one that runs on; a server that keeps no calls is called through `workspaceFunctionCallSync`, the sync route, which a caller may name itself. Apart: `workspaceFunctionStart(…, opts, { waitMs? })` → `{ id, status }` (`?wait=0` answers `running` at once), `workspaceFunctionCallStatus(url, repo, ws, id, opts, { waitMs?, signal? })` → `running`, `completed` with the result, `cancelled` naming who asked, or `lost`, `workspaceFunctionCallCancel(url, repo, ws, id, opts)`. The server keeps each call for its lifetime, an hour for a success or a cancel and a week for any other end: `workspaceCallList(url, repo, ws, opts, { function?, state?, before?, limit? })` — a page of them, the latest first, its `next` the next page's `before` (`FUNCTION_CALLS_PAGE_DEFAULT` 50, `…_MAX` 200), and `workspaceCallGet(url, repo, ws, id, opts)` — one whole: its function and package (`none` for a one-shot), its arguments (values by the objects they were stored as, datasets by the hashes they were pinned at), the inputs it read, its limits, its value's object, its log tails and when it is forgotten (`expiresAt`); `objectGet` reads a value by its hash. A one-shot started as a call (`oneShotCall`) is kept, polled, cancelled and listed as a function's call is, its program the caller's, under its one-shot grant. A workspace keeps no call that another made: `internal`, "keeps no call" |
|
|
21
|
+
| Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits, drafts }, opts)` — each argument `variant('value', beast2Bytes)`; within one request, keeping nothing, and reading no workspace, so its `drafts` are `[]` (one passed is `invalid`); the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts, { signal?, onStarted?, startWaitMs? })` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`, each `{ path, index, hash }`: `index` names the index of the record at `path` that an input read, `none` for a dataset itself, and every input of one record names the one hash it was pinned at), and the request's `drafts` unsaved edits laid over the records the function reads (below); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`), within one request, keeping nothing; `oneShotCall(url, repo, ws, request, opts, { signal?, onStarted?, startWaitMs? })` — the same, under the same rule, started as a kept call through the workspace (below), and run within one request on a server that keeps none (`oneShotStart` its start apart); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress?, signal? })` → `{ result, output, report }` — a program over a dataset's pieces, as a job it polls, under the same rule; `output` is `array`, `set`, `dict { merge }`, `fold { combine, zero }`, or `first { limit }` — an `array`'s first `limit` elements in input order, its pieces run in order and stopped once they hold them; `then`, the final function, takes the assembled output and then the arguments not partitioned, in order, and one of another arity is `invalid`, and how it ran: its pieces, the units it ran and the cache served, and its peak; a value over the call's result limit is `too_large`, naming its stored `output`, which `collectionOpen(url, repo, output, opts)` → `{ type, totalElements, page({ offset, limit }) }` reads a window at a time, fetching only the segments each window overlaps; aborting `signal` cancels the job, whose units the server stops (or `splitCallLaunch`, `splitCallStatus`, `splitCallCancel` — `cancelled`, or the status the job had already ended with — and `splitCallExplain`) |
|
|
22
|
+
| Calls through a workspace | A call through a workspace is the server's, and outlives the request that starts it: `workspaceFunctionCall` starts it, waits up to the server's window (20 s; `startWaitMs` less), and polls a call still running to its end, each poll held until it ends or a while passes, however long its timeout lets it run — the server's ceiling, ten minutes locally, unless its `limits` ask for less; aborting `signal` asks the server to stop it, and `onStarted(id)` is told the id of one that runs on; a server that keeps no calls is called through `workspaceFunctionCallSync`, the sync route, which a caller may name itself. Apart: `workspaceFunctionStart(…, opts, { waitMs? })` → `{ id, status }` (`?wait=0` answers `running` at once), `workspaceFunctionCallStatus(url, repo, ws, id, opts, { waitMs?, signal? })` → `running`, `completed` with the result, `cancelled` naming who asked, or `lost`, `workspaceFunctionCallCancel(url, repo, ws, id, opts)`. The server keeps each call for its lifetime, an hour for a success or a cancel and a week for any other end: `workspaceCallList(url, repo, ws, opts, { function?, state?, before?, limit? })` — a page of them, the latest first, its `next` the next page's `before` (`FUNCTION_CALLS_PAGE_DEFAULT` 50, `…_MAX` 200), and `workspaceCallGet(url, repo, ws, id, opts)` — one whole: its function and package (`none` for a one-shot), its arguments (values by the objects they were stored as, datasets by the hashes they were pinned at), its drafts (each `{ record, mutation, args }`: the mutation's arguments by the object they were stored as, `none` for a mutation that takes none; the commit each drafted record stood at is its result's `drafts`), the inputs it read (a drafted record's, and an index of it, at the drafted state, once its drafts are laid), its limits, its value's object, its log tails and when it is forgotten (`expiresAt`); `objectGet` reads a value by its hash. A one-shot started as a call (`oneShotCall`) is kept, polled, cancelled and listed as a function's call is, its program the caller's, under its one-shot grant. A workspace keeps no call that another made: `internal`, "keeps no call". **Drafts**: a function's call may pass `drafts`, unsaved edits of records the function lists (its signature's `drafts`), each `{ record, mutation, args }` — a mutation call's beast2 arguments, as `workspaceRecordMutate` takes them — and `[]` for none. The server lays them in the order given, each by the mutation's own program over the state the one before it left, and the function reads each drafted record, and its indexes, at the state that leaves, and a dataset argument naming one as saved, at the state the drafts were laid over; nothing is saved — the record and what reads it stay as they were for every reader — while the call keeps its drafts, for whoever may read it, and each draft's mutation runs as a Save's does, an execution recorded and cached. The result's `inputs` name the drafted state's hash for such a record, and its `drafts` (`{ record, commit }`) the commit each stood at, so a caller that holds another commit knows its edits were laid over newer saves. A draft found wrong — a record the function does not list, an unknown mutation, arguments of the wrong number or type — is an `invalid` result naming the draft, and nothing runs; one whose mutation fails or outlasts the call's time ends the call `failed` or `timed_out` before the function runs, the result's `draftEnd` naming it, which `functionDraftEnded(result)` reads as `{ record, mutation, place, of, how }` (`place` and `of` Integers, `how` `.failed` or `.timed_out`), or `null`, and the first line of `stderr` saying so for people (`draft jobs.patch (2 of 3) failed`); a call whose timeout ends once its drafts are all laid ends `timed_out`, its `draftEnd` saying they were laid, which `functionDraftsOutlasted(result)` reads, its first line `drafts laid (3 of 3), and the call's timeout ended before its function ran`. Read the end from `draftEnd` through these, never from the line, which a function's own stderr may begin the same way. One that no longer fits the record, which changed under the edit, is the outcome `conflict` (`{ record, mutation, place, of, message }`, the draft named by its place among the call's), the kept call's state `conflict`. A draft is passed only where its mutation may be committed: one the server refuses throws `ApiError` `permission_denied`, its `path` `draft <record>.<mutation>`, and nothing is started |
|
|
23
23
|
| Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
|
|
24
24
|
| From East | `Platform` and the `platform_*` functions (`platform_dataset_get`, `platform_dataflow_execute`, …): the same calls as platform functions, for an East program that drives a server |
|
|
25
25
|
|
|
@@ -13,6 +13,8 @@ Part of the `e3` skill.
|
|
|
13
13
|
| `e3 rack policy [repo] show\|unset <glob>\|reset` | Inspect rules, remove an exact pattern, or delete the saved policy. | `e3 rack policy . show` |
|
|
14
14
|
| `e3 dataflow run <repo> <ws> --rack\|--no-rack\|--rack-only` | Select rack routing for this run; rack-only disables capacity spill but keeps infrastructure fallback: work waits two minutes for an unhealthy rack to come back, and is leased once more after a failed transfer, before it runs locally. ❗ Refused for remote URLs. | `e3 dataflow run . dev --rack` |
|
|
15
15
|
| `e3 watch <source> <repo> <ws> --start --rack` | Reuses a rack session across watch runs and refreshes capacity; a second Ctrl-C stops the watch at once. ❗ `--rack` and `--rack-only` require `--start`. | `e3 watch src/main.ts . dev --start --rack` |
|
|
16
|
+
| `e3 run <repo> <pkg.task> <inputs…> -o <out> --rack\|--rack-only` | Runs one task ad hoc on a rack, whatever the policy selects (a rule naming the task still sizes it); a split task's pieces and merges are leased, its planning stays here. Writes the cache and history a local run writes, says where it ran and why, and ends with the rack summary. A run the cache answers opens no rack session. Ctrl-C leaves its leases and stops the output's write, the file at `-o` left as it was (exit 130). ❗ Refused for remote URLs. | `e3 run . acme-forecast.train ./sales.beast2 -o ./model.beast2 --rack` |
|
|
17
|
+
| `e3 call <repo> [-w <ws>] <fn> [args…] --rack\|--rack-only` · `e3 call <repo> --ir <file> [args…] --rack` | Runs one call on a rack, whatever the policy selects, named by `fn:<name>` (an IR file's by its name): its own lease, sized as a whole task, its value back inline or, past 192 KiB, as an object. What no rack can run runs here; `[RACK] fn:<name> → <rack>` or `[LOCAL] fn:<name> (<reason>)` goes to stderr. Its time is its `fn:<name>` rule's `--timeout` when a rule sets one, longer or shorter than the CLI's ten minutes, however long: its drafts and its function share it, the function's part counted from when its lease is queued. A call no rack may run is judged before a rack session opens, and runs here at once, starting no hub. Ctrl-C leaves the lease; a value that came as Ctrl-C did is still printed. ❗ Refused for remote URLs. | `e3 call . -w dev train_once @.inputs.sales --rack` |
|
|
16
18
|
| `e3 rack status\|list\|remove <id-or-label>\|approve <id-or-label>\|stop` | Inspect fleet, revoke a rack, admit the boot a quarantined rack last reported, or stop the shared hub. Status never starts an absent hub, and lists how its last leases ended, each with its reason. | `e3 rack status` |
|
|
17
19
|
| `e3 rack config [--listen <host:port>] [--allow <ip,...>] [--idle-exit <min>] [--advertise-url <origin>] [--tls\|--no-tls] [--require-approval\|--no-require-approval]` | Configure the shared listener (off by default; HTTPS with the hub's own certificate unless `--no-tls`); every rack and data URL is built on `--advertise-url` when set; zero idle-exit disables automatic shutdown; `--require-approval` quarantines a rack after each reboot until `e3 rack approve` admits it (enrollment admits the first boot). ❗ Plain HTTP only on the loopback or a private address, with `--allow`. | `e3 rack config --listen 10.0.0.5:7331` |
|
|
18
20
|
|
|
@@ -47,7 +49,11 @@ though it was leased again). On a rack that admits by
|
|
|
47
49
|
its resources a split task's pieces are queued together, booked by what each
|
|
48
50
|
reserves, and the rack's guard keeps them from running its host out of memory;
|
|
49
51
|
a first run's piece reserves e3's estimate, as a local run does, so it is as
|
|
50
|
-
wide as a later run and measures no piece first.
|
|
52
|
+
wide as a later run and measures no piece first. A function call goes to a rack
|
|
53
|
+
only from `e3 call --rack`, and is fit for one as a task is: its runner's tier,
|
|
54
|
+
the environment it declares, and its program, scanned for host access (`[LOCAL]
|
|
55
|
+
fn:read_config (host-coupled:fs_read_file)`). A local API server's calls,
|
|
56
|
+
mutations and intake stay local. Stock runner
|
|
51
57
|
configurations qualify, and so does a task that declares a `python` or `node`
|
|
52
58
|
environment, with the project platform packages its runner names; commands, a
|
|
53
59
|
platform package with no environment (`non-stock-platform:<name>`) and
|
|
@@ -65,7 +71,15 @@ x86_64-manylinux_2_28: building` then `[ENV] 3f9a2b1c0d4e built in 48 s: 4
|
|
|
65
71
|
layers, 2.1 GB (1 new)`. No container runtime on either machine; nothing is
|
|
66
72
|
compiled for the rack; a second run builds nothing, and a change to the
|
|
67
73
|
project's code makes one small new layer. The layers are kept in the
|
|
68
|
-
repository's `rack/env-layers`, which `e3 repo gc` sweeps.
|
|
74
|
+
repository's `rack/env-layers`, which `e3 repo gc` sweeps. On a rack that
|
|
75
|
+
composes environments over its tier (`environments-over-tier`), a python
|
|
76
|
+
environment is built over the tier's image, without an interpreter and
|
|
77
|
+
without what the image holds the same (`[ENV] … over py-datascience: building`,
|
|
78
|
+
`[ENV] … built in 41 s: 2 layers, 6.2 MB (2 new), 31 distributions from the
|
|
79
|
+
tier`); one whose lock leaves out the image's Python is built whole, and so is
|
|
80
|
+
one whose build over the image fails. torch
|
|
81
|
+
stays in the tier for a member that asks for it through
|
|
82
|
+
`elaraai-east-py-datascience[torch,lightning]` and lists no torch of its own. ❗ What cannot go
|
|
69
83
|
stays local, with its reason on the task's `[LOCAL]` line: `environment:tools`
|
|
70
84
|
/ `environment:image` (its kind), `environment:windows-host` (a Windows
|
|
71
85
|
launching machine), `no-rack-for-environment:<tier>` (no rack composes them),
|
|
@@ -171,7 +185,7 @@ reach. A data URL lasts 15 minutes, which a transfer may use whole; an object is
|
|
|
171
185
|
at most 5 GiB. The hub serves one rack eight transfers at once (one more is told
|
|
172
186
|
to ask again), and reads an object from a byte on (`Range`), so a transfer a slow
|
|
173
187
|
link cut short resumes; it keeps a rack's idle connection for 75 seconds.
|
|
174
|
-
A hub and the e3 that calls it speak control protocol
|
|
188
|
+
A hub and the e3 that calls it speak control protocol 5 exactly:
|
|
175
189
|
a hub an older e3 started is asked to exit once idle, and `e3 rack stop` with
|
|
176
190
|
that e3 replaces it now. A rack's agent and its source speak rack protocol 3:
|
|
177
191
|
an agent of another protocol is refused by name each time it asks, and
|
|
@@ -186,11 +200,11 @@ sessions of the same canonical repository and e3 release.
|
|
|
186
200
|
|
|
187
201
|
| Signature | Description | Example |
|
|
188
202
|
| --- | --- | --- |
|
|
189
|
-
| `openRackRunner({ repoPath, workspace, storage, label, budget?, settings?, policy?, rackOnly?, unreachableWaitMs?, onPlacement?, onEnvironment? })` | Opens a runner/session; preserve the caller's existing budget and process settings, use `rackSlots` to extend orchestrator width, and close in `finally`. Under `rackOnly`, work waits `unreachableWaitMs` (two minutes) for an unhealthy rack before it runs locally. `onPlacement` is told where each task or
|
|
203
|
+
| `openRackRunner({ repoPath, workspace, storage, label, budget?, settings?, policy?, rackOnly?, selected?, unreachableWaitMs?, onPlacement?, onEnvironment? })` | Opens a runner/session; preserve the caller's existing budget and process settings, use `rackSlots` to extend orchestrator width, and close in `finally`. The runner's `runDetached` leases a call (`{ functionName }` names it) for the time its spec's limits give it; `rackCallTimeoutMs(policy, name)` is the time a `fn:<name>` rule gives a call, or null. `selected` takes every task and call to a rack whatever the policy's rules and `enabled` say, as `e3 run`/`e3 call --rack` do. Under `rackOnly`, work waits `unreachableWaitMs` (two minutes) for an unhealthy rack before it runs locally. `onPlacement` is told where each task, unit or call runs; `onEnvironment` that an environment's layers are being built for a rack, and what the build came to; `unattached` says why no session opened. | `const rack = await openRackRunner({ repoPath, workspace: 'dev', storage, label: 'run', budget });` |
|
|
190
204
|
| `createDataflowRunnerFactory(options?)` | Pass to `createServer({ dataflowRunner })`; one session per API dataflow, using the server's shared budget. Explicit server opt-in overrides policy.enabled; rules still apply. | `const server = await createServer({ reposDir: './repos', dataflowRunner: createDataflowRunnerFactory() });` |
|
|
191
205
|
|
|
192
206
|
The API server's optional rack package is needed only for
|
|
193
207
|
`e3-api-server --repos ./repos --rack`. Portable handlers receive a runner
|
|
194
208
|
factory and do not import hub code. `tools` and `image` environments on a
|
|
195
|
-
rack,
|
|
196
|
-
embedded UI integration are deferred.
|
|
209
|
+
rack, a local API server's calls on a rack, split calls on a rack, automatic
|
|
210
|
+
reconnect after hub restart and embedded UI integration are deferred.
|
|
@@ -9,6 +9,7 @@ Secondary, interval and area, series and summary, nested series, and entries ind
|
|
|
9
9
|
- Series and summary indexes — `index.series`, `index.summary`
|
|
10
10
|
- Nested series indexes — `index.nested`
|
|
11
11
|
- Entries indexes — `index.entries(field)`
|
|
12
|
+
- Reading an index inside a task or a function — `e3.index(record.indexes.<name>)`, and a reader for each kind: `of`, `between`, `overlapping`, `events`, `totals`, `within`
|
|
12
13
|
|
|
13
14
|
## Secondary indexes — `index.on([...fields])`, `index.key(fn)`, `index.keys(fn)`
|
|
14
15
|
|
|
@@ -259,10 +260,10 @@ const assemblies = e3.record('assemblies', AssembliesType, {
|
|
|
259
260
|
|
|
260
261
|
| Builder (`index.nested.…`) | A field holding |
|
|
261
262
|
|---|---|
|
|
262
|
-
| `items({ from, to })`, `items({ at })` | `Dict<
|
|
263
|
+
| `items({ from, to })`, `items({ at })` | `Dict<K, Struct>`: things people drag, by id, of any key type but a DateTime; their times are DateTime or Option<DateTime> fields — no start files an item nowhere, no end (or one at or before its start) makes it an instant. `{ rollup: true }` files each under every ancestor too |
|
|
263
264
|
| `values({ rollup? })` | `Dict<DateTime, V>`: a value per instant. `rollup` keeps every parent's subtotals |
|
|
264
|
-
| `rows({ … })` | `Dict<
|
|
265
|
-
| `recursive()` | `Dict<
|
|
265
|
+
| `rows({ … })` | `Dict<K, Struct>`: child rows, keyed by any type but a DateTime (a role, a number, a String), their own collections declared the same way. `rows({})` declares child rows that nest nothing in time: the index names them among its ids and files nothing of them |
|
|
266
|
+
| `recursive()` | `Dict<K, Self>`: more rows like the one listing it, to any depth, keyed as the rows listing it are |
|
|
266
267
|
|
|
267
268
|
TypeScript types `collections` field by field from the record's type alone: a
|
|
268
269
|
misnamed time, or a builder on the wrong kind of field, is an error where it is
|
|
@@ -272,11 +273,22 @@ series index, which takes no `on` and no `value`.
|
|
|
272
273
|
|
|
273
274
|
**What it files.** Every entry is keyed `{ collection, row, path, run, level,
|
|
274
275
|
tile, item }` (e3-types' `NestedEntryKeyType`): the collection's place, the
|
|
275
|
-
field names from the row down (`members` then `load`); the row's key; the
|
|
276
|
-
keys from the row down to the node (
|
|
277
|
-
run; the level and tile; and the item. A place and a path are held as a key
|
|
276
|
+
field names from the row down (`members` then `load`); the row's key; the child
|
|
277
|
+
rows' keys from the row down to the node (none for the row, `m-3` for a member);
|
|
278
|
+
the run; the level and tile; and the item. A place and a path are held as a key
|
|
278
279
|
holds a list, folded from the last: `.step (key="members", rest=.step
|
|
279
|
-
(key="load", rest=.end))`.
|
|
280
|
+
(key="load", rest=.end))`.
|
|
281
|
+
|
|
282
|
+
A child row's key and an item's id are held in their own types, never as text.
|
|
283
|
+
They are the index's **ids**: one variant with a case for each collection of
|
|
284
|
+
child rows and of items, named by its place's field names joined with a dot
|
|
285
|
+
(`members`, `members.tasks`), each holding that collection's key type. So a
|
|
286
|
+
member's path is `.step (key=.members "m-3", rest=.end)`, a job's item
|
|
287
|
+
`.id .jobs "j-0042"`, and a crew's hours by role, a variant, sit under
|
|
288
|
+
`.step (key=.hours .operator, rest=.end)`. Keys sort in their types' own order.
|
|
289
|
+
A declared field's name holds no dot.
|
|
290
|
+
|
|
291
|
+
Each run of a node is one contiguous range a level:
|
|
280
292
|
|
|
281
293
|
| Run | Holds |
|
|
282
294
|
|---|---|
|
|
@@ -374,3 +386,182 @@ an insert, one whose value changed an update by the value's patch — and the
|
|
|
374
386
|
lean rows where it changed the row's other fields. After a patch, an edit or a
|
|
375
387
|
reduce alike, a maintained entries index, its lean rows and the indexes beside
|
|
376
388
|
them are what `e3 reindex` rebuilds, byte for byte.
|
|
389
|
+
|
|
390
|
+
## Reading an index inside a task or a function — `e3.index(record.indexes.<name>)`
|
|
391
|
+
|
|
392
|
+
A page reads an index through the API (`index=<name>`, `Data.bindPaged`). A
|
|
393
|
+
task, a stream task or a function reads one by taking it as an input:
|
|
394
|
+
`e3.index(record.indexes.<name>)` in its `inputs`. The body is given the
|
|
395
|
+
index's entries — the collection the record's state holds, `Dict<{ ik, k }, P>`,
|
|
396
|
+
opened lazily and frozen as any input is — where a body given the record gets
|
|
397
|
+
its rows in the record's own order, and reads every one to find a few.
|
|
398
|
+
|
|
399
|
+
```typescript
|
|
400
|
+
const JobType = StructType({ title: StringType, press: StringType });
|
|
401
|
+
const jobs = e3.record('jobs', DictType(StringType, JobType), {
|
|
402
|
+
initial: new Map(),
|
|
403
|
+
indexes: (index) => ({ jobs_by_press: index.on(['press']) }),
|
|
404
|
+
mutations: (mutation) => ({ patch: mutation.patch() }),
|
|
405
|
+
});
|
|
406
|
+
const byPress = e3.index(jobs.indexes.jobs_by_press);
|
|
407
|
+
|
|
408
|
+
// A function: how many jobs a press holds, read from the index alone.
|
|
409
|
+
const jobsOnPress = e3.function('jobs_on_press', { inputs: [byPress], args: [StringType], output: IntegerType },
|
|
410
|
+
($, jobsByPress, press) => jobsByPress.prefix({ ik: press }).size());
|
|
411
|
+
|
|
412
|
+
// More of a row than the index carries: the record beside it, read by the entry's `k`.
|
|
413
|
+
const titlesOnPress = e3.function('titles_on_press', { inputs: [byPress, jobs], args: [StringType], output: DictType(StringType, StringType) },
|
|
414
|
+
($, jobsByPress, jobs, press) => {
|
|
415
|
+
const titles = $.let(new Map(), DictType(StringType, StringType));
|
|
416
|
+
$.for(jobsByPress.prefix({ ik: press }), ($, _carried, key) => { $(titles.insert(key.k, jobs.get(key.k).title)); });
|
|
417
|
+
return titles;
|
|
418
|
+
});
|
|
419
|
+
|
|
420
|
+
// A task takes it the same way. Its body's parameter is the input's type.
|
|
421
|
+
const watched = e3.input('watched', StringType, variant('value', 'P1'));
|
|
422
|
+
const watchedJobs = e3.task('watched_jobs', [byPress, watched],
|
|
423
|
+
East.function([byPress.type, StringType], IntegerType, ($, jobsByPress, press) => jobsByPress.prefix({ ik: press }).size()));
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
**The entries.** `ik` is the index's key and `k` the row's, `ik` first, so
|
|
427
|
+
every entry of one index key is one contiguous run, in the record's key order,
|
|
428
|
+
and a range of index keys is one run too. `P` is what the index carries. The
|
|
429
|
+
input's `type` is the type the state holds, so a task's body declares its
|
|
430
|
+
parameter by it:
|
|
431
|
+
|
|
432
|
+
| Index | `ik` | `k` | Carries |
|
|
433
|
+
|---|---|---|---|
|
|
434
|
+
| `index.on(['press'])` | the field's value | the row's key | nothing (Null), or its `value` |
|
|
435
|
+
| `index.on(['press', 'due'])` | `{ press, due }` | the row's key | nothing, or its `value` |
|
|
436
|
+
| `index.key(fn)`, `index.keys(fn)` | what `fn` returns, or an element of its Set | the row's key | nothing, or its `value` |
|
|
437
|
+
| `index.entries('byService')` | the field's key | the row's key | that key's value |
|
|
438
|
+
| `index.interval`, `index.area`, `index.series` | the generated key above | the row's key | nothing for an interval; the row for an area or a series |
|
|
439
|
+
| `index.summary` | `{ on, key, tile }` | Null: an entry stands for every row of its key | `{ count, ms }` |
|
|
440
|
+
| `index.nested` | `{ collection, row, path, run, level, tile, item }` | the row's key | the item, the totals or the subtotal |
|
|
441
|
+
|
|
442
|
+
It takes the indexes e3 keeps beside one as well: a nested series index's span
|
|
443
|
+
(`machines.indexes.machines_in_time.nested.span`, keyed `{ collection, tile }`,
|
|
444
|
+
its `k` Null) and an entries index's lean rows
|
|
445
|
+
(`stations.indexes.stations_by_service.entries.rows`). Beside a nested series
|
|
446
|
+
index or an entries index, an index that carries nothing or the row carries the
|
|
447
|
+
lean row. Its `type` says so, and so does TypeScript: the body's parameter, and
|
|
448
|
+
what a read of it gives, are typed as the row.
|
|
449
|
+
|
|
450
|
+
**A reader for each kind.** A plain index's rows under a key are one run, which
|
|
451
|
+
`prefix` reads. The indexes a planning board reads are not one run: a series
|
|
452
|
+
index files each event once, at the level its length fits, an interval index
|
|
453
|
+
answers a window with a run of starts and one a tree node above the window's
|
|
454
|
+
first instant, an area index with a cell a level, and each files more under
|
|
455
|
+
those runs than a window holds. So an index input arrives, in an `e3.function`
|
|
456
|
+
with a spec and an `e3.streamTask`, with its kind's readers on it — it is still
|
|
457
|
+
the entries, and `prefix`, `range`, `size` and a keyed read work as before — and
|
|
458
|
+
`e3.index(def).read(entries)` gives the same over any other expression of the
|
|
459
|
+
index's type, a task's own function's parameter among them.
|
|
460
|
+
|
|
461
|
+
```typescript
|
|
462
|
+
// The hours a press is booked for in a window, from its series index alone.
|
|
463
|
+
const hoursOnPress = e3.function('hours_on_press', {
|
|
464
|
+
inputs: [e3.index(jobs.indexes.jobs_by_press_week)],
|
|
465
|
+
args: [StringType, DateTimeType, DateTimeType],
|
|
466
|
+
output: FloatType,
|
|
467
|
+
}, ($, jobsByPress, press, from, to) => {
|
|
468
|
+
const sheets = $.let(0.0);
|
|
469
|
+
jobsByPress.events({ key: press, from, to }).each($, ($, job) => { $.assign(sheets, sheets.add(job.sheets)); });
|
|
470
|
+
return sheets.divide(8000.0);
|
|
471
|
+
});
|
|
472
|
+
|
|
473
|
+
// The same rows as a Dict by the job's own key.
|
|
474
|
+
const held = jobsByPress.events({ key: press, from, to }).collect($);
|
|
475
|
+
|
|
476
|
+
// In a task, whose body is its own East function.
|
|
477
|
+
const byPress = e3.index(jobs.indexes.jobs_by_press_week);
|
|
478
|
+
East.function([byPress.type, StringType], IntegerType, ($, entries, press) =>
|
|
479
|
+
byPress.read(entries).events({ key: press, from, to }).collect($).size());
|
|
480
|
+
```
|
|
481
|
+
|
|
482
|
+
A reader answers a **read**: the rows its words say and no others, each once,
|
|
483
|
+
in the index's own order.
|
|
484
|
+
|
|
485
|
+
| Form | Gives | Holds in memory |
|
|
486
|
+
|---|---|---|
|
|
487
|
+
| `read.each($, ($, carried, key) => { … })` | the body run once a row: what the index carries for it, and the row's key. It takes no label and runs to its end: skip a row with `$.if`, leave the function with `$.return` | a segment of the index at a time |
|
|
488
|
+
| `read.collect($)` | a new Dict by the row's key of what `each` gives | the rows it collects |
|
|
489
|
+
|
|
490
|
+
| Declared by | Reader | Gives, each row once | Runs read |
|
|
491
|
+
|---|---|---|---|
|
|
492
|
+
| `index.on`, `index.key`, `index.keys` | `of(key)`: the rows under an index key, or under its first fields (`{ owner }` of `['owner', 'due']`). `between({ from?, to? })`: the rows from one index key up to another | what the index carries, and the row's key | one |
|
|
493
|
+
| `index.interval` | `overlapping({ on?, from, to })`: the rows that overlap `[from, to)`. `unset({ on? })`: the rows with no start | what the index carries, and the row's key | at most 55, in the index's order: the starts in the window, then one a tree node above its first instant |
|
|
494
|
+
| `index.series` | `events({ on?, key, from, to })`: the events on `key` that overlap `[from, to)` | the row, and its key | one a level, from the index's tile to the year, in the index's order of its levels |
|
|
495
|
+
| `index.summary` | `totals({ on?, key, from, to })`: each tile of `key` that overlaps `[from, to)` | `{ count, ms }`, and the tile's start in place of a row's key | one |
|
|
496
|
+
| `index.area` | `within({ on?, x, y, width, height })`: the rows whose box meets the window. `unplaced({ on? })`: the rows with no place | the row, and its key | at most 266, as a page's read of a viewport, in the index's order: a cell before the cells inside it |
|
|
497
|
+
| `index.entries` | `of(key)`: that key's entry on every row that holds one | the entry's value, and the row's key | one |
|
|
498
|
+
| `index.nested` | of one node of a row's tree, below | below | one, or one a level |
|
|
499
|
+
|
|
500
|
+
A nested series index's span and an entries index's lean rows are read as
|
|
501
|
+
plain indexes are, with `of` and `between`. A span's entry stands for every
|
|
502
|
+
row of its key and names none, so its read gives the index key,
|
|
503
|
+
`{ collection, tile }`, in a row's key's place.
|
|
504
|
+
|
|
505
|
+
**A nested series index's readers.** A nested series index files a row's tree
|
|
506
|
+
a node at a time, each node's entries in runs, so each reader reads one node:
|
|
507
|
+
`{ collection, row, path?, from, to }` — the collection's place (its field
|
|
508
|
+
names from the row down, `['units', 'jobs']`, a recursive field adding none),
|
|
509
|
+
the row's key, and the Dict keys from the row down to the node, the row itself
|
|
510
|
+
where `path` is left out. TypeScript offers the places the index declares, and
|
|
511
|
+
types what a read gives by the place.
|
|
512
|
+
|
|
513
|
+
```typescript
|
|
514
|
+
const inTime = e3.index(lines.indexes.lines_in_time);
|
|
515
|
+
const sheetsOnUnit = e3.function('sheets_on_unit', {
|
|
516
|
+
inputs: [inTime], args: [StringType, StringType, DateTimeType, DateTimeType], output: FloatType,
|
|
517
|
+
}, ($, linesInTime, line, unit, from, to) => {
|
|
518
|
+
const sheets = $.let(0.0);
|
|
519
|
+
linesInTime.items({ collection: ['units', 'jobs'], row: line, path: [variant('units', unit)], from, to })
|
|
520
|
+
.each($, ($, job) => { $.assign(sheets, sheets.add(job.sheets)); });
|
|
521
|
+
return sheets;
|
|
522
|
+
});
|
|
523
|
+
// The line's mean of its units' load a day, which the index keeps.
|
|
524
|
+
linesInTime.subtotals({ collection: ['units', 'load'], row: line, rollup: 'mean', from, to }).collect($);
|
|
525
|
+
```
|
|
526
|
+
|
|
527
|
+
| Reader | Gives, each once | Needs | Runs read |
|
|
528
|
+
|---|---|---|---|
|
|
529
|
+
| `items(node)` | the node's own items that overlap `[from, to)`, each with its id, in its collection's own key type; or its values in the window, each with its instant | | one a level for items timed `{ from, to }`; one for values and items timed `{ at }` |
|
|
530
|
+
| `subtree(node)` | the items of every node beneath the node that overlap the window, each with `{ path, id }`: the keys from the node down to the one holding it — a path, `end` or a `step` and the rest — and its id, each an id of the index's (`NestedUnderType(index)` is the type) | `items(…, { rollup: true })` | one a level |
|
|
531
|
+
| `totals(node)` | the node's totals of the collection in each summary tile that overlaps the window — `{ count, ms, values }`, `values` the sum, least, greatest and latest of numbers — with the tile's start | `summary` | one |
|
|
532
|
+
| `subtreeTotals(node)` | the same, of the items beneath the node | `rollup: true` and `summary` | one |
|
|
533
|
+
| `subtotals({ …node, rollup, level? })` | a parent's subtotal of its children's values in each tile of `level` that overlaps the window, with the tile's start. `rollup` is written as the collection declares it; `level` is the index's tile where left out, or a coarser one | `values({ rollup })` | one |
|
|
534
|
+
|
|
535
|
+
A node that holds values of its own draws them and keeps no subtotal, so its
|
|
536
|
+
`subtotals` is empty; a row or a node that is not there reads as empty too. A
|
|
537
|
+
place the index does not file, a collection not rolled up, a rollup it does not
|
|
538
|
+
keep, a level finer than its tile and totals of an index with no `summary` are
|
|
539
|
+
each refused where the reader is called, naming what the index does keep.
|
|
540
|
+
|
|
541
|
+
| Rule | |
|
|
542
|
+
|---|---|
|
|
543
|
+
| **Exact** | A series, a nested series or an area index files more under the runs a window reads than the window holds, and an event past a year under each year it touches: the reader tests each entry, and gives it once. A plain index whose rows fan out — `index.keys`, a Set field — gives a row once across the keys of a read, and holds the keys of the rows it gave to do so |
|
|
544
|
+
| **The window's edges** | `[from, to)`, for every reader with a span of time: an event that ends at `from` is outside; an instant at `from` is inside and one at `to` outside; an event with no end, or one at or before its start, is an instant. A window that ends at or before its start holds no row. A row with no start is in no window: a series index files it nowhere, and an interval index gives it by `unset`. An area's window holds its edges: a box that touches it is in it, and one of a negative size holds none. A zero is one edge whatever its sign: a width of `-0.0` is a width of none, as it is to a page |
|
|
545
|
+
| **Order** | A read gives its rows in the index's own order: one of several runs reads them in the order the index holds them, so the rows come as a walk of the index meets them. Across a series index's levels that is not time order — the index orders a level by its name: day, month, quarter, week, year — so sort what is collected where order matters |
|
|
546
|
+
| **Leading fields** | An index declared with `on` takes every leading field, exactly, as `on: { status: "open" }`, in every reader: TypeScript requires it, `unset` and `unplaced` included. One left out, one that leads nothing, or a value of another type is refused where the reader is called, naming the fields |
|
|
547
|
+
| **Its kind's readers** | TypeScript offers an index the readers of its kind. `e3.index(def).read` refuses another index's entries, naming both: by their type, and, where two indexes' entries have one type, by the readers the expression already carries — a body's parameter for the other index, or what the other's `read` was given. An expression that carries none is known by its type alone, so in a task's own function two parameters of one type are the author's to keep apart. `unset` and `unplaced` are refused for an index whose start, or place, is no Option field: every row then has one |
|
|
548
|
+
| **More of the row** | Where an index carries less than the row, take the record as an input beside it and read the row by its key, at the same state. Those keyed reads come in the index's order, not the record's, so an index that carries the row is the one to read many rows through |
|
|
549
|
+
| **Cost** | A read costs its runs, each a range read: its two ends through the pager's cache and what lies between. It never costs the index. Over 24,000 jobs: a key's rows decode 1 of the index's 20 segments, a week of an interval 12 of 61, a week of a press's series 2 of 20, a room of an area 4 of 19; over forty lines of four units, each read of one node of a nested series index 1 of 31 — the same on east-node, east-c and east-py |
|
|
550
|
+
| **The page's runs** | Each reader reads the runs a page's reader of its kind searches, less the tiles and the margin a page reads beyond a window to pan into |
|
|
551
|
+
|
|
552
|
+
**What a read costs.** A body reads the entries with `prefix` and `range`, by
|
|
553
|
+
key, by size and by a loop, as it reads any lazily opened Dict: a run costs the
|
|
554
|
+
segment each of its ends lands in and the segments between them, never the
|
|
555
|
+
index, and the size of a run costs its two ends alone. A body that uses the
|
|
556
|
+
input any other way decodes the index whole, as it would any input; a runner's
|
|
557
|
+
`-v` says which, an input at a time.
|
|
558
|
+
|
|
559
|
+
| Rule | |
|
|
560
|
+
|---|---|
|
|
561
|
+
| **One state** | The inputs of one record — the record, and any of its indexes — are read at one state of it, so a body never meets an index entry for a row it reads otherwise |
|
|
562
|
+
| **A task** | is decided again whenever its record changes, and runs again when the index it reads did: its execution is keyed by the index, so a write that leaves the index as it was is served by the cache |
|
|
563
|
+
| **A function** | reads the index as the record is stored when the call starts. Its result's `inputs` name the record, the index and the hash the record was pinned at, which its other inputs of that record name too |
|
|
564
|
+
| **Drafts** | A function may list the record in `drafts` whether it reads the record or an index of it: a call's drafts are laid over the record, and its indexes hold them as they would after the Save |
|
|
565
|
+
| **No state yet** | A record with no state makes a task wait and a call `invalid`, as it does for the record as an input. So does a state that holds no such index — one written outside the doors that build every index its record declares — the task's status naming the record and the index, and the run going on without it |
|
|
566
|
+
| **The package** | `e3.package` collects the index's record |
|
|
567
|
+
| **Refused where declared** | Anything but an index its record carries by name, naming what was given; a task's parameter that is not the input's `type`; an index inside `e3.partition` (every piece reads it whole: pass the `e3.index` itself); an index as an input of `e3.customTask`, whose command reads files; a function on the `custom` runtime, which takes no inputs |
|
|
@@ -226,8 +226,13 @@ and the task runs anywhere, with no python or Node package where it runs. To
|
|
|
226
226
|
add one, build it and list it under the name the app imports; e3 finds the
|
|
227
227
|
member in the uv / npm workspace and exports it itself (the **e3** skill).
|
|
228
228
|
|
|
229
|
-
**Build order / semantics.**
|
|
230
|
-
|
|
229
|
+
**Build order / semantics.** A python member's environment installs the
|
|
230
|
+
member's own locked dependencies, the runtime, and the stock platform packages
|
|
231
|
+
its task's runner names, each with the extras the workspace declares it with —
|
|
232
|
+
not what the workspace root or another member lists, nor the member's
|
|
233
|
+
development groups, so a heavy dependency costs only the members that use it;
|
|
234
|
+
node members must be
|
|
235
|
+
`tsc`-built before export
|
|
231
236
|
(the root `build` runs `--workspaces` first — both the `./platform` and the
|
|
232
237
|
`./functions` entries are read from `dist/`); C tools must be `make`-built
|
|
233
238
|
before `npm run start`. Add more packages later with the same flags — each new
|