@elaraai/east-codex-plugin 0.0.0-stage → 1.0.89
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.build/daemon/east-py-lsp.js +568 -0
- package/.build/daemon/lsp.js +3737 -0
- package/.build/daemon/server.js +3535 -0
- package/.build/hooks/codex-tools.js +584 -0
- package/.build/hooks/diagnose-bash.js +399 -0
- package/.build/hooks/diagnose.js +347 -0
- package/.build/hooks/pre-read.js +100 -0
- package/.build/hooks/pre-write.js +247 -0
- package/.build/hooks/session-start.js +263 -0
- package/.build/hooks/subagent-start.js +212 -0
- package/.build/mcp/server.js +23577 -0
- package/.build/status/report.js +2163 -0
- package/.codex-plugin/plugin.json +36 -0
- package/.lsp.json +21 -0
- package/.mcp.json +11 -0
- package/LICENSE.md +682 -0
- package/README.md +192 -2
- package/hooks/hooks.json +51 -0
- package/index.json +44926 -0
- package/package.json +36 -4
- package/skills/e3/SKILL.md +841 -0
- package/skills/e3-create/SKILL.md +265 -0
- package/skills/e3-ui/SKILL.md +2260 -0
- package/skills/e3-ui-cli/SKILL.md +332 -0
- package/skills/east/SKILL.md +690 -0
- package/skills/east-contribute/SKILL.md +348 -0
- package/skills/east-design/SKILL.md +233 -0
- package/skills/east-node-io/SKILL.md +143 -0
- package/skills/east-node-std/SKILL.md +289 -0
- package/skills/east-ontology/SKILL.md +181 -0
- package/skills/east-project/SKILL.md +189 -0
- package/skills/east-py/SKILL.md +2023 -0
- package/skills/east-py-datascience/SKILL.md +280 -0
- package/skills/east-py-io/SKILL.md +186 -0
- package/skills/east-py-std/SKILL.md +301 -0
- package/skills/east-ui/SKILL.md +1529 -0
|
@@ -0,0 +1,841 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: e3
|
|
3
|
+
description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui()."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# East Execution Engine (e3)
|
|
7
|
+
|
|
8
|
+
e3 runs East programs as a durable dataflow over content-addressed data: every
|
|
9
|
+
value is an object named by its hash, every task execution is cached by its
|
|
10
|
+
inputs, and a change re-runs only what it reaches. It is the platform's
|
|
11
|
+
**Compute** layer — and East + e3 solutions are decision-oriented: a dataflow
|
|
12
|
+
exists to put auditable evidence behind a business decision, not to move data
|
|
13
|
+
for its own sake.
|
|
14
|
+
|
|
15
|
+
## Before writing code — search the example index
|
|
16
|
+
|
|
17
|
+
Every East API has a tested example in the plugin's index — the index IS the
|
|
18
|
+
API reference, printed from each example's IR in TypeScript or python. Before
|
|
19
|
+
writing or changing East code:
|
|
20
|
+
|
|
21
|
+
1. Call `search_east_examples` for each capability you
|
|
22
|
+
are about to use — `language: "python"` for east-py, `"typescript"`
|
|
23
|
+
otherwise. Summaries come back first: id, signature, the inputs and the
|
|
24
|
+
expected result, a few hundred bytes each.
|
|
25
|
+
2. Fetch the one or two that match with `get_east_example`
|
|
26
|
+
and pattern your code on them.
|
|
27
|
+
3. Do not read `node_modules/@elaraai/**` or `*.examples.ts` files wholesale,
|
|
28
|
+
and do not reason from `.d.ts` signatures: the index holds the same
|
|
29
|
+
programs, exact and far cheaper, and the signatures omit the runtime rules
|
|
30
|
+
that make East code correct.
|
|
31
|
+
|
|
32
|
+
Nothing is injected for you; the search is the step.
|
|
33
|
+
|
|
34
|
+
## Quick Start
|
|
35
|
+
|
|
36
|
+
```typescript
|
|
37
|
+
// src/index.ts
|
|
38
|
+
import { East, StringType, variant } from '@elaraai/east';
|
|
39
|
+
import e3 from '@elaraai/e3';
|
|
40
|
+
|
|
41
|
+
const name = e3.input('name', StringType, variant('value', 'World'));
|
|
42
|
+
const greet = e3.task('greet', [name],
|
|
43
|
+
East.function([StringType], StringType, ($, n) => East.str`Hello, ${n}!`));
|
|
44
|
+
|
|
45
|
+
const pkg = e3.package('hello', '1.0.0', greet); // dependencies are collected
|
|
46
|
+
await e3.export(pkg, '/tmp/hello.zip');
|
|
47
|
+
export default pkg;
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
e3 repo create .
|
|
52
|
+
e3 workspace deploy . dev --from-source ./src/index.ts # or --from-zip /tmp/hello.zip
|
|
53
|
+
e3 dataflow run . dev
|
|
54
|
+
e3 dataset get . dev.greet # flat path: <ws>.<name>
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## Decision Tree
|
|
58
|
+
|
|
59
|
+
```
|
|
60
|
+
What do you need?
|
|
61
|
+
├─ Author a package (@elaraai/e3)
|
|
62
|
+
│ ├─ An input dataset → e3.input(name, type, source?) source: variant('value', v) | variant('file', path)
|
|
63
|
+
│ ├─ A task — two questions: does the output fit in memory, can the work be split over an input? (see "Which task?")
|
|
64
|
+
│ │ ├─ Fits, built in one pass → e3.task(name, [inputs], fn, config?) — returns the output
|
|
65
|
+
│ │ ├─ Too large, or built as you go → e3.streamTask(name, { inputs, output }, fn) — emits into e3.output.array | set | dict | fold
|
|
66
|
+
│ │ └─ Split over an input, in pieces → wrap that input: e3.partition(dataset, { by? })
|
|
67
|
+
│ ├─ A shell command → e3.customTask(name, [inputs], outputType, command)
|
|
68
|
+
│ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name, persists nothing
|
|
69
|
+
│ ├─ Audited state → e3.record + e3.mutation.reduce | edit | patch, e3.recordIndex
|
|
70
|
+
│ ├─ A record's type changes → e3.migration.value | rows | rekey(name, record, fn, { after? })
|
|
71
|
+
│ ├─ A UI task → ui(name, [inputs], fn) from @elaraai/e3-ui (the e3-ui skill)
|
|
72
|
+
│ ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
|
|
73
|
+
│ └─ Bundle and export → e3.package(name, version, ...items); e3.export(pkg, zipPath)
|
|
74
|
+
├─ Operate it from the shell → the e3 CLI (below): deploy, run, read and write datasets, mutate records, gc
|
|
75
|
+
└─ Drive it from code
|
|
76
|
+
├─ A server, over HTTP → @elaraai/e3-api-client
|
|
77
|
+
├─ A local repository → @elaraai/e3-core
|
|
78
|
+
├─ Serve repositories → e3-api-server --repos <dir>, or createServer()
|
|
79
|
+
└─ e3 in a browser, no server → @elaraai/e3-web: serveUnits, serveE3, createWebE3 ("Running e3 in a browser")
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
## Authoring a package
|
|
83
|
+
|
|
84
|
+
### Inputs — `e3.input(name, type, source?)`
|
|
85
|
+
|
|
86
|
+
A dataset at `.inputs.<name>`, addressed `<ws>.<name>` from the CLI. The third
|
|
87
|
+
argument is a source variant — a bare value is refused:
|
|
88
|
+
|
|
89
|
+
| Source | Meaning |
|
|
90
|
+
|---|---|
|
|
91
|
+
| `variant('value', v)` | Inline, carried in the package. |
|
|
92
|
+
| `variant('file', path)` | A beast2 file on the machine that **deploys**. The package carries the path (relative paths resolve against the working directory at export); deploy takes the file in as the value. |
|
|
93
|
+
| omitted | Unassigned until something sets it. |
|
|
94
|
+
|
|
95
|
+
```typescript
|
|
96
|
+
const name = e3.input('name', StringType, variant('value', 'default'));
|
|
97
|
+
const count = e3.input('count', IntegerType);
|
|
98
|
+
const table = e3.input('table', ArrayType(RowType), variant('file', './deliveries/TABLE.beast2'));
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
An inline value is a small constant written in the source. Rows computed on
|
|
102
|
+
the host as the package is built — a `Map` filled in a loop, a generated
|
|
103
|
+
array — would ride the package as a snapshot, so a dataset that is generated
|
|
104
|
+
or large is made where data is made: a task's output (an `e3.task` over a
|
|
105
|
+
small authored count, as e3-ui's paged examples generate their rows), a
|
|
106
|
+
`file` source, or a record.
|
|
107
|
+
|
|
108
|
+
A `file` source:
|
|
109
|
+
- is checked against the declared type at `e3.export` (drift is a build error
|
|
110
|
+
naming the input and the first differing field) and again at deploy, before
|
|
111
|
+
the workspace is touched;
|
|
112
|
+
- is hashed first, so a delivery the store already holds costs that one read;
|
|
113
|
+
- is taken in without being read whole or modified — so replace a delivery with
|
|
114
|
+
a new file, never edit it in place. A collection is taken in on the runners —
|
|
115
|
+
east-c, or east-node where there is no east-c — in pieces of its segments, as
|
|
116
|
+
many at once as the budget allows: each row is walked, the bytes the Writer
|
|
117
|
+
writes for it kept as they stand and any other row written again, so the
|
|
118
|
+
delivery is stored as the Writer's value whichever writer wrote it, and a new
|
|
119
|
+
delivery stores only the segments that changed. A deploy or an upload
|
|
120
|
+
stopped part way takes up again from the pieces it finished. Any other value
|
|
121
|
+
is taken in by reflink, hard link or one copy;
|
|
122
|
+
- may hold no segment of more than 64 MiB, the most a collection is read in at
|
|
123
|
+
once. A value encoded whole is one such segment, and so can be a segment of
|
|
124
|
+
wide rows from an older Writer, which bounded a segment by its element count
|
|
125
|
+
alone; either is refused, naming the fix: write it again with a current
|
|
126
|
+
Writer, whose segments stay under 8 MiB;
|
|
127
|
+
- is read on the machine that runs `e3 workspace deploy`, local repository or
|
|
128
|
+
not: against a server the CLI checks every delivery first, then streams each
|
|
129
|
+
after the deploy. A deploy made through the API leaves it unset;
|
|
130
|
+
`--skip-file-sources` defers it and prints the
|
|
131
|
+
`e3 dataset set <repo> <ws>.<name> --from-file <path>` that completes each.
|
|
132
|
+
|
|
133
|
+
Every door into a dataset — `dataset set`, the API, the transfer, a file
|
|
134
|
+
adopt — refuses bytes whose type is not **exactly** the declared type, since
|
|
135
|
+
runners decode by it.
|
|
136
|
+
|
|
137
|
+
### Tasks — `e3.task(name, inputs, fn, config?)`
|
|
138
|
+
|
|
139
|
+
Runs an East function once, as one unit, and returns its output at
|
|
140
|
+
`.tasks.<name>.output`; chain tasks with `task.output`. A collection it returns
|
|
141
|
+
is written segment by segment, but the body builds it whole: an output too large
|
|
142
|
+
to hold is emitted instead (`e3.streamTask`). `config` is
|
|
143
|
+
`{ runner?, environment?, role? }` — `role` is what e3-ui's `ui()` sets.
|
|
144
|
+
|
|
145
|
+
Task inputs, on every runtime (a mutation's state included):
|
|
146
|
+
- are **deeply frozen**: mutating one raises `cannot mutate a frozen value (task
|
|
147
|
+
inputs are immutable) — copy first`; derive a changed value from `.copy()`;
|
|
148
|
+
- open **lazily**, whatever their size: size, iteration and keyed gets read a
|
|
149
|
+
segment at a time, with the same semantics, and an operation that needs the
|
|
150
|
+
whole value decodes it once, when it first does. A runner's `decode: 'whole'`
|
|
151
|
+
decodes every input before the program runs instead — for a program whose
|
|
152
|
+
reads land at random across more segments than the runner keeps, which `-v`
|
|
153
|
+
names in the task's log. Only elements carrying a `Ref` or a function decode
|
|
154
|
+
whole either way.
|
|
155
|
+
|
|
156
|
+
Data that is not a dataset opens the same way inside a body —
|
|
157
|
+
`FileSystem.openBeast(T, path)` for a beast2 file on the runner's disk,
|
|
158
|
+
`blob.openBeast(T)` for bytes in hand — but the dataflow does not watch it:
|
|
159
|
+
anything a task should react to is an input.
|
|
160
|
+
|
|
161
|
+
| `config.runner` | Runs on |
|
|
162
|
+
|---|---|
|
|
163
|
+
| omitted | `{ runtime: 'east-node', platforms: ['@elaraai/east-node-std'] }` — every project has Node |
|
|
164
|
+
| `{ runtime: 'east-py', platforms: ['east-py-std', 'east-py-io', 'east-py-datascience'] }` | Python |
|
|
165
|
+
| `{ runtime: 'east-c', platforms: ['east-c-std'] }` | native, the lowest overhead |
|
|
166
|
+
| `{ runtime: 'custom', command: ['uv', 'run', 'east-py', 'run', '-p', 'east-py-std'] }` | any command: e3 appends `run`'s arguments (`-i <input>` each, `-o <output>`, the program's file) |
|
|
167
|
+
|
|
168
|
+
Platform names are typed per runtime — a typo, or another runtime's platform, is
|
|
169
|
+
a compile error; `{ custom: '<name>' }` names your own. A stock runtime is handed
|
|
170
|
+
one unit file (`exec <unit>`), and takes `decode: 'lazy' | 'whole'` (`'lazy'`
|
|
171
|
+
when omitted): how its units read their inputs — the unit's `decode`, as every
|
|
172
|
+
runner's `run --decode` takes it. It is the runner's, so a task, each piece of a
|
|
173
|
+
split task, a function, a mutation, a migration and an index build all read
|
|
174
|
+
their inputs as their runner says, and changing it re-runs the task.
|
|
175
|
+
|
|
176
|
+
```typescript
|
|
177
|
+
const greet = e3.task('greet', [name],
|
|
178
|
+
East.function([StringType], StringType, ($, n) => East.str`Hello, ${n}!`));
|
|
179
|
+
const shout = e3.task('shout', [greet.output],
|
|
180
|
+
East.function([StringType], StringType, ($, s) => s.upperCase()),
|
|
181
|
+
{ runner: { runtime: 'east-c', platforms: ['east-c-std'] } });
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
#### Your own code: platform functions, imported functions, environments
|
|
185
|
+
|
|
186
|
+
- **A project-owned platform function** (native code East cannot express): list
|
|
187
|
+
`{ custom: '<name>' }` in the runner's platforms. For east-node the name is
|
|
188
|
+
your package's **scoped name** (its `./platform` export, a
|
|
189
|
+
`PlatformFunction[]`); for east-py, the Python **module** (its top-level
|
|
190
|
+
`platform` list). The function's declared name, `"<project>.<fn>"`, must
|
|
191
|
+
byte-match its implementation. Authoring and wiring: the **east-project**,
|
|
192
|
+
**east** (`East.platform(...).implement(...)`) and **east-py**
|
|
193
|
+
(`@platform_function`) skills.
|
|
194
|
+
- **An East function written in python, or in another node package**:
|
|
195
|
+
`East.importFunction(pkg, name, FunctionType(...))` in the body. `e3.export`
|
|
196
|
+
resolves it and embeds its IR, so the task is pure IR on any runner, its
|
|
197
|
+
locations kept so an error or a profile names the function's own lines. A uv
|
|
198
|
+
workspace member is found by name in `uv.lock`, and its root module's
|
|
199
|
+
`east_functions` dict is exported with `east-py export-functions` (from the
|
|
200
|
+
project's `.venv`, else `EAST_PY` or PATH); an npm member is found by its
|
|
201
|
+
`package.json` name, and its built `./functions` export's `eastFunctions` is
|
|
202
|
+
exported with `east-node export-functions` (`EAST_NODE`). A package built
|
|
203
|
+
elsewhere is passed as its manifest — `{ functions: [...] }` / `--functions` —
|
|
204
|
+
which wins. The declared type must equal the exported one exactly, and each
|
|
205
|
+
platform function it calls must be provided by the task's runner, by name or
|
|
206
|
+
by family (`east-py-std` ≡ `@elaraai/east-node-std` ≡ `east-c-std`;
|
|
207
|
+
`east-py-io` ≡ `@elaraai/east-node-io`).
|
|
208
|
+
- **Environments** are derived at export from each `{ custom }` platform: the
|
|
209
|
+
providing workspace package's closure (`pyproject.toml` + `uv.lock` + sdists,
|
|
210
|
+
or `package.json` + lockfile + `npm pack`) is captured into the package and
|
|
211
|
+
materialized per repository, so the task runs where your tree never was. With
|
|
212
|
+
platform code split into packages (`create-e3 --python-packages=a,b`), editing
|
|
213
|
+
one re-runs only its tasks. An explicit `environment` overrides the
|
|
214
|
+
derivation: `{ python: { project } }`, `{ node: { project } }`,
|
|
215
|
+
`{ tools: { files: [...] } }` (prebuilt binaries on PATH, such as a C runner)
|
|
216
|
+
or `{ image: { digest: 'repo@sha256:<64 hex>' } }` (cloud; a tag is refused).
|
|
217
|
+
|
|
218
|
+
```python
|
|
219
|
+
# packages/pricing/src/pricing/__init__.py — the member's root module
|
|
220
|
+
score = East.function([Row], FloatType, lambda b, r: r.qty.to_float() * r.price)
|
|
221
|
+
east_functions = {"score": score}
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
```typescript
|
|
225
|
+
const score = East.importFunction('pricing', 'score', FunctionType([RowType], FloatType));
|
|
226
|
+
const total = e3.task('total', [rows],
|
|
227
|
+
East.function([ArrayType(RowType)], FloatType, ($, rs) => rs.map(($, r) => score(r)).sum()));
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
### Stream tasks — `e3.streamTask(name, spec, fn)`
|
|
231
|
+
|
|
232
|
+
A stream task **emits** its output instead of returning it, so its memory does
|
|
233
|
+
not grow with its output: emission order is free, and the runner sorts a set or
|
|
234
|
+
a dict in bounded runs and merges them. `spec` is `inputs` (datasets, in the
|
|
235
|
+
body's parameter order, some wrapped in `e3.partition`), `output` (an output
|
|
236
|
+
kind), `runner` (a stock runtime; `custom` is refused) and `environment`. The
|
|
237
|
+
body takes the inputs, then `emit`, and returns nothing. With no partitioned
|
|
238
|
+
input it is one unit with exact left-to-right semantics; a producer has no
|
|
239
|
+
inputs and emits what it reads from platform functions.
|
|
240
|
+
|
|
241
|
+
| Output kind | `emit` | The parts combine by |
|
|
242
|
+
|---|---|---|
|
|
243
|
+
| `e3.output.array(T)` | `emit(t)` | concatenation: emission order within a unit, units in input order |
|
|
244
|
+
| `e3.output.set(T)` | `emit(t)` | union: an element emitted twice is held once |
|
|
245
|
+
| `e3.output.dict(K, V, { merge? })` | `emit(k, v)` | key: a key emitted more than once folds with `merge($, key, a, b)`, in input order; without `merge` it fails the task, naming the key |
|
|
246
|
+
| `e3.output.fold(T, { zero, combine })` | `emit(t)` | every value, folded with `combine($, a, b)` from `zero`, in input order |
|
|
247
|
+
|
|
248
|
+
```typescript
|
|
249
|
+
const balances = e3.streamTask('balances', {
|
|
250
|
+
inputs: [events], // no partitioned input: one unit, in order
|
|
251
|
+
output: e3.output.array(FloatType),
|
|
252
|
+
}, ($, events, emit) => {
|
|
253
|
+
const balance = $.let(0.0);
|
|
254
|
+
$.for(events, ($, event) => {
|
|
255
|
+
$.assign(balance, balance.add(event.amount));
|
|
256
|
+
$(emit(balance));
|
|
257
|
+
});
|
|
258
|
+
});
|
|
259
|
+
|
|
260
|
+
const ingest = e3.streamTask('ingest', {
|
|
261
|
+
inputs: [], // a producer: emits in any order
|
|
262
|
+
output: e3.output.dict(StringType, RowType, {
|
|
263
|
+
merge: ($, _id, a, b) => East.greater(a.at, b.at).ifElse(($) => a, ($) => b), // the latest row wins
|
|
264
|
+
}),
|
|
265
|
+
}, ($, emit) => { /* page through a platform source, $(emit(row.id, row)) each */ });
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
### Splitting the work — `e3.partition(dataset, { by? })`
|
|
269
|
+
|
|
270
|
+
Wrap a stream task's input in `e3.partition` and the body runs once per
|
|
271
|
+
**piece** of it: each piece is a unit of its own — cached on its own, run in
|
|
272
|
+
parallel under the budget — typed as the whole dataset (a key range of a Set or
|
|
273
|
+
Dict, a position range of an Array), and the pieces' outputs combine as the
|
|
274
|
+
output kind says. Pieces are cut by content, most holding 64 to 100 MiB of
|
|
275
|
+
stored bytes, so an append or an insertion re-runs only the pieces it reaches.
|
|
276
|
+
An input not wrapped reaches every piece whole, opened lazily as every input is
|
|
277
|
+
(a keyed get reads only its segments); a change to it re-runs every piece.
|
|
278
|
+
|
|
279
|
+
```typescript
|
|
280
|
+
// sales: Dict<{ sku, id }, { amount, currency }>. A re-key: a key emitted in
|
|
281
|
+
// several pieces folds with `merge`. `rates` reaches every piece whole.
|
|
282
|
+
const bySku = e3.streamTask('by_sku', {
|
|
283
|
+
inputs: [e3.partition(sales), rates],
|
|
284
|
+
output: e3.output.dict(StringType, FloatType, { merge: ($, _sku, a, b) => a.add(b) }),
|
|
285
|
+
}, ($, sales, rates, emit) => {
|
|
286
|
+
$.for(sales, ($, sale, key) => {
|
|
287
|
+
$(emit(key.sku, sale.amount.multiply(rates.get(sale.currency))));
|
|
288
|
+
});
|
|
289
|
+
});
|
|
290
|
+
|
|
291
|
+
// Huge → small: a fold.
|
|
292
|
+
const revenue = e3.streamTask('revenue', {
|
|
293
|
+
inputs: [e3.partition(sales)],
|
|
294
|
+
output: e3.output.fold(FloatType, { zero: 0.0, combine: ($, a, b) => a.add(b) }),
|
|
295
|
+
}, ($, sales, emit) => {
|
|
296
|
+
$.for(sales, ($, sale) => { $(emit(sale.amount)); });
|
|
297
|
+
});
|
|
298
|
+
|
|
299
|
+
// Per entity, in order. postings: Dict<{ account, at, id }, Float>. Rows whose
|
|
300
|
+
// `by` fields are equal are never split across pieces, and a piece iterates in
|
|
301
|
+
// key order, so each account's rows arrive together, oldest first.
|
|
302
|
+
const statements = e3.streamTask('statements', {
|
|
303
|
+
inputs: [e3.partition(postings, { by: ['account'] })],
|
|
304
|
+
output: e3.output.array(StatementLineType), // { account, at, balance }
|
|
305
|
+
}, ($, postings, emit) => {
|
|
306
|
+
const account = $.let('');
|
|
307
|
+
const balance = $.let(0.0);
|
|
308
|
+
$.for(postings, ($, amount, key) => {
|
|
309
|
+
$.if(East.notEqual(key.account, account), ($) => { // the next account starts from zero
|
|
310
|
+
$.assign(account, key.account);
|
|
311
|
+
$.assign(balance, 0.0);
|
|
312
|
+
});
|
|
313
|
+
$.assign(balance, balance.add(amount));
|
|
314
|
+
$(emit({ account: key.account, at: key.at, balance }));
|
|
315
|
+
});
|
|
316
|
+
});
|
|
317
|
+
|
|
318
|
+
// Reconcile two same-keyed datasets: both partitioned, so cut at the same keys.
|
|
319
|
+
const change = e3.streamTask('change', {
|
|
320
|
+
inputs: [e3.partition(today), e3.partition(yesterday)], // both Dict<SaleKey, Float>
|
|
321
|
+
output: e3.output.dict(SaleKeyType, FloatType),
|
|
322
|
+
}, ($, today, yesterday, emit) => {
|
|
323
|
+
$.for(today, ($, amount, key) => {
|
|
324
|
+
$(emit(key, amount.subtract(yesterday.get(key, ($, _key) => 0.0))));
|
|
325
|
+
});
|
|
326
|
+
});
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
- `by` names leading key fields, in order: `['account']`, or
|
|
330
|
+
`['account', 'at.day']`, whose last entry reads the first field of `at`.
|
|
331
|
+
- Two or more partitioned inputs are cut at the same keys, so they must be Sets
|
|
332
|
+
or Dicts whose keys, or `by` fields, have the same types.
|
|
333
|
+
- **The author's contract, the only one:** `merge` and `combine` are
|
|
334
|
+
associative, `zero` is an identity of `combine`, and a partitioned body's
|
|
335
|
+
combined result does not depend on where its input was cut.
|
|
336
|
+
- Refused when the task is defined, naming it: `e3.partition` on an `e3.task`
|
|
337
|
+
input; a partitioned input that is not a collection; `by` on an Array, or
|
|
338
|
+
naming anything but leading key fields; co-partitioned inputs with no common
|
|
339
|
+
key.
|
|
340
|
+
|
|
341
|
+
#### Which task?
|
|
342
|
+
|
|
343
|
+
| Workload | Use |
|
|
344
|
+
|---|---|
|
|
345
|
+
| The output fits in memory, computed in one pass | `e3.task` |
|
|
346
|
+
| Global sequential state: running balances, replay, simulation | `e3.streamTask` with no partitioned input |
|
|
347
|
+
| Ingest from external sources | `e3.streamTask` with no inputs |
|
|
348
|
+
| Row-local derive, clean or validate over a huge input | `e3.partition` it; emit each row kept (`e3.output.array`, or a `dict` keyed as the input) |
|
|
349
|
+
| Enrich against a small reference, or sparse keyed reads of another huge dataset | `e3.partition` the big input; pass the other unwrapped |
|
|
350
|
+
| Aggregate huge → small (KPIs, counts, top-k) | `e3.partition` + `e3.output.fold` |
|
|
351
|
+
| Per-key totals, latest per key, a re-key (shuffle) | `e3.partition` + `e3.output.dict` with `merge` (a Set: `e3.output.set`) |
|
|
352
|
+
| Per entity: sequential within it, parallel across entities | `e3.partition(dataset, { by: [<leading key fields>] })` |
|
|
353
|
+
| Reconcile two same-keyed huge datasets | two `e3.partition` inputs |
|
|
354
|
+
| ML training, or work East cannot express | a platform function, or `e3.customTask` |
|
|
355
|
+
|
|
356
|
+
### Custom tasks, functions, packages
|
|
357
|
+
|
|
358
|
+
| Definition | What it is |
|
|
359
|
+
|---|---|
|
|
360
|
+
| `e3.customTask(name, inputs, outputType, command, { environment? })` | A task that runs a bash script: `command($, inputPaths, outputPath)` builds it from the staged beast2 input files, and it writes the output as a beast2 file. |
|
|
361
|
+
| `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API) and its result returned inline: not in the dataflow, and a call persists nothing. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A server bounds a call by a timeout and a 1 MiB result, so long work belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
|
|
362
|
+
| `e3.package(name, version, ...items)` | Bundles the items and everything they depend on — pass the last tasks, records and functions; a package passed in contributes its contents. The name and version must be valid file names. |
|
|
363
|
+
| `e3.export(pkg, zipPath, { functions? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself; `functions` lists manifests built elsewhere). |
|
|
364
|
+
|
|
365
|
+
```typescript
|
|
366
|
+
const process = e3.customTask('process', [rawData], StringType,
|
|
367
|
+
($, inputPaths, outputPath) => East.str`python script.py -i ${inputPaths.get(0n)} -o ${outputPath}`);
|
|
368
|
+
const add = e3.function('add',
|
|
369
|
+
East.function([IntegerType, IntegerType], IntegerType, ($, a, b) => a.add(b)));
|
|
370
|
+
await e3.export(e3.package('tools', '1.0.0', process, add), '/tmp/tools.zip');
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
### Records — audited state, written only through mutations
|
|
374
|
+
|
|
375
|
+
`e3.record(name, type, initialValue)` is a dataset at `.records.<name>` that
|
|
376
|
+
tasks read and react to like any input, but that only its **mutations** write:
|
|
377
|
+
a raw `dataset set` is refused. A mutation is pure, synchronous East — no async
|
|
378
|
+
body and no platform call, since the compare-and-swap retry runs it again
|
|
379
|
+
against fresher state — named by an identifier. Each commit records its parent,
|
|
380
|
+
the state, the mutation, its arguments, the actor and the time. Deploy mints a
|
|
381
|
+
`$init` commit from the initial value, and a redeploy keeps state and history.
|
|
382
|
+
The record's version vector carries the commit hash, so even a mutation that
|
|
383
|
+
leaves the state unchanged triggers downstream tasks.
|
|
384
|
+
|
|
385
|
+
| Form | Body | Use it when |
|
|
386
|
+
|---|---|---|
|
|
387
|
+
| `e3.mutation.reduce(name, rec, fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
|
|
388
|
+
| `e3.mutation.edit(name, rec, fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
|
|
389
|
+
| `e3.mutation.patch(rec, name = 'patch')` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
|
|
390
|
+
|
|
391
|
+
Every form commits a delta addressed by key, rewriting only the segments it
|
|
392
|
+
touches (a `reduce` of a record that is not a Dict or Set writes it whole), and
|
|
393
|
+
takes `{ runner? }`. In an `edit`, repeated edits of one key fold, a `set` of the
|
|
394
|
+
value a row already holds changes nothing, and a `delete` or `update` of a key
|
|
395
|
+
the record does not hold is a conflict naming the key: the caller's view of the
|
|
396
|
+
record is stale. `e3.mutation.editType(recordType)` is the type of `edit`.
|
|
397
|
+
|
|
398
|
+
```typescript
|
|
399
|
+
const PlanType = StructType({
|
|
400
|
+
title: StringType, owner: StringType, due: DateTimeType, resources: SetType(StringType),
|
|
401
|
+
});
|
|
402
|
+
const plans = e3.record('plans', DictType(StringType, PlanType), new Map());
|
|
403
|
+
|
|
404
|
+
const addPlan = e3.mutation.reduce('add_plan', plans,
|
|
405
|
+
East.function([plans.type, StringType, PlanType], plans.type, ($, state, id, plan) => {
|
|
406
|
+
const next = $.let(state.copy()); // the state is frozen: copy first
|
|
407
|
+
$(next.insert(id, plan));
|
|
408
|
+
return next;
|
|
409
|
+
}));
|
|
410
|
+
|
|
411
|
+
const reschedule = e3.mutation.edit('reschedule', plans,
|
|
412
|
+
East.function([plans.type, StringType, DateTimeType, e3.mutation.editType(plans.type)], NullType,
|
|
413
|
+
($, state, id, due, edit) => {
|
|
414
|
+
const plan = $.let(state.get(id)); // decodes one segment, not the record
|
|
415
|
+
$(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
|
|
416
|
+
}));
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
Costs count in segments: a Dict or Set record is cut by key and by each entry's
|
|
420
|
+
encoded size, so wide rows get segments of a few rows and a one-row edit stays
|
|
421
|
+
cheap. A view reads every field of the rows it shows, so keep a bulky payload it
|
|
422
|
+
does not display in a second record keyed the same way.
|
|
423
|
+
|
|
424
|
+
#### Secondary indexes — `e3.recordIndex(name, record, { on | key | keys, value? })`
|
|
425
|
+
|
|
426
|
+
A second collection over a Dict record, in another sort order, maintained in the
|
|
427
|
+
same commit, so a view by an attribute is a page rather than a scan. `on`
|
|
428
|
+
names row fields in index order: one field uses its value as the key, several
|
|
429
|
+
use a Struct in that order, and a Set field gives an entry per element. It
|
|
430
|
+
also carries field metadata for `Data.bindPaged(record, { indexes })` and
|
|
431
|
+
`Slice.bindRecord` (see **e3-ui**). `key` gives one computed entry per row;
|
|
432
|
+
`keys` returns a Set, one entry per element (a row
|
|
433
|
+
naming five resources appears five times, an empty set not at all); `value` is a
|
|
434
|
+
projection a view renders without touching the record. The functions are pure
|
|
435
|
+
and synchronous, since an index maintained commit by commit must equal its
|
|
436
|
+
rebuild to the byte. The name is an identifier, never `primary` (the record's own
|
|
437
|
+
order). Read through one with `index=<name>` on a page or key search; `e3
|
|
438
|
+
reindex` rebuilds. A deploy builds an index the package declares and the state
|
|
439
|
+
lacks, drops one it no longer declares, and runs nothing for the rest; a build
|
|
440
|
+
is a split task over the record.
|
|
441
|
+
|
|
442
|
+
```typescript
|
|
443
|
+
const byDue = e3.recordIndex('by_due', plans, { on: ['due'] });
|
|
444
|
+
const byOwnerDue = e3.recordIndex('by_owner_due', plans, { on: ['owner', 'due'] });
|
|
445
|
+
const byOwner = e3.recordIndex('by_owner', plans, {
|
|
446
|
+
key: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner),
|
|
447
|
+
value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
|
|
448
|
+
});
|
|
449
|
+
const byResource = e3.recordIndex('by_resource', plans, {
|
|
450
|
+
keys: East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources),
|
|
451
|
+
});
|
|
452
|
+
const pkg = e3.package('planning', '1.0.0',
|
|
453
|
+
plans, addPlan, reschedule, e3.mutation.patch(plans), byDue, byOwnerDue, byOwner, byResource);
|
|
454
|
+
```
|
|
455
|
+
|
|
456
|
+
For events that must remain visible while they overlap a selected time window,
|
|
457
|
+
declare `interval: { from: 'start', to: 'end' }`. Both fields must be DateTime
|
|
458
|
+
or Option<DateTime>. Optional `on` fields are exact leading choices:
|
|
459
|
+
|
|
460
|
+
```typescript
|
|
461
|
+
const byTime = e3.recordIndex('by_time', events, {
|
|
462
|
+
interval: { from: 'start', to: 'end' },
|
|
463
|
+
});
|
|
464
|
+
const byStatusTime = e3.recordIndex('by_status_time', events, {
|
|
465
|
+
on: ['status'], interval: { from: 'start', to: 'end' },
|
|
466
|
+
});
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
An interval index uses the ordinary index build, mutation and paging machinery.
|
|
470
|
+
It stores at most three entries per event per leading-field combination,
|
|
471
|
+
independent of duration. The record Slice adapter resolves a half-open
|
|
472
|
+
`[from, to)` window with at most 55 disjoint bounded queries per exact prefix;
|
|
473
|
+
each matching original record occurs once, including events that started
|
|
474
|
+
earlier or span the whole window. It never scans the record or guesses a
|
|
475
|
+
maximum duration. An event ending at `from` is outside; an instant at `from`
|
|
476
|
+
is inside and one at `to` is outside. A missing or non-positive end is an
|
|
477
|
+
instant. A missing start remains selectable as Unset/backlog.
|
|
478
|
+
|
|
479
|
+
Use `on: ['start']` only when the intended query is **starts within**, since it
|
|
480
|
+
cannot preserve carry-in events. When both declarations are available, a Slice
|
|
481
|
+
range on the start field uses the interval declaration. The generated interval
|
|
482
|
+
keys are query machinery; record edits and sessions always target the original
|
|
483
|
+
primary key. Calendar's `Calendar.events(binding, { slice, entries, … })`
|
|
484
|
+
requires a matching interval declaration and the primary `entries` binding to
|
|
485
|
+
confirm saves even after an edit leaves the selected window.
|
|
486
|
+
|
|
487
|
+
#### Migrations — how a record's type changes
|
|
488
|
+
|
|
489
|
+
| Form | Function | Runs as |
|
|
490
|
+
|---|---|---|
|
|
491
|
+
| `e3.migration.value(name, rec, fn, { after? })` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
|
|
492
|
+
| `e3.migration.rows(name, rec, fn, { after? })` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
|
|
493
|
+
| `e3.migration.rekey(name, rec, fn, { after? })` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
|
|
494
|
+
|
|
495
|
+
A record's steps are one chain: each names the step before it with `after` (a
|
|
496
|
+
step of the same record, leaving it as this step's input type), passing the last
|
|
497
|
+
step passes the chain, and the last step leaves the record as its declared type.
|
|
498
|
+
A step's name is an identifier, unique on the record, and a workspace records the
|
|
499
|
+
steps it has applied by name — so moving a step's code, or re-exporting it with a
|
|
500
|
+
newer e3, does not run it again, and an edit to an applied step's body is not
|
|
501
|
+
detected. The functions are pure, synchronous East.
|
|
502
|
+
|
|
503
|
+
```typescript
|
|
504
|
+
const RowV1Type = StructType({ title: StringType });
|
|
505
|
+
const RowV2Type = StructType({ title: StringType, owner: StringType });
|
|
506
|
+
const plans = e3.record('plans', DictType(StringType, RowV2Type), new Map());
|
|
507
|
+
|
|
508
|
+
const addOwner = e3.migration.rows('add_owner', plans,
|
|
509
|
+
East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' })));
|
|
510
|
+
// The next version adds its step after this one: e3.migration.value('…', plans, fn, { after: addOwner })
|
|
511
|
+
const pkg = e3.package('planning', '2.0.0', plans, addOwner);
|
|
512
|
+
```
|
|
513
|
+
|
|
514
|
+
What a deploy does with each record, decided before it writes anything:
|
|
515
|
+
|
|
516
|
+
| The workspace holds | Against the package's chain | The deploy |
|
|
517
|
+
|---|---|---|
|
|
518
|
+
| no state | — | mints `$init`; the whole chain counts as applied |
|
|
519
|
+
| state | every step applied, the type unchanged | keeps it, with a `$deploy` commit when the package changed |
|
|
520
|
+
| state | a proper prefix applied | runs the rest, a `$migrate:<name>` commit each |
|
|
521
|
+
| state | every step applied, but the type changed | refused: a type change needs a migration, or `--schema reset` |
|
|
522
|
+
| state | applied steps the chain does not start with | refused, naming the steps applied and those declared |
|
|
523
|
+
| a record the package no longer declares | — | refused unless `--allow-drop-records` |
|
|
524
|
+
|
|
525
|
+
`--schema <policy>` on `e3 workspace deploy` and `e3 watch`: `migrate` (the
|
|
526
|
+
default), `fail` (run no migration; refuse) or `reset` (reset the record to its
|
|
527
|
+
initial value, with a `$reset` commit). `--plan` prints what the deploy would do
|
|
528
|
+
to each record, index and input and writes nothing — from a zip or a source it
|
|
529
|
+
imports nothing, and a server plans only a package it holds. The steps run
|
|
530
|
+
before the deploy writes: a failed step leaves the workspace as it was, and a
|
|
531
|
+
deploy run again is served the steps that finished from the cache.
|
|
532
|
+
|
|
533
|
+
What a deploy does with each input — a dataset an `e3.input` declares, which
|
|
534
|
+
people set:
|
|
535
|
+
|
|
536
|
+
| The input now | Its type | The deploy |
|
|
537
|
+
|---|---|---|
|
|
538
|
+
| holds the value the deployed package gave it, or is new | — | the new package's value (`package`) |
|
|
539
|
+
| someone set it | — | the new package's value under `--inputs reset`, the default, and says it reset it (`reset`) |
|
|
540
|
+
| someone set it | the new package's | kept, under `--inputs keep-edited` (`keep`) |
|
|
541
|
+
| someone set it | changed | the new package's value, and says why (`reset`) |
|
|
542
|
+
| the deployed package took it from a file, and it holds a value | — | the new package's value under either policy, since a value set since cannot be told from the file's, and says so (`reset`) |
|
|
543
|
+
| the new package takes it from a file | — | its file, as every deploy does (`file`); a deploy that reads no file, a server's, leaves it unset for its client to send |
|
|
544
|
+
| the new package no longer declares it | — | gone, with its value (`drop`) |
|
|
545
|
+
|
|
546
|
+
## The e3 CLI
|
|
547
|
+
|
|
548
|
+
Every `<repo>` is a local path or a server URL, `http(s)://<host>/repos/<name>`;
|
|
549
|
+
where `[repo]` is optional it defaults to `$E3_REPO`, then `.`. Dataset paths are
|
|
550
|
+
flat, `<ws>.<name>` — an input or a task output, with "did you mean" suggestions;
|
|
551
|
+
a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
|
|
552
|
+
|
|
553
|
+
| Command | Does |
|
|
554
|
+
|---|---|
|
|
555
|
+
| **Repositories** | |
|
|
556
|
+
| `e3 repo create [repo] [--exist-ok]` | Create one (`--exist-ok`: succeed if it exists). |
|
|
557
|
+
| `e3 repo status [repo]` · `e3 repo remove [repo] [-r]` | Show objects, packages, workspaces · remove (`-r`: its workspaces first). |
|
|
558
|
+
| `e3 repo gc [repo] [--dry-run] [--keep-runs <n>] [--keep-days <d>] [--min-age <ms>]` | Drop old run history, then what nothing names (below). |
|
|
559
|
+
| `e3 repo list <server>` | The repositories a server holds. |
|
|
560
|
+
| **Packages** | |
|
|
561
|
+
| `e3 package import [repo] <zip>` · `export [repo] <pkg[@ver]> <zip>` | Import · export (`--quiet`: errors only). |
|
|
562
|
+
| `e3 package list [repo]` · `remove [repo] <pkg[@ver]>` | |
|
|
563
|
+
| **Workspaces** | |
|
|
564
|
+
| `e3 workspace create [repo] <ws>` · `list [repo]` · `status [repo] <ws>` · `remove [repo] <ws>` | `status` shows tasks, datasets and locks. |
|
|
565
|
+
| `e3 workspace copy [repo] <from> <to>` | Make `<to>` what `<from>` is now — its package and every input, record and task output, so nothing runs again — made, or replaced whole with its runs. It writes refs only, so its cost follows the number of datasets, never their size; a write to either afterwards leaves the other as it was. Refused while a dataflow runs in `<to>`. |
|
|
566
|
+
| `e3 workspace deploy [repo] <ws> <pkg[@ver]>` | Deploy an imported package. |
|
|
567
|
+
| `… --from-zip <zip>` · `… --from-source <src.ts> [--functions <manifest…>]` | Import the zip (or bundle the source) and deploy, creating the workspace. |
|
|
568
|
+
| `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on; `--memory`: what those may reserve (local). |
|
|
569
|
+
| `e3 workspace export [repo] <ws> <zip> [--name <n>] [--version <v>]` | The workspace's state as a package. |
|
|
570
|
+
| **Datasets** | |
|
|
571
|
+
| `e3 dataset get [repo] <ws.name> [-f east\|json\|beast2]` | Print a value. |
|
|
572
|
+
| `e3 dataset set [repo] <ws.name> <file> [--type <spec> \| --type-file <path>]` | Write from `.east`, `.beast2`, `.json` or `.csv` (JSON and CSV need the type). |
|
|
573
|
+
| `e3 dataset set [repo] <ws.name> --from-file <path.beast2> [-j <n>] [--memory <size>]` | Take a beast2 file in as the value, as a deploy takes a `file` source in, saying how far it has got; `-j`, `--memory`: the budget its intake units run under (local). |
|
|
574
|
+
| `e3 dataset list [repo] <ws> [-l]` · `status [repo] <ws.name>` · `find [repo] <ws> <pattern>` | Paths (`-l`: kind, type, status, size) · one dataset · by substring or glob. |
|
|
575
|
+
| **Running** | |
|
|
576
|
+
| `e3 dataflow run [repo] <ws> [--filter <task>] [--force \| --force-task <task>…] [-j <n>] [--memory <size>] [-v]` | Run what is stale, then print the outputs' paths. `--filter`: one task, by its exact name, and the tasks it depends on. `--force`: re-run every task the run runs (under `--filter`, its task); `--force-task`, repeated for each: re-run the tasks it names, whose dependents re-run when an output changes. A task the graph lacks, or one `--filter` leaves out, refuses the run. |
|
|
577
|
+
| `e3 task list [repo] <ws>` | Tasks with the status of each one's latest run. |
|
|
578
|
+
| `e3 task logs [repo] <ws.task> [-n <lines>] [--all] [--follow] [--execution <task>/<inputs>/<id>]` | The last 200 lines by default, of the latest attempt over the task's current inputs, or its latest run; `--follow` stays on that execution; `--execution`: one unit's log, as a split task's log names it (local). |
|
|
579
|
+
| `e3 run <repo> <pkg[@ver].task> [inputs.beast2…] -o <out> [--force] [-v] [-j <n>] [--memory <size>]` | Run one task ad hoc. |
|
|
580
|
+
| `e3 call <repo> <pkg[@ver].fn> [args…] [-o <out.beast2>] [-j <n>] [--memory <size>] [-v]` · `e3 call <repo> -w <ws> <fn> [args…]` | Call a function; each argument is an `.east` literal or a `.beast2`/`.json`/`.east` file, and with `-w` one written `@<keypath>` (`@.inputs.sales`) is that dataset of the workspace, pinned at its hash when the call starts. |
|
|
581
|
+
| **Records** (`-w <ws>` required) | |
|
|
582
|
+
| `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
|
|
583
|
+
| `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target; local). |
|
|
584
|
+
| `e3 reindex <repo> <record> -w <ws> [--index <name>] [-j <n>] [--memory <size>]` · `e3 compact <repo> <record> -w <ws>` | Rebuild indexes (local) · collapse the history to a `$compact` root, the state kept. |
|
|
585
|
+
| **Development** | |
|
|
586
|
+
| `e3 watch <src.ts> <repo> <ws> [--start] [--schema <p>] [--inputs <p>] [--abort-on-change] [--functions <manifest…>] [-j <n>] [--memory <size>]` | Redeploy on each change (and run, with `--start`). |
|
|
587
|
+
| `e3 convert [input] [--from <f>] [--to <f>] [--type <spec>] [-o <out>]` | Convert between `.east`, `.json` and `.beast2`. |
|
|
588
|
+
| `e3 completion install [--shell <s>]` · `uninstall` · `bash` \| `zsh` \| `fish` | Shell completion. |
|
|
589
|
+
| **Servers** | |
|
|
590
|
+
| `e3 auth login <server> [--no-browser]` · `logout <server>` · `status` · `token <server>` · `whoami [server]` | OAuth2 device flow; credentials per server (`token` prints one for curl). |
|
|
591
|
+
|
|
592
|
+
**The budget.** `-j`/`--jobs <n>` is the cores — runner processes in flight, a
|
|
593
|
+
task or a unit each — and `--memory <size>` (`8G`, `512M`) what they may reserve
|
|
594
|
+
between them. The defaults are `E3_JOBS` and `E3_MEMORY`, else what e3 may use:
|
|
595
|
+
its CPU affinity capped by a cgroup quota, and the cgroup's `memory.max` or
|
|
596
|
+
physical memory, less a reserve. A unit of a split task reserves the largest
|
|
597
|
+
peak a unit of its stage has reached in the run, so a stage runs its first unit
|
|
598
|
+
alone and then fans out; on Linux and macOS a guard stops the newest unit when
|
|
599
|
+
the runners together pass the budget, and runs it again once it fits. A deploy
|
|
600
|
+
takes its `file` sources in `-j` at a time, their pieces' intake units running
|
|
601
|
+
under the same budget. The commands that run East — `workspace deploy`,
|
|
602
|
+
`dataset set --from-file`, `dataflow run`, `run`, `call`, `mutate`, `reindex`
|
|
603
|
+
and `watch` — take the flags for a local repository; against a server the CLI
|
|
604
|
+
refuses them, since the server runs the work under its own budget
|
|
605
|
+
(`e3-api-server -j`, `--memory`).
|
|
606
|
+
|
|
607
|
+
**A deploy says how far it has got**, on stderr. Each `file` source prints a
|
|
608
|
+
line once it is in: its size, time and rate, and how it was taken in
|
|
609
|
+
(`unchanged, already in the store`, `taken in by east-c`, or `carried` for a
|
|
610
|
+
value that is not a collection), with why the first time a runner fell back to
|
|
611
|
+
another. A terminal also keeps a live line for the files in flight, with their
|
|
612
|
+
pieces, the rate and the time left across them. Against a server, the deploy line
|
|
613
|
+
says what the job is doing, such as migrating a record or building an index;
|
|
614
|
+
the deliveries then upload a few at a time, each printing a line once it is in,
|
|
615
|
+
and the live line says what the server's commit is doing with each in flight.
|
|
616
|
+
While a deploy runs, its lock carries the same progress for `e3-ui` or any
|
|
617
|
+
client to read (`workspaceLockStatus`). A workspace deployed for the first time
|
|
618
|
+
has no status until its deploy ends, so the lock is the only place to see it.
|
|
619
|
+
|
|
620
|
+
**`-v`** passes `-v` to the runners, which print to the task's logs, identically
|
|
621
|
+
on every runtime, how each input was read — opened lazily, and what reading it
|
|
622
|
+
came to, or decoded whole, and the resident memory that added — and a timing
|
|
623
|
+
and peak-memory block. It never changes hashes or caching (add `--force` to see
|
|
624
|
+
it for a cached task), and works against a server.
|
|
625
|
+
**Where a task's time goes.** `EAST_PROFILE=1` in the environment of a local
|
|
626
|
+
`e3 dataflow run` (or `run`, `call`, `mutate`) has the east-c and east-py
|
|
627
|
+
runners profile each unit — the runners inherit e3's environment — and the
|
|
628
|
+
task's log (`e3 task logs`) then ends with every East function and platform
|
|
629
|
+
function it called, by self time, with its calls and where it was written and
|
|
630
|
+
called: the python lines of a function `East.importFunction` linked in
|
|
631
|
+
included. Each input read lazily is an entry too, named as `-v` names it
|
|
632
|
+
(`input 1`): the time its reads spent decoding segments, a call each — keyed
|
|
633
|
+
reads at random beyond the segments the runner keeps, or a scan repeated,
|
|
634
|
+
which `-v`'s account counts — and not the reading function's own time.
|
|
635
|
+
`EAST_PROFILE_INTERVAL=300` adds the profile so far every five minutes while a
|
|
636
|
+
long task runs. east-node does not profile. Like `-v` it changes no hash: add
|
|
637
|
+
`--force` for a cached task.
|
|
638
|
+
**`E3_SCRATCH_DIR`** moves a local run's per-execution scratch directories
|
|
639
|
+
(default `<repo>/tmp/scratch`, on the object store's disk; on tmpfs, outputs sit
|
|
640
|
+
in memory until stored).
|
|
641
|
+
|
|
642
|
+
**`repo gc`** keeps each workspace's last 10 runs (`--keep-runs`), every run
|
|
643
|
+
from the last 7 days (`--keep-days`) and the run its current state came from,
|
|
644
|
+
with every execution those runs used; every execution a workspace's current
|
|
645
|
+
state is served from, so a re-run stays cached; every execution from the last 7
|
|
646
|
+
days; and whatever is still running. An attempt recorded running that the local
|
|
647
|
+
runner says cannot finish is recorded interrupted, and goes as any that ended.
|
|
648
|
+
It then removes the objects nothing names, and staging files older than
|
|
649
|
+
`--min-age` (60 s). `--dry-run` reports what would go.
|
|
650
|
+
|
|
651
|
+
A package zip names the release of e3 that exported it, and an import refuses a
|
|
652
|
+
zip a newer release exported, naming that release: import it with an e3 at least
|
|
653
|
+
as new as the SDK that exported it. A dataset upload names the transfer protocol
|
|
654
|
+
its e3 speaks, and a server refuses one that speaks another, naming both
|
|
655
|
+
releases and which of them to upgrade.
|
|
656
|
+
|
|
657
|
+
## Driving e3 from code
|
|
658
|
+
|
|
659
|
+
**`@elaraai/e3-api-client`** talks to a server over HTTP. Every call is
|
|
660
|
+
`(url, repo, …, options)`, `options` being
|
|
661
|
+
`{ token: string | null, retry?, verbose?, fetch? }` (`e3 auth token <server>`
|
|
662
|
+
prints a token); `fetch` is the `fetch` every request of the call goes
|
|
663
|
+
through, the global one unless given — an in-page e3's `e3.fetch` (below).
|
|
664
|
+
Dataset paths are `TreePath`s — an input definition's `.path`, for one. A
|
|
665
|
+
request a workspace's lock refuses throws `ApiError` `workspace_locked`,
|
|
666
|
+
naming the holder: its pid, boot id, command and when it took the lock.
|
|
667
|
+
|
|
668
|
+
| Area | Functions |
|
|
669
|
+
|---|---|
|
|
670
|
+
| Repositories | `repoList(url, opts)`, `repoCreate(url, name, opts)`, `repoRemove(url, name, opts)`, `repoStatus(url, repo, opts)`, `repoRecord` (its release and upgrades), `repoGc(url, repo, gcRequest, opts)` — a job it polls; `repoGcStart` and `repoGcStatus` apart |
|
|
671
|
+
| Packages | `packageList(url, repo, opts)`, `packageGet(url, repo, name, version, opts)`, `packageImport(url, repo, zipBytes, opts)`, `packageExport(url, repo, name, version, opts)` → zip bytes, `packageRemove` |
|
|
672
|
+
| Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
|
|
673
|
+
| Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
|
|
674
|
+
| Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
|
|
675
|
+
| Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts)` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress? })` — a program over a dataset's pieces, as a job it polls, under the same rule (or `splitCallLaunch`, `splitCallStatus` and `splitCallExplain`) |
|
|
676
|
+
| Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
|
|
677
|
+
| From East | `Platform` and the `platform_*` functions (`platform_dataset_get`, `platform_dataflow_execute`, …): the same calls as platform functions, for an East program that drives a server |
|
|
678
|
+
|
|
679
|
+
**`@elaraai/e3-core`** does the same on a local repository, given a storage
|
|
680
|
+
backend (`new LocalStorage()`) and the repository's path; work that runs East
|
|
681
|
+
takes a runner, `new LocalTaskRunner(repo)`.
|
|
682
|
+
|
|
683
|
+
| Area | Functions |
|
|
684
|
+
|---|---|
|
|
685
|
+
| Repositories | `repoInit(path)`, `repoFind(startPath?)`, `repositoryOpen(storage, repo)` (checks the repository and applies the upgrades it owes), `repositoryUpgradeStep(storage, repo, { budgetMs, waitMs? })` → `{ owed }` — a host's job applies them so, a part per run under its time limit, each part taking a step up where the last stopped, `repoGc(storage, repo, { dryRun?, minAge?, keepRuns?, keepDays?, retention?, executionAlive? })` — holding the repository still, or, with `retention: { windowMs }`, beside running work: an object goes once unreachable for the window and not written or re-referenced since; given a runner's `executionAlive`, an attempt recorded running that cannot finish is recorded interrupted and pruned; `repoGcStep(storage, repo, step, { windowMs, pruneMs?, markMs?, concurrency?, executionAlive?, … })` runs that one step at a time, each returning the next (`GcStepType`), `pruneMs` and `markMs` bounding a prune step and a mark step for a host with a time limit |
|
|
686
|
+
| Packages | `packageImport(storage, repo, zipPath)`, `packageExport(storage, repo, name, version, zipPath)`, `packageList`, `packageRemove` |
|
|
687
|
+
| Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
|
|
688
|
+
| Datasets | `workspaceGetDataset(storage, repo, ws, treePath)`, `workspaceSetDataset(storage, repo, ws, treePath, value, type)`, `datasetAdoptFile(storage, repo, ws, treePath, file, { runner, onProgress? })` → `{ hash, size, segments, rows, taken, runners? }`, `taken` being `known`, `carried` or `taken` (by the `runners` named) |
|
|
689
|
+
| Runs | `dataflowExecute(storage, repo, ws, options?)`; `LocalOrchestrator` to start, poll and cancel a run |
|
|
690
|
+
| Records | `recordMutate(storage, runner, repo, ws, record, mutation, args, { actor })`, `recordHistory`, `recordDescribe`, `recordCompact`, `recordReindex` |
|
|
691
|
+
|
|
692
|
+
`@elaraai/e3-core/test` exports the contract suites another storage backend runs
|
|
693
|
+
over itself.
|
|
694
|
+
|
|
695
|
+
**`e3-api-server`** serves repositories: `--repos <dir>` (each subdirectory a
|
|
696
|
+
repository) or `--repo <path>` (one, served as `default`), `-p`/`--port` (3000),
|
|
697
|
+
`-H`/`--host` (localhost), `--cors`, `-j`/`--memory` (the server's budget, for
|
|
698
|
+
every run and call it serves and every upload it takes in), and auth — `--oidc`, a built-in provider for `e3
|
|
699
|
+
auth login` (`--token-expiry`, `--refresh-token-expiry`), or an external JWT
|
|
700
|
+
issuer (`--auth-key`, `--auth-issuer`, `--auth-audience`). In code,
|
|
701
|
+
`createServer({ reposDir | singleRepoPath, port, host, … })` starts one, and its
|
|
702
|
+
route factories mount on another host.
|
|
703
|
+
|
|
704
|
+
## Running e3 in a browser — `@elaraai/e3-web`
|
|
705
|
+
|
|
706
|
+
e3-web runs e3 entirely in a page, with no server: a repository kept in the
|
|
707
|
+
browser's own storage (records in IndexedDB, objects in OPFS, locks through
|
|
708
|
+
Web Locks), every East program on a pool of Web Workers, and e3's whole API
|
|
709
|
+
answered in the page by e3-api-server's own routes. An app is three scripts:
|
|
710
|
+
|
|
711
|
+
```typescript
|
|
712
|
+
// unit.worker.ts — where East programs run: east-web-std answers for east-node-std's
|
|
713
|
+
// platform functions, and e3's own (e3-api-client's Platform) reach the e3 worker
|
|
714
|
+
import { serveUnits } from '@elaraai/e3-web/units';
|
|
715
|
+
serveUnits();
|
|
716
|
+
```
|
|
717
|
+
|
|
718
|
+
```typescript
|
|
719
|
+
// e3.worker.ts — e3 itself: its storage, orchestrator, runner and API
|
|
720
|
+
import { serveE3 } from '@elaraai/e3-web/worker';
|
|
721
|
+
serveE3({
|
|
722
|
+
units: () => new Worker(new URL('./unit.worker.ts', import.meta.url), { type: 'module' }),
|
|
723
|
+
});
|
|
724
|
+
```
|
|
725
|
+
|
|
726
|
+
```typescript
|
|
727
|
+
// main.ts — the page: e3-api-client's calls, every one answered by the e3 worker
|
|
728
|
+
import { createWebE3 } from '@elaraai/e3-web';
|
|
729
|
+
import {
|
|
730
|
+
dataflowExecute, packageImport, repoCreate, repoList, workspaceCreate, workspaceDeploy,
|
|
731
|
+
} from '@elaraai/e3-api-client';
|
|
732
|
+
|
|
733
|
+
const e3 = await createWebE3(new Worker(new URL('./e3.worker.ts', import.meta.url), { type: 'module' }));
|
|
734
|
+
const opts = { token: null, fetch: e3.fetch };
|
|
735
|
+
|
|
736
|
+
// The first visit makes the repository; later visits find it where they left it.
|
|
737
|
+
if (!(await repoList(e3.apiUrl, opts)).includes('default')) {
|
|
738
|
+
await repoCreate(e3.apiUrl, 'default', opts);
|
|
739
|
+
// The package zip e3.export wrote at build time, served beside the page
|
|
740
|
+
const zip = new Uint8Array(await (await fetch('/sales-planning-1.0.0.zip')).arrayBuffer());
|
|
741
|
+
const { name, version } = await packageImport(e3.apiUrl, 'default', zip, opts);
|
|
742
|
+
await workspaceCreate(e3.apiUrl, 'default', 'main', opts);
|
|
743
|
+
await workspaceDeploy(e3.apiUrl, 'default', 'main', `${name}@${version}`, opts);
|
|
744
|
+
await dataflowExecute(e3.apiUrl, 'default', 'main', {}, opts);
|
|
745
|
+
}
|
|
746
|
+
```
|
|
747
|
+
|
|
748
|
+
`e3.apiUrl` is `https://e3-web.invalid`, an origin reserved so that a request
|
|
749
|
+
sent without `e3.fetch` fails rather than reaching the network, and
|
|
750
|
+
`e3.fetch`, the global `fetch`'s signature, is answered by the e3 worker. Both
|
|
751
|
+
go to e3-api-client (`RequestOptions.fetch`) and to e3-ui-components'
|
|
752
|
+
`<E3Provider>` (`E3Config.fetch`: the **e3-ui** skill). An app's own platform
|
|
753
|
+
package is served beside the standard ones:
|
|
754
|
+
`serveUnits({ platforms: { '@acme/pricing': PricingPlatform } })`.
|
|
755
|
+
|
|
756
|
+
| `serveE3` option | Default | Effect |
|
|
757
|
+
|---|---|---|
|
|
758
|
+
| `units` | — | starts a unit worker: a Web Worker whose script calls `serveUnits()` |
|
|
759
|
+
| `persist` | `true` | `false` keeps repositories in memory, gone with the page |
|
|
760
|
+
| `name` | `'e3'` | the storage's name: e3 workers of one name share their repositories, in one tab or several |
|
|
761
|
+
| `identify` | none | each request's caller, `{ sub, email?, roles }`, from the request; a request to a repository whose caller it does not identify is answered 401. None: the page's one caller may do everything |
|
|
762
|
+
| `access` | `oneShotAccessByRoles()` with `identify`, else `any` | the grant a caller holds for a one-shot, a split call, or a function call naming its own runner |
|
|
763
|
+
| `wholeIntakeLimit` | 256 MiB | the largest delivery one intake unit takes in whole |
|
|
764
|
+
|
|
765
|
+
What works is the API, as the local server answers it: every API suite of
|
|
766
|
+
e3-api-tests passes against it in Chromium, as an admin and, in its reader
|
|
767
|
+
cases, as a reader. Paged reads and key search, function calls, one-shot and
|
|
768
|
+
split calls, record mutations, deploys that migrate records or build indexes,
|
|
769
|
+
uploads, package import and export, gc, and dataflow runs — tasks in parallel
|
|
770
|
+
up to the pool's width (the cores), split tasks in pieces, re-run reactively.
|
|
771
|
+
A repository outlives a reload, and two tabs share an origin's repositories as
|
|
772
|
+
two processes share a directory. A closed tab's locks are free; a run it left
|
|
773
|
+
is taken as one whose process died, the next run served what finished from
|
|
774
|
+
the cache; and a job or an upload's commit it left is recorded failed, never
|
|
775
|
+
run again.
|
|
776
|
+
|
|
777
|
+
What a browser cannot do — each fails naming it, never a partial answer:
|
|
778
|
+
|
|
779
|
+
- **Commands.** A custom task, or a task on the `custom` runtime, is recorded
|
|
780
|
+
`error`: "a browser runs no commands".
|
|
781
|
+
- **Platform functions with no browser meaning.** east-web-std has no
|
|
782
|
+
FileSystem, Env or large-JSON reader, and east-node-io and the python
|
|
783
|
+
packages have no browser equivalent: a unit calling one fails naming the
|
|
784
|
+
function, and one listing a package its worker does not serve fails naming
|
|
785
|
+
the package.
|
|
786
|
+
- **Segments on demand.** A unit's inputs are staged whole into its worker:
|
|
787
|
+
waiting for a segment needs a cross-origin isolated page, which GitHub Pages
|
|
788
|
+
cannot give. Split a large input into pieces with `e3.partition`, which
|
|
789
|
+
bounds a unit's memory anywhere; a delivery that cannot be cut into pieces
|
|
790
|
+
is refused above `wholeIntakeLimit`, naming the fix.
|
|
791
|
+
- **A memory budget.** Memory is not measured: the pool is as wide as the
|
|
792
|
+
cores, there is no `--memory`, and no record names a peak.
|
|
793
|
+
- **The CLI.** `e3` drives e3 over HTTP, which an in-page e3 does not serve.
|
|
794
|
+
|
|
795
|
+
## A local repository
|
|
796
|
+
|
|
797
|
+
```
|
|
798
|
+
repo/
|
|
799
|
+
├── repository.beast2 # the release that last wrote it, and the upgrades it has had
|
|
800
|
+
├── repository-upgrade.beast2 # while an upgrade is under way: where its last part stopped
|
|
801
|
+
├── metadata.beast2 # its name and status
|
|
802
|
+
├── objects/ # content-addressed: values, segments, manifests, programs
|
|
803
|
+
├── packages/ # package refs
|
|
804
|
+
├── workspaces/ # each workspace's state, dataset refs, and its latest run's state and events
|
|
805
|
+
├── dataflows/ # run records
|
|
806
|
+
├── executions/ # execution attempts: status, owner, logs
|
|
807
|
+
├── running/ # an index of the attempts recorded running, by task: what a status reads
|
|
808
|
+
├── runs/ # an index of each task's runs, its units left out: what its history and logs read
|
|
809
|
+
├── adoptions/ # the manifest each delivered file, or piece of one, became
|
|
810
|
+
├── locks/ # locks, their holders, and how far each says it has got
|
|
811
|
+
├── gc/ # gc beside running work: unreachable notes, a stepped run's parts
|
|
812
|
+
├── envs/ # built execution environments
|
|
813
|
+
└── tmp/ # scratch and staged uploads
|
|
814
|
+
```
|
|
815
|
+
|
|
816
|
+
An e3 opening a repository an older release wrote applies the upgrades it has
|
|
817
|
+
not had, in place; one a newer e3 upgraded is refused, naming that release.
|
|
818
|
+
|
|
819
|
+
**Caching.** An execution is keyed by its task — program, runner and
|
|
820
|
+
environment — and its inputs' hashes, so a task re-runs only when one of those
|
|
821
|
+
changes (`--force` bypasses the cache, and `--force-task <task>` for the tasks
|
|
822
|
+
it names: a task whose code e3 cannot see changed, such as a platform
|
|
823
|
+
function's, or one reading data from outside e3). A change to an input re-runs its
|
|
824
|
+
consumers, even during a run. A split task is cached a unit at a time: each piece
|
|
825
|
+
and each merge is its own execution, and pieces are cut by content, so an append
|
|
826
|
+
or an insertion re-runs only the pieces it reaches and the merges above them.
|
|
827
|
+
|
|
828
|
+
## Related skills
|
|
829
|
+
|
|
830
|
+
- **east** — the language task bodies are written in.
|
|
831
|
+
- **e3-ui** — UI tasks (`ui()`) and decision surfaces bound to workspace
|
|
832
|
+
datasets, and rendering them in an app (`<E3Provider>`), over a server or an
|
|
833
|
+
e3 in the page; **east-ui** — their components; **e3-ui-cli** — the terminal
|
|
834
|
+
UI.
|
|
835
|
+
- **e3-create** — scaffold a project (`npm create @elaraai/e3`);
|
|
836
|
+
**east-project** — its build, deploy, run and test lifecycle.
|
|
837
|
+
- **east-py** and **east-py-datascience** — python platform functions and ML
|
|
838
|
+
tasks; **east-node-std** and **east-node-io** — Node platform functions for
|
|
839
|
+
files, HTTP, databases and storage.
|
|
840
|
+
- **east-design** and **east-ontology** — plan the dataflow, and model the
|
|
841
|
+
business, before building.
|