@elaraai/east-claude-plugin 1.0.97 → 1.0.99
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.build/daemon/lsp.js +40 -3
- package/.build/daemon/server.js +40 -3
- package/.claude-plugin/plugin.json +1 -1
- package/index.json +1829 -632
- package/package.json +1 -1
- package/skills/e3/SKILL.md +395 -90
- package/skills/e3-ui/SKILL.md +1248 -277
- package/skills/e3-ui-cli/SKILL.md +2 -2
- package/skills/east/SKILL.md +69 -2
- package/skills/east-contribute/SKILL.md +0 -1
- package/skills/east-py/SKILL.md +53 -11
- package/skills/east-ui/SKILL.md +20 -43
package/skills/e3/SKILL.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: e3
|
|
3
|
-
description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations
|
|
3
|
+
description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations, declared whole in one e3.record(name, type, { initial, indexes, mutations, migrations }) call: mutation.reduce/edit/patch; index.on/key/keys, index.interval / index.area for reads by a time window or a 2D window, index.series / index.summary for a block of keys over a span of time tiles and their totals, and index.nested for what a record's rows nest — jobs, a value a day, child rows, trees — filed item by item, with totals and parents' subtotals; and migrated between versions (migration.value/rows/rekey, in order), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack enroll/policy, dataflow run --rack, watch --start --rack, optional e3-api-server --rack, (7) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui()."
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# East Execution Engine (e3)
|
|
@@ -66,8 +66,14 @@ What do you need?
|
|
|
66
66
|
│ │ └─ Split over an input, in pieces → wrap that input: e3.partition(dataset, { by? })
|
|
67
67
|
│ ├─ A shell command → e3.customTask(name, [inputs], outputType, command)
|
|
68
68
|
│ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name, persists nothing
|
|
69
|
-
│ ├─ Audited state → e3.record
|
|
70
|
-
│ ├─
|
|
69
|
+
│ ├─ Audited state → e3.record(name, type, { initial?, indexes?, mutations?, migrations? }), declared whole
|
|
70
|
+
│ │ ├─ Its writes → mutations: (mutation) => ({ name: mutation.reduce | edit | patch })
|
|
71
|
+
│ │ ├─ An order to page in → indexes: (index) => ({ name: index.on([...fields]) | index.key(fn) | index.keys(fn) })
|
|
72
|
+
│ │ ├─ Read by a time window or a 2D window → index.interval | index.area
|
|
73
|
+
│ │ ├─ Read a block of keys over time tiles, or their totals → index.series | index.summary
|
|
74
|
+
│ │ └─ Read what rows nest (jobs, a value a day, child rows, trees) a tile at a time, with totals and subtotals
|
|
75
|
+
│ │ → index.nested({ collections, tile, summary? })
|
|
76
|
+
│ ├─ A record's type changes → migrations: (migration) => [migration.value | rows | rekey(name, fn), …], in order
|
|
71
77
|
│ ├─ A UI task → ui(name, [inputs], fn) from @elaraai/e3-ui (the e3-ui skill)
|
|
72
78
|
│ ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
|
|
73
79
|
│ └─ Bundle and export → e3.package(name, version, ...items); e3.export(pkg, zipPath)
|
|
@@ -333,17 +339,26 @@ const shout = e3.task('shout', [greet.output],
|
|
|
333
339
|
- **An East function written in python, or in another node package**:
|
|
334
340
|
`East.importFunction(pkg, name, FunctionType(...))` in the body. `e3.export`
|
|
335
341
|
resolves it and embeds its IR, so the task is pure IR on any runner, its
|
|
336
|
-
locations kept so an error or a profile names the function's own lines
|
|
342
|
+
locations kept so an error or a profile names the function's own lines (so
|
|
343
|
+
moving them re-runs the task: **Caching**, below). A uv
|
|
337
344
|
workspace member is found by name in `uv.lock`, and its root module's
|
|
338
|
-
`east_functions` dict is exported with `east-py export-functions` (
|
|
339
|
-
project's `.venv`, else
|
|
340
|
-
`package.json` name, and its built `./functions` export's `eastFunctions`
|
|
341
|
-
exported with `east-node export-functions` (`EAST_NODE`).
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
345
|
+
`east_functions` dict is exported with `east-py export-functions` (`EAST_PY`
|
|
346
|
+
when set, else the project's `.venv`, else PATH); an npm member is found by
|
|
347
|
+
its `package.json` name, and its built `./functions` export's `eastFunctions`
|
|
348
|
+
is exported with `east-node export-functions` (`EAST_NODE`). Each member is
|
|
349
|
+
exported in one process however many tasks import from it, the members side
|
|
350
|
+
by side, as many at once as the export has cores (a deploy's `-j`,
|
|
351
|
+
`e3.export`'s `cores`). A python member's manifests are kept in
|
|
352
|
+
`node_modules/.cache/e3/functions` under the directory the export runs in,
|
|
353
|
+
keyed on its sources and those of the local packages it depends on, the
|
|
354
|
+
workspace's `uv.lock` and `pyproject.toml`, the east-py exporting it and the
|
|
355
|
+
functions and providers each manifest is for, so a redeploy whose python has
|
|
356
|
+
not changed runs no export; deleting the directory clears it, and a node
|
|
357
|
+
member is exported every time. A package built elsewhere is passed as its
|
|
358
|
+
manifest — `{ functions: [...] }` / `--functions` — which wins. The declared
|
|
359
|
+
type must equal the exported one exactly, and each platform function it calls
|
|
360
|
+
must be provided by the task's runner, by name or by family (`east-py-std` ≡
|
|
361
|
+
`@elaraai/east-node-std` ≡ `east-c-std`; `east-py-io` ≡ `@elaraai/east-node-io`).
|
|
347
362
|
- **Environments** are derived at export from each `{ custom }` platform: the
|
|
348
363
|
providing workspace package's closure (`pyproject.toml` + `uv.lock` + sdists,
|
|
349
364
|
or `package.json` + lockfile + `npm pack`) is captured into the package and
|
|
@@ -499,7 +514,7 @@ const change = e3.streamTask('change', {
|
|
|
499
514
|
| `e3.customTask(name, inputs, outputType, command, { environment? })` | A task that runs a bash script: `command($, inputPaths, outputPath)` builds it from the staged beast2 input files, and it writes the output as a beast2 file. |
|
|
500
515
|
| `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API) and its result returned inline: not in the dataflow, and a call persists nothing. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A server bounds a call by a timeout and a 1 MiB result, so long work belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
|
|
501
516
|
| `e3.package(name, version, ...items)` | Bundles the items and everything they depend on — pass the last tasks, records and functions; a package passed in contributes its contents. The name and version must be valid file names. |
|
|
502
|
-
| `e3.export(pkg, zipPath, { functions? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself; `functions` lists manifests built elsewhere). |
|
|
517
|
+
| `e3.export(pkg, zipPath, { functions?, cores? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself, `cores` packages at once, the CPUs by default; `functions` lists manifests built elsewhere). |
|
|
503
518
|
|
|
504
519
|
```typescript
|
|
505
520
|
const process = e3.customTask('process', [rawData], StringType,
|
|
@@ -511,15 +526,19 @@ await e3.export(e3.package('tools', '1.0.0', process, add), '/tmp/tools.zip');
|
|
|
511
526
|
|
|
512
527
|
### Records — audited state, written only through mutations
|
|
513
528
|
|
|
514
|
-
`e3.record(name, type,
|
|
515
|
-
tasks read and react to like any input, but
|
|
529
|
+
`e3.record(name, type, { initial?, indexes?, mutations?, migrations? })` is a
|
|
530
|
+
dataset at `.records.<name>` that tasks read and react to like any input, but
|
|
531
|
+
that only its **mutations** write:
|
|
516
532
|
a raw `dataset set` is refused. A mutation is pure, synchronous East — no async
|
|
517
533
|
body and no platform call, since the compare-and-swap retry runs it again
|
|
518
534
|
against fresher state — named by an identifier. Each commit records its parent,
|
|
519
535
|
the state, the mutation, its arguments, the actor and the time. Deploy mints a
|
|
520
536
|
`$init` commit from the initial value, and a redeploy keeps state and history.
|
|
521
|
-
|
|
522
|
-
|
|
537
|
+
A large initial value is given as a function that makes it —
|
|
538
|
+
`initial: () => East.compile(generate, [])(50_000n)` — called once, as the
|
|
539
|
+
package is exported, never where the declaration is only read (a page binding
|
|
540
|
+
the record). The record's version vector carries the commit hash, so even a
|
|
541
|
+
mutation that leaves the state unchanged triggers downstream tasks.
|
|
523
542
|
|
|
524
543
|
Declared with no initial value, a record is **unassigned** until its first
|
|
525
544
|
state is committed — `e3 seed <repo> <record> <file.beast2> -w <ws>` from a
|
|
@@ -530,38 +549,54 @@ and a mutation on it is refused until then. So a record whose first state is
|
|
|
530
549
|
data that arrives after the deploy — a customer list, a reference table —
|
|
531
550
|
never runs its dependents on an empty value first.
|
|
532
551
|
|
|
552
|
+
A record is **declared whole**: its type, its initial value and its parts, in
|
|
553
|
+
one call, never added to after. Each part comes from a builder the call hands
|
|
554
|
+
its declaration, typed by the record's type: `indexes` gets `index` and returns
|
|
555
|
+
the indexes by name, `mutations` gets `mutation` and returns the mutations by
|
|
556
|
+
name, and `migrations` gets `migration` and returns its steps in order. Each is
|
|
557
|
+
checked where the record is declared, every refusal naming the record and the
|
|
558
|
+
part, and the record carries each by its name — `plans.mutations.add_plan`,
|
|
559
|
+
`plans.indexes.by_due` — which a package, a binding (`Record.bind(plans)`,
|
|
560
|
+
`Data.bindPaged(plans, { indexes: ['by_due'] })`) and TypeScript all read.
|
|
561
|
+
|
|
533
562
|
| Form | Body | Use it when |
|
|
534
563
|
|---|---|---|
|
|
535
|
-
| `
|
|
536
|
-
| `
|
|
537
|
-
| `
|
|
564
|
+
| `mutation.reduce(fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
|
|
565
|
+
| `mutation.edit(fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
|
|
566
|
+
| `mutation.patch()` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
|
|
538
567
|
|
|
539
568
|
Every form commits a delta addressed by key, rewriting only the segments it
|
|
540
569
|
touches (a `reduce` of a record that is not a Dict or Set writes it whole), and
|
|
541
570
|
takes `{ runner? }`. In an `edit`, repeated edits of one key fold, a `set` of the
|
|
542
571
|
value a row already holds changes nothing, and a `delete` or `update` of a key
|
|
543
572
|
the record does not hold is a conflict naming the key: the caller's view of the
|
|
544
|
-
record is stale. `e3.mutation.editType(recordType)` is the type of `edit`.
|
|
573
|
+
record is stale. `e3.mutation.editType(recordType)` is the type of `edit`. A
|
|
574
|
+
mutation is named by its key, an identifier; a part's functions are typed by the
|
|
575
|
+
record's type, so name the type (`PlansType`) to write them, in the declaration
|
|
576
|
+
or beside it.
|
|
545
577
|
|
|
546
578
|
```typescript
|
|
547
579
|
const PlanType = StructType({
|
|
548
580
|
title: StringType, owner: StringType, due: DateTimeType, resources: SetType(StringType),
|
|
549
581
|
});
|
|
550
|
-
const
|
|
551
|
-
|
|
552
|
-
const
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
582
|
+
const PlansType = DictType(StringType, PlanType);
|
|
583
|
+
|
|
584
|
+
const plans = e3.record('plans', PlansType, {
|
|
585
|
+
initial: new Map(),
|
|
586
|
+
mutations: (mutation) => ({
|
|
587
|
+
add_plan: mutation.reduce(East.function([PlansType, StringType, PlanType], PlansType, ($, state, id, plan) => {
|
|
588
|
+
const next = $.let(state.copy()); // the state is frozen: copy first
|
|
589
|
+
$(next.insert(id, plan));
|
|
590
|
+
return next;
|
|
591
|
+
})),
|
|
592
|
+
reschedule: mutation.edit(East.function([PlansType, StringType, DateTimeType, e3.mutation.editType(PlansType)], NullType,
|
|
593
|
+
($, state, id, due, edit) => {
|
|
594
|
+
const plan = $.let(state.get(id)); // decodes one segment, not the record
|
|
595
|
+
$(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
|
|
596
|
+
})),
|
|
597
|
+
patch: mutation.patch(), // a view's save
|
|
598
|
+
}),
|
|
599
|
+
});
|
|
565
600
|
```
|
|
566
601
|
|
|
567
602
|
Costs count in segments: a Dict or Set record is cut by key and by each entry's
|
|
@@ -569,54 +604,82 @@ encoded size, so wide rows get segments of a few rows and a one-row edit stays
|
|
|
569
604
|
cheap. A view reads every field of the rows it shows, so keep a bulky payload it
|
|
570
605
|
does not display in a second record keyed the same way.
|
|
571
606
|
|
|
572
|
-
#### Secondary indexes — `
|
|
607
|
+
#### Secondary indexes — `index.on([...fields])`, `index.key(fn)`, `index.keys(fn)`
|
|
573
608
|
|
|
574
609
|
A second collection over a Dict record, in another sort order, maintained in the
|
|
575
|
-
same commit, so a view by an attribute is a page rather than a scan.
|
|
576
|
-
names row fields in index order: one field uses its value as
|
|
577
|
-
use a Struct in that order, and a Set field gives an entry per
|
|
578
|
-
also carries field metadata for `Data.bindPaged(record, { indexes })`
|
|
579
|
-
`Slice.bindRecord` (see **e3-ui**). `key` gives one computed entry
|
|
580
|
-
`keys` returns a Set, one entry per element (a row
|
|
581
|
-
|
|
582
|
-
projection a view renders without touching the record. The functions
|
|
583
|
-
and synchronous, since an index maintained commit by commit must equal
|
|
584
|
-
rebuild to the byte.
|
|
585
|
-
order). Read through one with `index=<name>` on a page or key search; `e3
|
|
610
|
+
same commit, so a view by an attribute is a page rather than a scan.
|
|
611
|
+
`index.on([...])` names row fields in index order: one field uses its value as
|
|
612
|
+
the key, several use a Struct in that order, and a Set field gives an entry per
|
|
613
|
+
element. It also carries field metadata for `Data.bindPaged(record, { indexes })`
|
|
614
|
+
and `Slice.bindRecord` (see **e3-ui**). `index.key(fn)` gives one computed entry
|
|
615
|
+
per row; `index.keys(fn)` returns a Set, one entry per element (a row naming five
|
|
616
|
+
resources appears five times, an empty set not at all); the `value` option of
|
|
617
|
+
each is a projection a view renders without touching the record. The functions
|
|
618
|
+
are pure and synchronous, since an index maintained commit by commit must equal
|
|
619
|
+
its rebuild to the byte. Its name — its key in `indexes` — is an identifier,
|
|
620
|
+
never `primary` (the record's own order). Read through one with `index=<name>` on a page or key search; `e3
|
|
586
621
|
reindex` rebuilds. A deploy builds an index the package declares and the state
|
|
587
622
|
lacks, drops one it no longer declares, and runs nothing for the rest; a build
|
|
588
623
|
is a split task over the record.
|
|
589
624
|
|
|
590
625
|
```typescript
|
|
591
|
-
const
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
626
|
+
const plans = e3.record('plans', PlansType, {
|
|
627
|
+
initial: new Map(),
|
|
628
|
+
indexes: (index) => ({
|
|
629
|
+
by_due: index.on(['due']),
|
|
630
|
+
by_owner_due: index.on(['owner', 'due']),
|
|
631
|
+
by_owner: index.key(East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner), {
|
|
632
|
+
value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
|
|
633
|
+
}),
|
|
634
|
+
by_resource: index.keys(East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources)),
|
|
635
|
+
}),
|
|
636
|
+
mutations: (mutation) => ({ /* add_plan, reschedule, patch — above */ }),
|
|
599
637
|
});
|
|
600
|
-
const pkg = e3.package('planning', '1.0.0',
|
|
601
|
-
plans, addPlan, reschedule, e3.mutation.patch(plans), byDue, byOwnerDue, byOwner, byResource);
|
|
638
|
+
const pkg = e3.package('planning', '1.0.0', plans); // the record whole: its indexes and mutations with it
|
|
602
639
|
```
|
|
603
640
|
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
641
|
+
#### Interval and area indexes — `index.interval`, `index.area`
|
|
642
|
+
|
|
643
|
+
These indexes answer more than an order — as do the series and summary indexes
|
|
644
|
+
below — each with a builder method of its own whose keys the SDK generates.
|
|
645
|
+
Underneath each is an ordinary index — the same build,
|
|
646
|
+
mutation, storage, paging and `e3 reindex` — and the record Slice reads a
|
|
647
|
+
window through either (see **e3-ui**). Their generated keys are query
|
|
648
|
+
machinery: edits and sessions always target the record's primary key.
|
|
649
|
+
|
|
650
|
+
A generated key is a struct whose own fields are e3's: `on` holds the
|
|
651
|
+
declaration's leading fields, by their names, at their key types (a Set field's
|
|
652
|
+
element type) — a struct with no fields when it lists none — and the kind's own
|
|
653
|
+
parts sit beside it. So no name is reserved: a row may name a field anything,
|
|
654
|
+
`area`, `tile` and `__area` among them, and an index may be led by it.
|
|
655
|
+
|
|
656
|
+
| Index | Key | e3's parts |
|
|
657
|
+
|---|---|---|
|
|
658
|
+
| interval | `{ on, node, at }` | the tree node an entry is filed at; the instant it is ordered by, at the `from` field's type |
|
|
659
|
+
| area | `{ on, area }` | the cell's Morton code times 32 plus its level; −1 for a row not yet placed |
|
|
660
|
+
| series | `{ on, key, level, tile }` | the value of the field `key` names; the level an event is filed at, and the start of that level's tile (#1433) |
|
|
661
|
+
| summary | `{ on, key, tile }` | the value of the field `key` names; the tile's start |
|
|
662
|
+
|
|
663
|
+
A definition's `on` is the leading fields alone; its kind's options say the
|
|
664
|
+
rest (`events.indexes.by_time.interval`, `racks.indexes.racks_by_area.area`,
|
|
665
|
+
`jobs.indexes.jobs_by_press_week.series`).
|
|
666
|
+
|
|
667
|
+
**`index.interval({ on?, from, to })`** keeps events visible while they
|
|
668
|
+
overlap a selected time window. `from` and `to` are the row's DateTime or
|
|
669
|
+
Option<DateTime> fields; optional `on` fields are exact leading choices:
|
|
607
670
|
|
|
608
671
|
```typescript
|
|
609
|
-
const
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
|
|
672
|
+
const events = e3.record('events', EventsType, {
|
|
673
|
+
initial: new Map(),
|
|
674
|
+
indexes: (index) => ({
|
|
675
|
+
by_time: index.interval({ from: 'start', to: 'end' }),
|
|
676
|
+
by_status_time: index.interval({ on: ['status'], from: 'start', to: 'end' }),
|
|
677
|
+
}),
|
|
614
678
|
});
|
|
615
679
|
```
|
|
616
680
|
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
independent of duration. The record Slice adapter resolves a half-open
|
|
681
|
+
Its key is `{ on, node, at }`. It stores at most three entries per event per
|
|
682
|
+
leading-field combination, independent of duration. The record Slice adapter resolves a half-open
|
|
620
683
|
`[from, to)` window with at most 55 disjoint bounded queries per exact prefix;
|
|
621
684
|
each matching original record occurs once, including events that started
|
|
622
685
|
earlier or span the whole window. It never scans the record or guesses a
|
|
@@ -632,17 +695,242 @@ primary key. Calendar's `Calendar.events(binding, { slice, entries, … })`
|
|
|
632
695
|
requires a matching interval declaration and the primary `entries` binding to
|
|
633
696
|
confirm saves even after an edit leaves the selected window.
|
|
634
697
|
|
|
698
|
+
**`index.area({ on?, x, y, width?, height?, at?, cell })`** reads the rows
|
|
699
|
+
whose box meets a 2D window — what a floor plan's viewport shows — the interval
|
|
700
|
+
index's twin:
|
|
701
|
+
|
|
702
|
+
```typescript
|
|
703
|
+
// RacksType: Dict<String, { hall: String, x: Option<Float>, y: Option<Float>, w: Float, h: Float }>
|
|
704
|
+
const racks = e3.record('racks', RacksType, {
|
|
705
|
+
initial: new Map(),
|
|
706
|
+
indexes: (index) => ({
|
|
707
|
+
racks_by_area: index.area({ x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01 }),
|
|
708
|
+
racks_by_hall_area: index.area({ on: ['hall'], x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01 }),
|
|
709
|
+
}),
|
|
710
|
+
});
|
|
711
|
+
```
|
|
712
|
+
|
|
713
|
+
| Option | The box's |
|
|
714
|
+
|---|---|
|
|
715
|
+
| `x`, `y` | place: two Float fields, or two Option<Float> fields whose `none` is a row not yet placed |
|
|
716
|
+
| `width`, `height` | size: two Float fields, or neither for a point |
|
|
717
|
+
| `at` | `'corner'`, the default — `x`, `y` is the box's minimum corner, as a zone's is — or `'centre'`, as an item's is |
|
|
718
|
+
| `cell` | finest cell: its side in world units, about the smallest thing drawn — 0.01 for a floor in metres, 1 for one in millimetres. It is rounded down to a power of two, so the keys' arithmetic is exact on every runtime, and has no default: a wrong one costs a silent tenfold read |
|
|
719
|
+
| `on` | exact leading fields, as the interval index's; a Set field fans out |
|
|
720
|
+
|
|
721
|
+
Each option takes only the row's fields of its type, so a wrong one does not
|
|
722
|
+
compile, and the definition checks them again, naming the option, the field and
|
|
723
|
+
its type. The domain is the square ±`cell`·2²⁷ world units around zero
|
|
724
|
+
(±1,048 km at a 0.01 cell, which rounds to 2⁻⁷), in 29 levels. Each row is
|
|
725
|
+
filed once, under the deepest level whose cell is at least twice its box's
|
|
726
|
+
larger side, in the cell holding its box's centre: the key
|
|
727
|
+
`{ on, area: Integer }`. A row not yet placed files under `area` −1, which the
|
|
728
|
+
record Slice reads as the unplaced clause; one outside the domain, not finite,
|
|
729
|
+
or larger than half of it files under the root, which every window reads. The index **carries its rows**: each
|
|
730
|
+
entry's projection is the whole row, so a window reads the index alone and no
|
|
731
|
+
record segment, and it takes no `value`. The record Slice reads a viewport,
|
|
732
|
+
grown by an eighth of its larger side on each side, as at most 266 disjoint
|
|
733
|
+
searches per exact prefix, sent in one request, which together hold every row
|
|
734
|
+
whose box meets it, each once — beside rows of the cells searched that it
|
|
735
|
+
misses, which the reader draws or not.
|
|
736
|
+
|
|
737
|
+
#### Series and summary indexes — `index.series`, `index.summary`
|
|
738
|
+
|
|
739
|
+
What a planning board reads: the events on a block of keys — a Plan's
|
|
740
|
+
resources — over a span of time, and, zoomed out past what a screen can draw,
|
|
741
|
+
how busy each key is. Both file a row's event by the key it is on and by time
|
|
742
|
+
tiles — `day`, `week` (ISO, from Monday), `month`, `quarter` or `year`, in UTC,
|
|
743
|
+
half-open — the key before the time, so a key's entries over any span are
|
|
744
|
+
contiguous: a series index's `{ on, key, level, tile }`, a summary index's
|
|
745
|
+
`{ on, key, tile }`.
|
|
746
|
+
|
|
747
|
+
```typescript
|
|
748
|
+
// JobsType: Dict<String, { press: String, start: Option<DateTime>, end: Option<DateTime>, … }>
|
|
749
|
+
const jobs = e3.record('jobs', JobsType, {
|
|
750
|
+
initial: new Map(),
|
|
751
|
+
indexes: (index) => ({
|
|
752
|
+
jobs_by_press_week: index.series({ key: 'press', from: 'start', to: 'end', tile: 'week' }),
|
|
753
|
+
jobs_by_press_month: index.summary({ key: 'press', from: 'start', to: 'end', tile: 'month' }),
|
|
754
|
+
}),
|
|
755
|
+
});
|
|
756
|
+
```
|
|
757
|
+
|
|
758
|
+
| Option | |
|
|
759
|
+
|---|---|
|
|
760
|
+
| `key` | the field naming what an event is on: any field but a Set |
|
|
761
|
+
| `from`, `to` | the event's start and end: DateTime or Option<DateTime> fields. A row with no start is filed nowhere; no end, or one at or before the start, is an instant |
|
|
762
|
+
| `tile` | a series index's finest level, and a summary index's tile: `day`, `week`, `month`, `quarter` or `year` |
|
|
763
|
+
| `on` | exact leading fields, before the key; a Set field fans out |
|
|
764
|
+
|
|
765
|
+
**A series index** is an ordinary index whose keys the SDK generates, and it
|
|
766
|
+
**carries its rows**, as the area index does. It files each event **once**, at
|
|
767
|
+
the finest level from its `tile` up to the year whose shortest tile holds the
|
|
768
|
+
event's length — a day, a week, 28 days, 90 days, 365 days — under that level's
|
|
769
|
+
tile holding its start; an instant at `tile`; past 365 days, under each year it
|
|
770
|
+
touches. An event filed at a level ends within the tile after the one holding
|
|
771
|
+
its start, so a reader reads a block of keys over any span from the index alone
|
|
772
|
+
as one range a key a level — that level's tiles from two before the span to one
|
|
773
|
+
after it — in one request, whatever the span's width. **A summary index** holds,
|
|
774
|
+
under each key and tile, `{ count, ms }`: how many rows touch the tile, and the
|
|
775
|
+
milliseconds of it they cover — a row's shares
|
|
776
|
+
add up to its length, and an instant's is none. Each entry stands for every row
|
|
777
|
+
of its key and names none, so its window's `key` is Null, a read of it joins
|
|
778
|
+
nothing, and a read that needs its rows — a joined read, a placement
|
|
779
|
+
(`datasetPlace`) or a membership read (`datasetWithin`) — is refused, naming the
|
|
780
|
+
index; the record's describe says it is a summary. A write
|
|
781
|
+
changes its totals by what its rows took away and gave — its program reads the
|
|
782
|
+
record, never the summary — and the apply adds that to each total, dropping one
|
|
783
|
+
whose count reaches zero; the totals are Integers, so a maintained summary is the
|
|
784
|
+
one `e3 reindex` sums, byte for byte, and a build split into pieces adds the
|
|
785
|
+
pieces' totals of a key. A change that would leave a total no rebuild sums — a
|
|
786
|
+
count below zero, or time with no row — fails the write, naming the total and
|
|
787
|
+
`e3 reindex`, which rebuilds it; a total past the largest Integer fails the
|
|
788
|
+
write or the build, naming a finer tile.
|
|
789
|
+
|
|
790
|
+
A series index files an event past a year under each year it touches, as many
|
|
791
|
+
as 4,096: one touching more is refused where it is written, naming it and its
|
|
792
|
+
span. A summary splits an event over its tiles, as many as 4,096 — eleven years
|
|
793
|
+
of days — and one touching more is refused the same way, naming the tile:
|
|
794
|
+
declare a coarser tile. Each option takes only the row's fields of its type, as
|
|
795
|
+
the area index's do, and the definition checks them again; a `value` is
|
|
796
|
+
refused, and so is an `on` field that is the key, the start or the end: the key
|
|
797
|
+
follows the leading fields, and the tiles are cut by the start and the end.
|
|
798
|
+
|
|
799
|
+
#### Nested series indexes — `index.nested`
|
|
800
|
+
|
|
801
|
+
What a planning board reads when a record's rows nest what it draws: a
|
|
802
|
+
machine's jobs and its load a day at a time, a hall's members, an assembly's
|
|
803
|
+
parts to any depth. One declaration a record lists every collection its rows
|
|
804
|
+
nest, each field with a builder saying what it holds. e3 files each item apart
|
|
805
|
+
from its row — once, by its collection, its row, its node and its time — keeps
|
|
806
|
+
each node's totals a summary tile at a time, and each parent's subtotals at
|
|
807
|
+
every level; and the record's other indexes carry the rows without those
|
|
808
|
+
collections, so a page of rows reads none of them and a window of items reads
|
|
809
|
+
no row.
|
|
810
|
+
|
|
811
|
+
```typescript
|
|
812
|
+
// MachinesType: Dict<String, { name, site, status, jobs: Dict<String, Job>, load: Dict<DateTime, Float> }>
|
|
813
|
+
const machines = e3.record('machines', MachinesType, {
|
|
814
|
+
initial: new Map(),
|
|
815
|
+
indexes: (index) => ({
|
|
816
|
+
machines_in_time: index.nested({
|
|
817
|
+
collections: {
|
|
818
|
+
jobs: index.nested.items({ from: 'start', to: 'end' }), // things people drag: by id, their times fields
|
|
819
|
+
load: index.nested.values(), // a value per instant: keyed by the instant
|
|
820
|
+
},
|
|
821
|
+
tile: 'week',
|
|
822
|
+
summary: 'month',
|
|
823
|
+
}),
|
|
824
|
+
machines_by_site: index.on(['site']), // pages each machine without its jobs and load
|
|
825
|
+
}),
|
|
826
|
+
});
|
|
827
|
+
// Each member's load, and each hall's mean of its members', at every level from the day up.
|
|
828
|
+
const halls = e3.record('halls', HallsType, {
|
|
829
|
+
indexes: (index) => ({
|
|
830
|
+
halls_in_time: index.nested({
|
|
831
|
+
collections: { members: index.nested.rows({ load: index.nested.values({ rollup: 'mean' }) }) },
|
|
832
|
+
tile: 'day',
|
|
833
|
+
summary: 'month',
|
|
834
|
+
}),
|
|
835
|
+
}),
|
|
836
|
+
});
|
|
837
|
+
// Parts to any depth, each part's jobs filed under every part above it too.
|
|
838
|
+
const assemblies = e3.record('assemblies', AssembliesType, {
|
|
839
|
+
indexes: (index) => ({
|
|
840
|
+
assemblies_in_time: index.nested({
|
|
841
|
+
collections: {
|
|
842
|
+
parts: index.nested.rows({
|
|
843
|
+
jobs: index.nested.items({ from: 'start', to: 'end' }, { rollup: true }),
|
|
844
|
+
parts: index.nested.recursive(),
|
|
845
|
+
}),
|
|
846
|
+
},
|
|
847
|
+
tile: 'week',
|
|
848
|
+
summary: 'month',
|
|
849
|
+
}),
|
|
850
|
+
}),
|
|
851
|
+
});
|
|
852
|
+
```
|
|
853
|
+
|
|
854
|
+
| Builder (`index.nested.…`) | A field holding |
|
|
855
|
+
|---|---|
|
|
856
|
+
| `items({ from, to })`, `items({ at })` | `Dict<String, Struct>`: things people drag, by id; their times are DateTime or Option<DateTime> fields — no start files an item nowhere, no end (or one at or before its start) makes it an instant. `{ rollup: true }` files each under every ancestor too |
|
|
857
|
+
| `values({ rollup? })` | `Dict<DateTime, V>`: a value per instant. `rollup` keeps every parent's subtotals |
|
|
858
|
+
| `rows({ … })` | `Dict<String, Struct>`: child rows, their own collections declared the same way |
|
|
859
|
+
| `recursive()` | `Dict<String, Self>`: more rows like the one listing it, to any depth |
|
|
860
|
+
|
|
861
|
+
TypeScript types `collections` field by field from the record's type alone: a
|
|
862
|
+
misnamed time, or a builder on the wrong kind of field, is an error where it is
|
|
863
|
+
written, and its message says what fits. The definition checks the same again,
|
|
864
|
+
in words, for a declaration TypeScript could not see. A record has one nested
|
|
865
|
+
series index, which takes no `on` and no `value`.
|
|
866
|
+
|
|
867
|
+
**What it files.** Every entry is keyed `{ collection, row, path, run, level,
|
|
868
|
+
tile, item }` (e3-types' `NestedEntryKeyType`): the collection's place, the
|
|
869
|
+
field names from the row down (`members` then `load`); the row's key; the Dict
|
|
870
|
+
keys from the row down to the node (`[]` for the row, `m-3` for a member); the
|
|
871
|
+
run; the level and tile; and the item. A place and a path are held as a key
|
|
872
|
+
holds a list, folded from the last: `.step (key="members", rest=.step
|
|
873
|
+
(key="load", rest=.end))`. Each run of a node is one contiguous range a level:
|
|
874
|
+
|
|
875
|
+
| Run | Holds |
|
|
876
|
+
|---|---|
|
|
877
|
+
| `items` | each item once, as a series index files an event — at the level its length fits, past a year under each year — carrying it (`.item .jobs (…)`); a value, and an `{ at }` item, at the declared tile alone |
|
|
878
|
+
| `totals` | where `summary` is declared, each node's `{ count, ms, values }` a summary tile at a time; for a Float or Integer collection, `values` is the sum, the least, the greatest and the last there, taken in key order |
|
|
879
|
+
| `subtotals(rollup)` | for `values({ rollup })`, every parent's subtotal by each rollup at each level from `tile` to the year, where a descendant's value touches the tile: a node holding values folds its own, a parent aggregates its children's, in key order |
|
|
880
|
+
| `subtree`, `subtreeTotals` | for `items(…, { rollup: true })`, each item under every ancestor too, `.under (path, id)`, and each ancestor's totals of them |
|
|
881
|
+
|
|
882
|
+
A rollup is `'sum'`, `'mean'` (a mean of means up the tree), `'min'`, `'max'`,
|
|
883
|
+
`'count'` (the leaf rows beneath that hold a value), a fold and an aggregate
|
|
884
|
+
named apart (`{ fold: 'last', aggregate: 'sum' }`, a balance), or several.
|
|
885
|
+
|
|
886
|
+
**Reading.** A reader searches the index's leading fields — `[collection, row,
|
|
887
|
+
path, run, level, tile]`, each as East prints it — as a range a row a level: a
|
|
888
|
+
block of fifty machines' jobs over three weeks is fifty times four ranges in one
|
|
889
|
+
`datasetFindKeys` request, and one `datasetGetRuns` read of the entries they
|
|
890
|
+
select, each job carried whole; a parent's week subtotals are one range; a
|
|
891
|
+
part's subtree is one range a level.
|
|
892
|
+
|
|
893
|
+
**The lean rows.** Beside a nested series index, every other index of the
|
|
894
|
+
record that carries nothing or the row carries the row without the listed
|
|
895
|
+
collections — each listed collection empty, child rows kept with theirs empty —
|
|
896
|
+
so a page of rows read without a join reads no nested item and no segment of
|
|
897
|
+
the record's own. An index with a `value` of its own carries its own. A
|
|
898
|
+
binding of the record's indexes lists the nested index by name beside any that
|
|
899
|
+
carry the lean rows — `Data.bindPaged(machines, { indexes: ['machines_by_site',
|
|
900
|
+
'machines_in_time'] })` (see **e3-ui**).
|
|
901
|
+
|
|
902
|
+
**Its span.** Beside every nested series index e3 keeps its span, an index of
|
|
903
|
+
the record's named `<index>_span` (e3-types' `nestedSpanName`), built with it
|
|
904
|
+
and never declared: under each collection's place and each month its items or
|
|
905
|
+
values touch (`NestedSpanKeyType`, `{ collection, tile }`), how many of them
|
|
906
|
+
touch it and the milliseconds of it its items cover, summed across the rows as a
|
|
907
|
+
summary index's totals are, and a month let go once nothing touches it. A
|
|
908
|
+
collection's first and last month held are its span across the record — what a
|
|
909
|
+
Plan's horizon covers for the series reading it — and its months' counts are
|
|
910
|
+
how much it holds in each, which the horizon's strip draws as its bars (see
|
|
911
|
+
**e3-ui**). Reading it is one search a collection, an exact match on its
|
|
912
|
+
place, and one read of the first and last entry of each. An index the record
|
|
913
|
+
declares under that name is refused.
|
|
914
|
+
|
|
915
|
+
**Writes.** A row a write updates by a patch is maintained by the patch: e3
|
|
916
|
+
walks it along the declaration — each collection's Dict patch, and each child
|
|
917
|
+
rows' — and compares the entries of only the items it names, with the totals of
|
|
918
|
+
the summary tiles they touch and the subtotals of the tiles they touch at each
|
|
919
|
+
level. A replace on the way, which a recursive type's patch always is, and an
|
|
920
|
+
inserted or deleted row are compared whole. Both ways give the same delta, and a maintained
|
|
921
|
+
index is the one `e3 reindex` rebuilds, byte for byte.
|
|
922
|
+
|
|
635
923
|
#### Migrations — how a record's type changes
|
|
636
924
|
|
|
637
925
|
| Form | Function | Runs as |
|
|
638
926
|
|---|---|---|
|
|
639
|
-
| `
|
|
640
|
-
| `
|
|
641
|
-
| `
|
|
927
|
+
| `migration.value(name, fn)` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
|
|
928
|
+
| `migration.rows(name, fn)` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
|
|
929
|
+
| `migration.rekey(name, fn)` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
|
|
642
930
|
|
|
643
|
-
A record's steps are one chain:
|
|
644
|
-
step
|
|
645
|
-
|
|
931
|
+
A record's steps are one chain: the list `migrations` returns, in order, each
|
|
932
|
+
step taking the record as the step before it leaves it, and the last leaving it
|
|
933
|
+
as its declared type — each checked where the record is declared.
|
|
646
934
|
A step's name is an identifier, unique on the record, and a workspace records the
|
|
647
935
|
steps it has applied by name — so moving a step's code, or re-exporting it with a
|
|
648
936
|
newer e3, does not run it again, and an edit to an applied step's body is not
|
|
@@ -651,12 +939,14 @@ detected. The functions are pure, synchronous East.
|
|
|
651
939
|
```typescript
|
|
652
940
|
const RowV1Type = StructType({ title: StringType });
|
|
653
941
|
const RowV2Type = StructType({ title: StringType, owner: StringType });
|
|
654
|
-
const plans = e3.record('plans', DictType(StringType, RowV2Type),
|
|
655
|
-
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
// The next version adds its step after this one
|
|
659
|
-
|
|
942
|
+
const plans = e3.record('plans', DictType(StringType, RowV2Type), {
|
|
943
|
+
initial: new Map(),
|
|
944
|
+
migrations: (migration) => [
|
|
945
|
+
migration.rows('add_owner', East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' }))),
|
|
946
|
+
// The next version adds its step here, after this one.
|
|
947
|
+
],
|
|
948
|
+
});
|
|
949
|
+
const pkg = e3.package('planning', '2.0.0', plans);
|
|
660
950
|
```
|
|
661
951
|
|
|
662
952
|
What a deploy does with each record, decided before it writes anything:
|
|
@@ -714,7 +1004,7 @@ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
|
|
|
714
1004
|
| `e3 workspace copy [repo] <from> <to>` | Make `<to>` what `<from>` is now — its package and every input, record and task output, so nothing runs again — made, or replaced whole with its runs. It writes refs only, so its cost follows the number of datasets, never their size; a write to either afterwards leaves the other as it was. Refused while a dataflow runs in `<to>`. |
|
|
715
1005
|
| `e3 workspace deploy [repo] <ws> <pkg[@ver]>` | Deploy an imported package. |
|
|
716
1006
|
| `… --from-zip <zip>` · `… --from-source <src.ts> [--functions <manifest…>]` | Import the zip (or bundle the source) and deploy, creating the workspace. |
|
|
717
|
-
| `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on; `--memory`: what those may reserve (local). |
|
|
1007
|
+
| `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on, and with `--from-source` the workspace packages exporting their functions at once; `--memory`: what those may reserve (local). |
|
|
718
1008
|
| `e3 workspace export [repo] <ws> <zip> [--name <n>] [--version <v>]` | The workspace's state as a package. |
|
|
719
1009
|
| **Datasets** | |
|
|
720
1010
|
| `e3 dataset get [repo] <ws.name> [-f east\|json\|beast2]` | Print a value. |
|
|
@@ -730,7 +1020,7 @@ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
|
|
|
730
1020
|
| **Records** (`-w <ws>` required) | |
|
|
731
1021
|
| `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
|
|
732
1022
|
| `e3 seed <repo> <record> <file.beast2> -w <ws> [-j <n>] [--memory <size>]` | Commit a record's first state from a beast2 file, as its `$init`: for a record declared with no initial value, unassigned until then (local). The file is taken in as a `file` source is, never read whole; a record that holds a state is refused. |
|
|
733
|
-
| `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target
|
|
1023
|
+
| `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target — a summary's totals added to as `±n`; local). |
|
|
734
1024
|
| `e3 reindex <repo> <record> -w <ws> [--index <name>] [-j <n>] [--memory <size>]` · `e3 compact <repo> <record> -w <ws>` | Rebuild indexes (local) · collapse the history to a `$compact` root, the state kept. |
|
|
735
1025
|
| **Development** | |
|
|
736
1026
|
| `e3 watch <src.ts> <repo> <ws> [--start] [--schema <p>] [--inputs <p>] [--abort-on-change] [--functions <manifest…>] [-j <n>] [--memory <size>]` | Redeploy on each change (and run, with `--start`). |
|
|
@@ -748,7 +1038,10 @@ peak a unit of its stage has reached in the run, so a stage runs its first unit
|
|
|
748
1038
|
alone and then fans out; on Linux and macOS a guard stops the newest unit when
|
|
749
1039
|
the runners together pass the budget, and runs it again once it fits. A deploy
|
|
750
1040
|
takes its `file` sources in `-j` at a time, their pieces' intake units running
|
|
751
|
-
under the same budget.
|
|
1041
|
+
under the same budget. A source deploy, or a `watch`, exports the workspace
|
|
1042
|
+
packages its imported functions come from `-j` at a time, a process each, on
|
|
1043
|
+
this machine; against a server, as many at once as this machine has CPUs. The
|
|
1044
|
+
commands that run East — `workspace deploy`,
|
|
752
1045
|
`dataset set --from-file`, `dataflow run`, `run`, `call`, `mutate`, `seed`,
|
|
753
1046
|
`reindex` and `watch` — take the flags for a local repository; against a server the CLI
|
|
754
1047
|
refuses them, since the server runs the work under its own budget
|
|
@@ -759,7 +1052,11 @@ line once it is in: its size, time and rate, and how it was taken in
|
|
|
759
1052
|
(`unchanged, already in the store`, `taken in by east-c`, or `carried` for a
|
|
760
1053
|
value that is not a collection), with why the first time a runner fell back to
|
|
761
1054
|
another. A terminal also keeps a live line for the files in flight, with their
|
|
762
|
-
pieces, the rate and the time left across them.
|
|
1055
|
+
pieces, the rate and the time left across them. A source deploy prints a line
|
|
1056
|
+
for each workspace package whose functions it links: `exported <n> function(s)
|
|
1057
|
+
of <pkg> (east-py export-functions)`, or `reused <n> function(s) of <pkg>,
|
|
1058
|
+
unchanged since its last export` when the export cache held them. Against a
|
|
1059
|
+
server, the deploy line
|
|
763
1060
|
says what the job is doing, such as migrating a record or building an index;
|
|
764
1061
|
the deliveries then upload a few at a time, each printing a line once it is in,
|
|
765
1062
|
and the live line says what the server's commit is doing with each in flight.
|
|
@@ -819,8 +1116,8 @@ naming the holder: its pid, boot id, command and when it took the lock.
|
|
|
819
1116
|
|---|---|
|
|
820
1117
|
| Repositories | `repoList(url, opts)`, `repoCreate(url, name, opts)`, `repoRemove(url, name, opts)`, `repoStatus(url, repo, opts)`, `repoRecord` (its release and upgrades), `repoGc(url, repo, gcRequest, opts)` — a job it polls; `repoGcStart` and `repoGcStatus` apart |
|
|
821
1118
|
| Packages | `packageList(url, repo, opts)`, `packageGet(url, repo, name, version, opts)`, `packageImport(url, repo, zipBytes, opts)`, `packageExport(url, repo, name, version, opts)` → zip bytes, `packageRemove` |
|
|
822
|
-
| Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
|
|
823
|
-
| Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
|
|
1119
|
+
| Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceBundle(url, repo, ws, opts)` → the workspace as it is, as a bundle's bytes — its package, every dataset's ref, the executions its tasks' outputs come from, and every object those reach — and `workspaceBundleImport(url, repo, ws, bundle, opts)` → the workspace made from one under a name no workspace has, in another repository or another e3 (`workspace_exists`, `package_exists`, `bundle_invalid`), its tasks up to date with no run; a host mounts these routes where a bundle fits in a request: the local server, and e3-web, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
|
|
1120
|
+
| Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, and several reads in one request (at most `DATASET_BATCH_MAX`, 10,000, each at one content hash, in a body of at most 16 MiB: a larger one is refused 413, `payload_too_large`): `datasetFindKeys(…, { queries, index?, hash? }, opts)` — key searches, `datasetGetRows(…, { keys, hash? }, opts)` — a Dict's rows by `.east` key, each with its position — or `datasetGetRows(…, { keysOf: { path, index, runs, hash }, hash? }, opts)`, by the keys the entries of some runs of another record's index name, which the server reads from the index so none is sent (a drawing's details for its window's cells), a stale `keysOf.hash` thrown as `RunsHashMismatchError` naming the record's content — and `datasetGetRuns(…, { runs, hash, index?, join? }, opts)` — the entries of several runs as one window, a rows or runs answer holding the first `count` the page budget reaches (by runs, of `runEntries`); and where a narrowing's rows sit in a record, from its index alone: `datasetPlace(…, { index, runs, offset?, limit?, hash }, opts)` → `{ segments, placed, runEntries, totalElements, hash }` — how many of the runs' entries each record segment holds (`{ segment, offset, count, rows }`), for the first `placed` of a window from `offset` (the next window from `offset + placed`), and `datasetWithin(…, { index, runs, keys, hash }, opts)` → `{ within, hash }` — whether each index entry key (`(ik=…, k=…)`) lies in one of the runs, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
|
|
824
1121
|
| Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
|
|
825
1122
|
| Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts)` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress? })` — a program over a dataset's pieces, as a job it polls, under the same rule (or `splitCallLaunch`, `splitCallStatus` and `splitCallExplain`) |
|
|
826
1123
|
| Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
|
|
@@ -834,7 +1131,7 @@ takes a runner, `new LocalTaskRunner(repo)`.
|
|
|
834
1131
|
|---|---|
|
|
835
1132
|
| Repositories | `repoInit(path)`, `repoFind(startPath?)`, `repositoryOpen(storage, repo)` (checks the repository and applies the upgrades it owes), `repositoryUpgradeStep(storage, repo, { budgetMs, waitMs? })` → `{ owed }` — a host's job applies them so, a part per run under its time limit, each part taking a step up where the last stopped, `repoGc(storage, repo, { dryRun?, minAge?, keepRuns?, keepDays?, retention?, executionAlive? })` — holding the repository still, or, with `retention: { windowMs }`, beside running work: an object goes once unreachable for the window and not written or re-referenced since; given a runner's `executionAlive`, an attempt recorded running that cannot finish is recorded interrupted and pruned; `repoGcStep(storage, repo, step, { windowMs, pruneMs?, markMs?, concurrency?, executionAlive?, … })` runs that one step at a time, each returning the next (`GcStepType`), `pruneMs` and `markMs` bounding a prune step and a mark step for a host with a time limit |
|
|
836
1133
|
| Packages | `packageImport(storage, repo, zipPath)`, `packageExport(storage, repo, name, version, zipPath)`, `packageList`, `packageRemove` |
|
|
837
|
-
| Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
|
|
1134
|
+
| Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceBundleExport(storage, repo, ws, writable, { have? })` — the workspace as a bundle to a WHATWG stream, leaving out the objects `have` says the receiving repository holds — and `workspaceBundleImport(storage, repo, ws, source)` — the workspace made from a bundle (a stream or chunks) under a name no workspace has, its objects stored as they come and its refs and record written once the repository holds everything they name, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
|
|
838
1135
|
| Datasets | `workspaceGetDataset(storage, repo, ws, treePath)`, `workspaceSetDataset(storage, repo, ws, treePath, value, type)`, `datasetAdoptFile(storage, repo, ws, treePath, file, { runner, onProgress? })` → `{ hash, size, segments, rows, taken, runners? }`, `taken` being `known`, `carried` or `taken` (by the `runners` named); `deliveryAdoptStored(storage, repo, ws, { address, type }, sha256, { runner?, onProgress?, signal? })` — a delivery the store holds taken in as a stored value of the type, naming no dataset, for a record's first state |
|
|
839
1136
|
| Runs | `dataflowExecute(storage, repo, ws, options?)`; `LocalOrchestrator` to start, poll and cancel a run |
|
|
840
1137
|
| Records | `recordMutate(storage, runner, repo, ws, record, mutation, args, { actor })`, `recordHistory`, `recordDescribe`, `recordCompact`, `recordReindex`; `recordSystemCommit(storage, runner, repo, ws, record, { name, target, actor, args? })` — `$rollback` to a commit, `$restore` of a state at the record's migrations, and `$init` on a record declared with no initial value: its first state, from a stored value, with the whole chain applied |
|
|
@@ -906,6 +1203,7 @@ package is served beside the standard ones:
|
|
|
906
1203
|
| `serveE3` option | Default | Effect |
|
|
907
1204
|
|---|---|---|
|
|
908
1205
|
| `units` | — | starts a unit worker: a Web Worker whose script calls `serveUnits()` |
|
|
1206
|
+
| `width` | the cores | the most units that run at once, and so the most unit workers, each with a heap of its own |
|
|
909
1207
|
| `persist` | `true` | `false` keeps repositories in memory, gone with the page |
|
|
910
1208
|
| `name` | `'e3'` | the storage's name: e3 workers of one name share their repositories, in one tab or several |
|
|
911
1209
|
| `identify` | none | each request's caller, `{ sub, email?, roles }`, from the request; a request to a repository whose caller it does not identify is answered 401. None: the page's one caller may do everything |
|
|
@@ -970,7 +1268,14 @@ not had, in place; one a newer e3 upgraded is refused, naming that release.
|
|
|
970
1268
|
environment — and its inputs' hashes, so a task re-runs only when one of those
|
|
971
1269
|
changes (`--force` bypasses the cache, and `--force-task <task>` for the tasks
|
|
972
1270
|
it names: a task whose code e3 cannot see changed, such as a platform
|
|
973
|
-
function's, or one reading data from outside e3). A
|
|
1271
|
+
function's, or one reading data from outside e3). A task's program holds the
|
|
1272
|
+
source locations of its body and of every function it embeds, an imported
|
|
1273
|
+
python or node function's too, since its errors, a caught error's `stack` and
|
|
1274
|
+
the closures it returns name them: an edit that moves one of those lines — a
|
|
1275
|
+
comment or an import added above a function — re-runs the task. One that leaves
|
|
1276
|
+
them where they were — another function, the end of a module, another package —
|
|
1277
|
+
re-runs only the tasks whose environment it is in, their runner's `{ custom }`
|
|
1278
|
+
packages. A change to an input re-runs its
|
|
974
1279
|
consumers, even during a run. A split task is cached a unit at a time: each piece
|
|
975
1280
|
and each merge is its own execution, and pieces are cut by content, so an append
|
|
976
1281
|
or an insertion re-runs only the pieces it reaches and the merges above them.
|