@elaraai/east-claude-plugin 1.0.97 → 1.0.99

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: e3
3
- description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack enroll/policy, dataflow run --rack, watch --start --rack, optional e3-api-server --rack, (7) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui()."
3
+ description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations, declared whole in one e3.record(name, type, { initial, indexes, mutations, migrations }) call: mutation.reduce/edit/patch; index.on/key/keys, index.interval / index.area for reads by a time window or a 2D window, index.series / index.summary for a block of keys over a span of time tiles and their totals, and index.nested for what a record's rows nest — jobs, a value a day, child rows, trees — filed item by item, with totals and parents' subtotals; and migrated between versions (migration.value/rows/rekey, in order), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack enroll/policy, dataflow run --rack, watch --start --rack, optional e3-api-server --rack, (7) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui()."
4
4
  ---
5
5
 
6
6
  # East Execution Engine (e3)
@@ -66,8 +66,14 @@ What do you need?
66
66
  │ │ └─ Split over an input, in pieces → wrap that input: e3.partition(dataset, { by? })
67
67
  │ ├─ A shell command → e3.customTask(name, [inputs], outputType, command)
68
68
  │ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name, persists nothing
69
- │ ├─ Audited state → e3.record + e3.mutation.reduce | edit | patch, e3.recordIndex
70
- │ ├─ A record's type changes → e3.migration.value | rows | rekey(name, record, fn, { after? })
69
+ │ ├─ Audited state → e3.record(name, type, { initial?, indexes?, mutations?, migrations? }), declared whole
70
+ │ │ ├─ Its writes → mutations: (mutation) => ({ name: mutation.reduce | edit | patch })
71
+ │ │ ├─ An order to page in → indexes: (index) => ({ name: index.on([...fields]) | index.key(fn) | index.keys(fn) })
72
+ │ │ ├─ Read by a time window or a 2D window → index.interval | index.area
73
+ │ │ ├─ Read a block of keys over time tiles, or their totals → index.series | index.summary
74
+ │ │ └─ Read what rows nest (jobs, a value a day, child rows, trees) a tile at a time, with totals and subtotals
75
+ │ │ → index.nested({ collections, tile, summary? })
76
+ │ ├─ A record's type changes → migrations: (migration) => [migration.value | rows | rekey(name, fn), …], in order
71
77
  │ ├─ A UI task → ui(name, [inputs], fn) from @elaraai/e3-ui (the e3-ui skill)
72
78
  │ ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
73
79
  │ └─ Bundle and export → e3.package(name, version, ...items); e3.export(pkg, zipPath)
@@ -333,17 +339,26 @@ const shout = e3.task('shout', [greet.output],
333
339
  - **An East function written in python, or in another node package**:
334
340
  `East.importFunction(pkg, name, FunctionType(...))` in the body. `e3.export`
335
341
  resolves it and embeds its IR, so the task is pure IR on any runner, its
336
- locations kept so an error or a profile names the function's own lines. A uv
342
+ locations kept so an error or a profile names the function's own lines (so
343
+ moving them re-runs the task: **Caching**, below). A uv
337
344
  workspace member is found by name in `uv.lock`, and its root module's
338
- `east_functions` dict is exported with `east-py export-functions` (from the
339
- project's `.venv`, else `EAST_PY` or PATH); an npm member is found by its
340
- `package.json` name, and its built `./functions` export's `eastFunctions` is
341
- exported with `east-node export-functions` (`EAST_NODE`). A package built
342
- elsewhere is passed as its manifest — `{ functions: [...] }` / `--functions` —
343
- which wins. The declared type must equal the exported one exactly, and each
344
- platform function it calls must be provided by the task's runner, by name or
345
- by family (`east-py-std` ≡ `@elaraai/east-node-std` ≡ `east-c-std`;
346
- `east-py-io` ≡ `@elaraai/east-node-io`).
345
+ `east_functions` dict is exported with `east-py export-functions` (`EAST_PY`
346
+ when set, else the project's `.venv`, else PATH); an npm member is found by
347
+ its `package.json` name, and its built `./functions` export's `eastFunctions`
348
+ is exported with `east-node export-functions` (`EAST_NODE`). Each member is
349
+ exported in one process however many tasks import from it, the members side
350
+ by side, as many at once as the export has cores (a deploy's `-j`,
351
+ `e3.export`'s `cores`). A python member's manifests are kept in
352
+ `node_modules/.cache/e3/functions` under the directory the export runs in,
353
+ keyed on its sources and those of the local packages it depends on, the
354
+ workspace's `uv.lock` and `pyproject.toml`, the east-py exporting it and the
355
+ functions and providers each manifest is for, so a redeploy whose python has
356
+ not changed runs no export; deleting the directory clears it, and a node
357
+ member is exported every time. A package built elsewhere is passed as its
358
+ manifest — `{ functions: [...] }` / `--functions` — which wins. The declared
359
+ type must equal the exported one exactly, and each platform function it calls
360
+ must be provided by the task's runner, by name or by family (`east-py-std` ≡
361
+ `@elaraai/east-node-std` ≡ `east-c-std`; `east-py-io` ≡ `@elaraai/east-node-io`).
347
362
  - **Environments** are derived at export from each `{ custom }` platform: the
348
363
  providing workspace package's closure (`pyproject.toml` + `uv.lock` + sdists,
349
364
  or `package.json` + lockfile + `npm pack`) is captured into the package and
@@ -499,7 +514,7 @@ const change = e3.streamTask('change', {
499
514
  | `e3.customTask(name, inputs, outputType, command, { environment? })` | A task that runs a bash script: `command($, inputPaths, outputPath)` builds it from the staged beast2 input files, and it writes the output as a beast2 file. |
500
515
  | `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API) and its result returned inline: not in the dataflow, and a call persists nothing. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A server bounds a call by a timeout and a 1 MiB result, so long work belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
501
516
  | `e3.package(name, version, ...items)` | Bundles the items and everything they depend on — pass the last tasks, records and functions; a package passed in contributes its contents. The name and version must be valid file names. |
502
- | `e3.export(pkg, zipPath, { functions? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself; `functions` lists manifests built elsewhere). |
517
+ | `e3.export(pkg, zipPath, { functions?, cores? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself, `cores` packages at once, the CPUs by default; `functions` lists manifests built elsewhere). |
503
518
 
504
519
  ```typescript
505
520
  const process = e3.customTask('process', [rawData], StringType,
@@ -511,15 +526,19 @@ await e3.export(e3.package('tools', '1.0.0', process, add), '/tmp/tools.zip');
511
526
 
512
527
  ### Records — audited state, written only through mutations
513
528
 
514
- `e3.record(name, type, initialValue?)` is a dataset at `.records.<name>` that
515
- tasks read and react to like any input, but that only its **mutations** write:
529
+ `e3.record(name, type, { initial?, indexes?, mutations?, migrations? })` is a
530
+ dataset at `.records.<name>` that tasks read and react to like any input, but
531
+ that only its **mutations** write:
516
532
  a raw `dataset set` is refused. A mutation is pure, synchronous East — no async
517
533
  body and no platform call, since the compare-and-swap retry runs it again
518
534
  against fresher state — named by an identifier. Each commit records its parent,
519
535
  the state, the mutation, its arguments, the actor and the time. Deploy mints a
520
536
  `$init` commit from the initial value, and a redeploy keeps state and history.
521
- The record's version vector carries the commit hash, so even a mutation that
522
- leaves the state unchanged triggers downstream tasks.
537
+ A large initial value is given as a function that makes it —
538
+ `initial: () => East.compile(generate, [])(50_000n)` — called once, as the
539
+ package is exported, never where the declaration is only read (a page binding
540
+ the record). The record's version vector carries the commit hash, so even a
541
+ mutation that leaves the state unchanged triggers downstream tasks.
523
542
 
524
543
  Declared with no initial value, a record is **unassigned** until its first
525
544
  state is committed — `e3 seed <repo> <record> <file.beast2> -w <ws>` from a
@@ -530,38 +549,54 @@ and a mutation on it is refused until then. So a record whose first state is
530
549
  data that arrives after the deploy — a customer list, a reference table —
531
550
  never runs its dependents on an empty value first.
532
551
 
552
+ A record is **declared whole**: its type, its initial value and its parts, in
553
+ one call, never added to after. Each part comes from a builder the call hands
554
+ its declaration, typed by the record's type: `indexes` gets `index` and returns
555
+ the indexes by name, `mutations` gets `mutation` and returns the mutations by
556
+ name, and `migrations` gets `migration` and returns its steps in order. Each is
557
+ checked where the record is declared, every refusal naming the record and the
558
+ part, and the record carries each by its name — `plans.mutations.add_plan`,
559
+ `plans.indexes.by_due` — which a package, a binding (`Record.bind(plans)`,
560
+ `Data.bindPaged(plans, { indexes: ['by_due'] })`) and TypeScript all read.
561
+
533
562
  | Form | Body | Use it when |
534
563
  |---|---|---|
535
- | `e3.mutation.reduce(name, rec, fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
536
- | `e3.mutation.edit(name, rec, fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
537
- | `e3.mutation.patch(rec, name = 'patch')` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
564
+ | `mutation.reduce(fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
565
+ | `mutation.edit(fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
566
+ | `mutation.patch()` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
538
567
 
539
568
  Every form commits a delta addressed by key, rewriting only the segments it
540
569
  touches (a `reduce` of a record that is not a Dict or Set writes it whole), and
541
570
  takes `{ runner? }`. In an `edit`, repeated edits of one key fold, a `set` of the
542
571
  value a row already holds changes nothing, and a `delete` or `update` of a key
543
572
  the record does not hold is a conflict naming the key: the caller's view of the
544
- record is stale. `e3.mutation.editType(recordType)` is the type of `edit`.
573
+ record is stale. `e3.mutation.editType(recordType)` is the type of `edit`. A
574
+ mutation is named by its key, an identifier; a part's functions are typed by the
575
+ record's type, so name the type (`PlansType`) to write them, in the declaration
576
+ or beside it.
545
577
 
546
578
  ```typescript
547
579
  const PlanType = StructType({
548
580
  title: StringType, owner: StringType, due: DateTimeType, resources: SetType(StringType),
549
581
  });
550
- const plans = e3.record('plans', DictType(StringType, PlanType), new Map());
551
-
552
- const addPlan = e3.mutation.reduce('add_plan', plans,
553
- East.function([plans.type, StringType, PlanType], plans.type, ($, state, id, plan) => {
554
- const next = $.let(state.copy()); // the state is frozen: copy first
555
- $(next.insert(id, plan));
556
- return next;
557
- }));
558
-
559
- const reschedule = e3.mutation.edit('reschedule', plans,
560
- East.function([plans.type, StringType, DateTimeType, e3.mutation.editType(plans.type)], NullType,
561
- ($, state, id, due, edit) => {
562
- const plan = $.let(state.get(id)); // decodes one segment, not the record
563
- $(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
564
- }));
582
+ const PlansType = DictType(StringType, PlanType);
583
+
584
+ const plans = e3.record('plans', PlansType, {
585
+ initial: new Map(),
586
+ mutations: (mutation) => ({
587
+ add_plan: mutation.reduce(East.function([PlansType, StringType, PlanType], PlansType, ($, state, id, plan) => {
588
+ const next = $.let(state.copy()); // the state is frozen: copy first
589
+ $(next.insert(id, plan));
590
+ return next;
591
+ })),
592
+ reschedule: mutation.edit(East.function([PlansType, StringType, DateTimeType, e3.mutation.editType(PlansType)], NullType,
593
+ ($, state, id, due, edit) => {
594
+ const plan = $.let(state.get(id)); // decodes one segment, not the record
595
+ $(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
596
+ })),
597
+ patch: mutation.patch(), // a view's save
598
+ }),
599
+ });
565
600
  ```
566
601
 
567
602
  Costs count in segments: a Dict or Set record is cut by key and by each entry's
@@ -569,54 +604,82 @@ encoded size, so wide rows get segments of a few rows and a one-row edit stays
569
604
  cheap. A view reads every field of the rows it shows, so keep a bulky payload it
570
605
  does not display in a second record keyed the same way.
571
606
 
572
- #### Secondary indexes — `e3.recordIndex(name, record, { on | key | keys, value? })`
607
+ #### Secondary indexes — `index.on([...fields])`, `index.key(fn)`, `index.keys(fn)`
573
608
 
574
609
  A second collection over a Dict record, in another sort order, maintained in the
575
- same commit, so a view by an attribute is a page rather than a scan. `on`
576
- names row fields in index order: one field uses its value as the key, several
577
- use a Struct in that order, and a Set field gives an entry per element. It
578
- also carries field metadata for `Data.bindPaged(record, { indexes })` and
579
- `Slice.bindRecord` (see **e3-ui**). `key` gives one computed entry per row;
580
- `keys` returns a Set, one entry per element (a row
581
- naming five resources appears five times, an empty set not at all); `value` is a
582
- projection a view renders without touching the record. The functions are pure
583
- and synchronous, since an index maintained commit by commit must equal its
584
- rebuild to the byte. The name is an identifier, never `primary` (the record's own
585
- order). Read through one with `index=<name>` on a page or key search; `e3
610
+ same commit, so a view by an attribute is a page rather than a scan.
611
+ `index.on([...])` names row fields in index order: one field uses its value as
612
+ the key, several use a Struct in that order, and a Set field gives an entry per
613
+ element. It also carries field metadata for `Data.bindPaged(record, { indexes })`
614
+ and `Slice.bindRecord` (see **e3-ui**). `index.key(fn)` gives one computed entry
615
+ per row; `index.keys(fn)` returns a Set, one entry per element (a row naming five
616
+ resources appears five times, an empty set not at all); the `value` option of
617
+ each is a projection a view renders without touching the record. The functions
618
+ are pure and synchronous, since an index maintained commit by commit must equal
619
+ its rebuild to the byte. Its name — its key in `indexes` — is an identifier,
620
+ never `primary` (the record's own order). Read through one with `index=<name>` on a page or key search; `e3
586
621
  reindex` rebuilds. A deploy builds an index the package declares and the state
587
622
  lacks, drops one it no longer declares, and runs nothing for the rest; a build
588
623
  is a split task over the record.
589
624
 
590
625
  ```typescript
591
- const byDue = e3.recordIndex('by_due', plans, { on: ['due'] });
592
- const byOwnerDue = e3.recordIndex('by_owner_due', plans, { on: ['owner', 'due'] });
593
- const byOwner = e3.recordIndex('by_owner', plans, {
594
- key: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner),
595
- value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
596
- });
597
- const byResource = e3.recordIndex('by_resource', plans, {
598
- keys: East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources),
626
+ const plans = e3.record('plans', PlansType, {
627
+ initial: new Map(),
628
+ indexes: (index) => ({
629
+ by_due: index.on(['due']),
630
+ by_owner_due: index.on(['owner', 'due']),
631
+ by_owner: index.key(East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner), {
632
+ value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
633
+ }),
634
+ by_resource: index.keys(East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources)),
635
+ }),
636
+ mutations: (mutation) => ({ /* add_plan, reschedule, patch — above */ }),
599
637
  });
600
- const pkg = e3.package('planning', '1.0.0',
601
- plans, addPlan, reschedule, e3.mutation.patch(plans), byDue, byOwnerDue, byOwner, byResource);
638
+ const pkg = e3.package('planning', '1.0.0', plans); // the record whole: its indexes and mutations with it
602
639
  ```
603
640
 
604
- For events that must remain visible while they overlap a selected time window,
605
- declare `interval: { from: 'start', to: 'end' }`. Both fields must be DateTime
606
- or Option<DateTime>. Optional `on` fields are exact leading choices:
641
+ #### Interval and area indexes — `index.interval`, `index.area`
642
+
643
+ These indexes answer more than an order — as do the series and summary indexes
644
+ below — each with a builder method of its own whose keys the SDK generates.
645
+ Underneath each is an ordinary index — the same build,
646
+ mutation, storage, paging and `e3 reindex` — and the record Slice reads a
647
+ window through either (see **e3-ui**). Their generated keys are query
648
+ machinery: edits and sessions always target the record's primary key.
649
+
650
+ A generated key is a struct whose own fields are e3's: `on` holds the
651
+ declaration's leading fields, by their names, at their key types (a Set field's
652
+ element type) — a struct with no fields when it lists none — and the kind's own
653
+ parts sit beside it. So no name is reserved: a row may name a field anything,
654
+ `area`, `tile` and `__area` among them, and an index may be led by it.
655
+
656
+ | Index | Key | e3's parts |
657
+ |---|---|---|
658
+ | interval | `{ on, node, at }` | the tree node an entry is filed at; the instant it is ordered by, at the `from` field's type |
659
+ | area | `{ on, area }` | the cell's Morton code times 32 plus its level; −1 for a row not yet placed |
660
+ | series | `{ on, key, level, tile }` | the value of the field `key` names; the level an event is filed at, and the start of that level's tile (#1433) |
661
+ | summary | `{ on, key, tile }` | the value of the field `key` names; the tile's start |
662
+
663
+ A definition's `on` is the leading fields alone; its kind's options say the
664
+ rest (`events.indexes.by_time.interval`, `racks.indexes.racks_by_area.area`,
665
+ `jobs.indexes.jobs_by_press_week.series`).
666
+
667
+ **`index.interval({ on?, from, to })`** keeps events visible while they
668
+ overlap a selected time window. `from` and `to` are the row's DateTime or
669
+ Option<DateTime> fields; optional `on` fields are exact leading choices:
607
670
 
608
671
  ```typescript
609
- const byTime = e3.recordIndex('by_time', events, {
610
- interval: { from: 'start', to: 'end' },
611
- });
612
- const byStatusTime = e3.recordIndex('by_status_time', events, {
613
- on: ['status'], interval: { from: 'start', to: 'end' },
672
+ const events = e3.record('events', EventsType, {
673
+ initial: new Map(),
674
+ indexes: (index) => ({
675
+ by_time: index.interval({ from: 'start', to: 'end' }),
676
+ by_status_time: index.interval({ on: ['status'], from: 'start', to: 'end' }),
677
+ }),
614
678
  });
615
679
  ```
616
680
 
617
- An interval index uses the ordinary index build, mutation and paging machinery.
618
- It stores at most three entries per event per leading-field combination,
619
- independent of duration. The record Slice adapter resolves a half-open
681
+ Its key is `{ on, node, at }`. It stores at most three entries per event per
682
+ leading-field combination, independent of duration. The record Slice adapter resolves a half-open
620
683
  `[from, to)` window with at most 55 disjoint bounded queries per exact prefix;
621
684
  each matching original record occurs once, including events that started
622
685
  earlier or span the whole window. It never scans the record or guesses a
@@ -632,17 +695,242 @@ primary key. Calendar's `Calendar.events(binding, { slice, entries, … })`
632
695
  requires a matching interval declaration and the primary `entries` binding to
633
696
  confirm saves even after an edit leaves the selected window.
634
697
 
698
+ **`index.area({ on?, x, y, width?, height?, at?, cell })`** reads the rows
699
+ whose box meets a 2D window — what a floor plan's viewport shows — the interval
700
+ index's twin:
701
+
702
+ ```typescript
703
+ // RacksType: Dict<String, { hall: String, x: Option<Float>, y: Option<Float>, w: Float, h: Float }>
704
+ const racks = e3.record('racks', RacksType, {
705
+ initial: new Map(),
706
+ indexes: (index) => ({
707
+ racks_by_area: index.area({ x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01 }),
708
+ racks_by_hall_area: index.area({ on: ['hall'], x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01 }),
709
+ }),
710
+ });
711
+ ```
712
+
713
+ | Option | The box's |
714
+ |---|---|
715
+ | `x`, `y` | place: two Float fields, or two Option<Float> fields whose `none` is a row not yet placed |
716
+ | `width`, `height` | size: two Float fields, or neither for a point |
717
+ | `at` | `'corner'`, the default — `x`, `y` is the box's minimum corner, as a zone's is — or `'centre'`, as an item's is |
718
+ | `cell` | finest cell: its side in world units, about the smallest thing drawn — 0.01 for a floor in metres, 1 for one in millimetres. It is rounded down to a power of two, so the keys' arithmetic is exact on every runtime, and has no default: a wrong one costs a silent tenfold read |
719
+ | `on` | exact leading fields, as the interval index's; a Set field fans out |
720
+
721
+ Each option takes only the row's fields of its type, so a wrong one does not
722
+ compile, and the definition checks them again, naming the option, the field and
723
+ its type. The domain is the square ±`cell`·2²⁷ world units around zero
724
+ (±1,048 km at a 0.01 cell, which rounds to 2⁻⁷), in 29 levels. Each row is
725
+ filed once, under the deepest level whose cell is at least twice its box's
726
+ larger side, in the cell holding its box's centre: the key
727
+ `{ on, area: Integer }`. A row not yet placed files under `area` −1, which the
728
+ record Slice reads as the unplaced clause; one outside the domain, not finite,
729
+ or larger than half of it files under the root, which every window reads. The index **carries its rows**: each
730
+ entry's projection is the whole row, so a window reads the index alone and no
731
+ record segment, and it takes no `value`. The record Slice reads a viewport,
732
+ grown by an eighth of its larger side on each side, as at most 266 disjoint
733
+ searches per exact prefix, sent in one request, which together hold every row
734
+ whose box meets it, each once — beside rows of the cells searched that it
735
+ misses, which the reader draws or not.
736
+
737
+ #### Series and summary indexes — `index.series`, `index.summary`
738
+
739
+ What a planning board reads: the events on a block of keys — a Plan's
740
+ resources — over a span of time, and, zoomed out past what a screen can draw,
741
+ how busy each key is. Both file a row's event by the key it is on and by time
742
+ tiles — `day`, `week` (ISO, from Monday), `month`, `quarter` or `year`, in UTC,
743
+ half-open — the key before the time, so a key's entries over any span are
744
+ contiguous: a series index's `{ on, key, level, tile }`, a summary index's
745
+ `{ on, key, tile }`.
746
+
747
+ ```typescript
748
+ // JobsType: Dict<String, { press: String, start: Option<DateTime>, end: Option<DateTime>, … }>
749
+ const jobs = e3.record('jobs', JobsType, {
750
+ initial: new Map(),
751
+ indexes: (index) => ({
752
+ jobs_by_press_week: index.series({ key: 'press', from: 'start', to: 'end', tile: 'week' }),
753
+ jobs_by_press_month: index.summary({ key: 'press', from: 'start', to: 'end', tile: 'month' }),
754
+ }),
755
+ });
756
+ ```
757
+
758
+ | Option | |
759
+ |---|---|
760
+ | `key` | the field naming what an event is on: any field but a Set |
761
+ | `from`, `to` | the event's start and end: DateTime or Option<DateTime> fields. A row with no start is filed nowhere; no end, or one at or before the start, is an instant |
762
+ | `tile` | a series index's finest level, and a summary index's tile: `day`, `week`, `month`, `quarter` or `year` |
763
+ | `on` | exact leading fields, before the key; a Set field fans out |
764
+
765
+ **A series index** is an ordinary index whose keys the SDK generates, and it
766
+ **carries its rows**, as the area index does. It files each event **once**, at
767
+ the finest level from its `tile` up to the year whose shortest tile holds the
768
+ event's length — a day, a week, 28 days, 90 days, 365 days — under that level's
769
+ tile holding its start; an instant at `tile`; past 365 days, under each year it
770
+ touches. An event filed at a level ends within the tile after the one holding
771
+ its start, so a reader reads a block of keys over any span from the index alone
772
+ as one range a key a level — that level's tiles from two before the span to one
773
+ after it — in one request, whatever the span's width. **A summary index** holds,
774
+ under each key and tile, `{ count, ms }`: how many rows touch the tile, and the
775
+ milliseconds of it they cover — a row's shares
776
+ add up to its length, and an instant's is none. Each entry stands for every row
777
+ of its key and names none, so its window's `key` is Null, a read of it joins
778
+ nothing, and a read that needs its rows — a joined read, a placement
779
+ (`datasetPlace`) or a membership read (`datasetWithin`) — is refused, naming the
780
+ index; the record's describe says it is a summary. A write
781
+ changes its totals by what its rows took away and gave — its program reads the
782
+ record, never the summary — and the apply adds that to each total, dropping one
783
+ whose count reaches zero; the totals are Integers, so a maintained summary is the
784
+ one `e3 reindex` sums, byte for byte, and a build split into pieces adds the
785
+ pieces' totals of a key. A change that would leave a total no rebuild sums — a
786
+ count below zero, or time with no row — fails the write, naming the total and
787
+ `e3 reindex`, which rebuilds it; a total past the largest Integer fails the
788
+ write or the build, naming a finer tile.
789
+
790
+ A series index files an event past a year under each year it touches, as many
791
+ as 4,096: one touching more is refused where it is written, naming it and its
792
+ span. A summary splits an event over its tiles, as many as 4,096 — eleven years
793
+ of days — and one touching more is refused the same way, naming the tile:
794
+ declare a coarser tile. Each option takes only the row's fields of its type, as
795
+ the area index's do, and the definition checks them again; a `value` is
796
+ refused, and so is an `on` field that is the key, the start or the end: the key
797
+ follows the leading fields, and the tiles are cut by the start and the end.
798
+
799
+ #### Nested series indexes — `index.nested`
800
+
801
+ What a planning board reads when a record's rows nest what it draws: a
802
+ machine's jobs and its load a day at a time, a hall's members, an assembly's
803
+ parts to any depth. One declaration a record lists every collection its rows
804
+ nest, each field with a builder saying what it holds. e3 files each item apart
805
+ from its row — once, by its collection, its row, its node and its time — keeps
806
+ each node's totals a summary tile at a time, and each parent's subtotals at
807
+ every level; and the record's other indexes carry the rows without those
808
+ collections, so a page of rows reads none of them and a window of items reads
809
+ no row.
810
+
811
+ ```typescript
812
+ // MachinesType: Dict<String, { name, site, status, jobs: Dict<String, Job>, load: Dict<DateTime, Float> }>
813
+ const machines = e3.record('machines', MachinesType, {
814
+ initial: new Map(),
815
+ indexes: (index) => ({
816
+ machines_in_time: index.nested({
817
+ collections: {
818
+ jobs: index.nested.items({ from: 'start', to: 'end' }), // things people drag: by id, their times fields
819
+ load: index.nested.values(), // a value per instant: keyed by the instant
820
+ },
821
+ tile: 'week',
822
+ summary: 'month',
823
+ }),
824
+ machines_by_site: index.on(['site']), // pages each machine without its jobs and load
825
+ }),
826
+ });
827
+ // Each member's load, and each hall's mean of its members', at every level from the day up.
828
+ const halls = e3.record('halls', HallsType, {
829
+ indexes: (index) => ({
830
+ halls_in_time: index.nested({
831
+ collections: { members: index.nested.rows({ load: index.nested.values({ rollup: 'mean' }) }) },
832
+ tile: 'day',
833
+ summary: 'month',
834
+ }),
835
+ }),
836
+ });
837
+ // Parts to any depth, each part's jobs filed under every part above it too.
838
+ const assemblies = e3.record('assemblies', AssembliesType, {
839
+ indexes: (index) => ({
840
+ assemblies_in_time: index.nested({
841
+ collections: {
842
+ parts: index.nested.rows({
843
+ jobs: index.nested.items({ from: 'start', to: 'end' }, { rollup: true }),
844
+ parts: index.nested.recursive(),
845
+ }),
846
+ },
847
+ tile: 'week',
848
+ summary: 'month',
849
+ }),
850
+ }),
851
+ });
852
+ ```
853
+
854
+ | Builder (`index.nested.…`) | A field holding |
855
+ |---|---|
856
+ | `items({ from, to })`, `items({ at })` | `Dict<String, Struct>`: things people drag, by id; their times are DateTime or Option<DateTime> fields — no start files an item nowhere, no end (or one at or before its start) makes it an instant. `{ rollup: true }` files each under every ancestor too |
857
+ | `values({ rollup? })` | `Dict<DateTime, V>`: a value per instant. `rollup` keeps every parent's subtotals |
858
+ | `rows({ … })` | `Dict<String, Struct>`: child rows, their own collections declared the same way |
859
+ | `recursive()` | `Dict<String, Self>`: more rows like the one listing it, to any depth |
860
+
861
+ TypeScript types `collections` field by field from the record's type alone: a
862
+ misnamed time, or a builder on the wrong kind of field, is an error where it is
863
+ written, and its message says what fits. The definition checks the same again,
864
+ in words, for a declaration TypeScript could not see. A record has one nested
865
+ series index, which takes no `on` and no `value`.
866
+
867
+ **What it files.** Every entry is keyed `{ collection, row, path, run, level,
868
+ tile, item }` (e3-types' `NestedEntryKeyType`): the collection's place, the
869
+ field names from the row down (`members` then `load`); the row's key; the Dict
870
+ keys from the row down to the node (`[]` for the row, `m-3` for a member); the
871
+ run; the level and tile; and the item. A place and a path are held as a key
872
+ holds a list, folded from the last: `.step (key="members", rest=.step
873
+ (key="load", rest=.end))`. Each run of a node is one contiguous range a level:
874
+
875
+ | Run | Holds |
876
+ |---|---|
877
+ | `items` | each item once, as a series index files an event — at the level its length fits, past a year under each year — carrying it (`.item .jobs (…)`); a value, and an `{ at }` item, at the declared tile alone |
878
+ | `totals` | where `summary` is declared, each node's `{ count, ms, values }` a summary tile at a time; for a Float or Integer collection, `values` is the sum, the least, the greatest and the last there, taken in key order |
879
+ | `subtotals(rollup)` | for `values({ rollup })`, every parent's subtotal by each rollup at each level from `tile` to the year, where a descendant's value touches the tile: a node holding values folds its own, a parent aggregates its children's, in key order |
880
+ | `subtree`, `subtreeTotals` | for `items(…, { rollup: true })`, each item under every ancestor too, `.under (path, id)`, and each ancestor's totals of them |
881
+
882
+ A rollup is `'sum'`, `'mean'` (a mean of means up the tree), `'min'`, `'max'`,
883
+ `'count'` (the leaf rows beneath that hold a value), a fold and an aggregate
884
+ named apart (`{ fold: 'last', aggregate: 'sum' }`, a balance), or several.
885
+
886
+ **Reading.** A reader searches the index's leading fields — `[collection, row,
887
+ path, run, level, tile]`, each as East prints it — as a range a row a level: a
888
+ block of fifty machines' jobs over three weeks is fifty times four ranges in one
889
+ `datasetFindKeys` request, and one `datasetGetRuns` read of the entries they
890
+ select, each job carried whole; a parent's week subtotals are one range; a
891
+ part's subtree is one range a level.
892
+
893
+ **The lean rows.** Beside a nested series index, every other index of the
894
+ record that carries nothing or the row carries the row without the listed
895
+ collections — each listed collection empty, child rows kept with theirs empty —
896
+ so a page of rows read without a join reads no nested item and no segment of
897
+ the record's own. An index with a `value` of its own carries its own. A
898
+ binding of the record's indexes lists the nested index by name beside any that
899
+ carry the lean rows — `Data.bindPaged(machines, { indexes: ['machines_by_site',
900
+ 'machines_in_time'] })` (see **e3-ui**).
901
+
902
+ **Its span.** Beside every nested series index e3 keeps its span, an index of
903
+ the record's named `<index>_span` (e3-types' `nestedSpanName`), built with it
904
+ and never declared: under each collection's place and each month its items or
905
+ values touch (`NestedSpanKeyType`, `{ collection, tile }`), how many of them
906
+ touch it and the milliseconds of it its items cover, summed across the rows as a
907
+ summary index's totals are, and a month let go once nothing touches it. A
908
+ collection's first and last month held are its span across the record — what a
909
+ Plan's horizon covers for the series reading it — and its months' counts are
910
+ how much it holds in each, which the horizon's strip draws as its bars (see
911
+ **e3-ui**). Reading it is one search a collection, an exact match on its
912
+ place, and one read of the first and last entry of each. An index the record
913
+ declares under that name is refused.
914
+
915
+ **Writes.** A row a write updates by a patch is maintained by the patch: e3
916
+ walks it along the declaration — each collection's Dict patch, and each child
917
+ rows' — and compares the entries of only the items it names, with the totals of
918
+ the summary tiles they touch and the subtotals of the tiles they touch at each
919
+ level. A replace on the way, which a recursive type's patch always is, and an
920
+ inserted or deleted row are compared whole. Both ways give the same delta, and a maintained
921
+ index is the one `e3 reindex` rebuilds, byte for byte.
922
+
635
923
  #### Migrations — how a record's type changes
636
924
 
637
925
  | Form | Function | Runs as |
638
926
  |---|---|---|
639
- | `e3.migration.value(name, rec, fn, { after? })` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
640
- | `e3.migration.rows(name, rec, fn, { after? })` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
641
- | `e3.migration.rekey(name, rec, fn, { after? })` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
927
+ | `migration.value(name, fn)` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
928
+ | `migration.rows(name, fn)` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
929
+ | `migration.rekey(name, fn)` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
642
930
 
643
- A record's steps are one chain: each names the step before it with `after` (a
644
- step of the same record, leaving it as this step's input type), passing the last
645
- step passes the chain, and the last step leaves the record as its declared type.
931
+ A record's steps are one chain: the list `migrations` returns, in order, each
932
+ step taking the record as the step before it leaves it, and the last leaving it
933
+ as its declared type — each checked where the record is declared.
646
934
  A step's name is an identifier, unique on the record, and a workspace records the
647
935
  steps it has applied by name — so moving a step's code, or re-exporting it with a
648
936
  newer e3, does not run it again, and an edit to an applied step's body is not
@@ -651,12 +939,14 @@ detected. The functions are pure, synchronous East.
651
939
  ```typescript
652
940
  const RowV1Type = StructType({ title: StringType });
653
941
  const RowV2Type = StructType({ title: StringType, owner: StringType });
654
- const plans = e3.record('plans', DictType(StringType, RowV2Type), new Map());
655
-
656
- const addOwner = e3.migration.rows('add_owner', plans,
657
- East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' })));
658
- // The next version adds its step after this one: e3.migration.value('…', plans, fn, { after: addOwner })
659
- const pkg = e3.package('planning', '2.0.0', plans, addOwner);
942
+ const plans = e3.record('plans', DictType(StringType, RowV2Type), {
943
+ initial: new Map(),
944
+ migrations: (migration) => [
945
+ migration.rows('add_owner', East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' }))),
946
+ // The next version adds its step here, after this one.
947
+ ],
948
+ });
949
+ const pkg = e3.package('planning', '2.0.0', plans);
660
950
  ```
661
951
 
662
952
  What a deploy does with each record, decided before it writes anything:
@@ -714,7 +1004,7 @@ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
714
1004
  | `e3 workspace copy [repo] <from> <to>` | Make `<to>` what `<from>` is now — its package and every input, record and task output, so nothing runs again — made, or replaced whole with its runs. It writes refs only, so its cost follows the number of datasets, never their size; a write to either afterwards leaves the other as it was. Refused while a dataflow runs in `<to>`. |
715
1005
  | `e3 workspace deploy [repo] <ws> <pkg[@ver]>` | Deploy an imported package. |
716
1006
  | `… --from-zip <zip>` · `… --from-source <src.ts> [--functions <manifest…>]` | Import the zip (or bundle the source) and deploy, creating the workspace. |
717
- | `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on; `--memory`: what those may reserve (local). |
1007
+ | `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on, and with `--from-source` the workspace packages exporting their functions at once; `--memory`: what those may reserve (local). |
718
1008
  | `e3 workspace export [repo] <ws> <zip> [--name <n>] [--version <v>]` | The workspace's state as a package. |
719
1009
  | **Datasets** | |
720
1010
  | `e3 dataset get [repo] <ws.name> [-f east\|json\|beast2]` | Print a value. |
@@ -730,7 +1020,7 @@ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
730
1020
  | **Records** (`-w <ws>` required) | |
731
1021
  | `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
732
1022
  | `e3 seed <repo> <record> <file.beast2> -w <ws> [-j <n>] [--memory <size>]` | Commit a record's first state from a beast2 file, as its `$init`: for a record declared with no initial value, unassigned until then (local). The file is taken in as a `file` source is, never read whole; a record that holds a state is refused. |
733
- | `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target; local). |
1023
+ | `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target — a summary's totals added to as `±n`; local). |
734
1024
  | `e3 reindex <repo> <record> -w <ws> [--index <name>] [-j <n>] [--memory <size>]` · `e3 compact <repo> <record> -w <ws>` | Rebuild indexes (local) · collapse the history to a `$compact` root, the state kept. |
735
1025
  | **Development** | |
736
1026
  | `e3 watch <src.ts> <repo> <ws> [--start] [--schema <p>] [--inputs <p>] [--abort-on-change] [--functions <manifest…>] [-j <n>] [--memory <size>]` | Redeploy on each change (and run, with `--start`). |
@@ -748,7 +1038,10 @@ peak a unit of its stage has reached in the run, so a stage runs its first unit
748
1038
  alone and then fans out; on Linux and macOS a guard stops the newest unit when
749
1039
  the runners together pass the budget, and runs it again once it fits. A deploy
750
1040
  takes its `file` sources in `-j` at a time, their pieces' intake units running
751
- under the same budget. The commands that run East — `workspace deploy`,
1041
+ under the same budget. A source deploy, or a `watch`, exports the workspace
1042
+ packages its imported functions come from `-j` at a time, a process each, on
1043
+ this machine; against a server, as many at once as this machine has CPUs. The
1044
+ commands that run East — `workspace deploy`,
752
1045
  `dataset set --from-file`, `dataflow run`, `run`, `call`, `mutate`, `seed`,
753
1046
  `reindex` and `watch` — take the flags for a local repository; against a server the CLI
754
1047
  refuses them, since the server runs the work under its own budget
@@ -759,7 +1052,11 @@ line once it is in: its size, time and rate, and how it was taken in
759
1052
  (`unchanged, already in the store`, `taken in by east-c`, or `carried` for a
760
1053
  value that is not a collection), with why the first time a runner fell back to
761
1054
  another. A terminal also keeps a live line for the files in flight, with their
762
- pieces, the rate and the time left across them. Against a server, the deploy line
1055
+ pieces, the rate and the time left across them. A source deploy prints a line
1056
+ for each workspace package whose functions it links: `exported <n> function(s)
1057
+ of <pkg> (east-py export-functions)`, or `reused <n> function(s) of <pkg>,
1058
+ unchanged since its last export` when the export cache held them. Against a
1059
+ server, the deploy line
763
1060
  says what the job is doing, such as migrating a record or building an index;
764
1061
  the deliveries then upload a few at a time, each printing a line once it is in,
765
1062
  and the live line says what the server's commit is doing with each in flight.
@@ -819,8 +1116,8 @@ naming the holder: its pid, boot id, command and when it took the lock.
819
1116
  |---|---|
820
1117
  | Repositories | `repoList(url, opts)`, `repoCreate(url, name, opts)`, `repoRemove(url, name, opts)`, `repoStatus(url, repo, opts)`, `repoRecord` (its release and upgrades), `repoGc(url, repo, gcRequest, opts)` — a job it polls; `repoGcStart` and `repoGcStatus` apart |
821
1118
  | Packages | `packageList(url, repo, opts)`, `packageGet(url, repo, name, version, opts)`, `packageImport(url, repo, zipBytes, opts)`, `packageExport(url, repo, name, version, opts)` → zip bytes, `packageRemove` |
822
- | Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
823
- | Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
1119
+ | Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceBundle(url, repo, ws, opts)` → the workspace as it is, as a bundle's bytes — its package, every dataset's ref, the executions its tasks' outputs come from, and every object those reach — and `workspaceBundleImport(url, repo, ws, bundle, opts)` → the workspace made from one under a name no workspace has, in another repository or another e3 (`workspace_exists`, `package_exists`, `bundle_invalid`), its tasks up to date with no run; a host mounts these routes where a bundle fits in a request: the local server, and e3-web, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
1120
+ | Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, and several reads in one request (at most `DATASET_BATCH_MAX`, 10,000, each at one content hash, in a body of at most 16 MiB: a larger one is refused 413, `payload_too_large`): `datasetFindKeys(…, { queries, index?, hash? }, opts)` — key searches, `datasetGetRows(…, { keys, hash? }, opts)` — a Dict's rows by `.east` key, each with its position — or `datasetGetRows(…, { keysOf: { path, index, runs, hash }, hash? }, opts)`, by the keys the entries of some runs of another record's index name, which the server reads from the index so none is sent (a drawing's details for its window's cells), a stale `keysOf.hash` thrown as `RunsHashMismatchError` naming the record's content — and `datasetGetRuns(…, { runs, hash, index?, join? }, opts)` — the entries of several runs as one window, a rows or runs answer holding the first `count` the page budget reaches (by runs, of `runEntries`); and where a narrowing's rows sit in a record, from its index alone: `datasetPlace(…, { index, runs, offset?, limit?, hash }, opts)` → `{ segments, placed, runEntries, totalElements, hash }` — how many of the runs' entries each record segment holds (`{ segment, offset, count, rows }`), for the first `placed` of a window from `offset` (the next window from `offset + placed`), and `datasetWithin(…, { index, runs, keys, hash }, opts)` → `{ within, hash }` — whether each index entry key (`(ik=…, k=…)`) lies in one of the runs, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
824
1121
  | Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
825
1122
  | Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts)` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress? })` — a program over a dataset's pieces, as a job it polls, under the same rule (or `splitCallLaunch`, `splitCallStatus` and `splitCallExplain`) |
826
1123
  | Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
@@ -834,7 +1131,7 @@ takes a runner, `new LocalTaskRunner(repo)`.
834
1131
  |---|---|
835
1132
  | Repositories | `repoInit(path)`, `repoFind(startPath?)`, `repositoryOpen(storage, repo)` (checks the repository and applies the upgrades it owes), `repositoryUpgradeStep(storage, repo, { budgetMs, waitMs? })` → `{ owed }` — a host's job applies them so, a part per run under its time limit, each part taking a step up where the last stopped, `repoGc(storage, repo, { dryRun?, minAge?, keepRuns?, keepDays?, retention?, executionAlive? })` — holding the repository still, or, with `retention: { windowMs }`, beside running work: an object goes once unreachable for the window and not written or re-referenced since; given a runner's `executionAlive`, an attempt recorded running that cannot finish is recorded interrupted and pruned; `repoGcStep(storage, repo, step, { windowMs, pruneMs?, markMs?, concurrency?, executionAlive?, … })` runs that one step at a time, each returning the next (`GcStepType`), `pruneMs` and `markMs` bounding a prune step and a mark step for a host with a time limit |
836
1133
  | Packages | `packageImport(storage, repo, zipPath)`, `packageExport(storage, repo, name, version, zipPath)`, `packageList`, `packageRemove` |
837
- | Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
1134
+ | Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceBundleExport(storage, repo, ws, writable, { have? })` — the workspace as a bundle to a WHATWG stream, leaving out the objects `have` says the receiving repository holds — and `workspaceBundleImport(storage, repo, ws, source)` — the workspace made from a bundle (a stream or chunks) under a name no workspace has, its objects stored as they come and its refs and record written once the repository holds everything they name, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
838
1135
  | Datasets | `workspaceGetDataset(storage, repo, ws, treePath)`, `workspaceSetDataset(storage, repo, ws, treePath, value, type)`, `datasetAdoptFile(storage, repo, ws, treePath, file, { runner, onProgress? })` → `{ hash, size, segments, rows, taken, runners? }`, `taken` being `known`, `carried` or `taken` (by the `runners` named); `deliveryAdoptStored(storage, repo, ws, { address, type }, sha256, { runner?, onProgress?, signal? })` — a delivery the store holds taken in as a stored value of the type, naming no dataset, for a record's first state |
839
1136
  | Runs | `dataflowExecute(storage, repo, ws, options?)`; `LocalOrchestrator` to start, poll and cancel a run |
840
1137
  | Records | `recordMutate(storage, runner, repo, ws, record, mutation, args, { actor })`, `recordHistory`, `recordDescribe`, `recordCompact`, `recordReindex`; `recordSystemCommit(storage, runner, repo, ws, record, { name, target, actor, args? })` — `$rollback` to a commit, `$restore` of a state at the record's migrations, and `$init` on a record declared with no initial value: its first state, from a stored value, with the whole chain applied |
@@ -906,6 +1203,7 @@ package is served beside the standard ones:
906
1203
  | `serveE3` option | Default | Effect |
907
1204
  |---|---|---|
908
1205
  | `units` | — | starts a unit worker: a Web Worker whose script calls `serveUnits()` |
1206
+ | `width` | the cores | the most units that run at once, and so the most unit workers, each with a heap of its own |
909
1207
  | `persist` | `true` | `false` keeps repositories in memory, gone with the page |
910
1208
  | `name` | `'e3'` | the storage's name: e3 workers of one name share their repositories, in one tab or several |
911
1209
  | `identify` | none | each request's caller, `{ sub, email?, roles }`, from the request; a request to a repository whose caller it does not identify is answered 401. None: the page's one caller may do everything |
@@ -970,7 +1268,14 @@ not had, in place; one a newer e3 upgraded is refused, naming that release.
970
1268
  environment — and its inputs' hashes, so a task re-runs only when one of those
971
1269
  changes (`--force` bypasses the cache, and `--force-task <task>` for the tasks
972
1270
  it names: a task whose code e3 cannot see changed, such as a platform
973
- function's, or one reading data from outside e3). A change to an input re-runs its
1271
+ function's, or one reading data from outside e3). A task's program holds the
1272
+ source locations of its body and of every function it embeds, an imported
1273
+ python or node function's too, since its errors, a caught error's `stack` and
1274
+ the closures it returns name them: an edit that moves one of those lines — a
1275
+ comment or an import added above a function — re-runs the task. One that leaves
1276
+ them where they were — another function, the end of a module, another package —
1277
+ re-runs only the tasks whose environment it is in, their runner's `{ custom }`
1278
+ packages. A change to an input re-runs its
974
1279
  consumers, even during a run. A split task is cached a unit at a time: each piece
975
1280
  and each merge is its own execution, and pieces are cut by content, so an append
976
1281
  or an insertion re-runs only the pieces it reaches and the merges above them.