@elaraai/east-codex-plugin 1.0.97 → 1.0.99

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,11 +1,11 @@
1
1
  ---
2
2
  name: e3
3
- description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack… See the detailed scope below."
3
+ description: "East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations, declared whole in one e3.record(name, type, { initial, indexes, mutations, migrations }) call: mutation.reduce/edit/patch; index.on/key/keys, index.interval / index.area for reads by a time window or a 2D window, index.series / index.summary for a block of keys over a span of time tiles and their totals, and index.nested for what a record's rows nest — jobs, a value a day, child rows, trees — filed item by item, with totals and parents' subtotals; and migrated between versions (migration.value/rows/rekey, in order), (3) the e3 CLI (repo, package,… See the detailed scope below."
4
4
  ---
5
5
 
6
6
  ## Detailed skill scope
7
7
 
8
- East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations (e3.record, e3.mutation.reduce/edit/patch, e3.recordIndex) and migrated between versions (e3.migration.value/rows/rekey), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack enroll/policy, dataflow run --rack, watch --start --rack, optional e3-api-server --rack, (7) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui().
8
+ East Execution Engine (e3) - durable, content-addressed dataflow for East programs. Use when: (1) authoring a package with @elaraai/e3 (e3.input, e3.task, e3.streamTask with e3.output and e3.partition for bounded memory and work split over an input, e3.customTask, e3.function, e3.package, e3.export, East.importFunction), (2) records - audited state written only through mutations, declared whole in one e3.record(name, type, { initial, indexes, mutations, migrations }) call: mutation.reduce/edit/patch; index.on/key/keys, index.interval / index.area for reads by a time window or a 2D window, index.series / index.summary for a block of keys over a span of time tiles and their totals, and index.nested for what a record's rows nest — jobs, a value a day, child rows, trees — filed item by item, with totals and parents' subtotals; and migrated between versions (migration.value/rows/rekey, in order), (3) the e3 CLI (repo, package, workspace, dataset, task, dataflow run, run, call, mutate, seed, history, reindex, compact, watch, convert, auth) and the cores and memory a run may use (-j, --memory), (4) driving e3 from code: @elaraai/e3-api-client against a server, @elaraai/e3-core on a local repository, e3-api-server, (5) caching, reactive re-runs and garbage collection, (6) delegating local task bodies to a rack: e3 rack enroll/policy, dataflow run --rack, watch --start --rack, optional e3-api-server --rack, (7) e3 in a browser, no server: @elaraai/e3-web (serveUnits, serveE3, createWebE3) and what a browser cannot do. UI tasks are e3-ui's ui().
9
9
 
10
10
  # East Execution Engine (e3)
11
11
 
@@ -70,8 +70,14 @@ What do you need?
70
70
  │ │ └─ Split over an input, in pieces → wrap that input: e3.partition(dataset, { by? })
71
71
  │ ├─ A shell command → e3.customTask(name, [inputs], outputType, command)
72
72
  │ ├─ A named function (RPC) → e3.function(name, fn, config?) — called by name, persists nothing
73
- │ ├─ Audited state → e3.record + e3.mutation.reduce | edit | patch, e3.recordIndex
74
- │ ├─ A record's type changes → e3.migration.value | rows | rekey(name, record, fn, { after? })
73
+ │ ├─ Audited state → e3.record(name, type, { initial?, indexes?, mutations?, migrations? }), declared whole
74
+ │ │ ├─ Its writes → mutations: (mutation) => ({ name: mutation.reduce | edit | patch })
75
+ │ │ ├─ An order to page in → indexes: (index) => ({ name: index.on([...fields]) | index.key(fn) | index.keys(fn) })
76
+ │ │ ├─ Read by a time window or a 2D window → index.interval | index.area
77
+ │ │ ├─ Read a block of keys over time tiles, or their totals → index.series | index.summary
78
+ │ │ └─ Read what rows nest (jobs, a value a day, child rows, trees) a tile at a time, with totals and subtotals
79
+ │ │ → index.nested({ collections, tile, summary? })
80
+ │ ├─ A record's type changes → migrations: (migration) => [migration.value | rows | rekey(name, fn), …], in order
75
81
  │ ├─ A UI task → ui(name, [inputs], fn) from @elaraai/e3-ui (the e3-ui skill)
76
82
  │ ├─ A function written in python or another node package → East.importFunction(pkg, name, FunctionType) in the body
77
83
  │ └─ Bundle and export → e3.package(name, version, ...items); e3.export(pkg, zipPath)
@@ -337,17 +343,26 @@ const shout = e3.task('shout', [greet.output],
337
343
  - **An East function written in python, or in another node package**:
338
344
  `East.importFunction(pkg, name, FunctionType(...))` in the body. `e3.export`
339
345
  resolves it and embeds its IR, so the task is pure IR on any runner, its
340
- locations kept so an error or a profile names the function's own lines. A uv
346
+ locations kept so an error or a profile names the function's own lines (so
347
+ moving them re-runs the task: **Caching**, below). A uv
341
348
  workspace member is found by name in `uv.lock`, and its root module's
342
- `east_functions` dict is exported with `east-py export-functions` (from the
343
- project's `.venv`, else `EAST_PY` or PATH); an npm member is found by its
344
- `package.json` name, and its built `./functions` export's `eastFunctions` is
345
- exported with `east-node export-functions` (`EAST_NODE`). A package built
346
- elsewhere is passed as its manifest — `{ functions: [...] }` / `--functions` —
347
- which wins. The declared type must equal the exported one exactly, and each
348
- platform function it calls must be provided by the task's runner, by name or
349
- by family (`east-py-std` ≡ `@elaraai/east-node-std` ≡ `east-c-std`;
350
- `east-py-io` ≡ `@elaraai/east-node-io`).
349
+ `east_functions` dict is exported with `east-py export-functions` (`EAST_PY`
350
+ when set, else the project's `.venv`, else PATH); an npm member is found by
351
+ its `package.json` name, and its built `./functions` export's `eastFunctions`
352
+ is exported with `east-node export-functions` (`EAST_NODE`). Each member is
353
+ exported in one process however many tasks import from it, the members side
354
+ by side, as many at once as the export has cores (a deploy's `-j`,
355
+ `e3.export`'s `cores`). A python member's manifests are kept in
356
+ `node_modules/.cache/e3/functions` under the directory the export runs in,
357
+ keyed on its sources and those of the local packages it depends on, the
358
+ workspace's `uv.lock` and `pyproject.toml`, the east-py exporting it and the
359
+ functions and providers each manifest is for, so a redeploy whose python has
360
+ not changed runs no export; deleting the directory clears it, and a node
361
+ member is exported every time. A package built elsewhere is passed as its
362
+ manifest — `{ functions: [...] }` / `--functions` — which wins. The declared
363
+ type must equal the exported one exactly, and each platform function it calls
364
+ must be provided by the task's runner, by name or by family (`east-py-std` ≡
365
+ `@elaraai/east-node-std` ≡ `east-c-std`; `east-py-io` ≡ `@elaraai/east-node-io`).
351
366
  - **Environments** are derived at export from each `{ custom }` platform: the
352
367
  providing workspace package's closure (`pyproject.toml` + `uv.lock` + sdists,
353
368
  or `package.json` + lockfile + `npm pack`) is captured into the package and
@@ -503,7 +518,7 @@ const change = e3.streamTask('change', {
503
518
  | `e3.customTask(name, inputs, outputType, command, { environment? })` | A task that runs a bash script: `command($, inputPaths, outputPath)` builds it from the staged beast2 input files, and it writes the output as a beast2 file. |
504
519
  | `e3.function(name, fn, { runner?, environment? })` | A named function, called by name with arguments (`e3 call`, the API) and its result returned inline: not in the dataflow, and a call persists nothing. An argument is a value, or in a call through a workspace one of its datasets, read where it is stored. A server bounds a call by a timeout and a 1 MiB result, so long work belongs in a task. A `custom` runner must speak the runner CLI (`<command> -i <arg>… -o <out> <ir>`). |
505
520
  | `e3.package(name, version, ...items)` | Bundles the items and everything they depend on — pass the last tasks, records and functions; a package passed in contributes its contents. The name and version must be valid file names. |
506
- | `e3.export(pkg, zipPath, { functions? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself; `functions` lists manifests built elsewhere). |
521
+ | `e3.export(pkg, zipPath, { functions?, cores? })` | Writes the package zip, resolving every `East.importFunction` (a workspace package is exported by the export itself, `cores` packages at once, the CPUs by default; `functions` lists manifests built elsewhere). |
507
522
 
508
523
  ```typescript
509
524
  const process = e3.customTask('process', [rawData], StringType,
@@ -515,15 +530,19 @@ await e3.export(e3.package('tools', '1.0.0', process, add), '/tmp/tools.zip');
515
530
 
516
531
  ### Records — audited state, written only through mutations
517
532
 
518
- `e3.record(name, type, initialValue?)` is a dataset at `.records.<name>` that
519
- tasks read and react to like any input, but that only its **mutations** write:
533
+ `e3.record(name, type, { initial?, indexes?, mutations?, migrations? })` is a
534
+ dataset at `.records.<name>` that tasks read and react to like any input, but
535
+ that only its **mutations** write:
520
536
  a raw `dataset set` is refused. A mutation is pure, synchronous East — no async
521
537
  body and no platform call, since the compare-and-swap retry runs it again
522
538
  against fresher state — named by an identifier. Each commit records its parent,
523
539
  the state, the mutation, its arguments, the actor and the time. Deploy mints a
524
540
  `$init` commit from the initial value, and a redeploy keeps state and history.
525
- The record's version vector carries the commit hash, so even a mutation that
526
- leaves the state unchanged triggers downstream tasks.
541
+ A large initial value is given as a function that makes it —
542
+ `initial: () => East.compile(generate, [])(50_000n)` — called once, as the
543
+ package is exported, never where the declaration is only read (a page binding
544
+ the record). The record's version vector carries the commit hash, so even a
545
+ mutation that leaves the state unchanged triggers downstream tasks.
527
546
 
528
547
  Declared with no initial value, a record is **unassigned** until its first
529
548
  state is committed — `e3 seed <repo> <record> <file.beast2> -w <ws>` from a
@@ -534,38 +553,54 @@ and a mutation on it is refused until then. So a record whose first state is
534
553
  data that arrives after the deploy — a customer list, a reference table —
535
554
  never runs its dependents on an empty value first.
536
555
 
556
+ A record is **declared whole**: its type, its initial value and its parts, in
557
+ one call, never added to after. Each part comes from a builder the call hands
558
+ its declaration, typed by the record's type: `indexes` gets `index` and returns
559
+ the indexes by name, `mutations` gets `mutation` and returns the mutations by
560
+ name, and `migrations` gets `migration` and returns its steps in order. Each is
561
+ checked where the record is declared, every refusal naming the record and the
562
+ part, and the record carries each by its name — `plans.mutations.add_plan`,
563
+ `plans.indexes.by_due` — which a package, a binding (`Record.bind(plans)`,
564
+ `Data.bindPaged(plans, { indexes: ['by_due'] })`) and TypeScript all read.
565
+
537
566
  | Form | Body | Use it when |
538
567
  |---|---|---|
539
- | `e3.mutation.reduce(name, rec, fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
540
- | `e3.mutation.edit(name, rec, fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
541
- | `e3.mutation.patch(rec, name = 'patch')` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
568
+ | `mutation.reduce(fn)` | `(state, …args) => state`, over the whole state | the rule is over the whole state, or the record is small; any record type |
569
+ | `mutation.edit(fn)` | `(state, …args, edit) => Null`: reads lazily, writes through `edit.set(k, v)`, `edit.delete(k)`, `edit.update(k, patch)` | server-side logic touches a few entries of a large Dict or Set record |
570
+ | `mutation.patch()` | none: the argument is a `PatchType(state)` | a view's save, or an integration sending diffs; on a record without indexes no program runs at all |
542
571
 
543
572
  Every form commits a delta addressed by key, rewriting only the segments it
544
573
  touches (a `reduce` of a record that is not a Dict or Set writes it whole), and
545
574
  takes `{ runner? }`. In an `edit`, repeated edits of one key fold, a `set` of the
546
575
  value a row already holds changes nothing, and a `delete` or `update` of a key
547
576
  the record does not hold is a conflict naming the key: the caller's view of the
548
- record is stale. `e3.mutation.editType(recordType)` is the type of `edit`.
577
+ record is stale. `e3.mutation.editType(recordType)` is the type of `edit`. A
578
+ mutation is named by its key, an identifier; a part's functions are typed by the
579
+ record's type, so name the type (`PlansType`) to write them, in the declaration
580
+ or beside it.
549
581
 
550
582
  ```typescript
551
583
  const PlanType = StructType({
552
584
  title: StringType, owner: StringType, due: DateTimeType, resources: SetType(StringType),
553
585
  });
554
- const plans = e3.record('plans', DictType(StringType, PlanType), new Map());
555
-
556
- const addPlan = e3.mutation.reduce('add_plan', plans,
557
- East.function([plans.type, StringType, PlanType], plans.type, ($, state, id, plan) => {
558
- const next = $.let(state.copy()); // the state is frozen: copy first
559
- $(next.insert(id, plan));
560
- return next;
561
- }));
562
-
563
- const reschedule = e3.mutation.edit('reschedule', plans,
564
- East.function([plans.type, StringType, DateTimeType, e3.mutation.editType(plans.type)], NullType,
565
- ($, state, id, due, edit) => {
566
- const plan = $.let(state.get(id)); // decodes one segment, not the record
567
- $(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
568
- }));
586
+ const PlansType = DictType(StringType, PlanType);
587
+
588
+ const plans = e3.record('plans', PlansType, {
589
+ initial: new Map(),
590
+ mutations: (mutation) => ({
591
+ add_plan: mutation.reduce(East.function([PlansType, StringType, PlanType], PlansType, ($, state, id, plan) => {
592
+ const next = $.let(state.copy()); // the state is frozen: copy first
593
+ $(next.insert(id, plan));
594
+ return next;
595
+ })),
596
+ reschedule: mutation.edit(East.function([PlansType, StringType, DateTimeType, e3.mutation.editType(PlansType)], NullType,
597
+ ($, state, id, due, edit) => {
598
+ const plan = $.let(state.get(id)); // decodes one segment, not the record
599
+ $(edit.set(id, { title: plan.title, owner: plan.owner, due, resources: plan.resources }));
600
+ })),
601
+ patch: mutation.patch(), // a view's save
602
+ }),
603
+ });
569
604
  ```
570
605
 
571
606
  Costs count in segments: a Dict or Set record is cut by key and by each entry's
@@ -573,54 +608,82 @@ encoded size, so wide rows get segments of a few rows and a one-row edit stays
573
608
  cheap. A view reads every field of the rows it shows, so keep a bulky payload it
574
609
  does not display in a second record keyed the same way.
575
610
 
576
- #### Secondary indexes — `e3.recordIndex(name, record, { on | key | keys, value? })`
611
+ #### Secondary indexes — `index.on([...fields])`, `index.key(fn)`, `index.keys(fn)`
577
612
 
578
613
  A second collection over a Dict record, in another sort order, maintained in the
579
- same commit, so a view by an attribute is a page rather than a scan. `on`
580
- names row fields in index order: one field uses its value as the key, several
581
- use a Struct in that order, and a Set field gives an entry per element. It
582
- also carries field metadata for `Data.bindPaged(record, { indexes })` and
583
- `Slice.bindRecord` (see **e3-ui**). `key` gives one computed entry per row;
584
- `keys` returns a Set, one entry per element (a row
585
- naming five resources appears five times, an empty set not at all); `value` is a
586
- projection a view renders without touching the record. The functions are pure
587
- and synchronous, since an index maintained commit by commit must equal its
588
- rebuild to the byte. The name is an identifier, never `primary` (the record's own
589
- order). Read through one with `index=<name>` on a page or key search; `e3
614
+ same commit, so a view by an attribute is a page rather than a scan.
615
+ `index.on([...])` names row fields in index order: one field uses its value as
616
+ the key, several use a Struct in that order, and a Set field gives an entry per
617
+ element. It also carries field metadata for `Data.bindPaged(record, { indexes })`
618
+ and `Slice.bindRecord` (see **e3-ui**). `index.key(fn)` gives one computed entry
619
+ per row; `index.keys(fn)` returns a Set, one entry per element (a row naming five
620
+ resources appears five times, an empty set not at all); the `value` option of
621
+ each is a projection a view renders without touching the record. The functions
622
+ are pure and synchronous, since an index maintained commit by commit must equal
623
+ its rebuild to the byte. Its name — its key in `indexes` — is an identifier,
624
+ never `primary` (the record's own order). Read through one with `index=<name>` on a page or key search; `e3
590
625
  reindex` rebuilds. A deploy builds an index the package declares and the state
591
626
  lacks, drops one it no longer declares, and runs nothing for the rest; a build
592
627
  is a split task over the record.
593
628
 
594
629
  ```typescript
595
- const byDue = e3.recordIndex('by_due', plans, { on: ['due'] });
596
- const byOwnerDue = e3.recordIndex('by_owner_due', plans, { on: ['owner', 'due'] });
597
- const byOwner = e3.recordIndex('by_owner', plans, {
598
- key: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner),
599
- value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
600
- });
601
- const byResource = e3.recordIndex('by_resource', plans, {
602
- keys: East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources),
630
+ const plans = e3.record('plans', PlansType, {
631
+ initial: new Map(),
632
+ indexes: (index) => ({
633
+ by_due: index.on(['due']),
634
+ by_owner_due: index.on(['owner', 'due']),
635
+ by_owner: index.key(East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.owner), {
636
+ value: East.function([StringType, PlanType], StringType, ($, _id, plan) => plan.title),
637
+ }),
638
+ by_resource: index.keys(East.function([StringType, PlanType], SetType(StringType), ($, _id, plan) => plan.resources)),
639
+ }),
640
+ mutations: (mutation) => ({ /* add_plan, reschedule, patch — above */ }),
603
641
  });
604
- const pkg = e3.package('planning', '1.0.0',
605
- plans, addPlan, reschedule, e3.mutation.patch(plans), byDue, byOwnerDue, byOwner, byResource);
642
+ const pkg = e3.package('planning', '1.0.0', plans); // the record whole: its indexes and mutations with it
606
643
  ```
607
644
 
608
- For events that must remain visible while they overlap a selected time window,
609
- declare `interval: { from: 'start', to: 'end' }`. Both fields must be DateTime
610
- or Option<DateTime>. Optional `on` fields are exact leading choices:
645
+ #### Interval and area indexes — `index.interval`, `index.area`
646
+
647
+ These indexes answer more than an order — as do the series and summary indexes
648
+ below — each with a builder method of its own whose keys the SDK generates.
649
+ Underneath each is an ordinary index — the same build,
650
+ mutation, storage, paging and `e3 reindex` — and the record Slice reads a
651
+ window through either (see **e3-ui**). Their generated keys are query
652
+ machinery: edits and sessions always target the record's primary key.
653
+
654
+ A generated key is a struct whose own fields are e3's: `on` holds the
655
+ declaration's leading fields, by their names, at their key types (a Set field's
656
+ element type) — a struct with no fields when it lists none — and the kind's own
657
+ parts sit beside it. So no name is reserved: a row may name a field anything,
658
+ `area`, `tile` and `__area` among them, and an index may be led by it.
659
+
660
+ | Index | Key | e3's parts |
661
+ |---|---|---|
662
+ | interval | `{ on, node, at }` | the tree node an entry is filed at; the instant it is ordered by, at the `from` field's type |
663
+ | area | `{ on, area }` | the cell's Morton code times 32 plus its level; −1 for a row not yet placed |
664
+ | series | `{ on, key, level, tile }` | the value of the field `key` names; the level an event is filed at, and the start of that level's tile (#1433) |
665
+ | summary | `{ on, key, tile }` | the value of the field `key` names; the tile's start |
666
+
667
+ A definition's `on` is the leading fields alone; its kind's options say the
668
+ rest (`events.indexes.by_time.interval`, `racks.indexes.racks_by_area.area`,
669
+ `jobs.indexes.jobs_by_press_week.series`).
670
+
671
+ **`index.interval({ on?, from, to })`** keeps events visible while they
672
+ overlap a selected time window. `from` and `to` are the row's DateTime or
673
+ Option<DateTime> fields; optional `on` fields are exact leading choices:
611
674
 
612
675
  ```typescript
613
- const byTime = e3.recordIndex('by_time', events, {
614
- interval: { from: 'start', to: 'end' },
615
- });
616
- const byStatusTime = e3.recordIndex('by_status_time', events, {
617
- on: ['status'], interval: { from: 'start', to: 'end' },
676
+ const events = e3.record('events', EventsType, {
677
+ initial: new Map(),
678
+ indexes: (index) => ({
679
+ by_time: index.interval({ from: 'start', to: 'end' }),
680
+ by_status_time: index.interval({ on: ['status'], from: 'start', to: 'end' }),
681
+ }),
618
682
  });
619
683
  ```
620
684
 
621
- An interval index uses the ordinary index build, mutation and paging machinery.
622
- It stores at most three entries per event per leading-field combination,
623
- independent of duration. The record Slice adapter resolves a half-open
685
+ Its key is `{ on, node, at }`. It stores at most three entries per event per
686
+ leading-field combination, independent of duration. The record Slice adapter resolves a half-open
624
687
  `[from, to)` window with at most 55 disjoint bounded queries per exact prefix;
625
688
  each matching original record occurs once, including events that started
626
689
  earlier or span the whole window. It never scans the record or guesses a
@@ -636,17 +699,242 @@ primary key. Calendar's `Calendar.events(binding, { slice, entries, … })`
636
699
  requires a matching interval declaration and the primary `entries` binding to
637
700
  confirm saves even after an edit leaves the selected window.
638
701
 
702
+ **`index.area({ on?, x, y, width?, height?, at?, cell })`** reads the rows
703
+ whose box meets a 2D window — what a floor plan's viewport shows — the interval
704
+ index's twin:
705
+
706
+ ```typescript
707
+ // RacksType: Dict<String, { hall: String, x: Option<Float>, y: Option<Float>, w: Float, h: Float }>
708
+ const racks = e3.record('racks', RacksType, {
709
+ initial: new Map(),
710
+ indexes: (index) => ({
711
+ racks_by_area: index.area({ x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01 }),
712
+ racks_by_hall_area: index.area({ on: ['hall'], x: 'x', y: 'y', width: 'w', height: 'h', at: 'centre', cell: 0.01 }),
713
+ }),
714
+ });
715
+ ```
716
+
717
+ | Option | The box's |
718
+ |---|---|
719
+ | `x`, `y` | place: two Float fields, or two Option<Float> fields whose `none` is a row not yet placed |
720
+ | `width`, `height` | size: two Float fields, or neither for a point |
721
+ | `at` | `'corner'`, the default — `x`, `y` is the box's minimum corner, as a zone's is — or `'centre'`, as an item's is |
722
+ | `cell` | finest cell: its side in world units, about the smallest thing drawn — 0.01 for a floor in metres, 1 for one in millimetres. It is rounded down to a power of two, so the keys' arithmetic is exact on every runtime, and has no default: a wrong one costs a silent tenfold read |
723
+ | `on` | exact leading fields, as the interval index's; a Set field fans out |
724
+
725
+ Each option takes only the row's fields of its type, so a wrong one does not
726
+ compile, and the definition checks them again, naming the option, the field and
727
+ its type. The domain is the square ±`cell`·2²⁷ world units around zero
728
+ (±1,048 km at a 0.01 cell, which rounds to 2⁻⁷), in 29 levels. Each row is
729
+ filed once, under the deepest level whose cell is at least twice its box's
730
+ larger side, in the cell holding its box's centre: the key
731
+ `{ on, area: Integer }`. A row not yet placed files under `area` −1, which the
732
+ record Slice reads as the unplaced clause; one outside the domain, not finite,
733
+ or larger than half of it files under the root, which every window reads. The index **carries its rows**: each
734
+ entry's projection is the whole row, so a window reads the index alone and no
735
+ record segment, and it takes no `value`. The record Slice reads a viewport,
736
+ grown by an eighth of its larger side on each side, as at most 266 disjoint
737
+ searches per exact prefix, sent in one request, which together hold every row
738
+ whose box meets it, each once — beside rows of the cells searched that it
739
+ misses, which the reader draws or not.
740
+
741
+ #### Series and summary indexes — `index.series`, `index.summary`
742
+
743
+ What a planning board reads: the events on a block of keys — a Plan's
744
+ resources — over a span of time, and, zoomed out past what a screen can draw,
745
+ how busy each key is. Both file a row's event by the key it is on and by time
746
+ tiles — `day`, `week` (ISO, from Monday), `month`, `quarter` or `year`, in UTC,
747
+ half-open — the key before the time, so a key's entries over any span are
748
+ contiguous: a series index's `{ on, key, level, tile }`, a summary index's
749
+ `{ on, key, tile }`.
750
+
751
+ ```typescript
752
+ // JobsType: Dict<String, { press: String, start: Option<DateTime>, end: Option<DateTime>, … }>
753
+ const jobs = e3.record('jobs', JobsType, {
754
+ initial: new Map(),
755
+ indexes: (index) => ({
756
+ jobs_by_press_week: index.series({ key: 'press', from: 'start', to: 'end', tile: 'week' }),
757
+ jobs_by_press_month: index.summary({ key: 'press', from: 'start', to: 'end', tile: 'month' }),
758
+ }),
759
+ });
760
+ ```
761
+
762
+ | Option | |
763
+ |---|---|
764
+ | `key` | the field naming what an event is on: any field but a Set |
765
+ | `from`, `to` | the event's start and end: DateTime or Option<DateTime> fields. A row with no start is filed nowhere; no end, or one at or before the start, is an instant |
766
+ | `tile` | a series index's finest level, and a summary index's tile: `day`, `week`, `month`, `quarter` or `year` |
767
+ | `on` | exact leading fields, before the key; a Set field fans out |
768
+
769
+ **A series index** is an ordinary index whose keys the SDK generates, and it
770
+ **carries its rows**, as the area index does. It files each event **once**, at
771
+ the finest level from its `tile` up to the year whose shortest tile holds the
772
+ event's length — a day, a week, 28 days, 90 days, 365 days — under that level's
773
+ tile holding its start; an instant at `tile`; past 365 days, under each year it
774
+ touches. An event filed at a level ends within the tile after the one holding
775
+ its start, so a reader reads a block of keys over any span from the index alone
776
+ as one range a key a level — that level's tiles from two before the span to one
777
+ after it — in one request, whatever the span's width. **A summary index** holds,
778
+ under each key and tile, `{ count, ms }`: how many rows touch the tile, and the
779
+ milliseconds of it they cover — a row's shares
780
+ add up to its length, and an instant's is none. Each entry stands for every row
781
+ of its key and names none, so its window's `key` is Null, a read of it joins
782
+ nothing, and a read that needs its rows — a joined read, a placement
783
+ (`datasetPlace`) or a membership read (`datasetWithin`) — is refused, naming the
784
+ index; the record's describe says it is a summary. A write
785
+ changes its totals by what its rows took away and gave — its program reads the
786
+ record, never the summary — and the apply adds that to each total, dropping one
787
+ whose count reaches zero; the totals are Integers, so a maintained summary is the
788
+ one `e3 reindex` sums, byte for byte, and a build split into pieces adds the
789
+ pieces' totals of a key. A change that would leave a total no rebuild sums — a
790
+ count below zero, or time with no row — fails the write, naming the total and
791
+ `e3 reindex`, which rebuilds it; a total past the largest Integer fails the
792
+ write or the build, naming a finer tile.
793
+
794
+ A series index files an event past a year under each year it touches, as many
795
+ as 4,096: one touching more is refused where it is written, naming it and its
796
+ span. A summary splits an event over its tiles, as many as 4,096 — eleven years
797
+ of days — and one touching more is refused the same way, naming the tile:
798
+ declare a coarser tile. Each option takes only the row's fields of its type, as
799
+ the area index's do, and the definition checks them again; a `value` is
800
+ refused, and so is an `on` field that is the key, the start or the end: the key
801
+ follows the leading fields, and the tiles are cut by the start and the end.
802
+
803
+ #### Nested series indexes — `index.nested`
804
+
805
+ What a planning board reads when a record's rows nest what it draws: a
806
+ machine's jobs and its load a day at a time, a hall's members, an assembly's
807
+ parts to any depth. One declaration a record lists every collection its rows
808
+ nest, each field with a builder saying what it holds. e3 files each item apart
809
+ from its row — once, by its collection, its row, its node and its time — keeps
810
+ each node's totals a summary tile at a time, and each parent's subtotals at
811
+ every level; and the record's other indexes carry the rows without those
812
+ collections, so a page of rows reads none of them and a window of items reads
813
+ no row.
814
+
815
+ ```typescript
816
+ // MachinesType: Dict<String, { name, site, status, jobs: Dict<String, Job>, load: Dict<DateTime, Float> }>
817
+ const machines = e3.record('machines', MachinesType, {
818
+ initial: new Map(),
819
+ indexes: (index) => ({
820
+ machines_in_time: index.nested({
821
+ collections: {
822
+ jobs: index.nested.items({ from: 'start', to: 'end' }), // things people drag: by id, their times fields
823
+ load: index.nested.values(), // a value per instant: keyed by the instant
824
+ },
825
+ tile: 'week',
826
+ summary: 'month',
827
+ }),
828
+ machines_by_site: index.on(['site']), // pages each machine without its jobs and load
829
+ }),
830
+ });
831
+ // Each member's load, and each hall's mean of its members', at every level from the day up.
832
+ const halls = e3.record('halls', HallsType, {
833
+ indexes: (index) => ({
834
+ halls_in_time: index.nested({
835
+ collections: { members: index.nested.rows({ load: index.nested.values({ rollup: 'mean' }) }) },
836
+ tile: 'day',
837
+ summary: 'month',
838
+ }),
839
+ }),
840
+ });
841
+ // Parts to any depth, each part's jobs filed under every part above it too.
842
+ const assemblies = e3.record('assemblies', AssembliesType, {
843
+ indexes: (index) => ({
844
+ assemblies_in_time: index.nested({
845
+ collections: {
846
+ parts: index.nested.rows({
847
+ jobs: index.nested.items({ from: 'start', to: 'end' }, { rollup: true }),
848
+ parts: index.nested.recursive(),
849
+ }),
850
+ },
851
+ tile: 'week',
852
+ summary: 'month',
853
+ }),
854
+ }),
855
+ });
856
+ ```
857
+
858
+ | Builder (`index.nested.…`) | A field holding |
859
+ |---|---|
860
+ | `items({ from, to })`, `items({ at })` | `Dict<String, Struct>`: things people drag, by id; their times are DateTime or Option<DateTime> fields — no start files an item nowhere, no end (or one at or before its start) makes it an instant. `{ rollup: true }` files each under every ancestor too |
861
+ | `values({ rollup? })` | `Dict<DateTime, V>`: a value per instant. `rollup` keeps every parent's subtotals |
862
+ | `rows({ … })` | `Dict<String, Struct>`: child rows, their own collections declared the same way |
863
+ | `recursive()` | `Dict<String, Self>`: more rows like the one listing it, to any depth |
864
+
865
+ TypeScript types `collections` field by field from the record's type alone: a
866
+ misnamed time, or a builder on the wrong kind of field, is an error where it is
867
+ written, and its message says what fits. The definition checks the same again,
868
+ in words, for a declaration TypeScript could not see. A record has one nested
869
+ series index, which takes no `on` and no `value`.
870
+
871
+ **What it files.** Every entry is keyed `{ collection, row, path, run, level,
872
+ tile, item }` (e3-types' `NestedEntryKeyType`): the collection's place, the
873
+ field names from the row down (`members` then `load`); the row's key; the Dict
874
+ keys from the row down to the node (`[]` for the row, `m-3` for a member); the
875
+ run; the level and tile; and the item. A place and a path are held as a key
876
+ holds a list, folded from the last: `.step (key="members", rest=.step
877
+ (key="load", rest=.end))`. Each run of a node is one contiguous range a level:
878
+
879
+ | Run | Holds |
880
+ |---|---|
881
+ | `items` | each item once, as a series index files an event — at the level its length fits, past a year under each year — carrying it (`.item .jobs (…)`); a value, and an `{ at }` item, at the declared tile alone |
882
+ | `totals` | where `summary` is declared, each node's `{ count, ms, values }` a summary tile at a time; for a Float or Integer collection, `values` is the sum, the least, the greatest and the last there, taken in key order |
883
+ | `subtotals(rollup)` | for `values({ rollup })`, every parent's subtotal by each rollup at each level from `tile` to the year, where a descendant's value touches the tile: a node holding values folds its own, a parent aggregates its children's, in key order |
884
+ | `subtree`, `subtreeTotals` | for `items(…, { rollup: true })`, each item under every ancestor too, `.under (path, id)`, and each ancestor's totals of them |
885
+
886
+ A rollup is `'sum'`, `'mean'` (a mean of means up the tree), `'min'`, `'max'`,
887
+ `'count'` (the leaf rows beneath that hold a value), a fold and an aggregate
888
+ named apart (`{ fold: 'last', aggregate: 'sum' }`, a balance), or several.
889
+
890
+ **Reading.** A reader searches the index's leading fields — `[collection, row,
891
+ path, run, level, tile]`, each as East prints it — as a range a row a level: a
892
+ block of fifty machines' jobs over three weeks is fifty times four ranges in one
893
+ `datasetFindKeys` request, and one `datasetGetRuns` read of the entries they
894
+ select, each job carried whole; a parent's week subtotals are one range; a
895
+ part's subtree is one range a level.
896
+
897
+ **The lean rows.** Beside a nested series index, every other index of the
898
+ record that carries nothing or the row carries the row without the listed
899
+ collections — each listed collection empty, child rows kept with theirs empty —
900
+ so a page of rows read without a join reads no nested item and no segment of
901
+ the record's own. An index with a `value` of its own carries its own. A
902
+ binding of the record's indexes lists the nested index by name beside any that
903
+ carry the lean rows — `Data.bindPaged(machines, { indexes: ['machines_by_site',
904
+ 'machines_in_time'] })` (see **e3-ui**).
905
+
906
+ **Its span.** Beside every nested series index e3 keeps its span, an index of
907
+ the record's named `<index>_span` (e3-types' `nestedSpanName`), built with it
908
+ and never declared: under each collection's place and each month its items or
909
+ values touch (`NestedSpanKeyType`, `{ collection, tile }`), how many of them
910
+ touch it and the milliseconds of it its items cover, summed across the rows as a
911
+ summary index's totals are, and a month let go once nothing touches it. A
912
+ collection's first and last month held are its span across the record — what a
913
+ Plan's horizon covers for the series reading it — and its months' counts are
914
+ how much it holds in each, which the horizon's strip draws as its bars (see
915
+ **e3-ui**). Reading it is one search a collection, an exact match on its
916
+ place, and one read of the first and last entry of each. An index the record
917
+ declares under that name is refused.
918
+
919
+ **Writes.** A row a write updates by a patch is maintained by the patch: e3
920
+ walks it along the declaration — each collection's Dict patch, and each child
921
+ rows' — and compares the entries of only the items it names, with the totals of
922
+ the summary tiles they touch and the subtotals of the tiles they touch at each
923
+ level. A replace on the way, which a recursive type's patch always is, and an
924
+ inserted or deleted row are compared whole. Both ways give the same delta, and a maintained
925
+ index is the one `e3 reindex` rebuilds, byte for byte.
926
+
639
927
  #### Migrations — how a record's type changes
640
928
 
641
929
  | Form | Function | Runs as |
642
930
  |---|---|---|
643
- | `e3.migration.value(name, rec, fn, { after? })` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
644
- | `e3.migration.rows(name, rec, fn, { after? })` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
645
- | `e3.migration.rekey(name, rec, fn, { after? })` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
931
+ | `migration.value(name, fn)` | `(Old) => New`, over any record | one unit; its runner opens the state lazily |
932
+ | `migration.rows(name, fn)` | a Dict's rows, `(K, V1) => V2` with the keys kept, or an Array's elements, `(T1) => T2` | a task split over the state: a piece at a time, in parallel, each piece cached |
933
+ | `migration.rekey(name, fn)` | a Dict's entries, `(K1, V1) => { key: K2, value: V2 }`, or a Set's elements, `(T1) => T2` | the same split task, its pieces merged by key: two Dict rows landing on one key fail the deploy, naming it; two Set elements are one |
646
934
 
647
- A record's steps are one chain: each names the step before it with `after` (a
648
- step of the same record, leaving it as this step's input type), passing the last
649
- step passes the chain, and the last step leaves the record as its declared type.
935
+ A record's steps are one chain: the list `migrations` returns, in order, each
936
+ step taking the record as the step before it leaves it, and the last leaving it
937
+ as its declared type — each checked where the record is declared.
650
938
  A step's name is an identifier, unique on the record, and a workspace records the
651
939
  steps it has applied by name — so moving a step's code, or re-exporting it with a
652
940
  newer e3, does not run it again, and an edit to an applied step's body is not
@@ -655,12 +943,14 @@ detected. The functions are pure, synchronous East.
655
943
  ```typescript
656
944
  const RowV1Type = StructType({ title: StringType });
657
945
  const RowV2Type = StructType({ title: StringType, owner: StringType });
658
- const plans = e3.record('plans', DictType(StringType, RowV2Type), new Map());
659
-
660
- const addOwner = e3.migration.rows('add_owner', plans,
661
- East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' })));
662
- // The next version adds its step after this one: e3.migration.value('…', plans, fn, { after: addOwner })
663
- const pkg = e3.package('planning', '2.0.0', plans, addOwner);
946
+ const plans = e3.record('plans', DictType(StringType, RowV2Type), {
947
+ initial: new Map(),
948
+ migrations: (migration) => [
949
+ migration.rows('add_owner', East.function([StringType, RowV1Type], RowV2Type, ($, _id, row) => ({ title: row.title, owner: 'unassigned' }))),
950
+ // The next version adds its step here, after this one.
951
+ ],
952
+ });
953
+ const pkg = e3.package('planning', '2.0.0', plans);
664
954
  ```
665
955
 
666
956
  What a deploy does with each record, decided before it writes anything:
@@ -718,7 +1008,7 @@ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
718
1008
  | `e3 workspace copy [repo] <from> <to>` | Make `<to>` what `<from>` is now — its package and every input, record and task output, so nothing runs again — made, or replaced whole with its runs. It writes refs only, so its cost follows the number of datasets, never their size; a write to either afterwards leaves the other as it was. Refused while a dataflow runs in `<to>`. |
719
1009
  | `e3 workspace deploy [repo] <ws> <pkg[@ver]>` | Deploy an imported package. |
720
1010
  | `… --from-zip <zip>` · `… --from-source <src.ts> [--functions <manifest…>]` | Import the zip (or bundle the source) and deploy, creating the workspace. |
721
- | `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on; `--memory`: what those may reserve (local). |
1011
+ | `… [--schema <policy>] [--inputs <policy>] [--allow-drop-records] [--plan] [--skip-file-sources] [-j <n>] [--memory <size>] [--quiet]` | A record's policy, an input's, and `file` sources; says how far it has got (below), `--quiet` aside. `-j`: the `file` sources taken in at once, and the runner processes their intake units, the migrations and the index builds run on, and with `--from-source` the workspace packages exporting their functions at once; `--memory`: what those may reserve (local). |
722
1012
  | `e3 workspace export [repo] <ws> <zip> [--name <n>] [--version <v>]` | The workspace's state as a package. |
723
1013
  | **Datasets** | |
724
1014
  | `e3 dataset get [repo] <ws.name> [-f east\|json\|beast2]` | Print a value. |
@@ -734,7 +1024,7 @@ a task is `<ws>.<task>` and a mutation `<record>.<mutation>`.
734
1024
  | **Records** (`-w <ws>` required) | |
735
1025
  | `e3 mutate <repo> <record.mutation> [args…] -w <ws> [-j <n>] [--memory <size>] [-v]` | Apply a mutation. |
736
1026
  | `e3 seed <repo> <record> <file.beast2> -w <ws> [-j <n>] [--memory <size>]` | Commit a record's first state from a beast2 file, as its `$init`: for a record declared with no initial value, unassigned until then (local). The file is taken in as a `file` source is, never read whole; a record that holds a state is refused. |
737
- | `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target; local). |
1027
+ | `e3 history <repo> <record> -w <ws> [--limit <n>] [--from <hash>] [--delta]` | Commits, newest first (`--delta`: what each changed, per target — a summary's totals added to as `±n`; local). |
738
1028
  | `e3 reindex <repo> <record> -w <ws> [--index <name>] [-j <n>] [--memory <size>]` · `e3 compact <repo> <record> -w <ws>` | Rebuild indexes (local) · collapse the history to a `$compact` root, the state kept. |
739
1029
  | **Development** | |
740
1030
  | `e3 watch <src.ts> <repo> <ws> [--start] [--schema <p>] [--inputs <p>] [--abort-on-change] [--functions <manifest…>] [-j <n>] [--memory <size>]` | Redeploy on each change (and run, with `--start`). |
@@ -752,7 +1042,10 @@ peak a unit of its stage has reached in the run, so a stage runs its first unit
752
1042
  alone and then fans out; on Linux and macOS a guard stops the newest unit when
753
1043
  the runners together pass the budget, and runs it again once it fits. A deploy
754
1044
  takes its `file` sources in `-j` at a time, their pieces' intake units running
755
- under the same budget. The commands that run East — `workspace deploy`,
1045
+ under the same budget. A source deploy, or a `watch`, exports the workspace
1046
+ packages its imported functions come from `-j` at a time, a process each, on
1047
+ this machine; against a server, as many at once as this machine has CPUs. The
1048
+ commands that run East — `workspace deploy`,
756
1049
  `dataset set --from-file`, `dataflow run`, `run`, `call`, `mutate`, `seed`,
757
1050
  `reindex` and `watch` — take the flags for a local repository; against a server the CLI
758
1051
  refuses them, since the server runs the work under its own budget
@@ -763,7 +1056,11 @@ line once it is in: its size, time and rate, and how it was taken in
763
1056
  (`unchanged, already in the store`, `taken in by east-c`, or `carried` for a
764
1057
  value that is not a collection), with why the first time a runner fell back to
765
1058
  another. A terminal also keeps a live line for the files in flight, with their
766
- pieces, the rate and the time left across them. Against a server, the deploy line
1059
+ pieces, the rate and the time left across them. A source deploy prints a line
1060
+ for each workspace package whose functions it links: `exported <n> function(s)
1061
+ of <pkg> (east-py export-functions)`, or `reused <n> function(s) of <pkg>,
1062
+ unchanged since its last export` when the export cache held them. Against a
1063
+ server, the deploy line
767
1064
  says what the job is doing, such as migrating a record or building an index;
768
1065
  the deliveries then upload a few at a time, each printing a line once it is in,
769
1066
  and the live line says what the server's commit is doing with each in flight.
@@ -823,8 +1120,8 @@ naming the holder: its pid, boot id, command and when it took the lock.
823
1120
  |---|---|
824
1121
  | Repositories | `repoList(url, opts)`, `repoCreate(url, name, opts)`, `repoRemove(url, name, opts)`, `repoStatus(url, repo, opts)`, `repoRecord` (its release and upgrades), `repoGc(url, repo, gcRequest, opts)` — a job it polls; `repoGcStart` and `repoGcStatus` apart |
825
1122
  | Packages | `packageList(url, repo, opts)`, `packageGet(url, repo, name, version, opts)`, `packageImport(url, repo, zipBytes, opts)`, `packageExport(url, repo, name, version, opts)` → zip bytes, `packageRemove` |
826
- | Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
827
- | Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
1123
+ | Workspaces | `workspaceList`, `workspaceCreate(url, repo, ws, opts)`, `workspaceCopy(url, repo, from, to, opts)` → the target, made or replaced whole, `workspaceBundle(url, repo, ws, opts)` → the workspace as it is, as a bundle's bytes — its package, every dataset's ref, the executions its tasks' outputs come from, and every object those reach — and `workspaceBundleImport(url, repo, ws, bundle, opts)` → the workspace made from one under a name no workspace has, in another repository or another e3 (`workspace_exists`, `package_exists`, `bundle_invalid`), its tasks up to date with no run; a host mounts these routes where a bundle fits in a request: the local server, and e3-web, `workspaceGet`, `workspaceStatus(url, repo, ws, opts, { paths? })` — the whole workspace's, or the datasets named and the tasks producing them, `workspaceStatusSince(url, repo, ws, etag, opts, { paths? })` → `{ status, etag }`, or `null` while the status answered under `etag` is current (a poll sends the last answer's tag back; `null` asks in full), `workspaceLockStatus(url, repo, ws, opts)` → what holds the workspace and how far it says it has got (a deploy's files and records), or `null`, `workspaceRemove`, `workspaceDeploy(url, repo, ws, 'pkg@ver', opts, { schema?, inputs?, allowDropRecords?, plan?, onProgress? })` — a job it polls, whose progress while `deploying` is the deploy's own, answering what it did with each record, index and input, `workspaceExport(url, repo, ws, opts, { name?, version? })` → zip bytes. A poll's request takes `pollRequestOptions(opts)`: one attempt, its next poll being its retry |
1124
+ | Datasets | `datasetGet(url, repo, ws, path, opts)` → `{ data, hash, size }` (a collection downloads as its segments), `datasetGetStream`, `datasetGetPage(…, window, opts)`, `datasetFindKey(…, query, opts)`, and several reads in one request (at most `DATASET_BATCH_MAX`, 10,000, each at one content hash, in a body of at most 16 MiB: a larger one is refused 413, `payload_too_large`): `datasetFindKeys(…, { queries, index?, hash? }, opts)` — key searches, `datasetGetRows(…, { keys, hash? }, opts)` — a Dict's rows by `.east` key, each with its position — or `datasetGetRows(…, { keysOf: { path, index, runs, hash }, hash? }, opts)`, by the keys the entries of some runs of another record's index name, which the server reads from the index so none is sent (a drawing's details for its window's cells), a stale `keysOf.hash` thrown as `RunsHashMismatchError` naming the record's content — and `datasetGetRuns(…, { runs, hash, index?, join? }, opts)` — the entries of several runs as one window, a rows or runs answer holding the first `count` the page budget reaches (by runs, of `runEntries`); and where a narrowing's rows sit in a record, from its index alone: `datasetPlace(…, { index, runs, offset?, limit?, hash }, opts)` → `{ segments, placed, runEntries, totalElements, hash }` — how many of the runs' entries each record segment holds (`{ segment, offset, count, rows }`), for the first `placed` of a window from `offset` (the next window from `offset + placed`), and `datasetWithin(…, { index, runs, keys, hash }, opts)` → `{ within, hash }` — whether each index entry key (`(ik=…, k=…)`) lies in one of the runs, `datasetSet(url, repo, ws, path, beast2Bytes, opts)`, `datasetSetStream(url, repo, ws, path, { size, hash, slice }, opts, { onCommitProgress? })` — a file of any size, the server's commit saying how far it has taken it in, `datasetList`, `datasetListAt`, `datasetListRecursive`, `datasetListWithStatus`, `datasetGetStatus` |
828
1125
  | Runs and tasks | `dataflowExecute(url, repo, ws, { force?, filter? }, opts, { pollInterval?, timeout? })` → the result (or `dataflowExecuteLaunch` and `dataflowExecutePoll`) — `force` is `true` or the names of the tasks to re-run, `filter` one task's exact name, `dataflowCancel` — with nothing running, `dataflow_error` ("No active execution for this workspace"), `dataflowGraph`, `dataflowBudget`, `taskList`, `taskGet`, `taskExecutionList(url, repo, ws, task, opts, { limit?, before? })` — a page of the task's runs, the latest first, each naming its `executionId`, the next page the runs `before` the last one's (100 a page unless `limit` says, 1,000 at most), `taskLogs(url, repo, ws, task, { stream?, offset?, limit?, execution? }, opts)` — a chunk of the log of the task's current execution, or of the one `execution` names (`{ inputsHash, executionId }`), the chunk naming its execution and whether it has `ended` |
829
1126
  | Functions | `functionList`, `functionDescribe`, `functionCall(url, repo, pkg, version, fn, { args, runner, limits }, opts)` — each argument `variant('value', beast2Bytes)`; the function runs on its own runner for any caller (`runner: none`); a runner the call names is never `custom`, and loads a platform package the function's does not only for an elevated caller; `workspaceFunctionList`, `…Describe`, `…Call(url, repo, ws, fn, request, opts)` — where an argument may also be `variant('dataset', path)`, a dataset of the workspace pinned at its hash, which the result names (`inputs`); `oneShotExecute` — a reader runs a platform-free one (stock runner, `platforms: []`, no platform call), and its result names the datasets it read (`inputs`); `splitCall(url, repo, ws, { bodyIr, args, output, then, runner, limits }, opts, { onProgress? })` — a program over a dataset's pieces, as a job it polls, under the same rule (or `splitCallLaunch`, `splitCallStatus` and `splitCallExplain`) |
830
1127
  | Records | `workspaceRecordDescribe`, `workspaceRecordMutate(url, repo, ws, record, mutation, { args, actor, limits }, opts, idempotencyKey?)`, `workspaceRecordHistory(url, repo, ws, record, limit, opts, from?)`, `workspaceRecordCompact` |
@@ -838,7 +1135,7 @@ takes a runner, `new LocalTaskRunner(repo)`.
838
1135
  |---|---|
839
1136
  | Repositories | `repoInit(path)`, `repoFind(startPath?)`, `repositoryOpen(storage, repo)` (checks the repository and applies the upgrades it owes), `repositoryUpgradeStep(storage, repo, { budgetMs, waitMs? })` → `{ owed }` — a host's job applies them so, a part per run under its time limit, each part taking a step up where the last stopped, `repoGc(storage, repo, { dryRun?, minAge?, keepRuns?, keepDays?, retention?, executionAlive? })` — holding the repository still, or, with `retention: { windowMs }`, beside running work: an object goes once unreachable for the window and not written or re-referenced since; given a runner's `executionAlive`, an attempt recorded running that cannot finish is recorded interrupted and pruned; `repoGcStep(storage, repo, step, { windowMs, pruneMs?, markMs?, concurrency?, executionAlive?, … })` runs that one step at a time, each returning the next (`GcStepType`), `pruneMs` and `markMs` bounding a prune step and a mark step for a host with a time limit |
840
1137
  | Packages | `packageImport(storage, repo, zipPath)`, `packageExport(storage, repo, name, version, zipPath)`, `packageList`, `packageRemove` |
841
- | Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
1138
+ | Workspaces | `workspaceCreate(storage, repo, ws)`, `workspaceCopy(storage, repo, from, to)`, `workspaceBundleExport(storage, repo, ws, writable, { have? })` — the workspace as a bundle to a WHATWG stream, leaving out the objects `have` says the receiving repository holds — and `workspaceBundleImport(storage, repo, ws, source)` — the workspace made from a bundle (a stream or chunks) under a name no workspace has, its objects stored as they come and its refs and record written once the repository holds everything they name, `workspaceDeploy(storage, repo, ws, pkgName, pkgVersion, options?)` (`runner`: its migrations, index builds and intake units; `schema` and `inputs`: its policies; `onRecordPlan`, `onRecordIndex`, `onInputPlan`: what it decided; `sourceConcurrency`: the `file` sources taken in at once; `onSourceProgress`, `onDeployProgress`: how far it has got), `workspaceExport(storage, repo, ws, zipPath, name?, version?)`, `workspaceStatus(storage, runner, repo, ws)`, `workspaceStatusRevision(storage, repo, ws)` — a token that moves whenever the status might, which the API's `ETag` names, `workspaceLockStatus(storage, repo, ws)`, `workspaceListStates(storage, repo)` — every workspace with its state in one read of the store, `workspaceRemove(storage, repo, ws, { sweep? })` — the record and what a store removes at once go in one call, and the rest is swept in bounded batches unless `sweep: false` leaves them to `workspaceSweep(storage, repo, cursor?)`, which a host with a time limit runs from a job |
842
1139
  | Datasets | `workspaceGetDataset(storage, repo, ws, treePath)`, `workspaceSetDataset(storage, repo, ws, treePath, value, type)`, `datasetAdoptFile(storage, repo, ws, treePath, file, { runner, onProgress? })` → `{ hash, size, segments, rows, taken, runners? }`, `taken` being `known`, `carried` or `taken` (by the `runners` named); `deliveryAdoptStored(storage, repo, ws, { address, type }, sha256, { runner?, onProgress?, signal? })` — a delivery the store holds taken in as a stored value of the type, naming no dataset, for a record's first state |
843
1140
  | Runs | `dataflowExecute(storage, repo, ws, options?)`; `LocalOrchestrator` to start, poll and cancel a run |
844
1141
  | Records | `recordMutate(storage, runner, repo, ws, record, mutation, args, { actor })`, `recordHistory`, `recordDescribe`, `recordCompact`, `recordReindex`; `recordSystemCommit(storage, runner, repo, ws, record, { name, target, actor, args? })` — `$rollback` to a commit, `$restore` of a state at the record's migrations, and `$init` on a record declared with no initial value: its first state, from a stored value, with the whole chain applied |
@@ -910,6 +1207,7 @@ package is served beside the standard ones:
910
1207
  | `serveE3` option | Default | Effect |
911
1208
  |---|---|---|
912
1209
  | `units` | — | starts a unit worker: a Web Worker whose script calls `serveUnits()` |
1210
+ | `width` | the cores | the most units that run at once, and so the most unit workers, each with a heap of its own |
913
1211
  | `persist` | `true` | `false` keeps repositories in memory, gone with the page |
914
1212
  | `name` | `'e3'` | the storage's name: e3 workers of one name share their repositories, in one tab or several |
915
1213
  | `identify` | none | each request's caller, `{ sub, email?, roles }`, from the request; a request to a repository whose caller it does not identify is answered 401. None: the page's one caller may do everything |
@@ -974,7 +1272,14 @@ not had, in place; one a newer e3 upgraded is refused, naming that release.
974
1272
  environment — and its inputs' hashes, so a task re-runs only when one of those
975
1273
  changes (`--force` bypasses the cache, and `--force-task <task>` for the tasks
976
1274
  it names: a task whose code e3 cannot see changed, such as a platform
977
- function's, or one reading data from outside e3). A change to an input re-runs its
1275
+ function's, or one reading data from outside e3). A task's program holds the
1276
+ source locations of its body and of every function it embeds, an imported
1277
+ python or node function's too, since its errors, a caught error's `stack` and
1278
+ the closures it returns name them: an edit that moves one of those lines — a
1279
+ comment or an import added above a function — re-runs the task. One that leaves
1280
+ them where they were — another function, the end of a module, another package —
1281
+ re-runs only the tasks whose environment it is in, their runner's `{ custom }`
1282
+ packages. A change to an input re-runs its
978
1283
  consumers, even during a run. A split task is cached a unit at a time: each piece
979
1284
  and each merge is its own execution, and pieces are cut by content, so an append
980
1285
  or an insertion re-runs only the pieces it reaches and the merges above them.