@elaraai/east-codex-plugin 1.0.99 → 1.0.101
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.build/mcp/server.js +3 -2
- package/.codex-plugin/plugin.json +1 -1
- package/index.json +12207 -216
- package/package.json +1 -1
- package/skills/e3/SKILL.md +37 -809
- package/skills/e3/reference/browser.md +93 -0
- package/skills/e3/reference/cli.md +123 -0
- package/skills/e3/reference/from-code.md +48 -0
- package/skills/e3/reference/rack.md +139 -0
- package/skills/e3/reference/record-indexes.md +326 -0
- package/skills/e3/reference/record-migrations.md +63 -0
- package/skills/e3/reference/repository.md +25 -0
- package/skills/e3-ui/SKILL.md +273 -3623
- package/skills/e3-ui-cli/SKILL.md +1 -5
- package/skills/east/SKILL.md +31 -464
- package/skills/east/reference/beast2.md +71 -0
- package/skills/east/reference/cross-language-and-json-schema.md +87 -0
- package/skills/east/reference/queries.md +46 -0
- package/skills/east/reference/runners-and-ir.md +148 -0
- package/skills/east/reference/trees-and-recursion.md +56 -0
- package/skills/east/reference/values-in-typescript.md +62 -0
- package/skills/east-py/SKILL.md +29 -1652
- package/skills/east-py/reference/beast2.md +108 -0
- package/skills/east-py/reference/cli.md +51 -0
- package/skills/east-py/reference/columns-and-platform-functions.md +87 -0
- package/skills/east-py/reference/control-flow-and-queries.md +147 -0
- package/skills/east-py/reference/cross-language.md +70 -0
- package/skills/east-py/reference/ir-and-diagnostics.md +150 -0
- package/skills/east-py/reference/patterns.md +369 -0
- package/skills/east-py/reference/performance.md +70 -0
- package/skills/east-py/reference/value-methods.md +306 -0
- package/skills/east-py/reference/values-construction-and-json.md +88 -0
- package/skills/east-py/reference/writing-east-functions.md +275 -0
- package/skills/east-py-datascience/SKILL.md +12 -33
- package/skills/east-py-io/SKILL.md +1 -5
- package/skills/east-py-std/SKILL.md +1 -5
- package/skills/east-ui/SKILL.md +220 -1332
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
# Beast2 streaming: collections larger than memory
|
|
2
|
+
|
|
3
|
+
`write_beast2_file` / `open_beast2_file`, parallel writes and splicing, the writers and page readers. Part of the `east-py` skill.
|
|
4
|
+
|
|
5
|
+
## Beast2 streaming — bounded-memory collections (`from east.serialization.beast2 import ...`)
|
|
6
|
+
|
|
7
|
+
Beast2 v5 encodes a large Array/Set/Dict as an append-only stream of
|
|
8
|
+
independently decodable segments: writer memory is one segment, never the whole
|
|
9
|
+
collection, and the decoders read v4 and v5 through the same entry points. The
|
|
10
|
+
canonical writers — the managed file writer, `Beast2ElementWriter`,
|
|
11
|
+
`encode_beast2_paged_for` — cut segments by one content-defined rule, the same in
|
|
12
|
+
TypeScript and east-c, so a value's bytes are its own whoever writes them;
|
|
13
|
+
`Beast2Writer` makes each batch you give it a segment instead.
|
|
14
|
+
|
|
15
|
+
**Managed files — start here (`open_beast2_file` / `write_beast2_file`).** Path
|
|
16
|
+
in, East values out. The file is self-describing, so a read needs no declared
|
|
17
|
+
type (a write does; a type declared on a read is validated at open). The file
|
|
18
|
+
object owns the fd and the mmap (closed on `with`-exit), east-c does every byte,
|
|
19
|
+
and the segments are the canonical ones — no buffers, iterators or batch sizes in
|
|
20
|
+
your code. The read flavour mirrors the root collection's read surface name for
|
|
21
|
+
name.
|
|
22
|
+
|
|
23
|
+
| Signature | Description |
|
|
24
|
+
|-----------|-------------|
|
|
25
|
+
| `write_beast2_file(path, T, value, *, codec="deflate")` | One call writes a collection of any size as one indexed v5 file — the value's canonical blob: segments where the content-defined cut rule places them, bounded in elements and bytes (wide rows never pile into one), key-disjoint for a Dict/Set. The bytes TypeScript and east-c write, and what `encode_beast2_paged_for(T)(value)` returns |
|
|
26
|
+
| `open_beast2_file(path, T, mode="w", *, codec=)` | The streaming writer: `.write()` takes East collections or python `list`/`dict`/`set` batches of any size, which add up to one canonical blob (a Set/Dict batch continues strictly ascending from the last); `.segments` counts the segments closed so far |
|
|
27
|
+
| `open_beast2_file(path, T=None, *, project=None)` | The read: `Beast2ArrayFile` / `Beast2DictFile` / `Beast2SetFile`, a READ-ONLY East collection VALUE subclassing `EastArray`/`EastDict`/`EastSet` — `isinstance`/`type_of` answer, every eager method works (streamed, below), mutation raises, and it binds into functions or passes into compiled calls by reference, where keyed reads answer from the pager, one frame per hit or miss, through a byte-budgeted segment cache (`EAST_PAGED_CACHE_BYTES`); `close()` defers while a bind holds it. `T` is optional (the header supplies it, also as `f.wire_type`); a declared `T` is checked against the header at open, never decoded as garbage |
|
|
28
|
+
| `read_beast2_type(source) -> EastType` | The root type of any beast2 blob (v4 or v5), from a path or a buffer, nothing decoded — for a file you know nothing about |
|
|
29
|
+
| `write_beast2_file_parallel(path, T, partitions, produce, *, processes=, strategy="auto", codec=, keep_shards=False, verify=False)` | N workers, ONE file: `produce(partition)` returns that partition's batches (or one collection); each worker writes a shard cut canonically from its start, and the shards splice **in partition order** as they finish. `strategy="auto"` forks on Linux/macOS — what `produce` closes over is inherited copy-on-write, so build the expensive context before the call (and before starting threads) — and runs inline on Windows, byte-identically. Any worker failure fails the call with the worker's traceback and leaves nothing behind |
|
|
30
|
+
| `splice_beast2_files(path, T, sources, *, verify=False) -> (segments, elements)` | Merges indexed v5 files by **byte copy** — east-c parses the geometry, `os.sendfile` moves the frames, nothing re-encodes. `sources` may be a lazy generator (spliced as they complete, in row order). Each must be v5, indexed and self-contained with an identical type section; a refusal names the path and leaves no destination. `verify=True` re-walks the result with east-c's strict reader |
|
|
31
|
+
| `f.load()` | The whole collection, decoded in east-c off the mmap — input memory stays one segment at any size; also the mutable escape hatch, like `f.copy()` |
|
|
32
|
+
| `f.segments()` | DEPRECATED — the file IS its collection value; still works (with a warning) for per-batch migration code |
|
|
33
|
+
| `len(f)` · `f.segment_count` · `f.self_contained` · `f.indexed` | O(1) from the trailing index, exact for every root kind |
|
|
34
|
+
| Array: `f[i]` / `f[a:b]` · `f.get(i)` ❗bounds · `f.get_or_default(i, d)` · `f.try_get(i)` · `f.has(i)` · `f.slice(a, b)` · `f.get_keys(rows)` | `EastArray`'s names, signatures and errors; a point read decodes only the owning segment, `get_keys` each owning segment once |
|
|
35
|
+
| Dict: `f[k]` ❗KeyError · `f.get(k[, default \| fn(k)])` · `f.get_or_default(k, d)` · `f.try_get(k)` · `f.has(k)` / `k in f` · `f.get_keys(keys, fill)` · `f.items()` / `values()` / iteration (streaming) · `f.keys()` (the Set) · `f.size()` — Set: `x in f` / `f.has(x)` | east-c binary-searches the segment *fences* (each segment's first key, from a bounded probe, cached), then decodes ONLY the owning segment (a small LRU keeps hot ones); `get_keys` merges the sorted keys against the fences and calls `fill` per missing key. The first keyed read verifies the fences: a corrupt or pre-contract blob raises `segments are not disjoint ascending key ranges` instead of reporting false misses |
|
|
36
|
+
| Array sorted search: `f.find_sorted_first/last(target)` → a global index · `f.find_sorted_range(target)` → `{start, end}` | `EastArray`'s contract over the whole file, only the boundary segment decoded; no `key=` projection (the file pages in element order) — pair with `f.slice(start, end)` |
|
|
37
|
+
| Compute: `map` `filter` `filter_map` `first_map` `reduce` `scan` `map_reduce` `sum` `mean` `maximum` `minimum` `every` `some` `find_*` `is_sorted` `to_set` `unique` `to_dict` `to_array` `to_columns` `map_batches` `string_join` `flat_map` (Array) / `flatten_to_array` (Set, Dict) `flatten_to_set` `flatten_to_dict` `for_each`, the whole `group_*` family, Set algebra | Each segment runs the ordinary eager method, and partials combine in east-c in stream order: order-dependent folds thread ONE accumulator and grouped folds seed each segment from the running per-group accumulators, so results equal `load()` exactly, float order included. Array `(el, idx)` callbacks, and the indices `find_*` / `group_find_*` report, are GLOBAL; `first_map`/`some`/`every`/`is_superset_of` stop decoding at the answer. Dict/Set compute streams disjointness-verified segments. A re-keyed collision (`to_dict`, `flatten_to_dict`, `group_to_dicts`) combines left-associatively in stream order — use an associative `combine`. `sort`/`reverse`/`copy`/`concat`/`union` materialize the collection — `load()` first |
|
|
38
|
+
| Column projection — INFERRED on the compute family; EXPLICIT with `open_beast2_file(path, project=NARROW)` | The compute family builds its callbacks FIRST and decodes each segment to just the struct fields their IR reads (the rest are hopped over, never built — building values, not walking bytes, is what decoding costs). Fields subset by name at any depth; a subtree used any other way stays whole, so results never change; Dict keys and Set elements never narrow. Runner-opened task inputs infer the same from the body's loop IR. What cannot be inferred decodes whole and is COUNTED in `eager_stats()` (`beast2_segments_projected/whole`, `beast2_projection_declined_*` by reason) — a projection that silently stopped applying would be an invisible cliff. The explicit form serves the subset to EVERY read (point reads, keyed gets, `load()`); `project` must be a subset of the wire type (a missing field raises `ValueError` naming it), a declared `T` keeps its meaning, and `find_sorted_*` refuse under it. A segment decoded under one mask is never served to a read needing more. No wire change |
|
|
39
|
+
| Degraded blobs | v4 → refused (`decode_beast2_with_header_for` still decodes it whole); index-less v5 → `load()` works, random access refuses; not self-contained → point reads refuse |
|
|
40
|
+
| **Manifest directories** — one object per segment plus a manifest naming them: how e3 stores a collection, and stages a task input |
|
|
41
|
+
| `Beast2ManifestWriter(T, path, *, codec="deflate")` | The canonical writer (a context manager): `.add(el)` / `.add_all(batch)`, as `Beast2ElementWriter` takes them; each segment goes to `<path>.segments/<sha256>.beast2`, a standalone blob under the shared header, and `.close()` writes the manifest to `path` — the directory TypeScript's `Beast2ManifestWriter` and east-c write. A writer left by an exception writes no manifest; `.segments` counts them; east-c does every byte, hashing included |
|
|
42
|
+
| `load_beast2_manifest(path, T=None)` ❗ | The whole collection, segment by segment in east-c — the value its segments spliced into one blob decode to. `ValueError` when `path` holds no manifest, the types differ, or a segment is missing or malformed |
|
|
43
|
+
| `read_beast2_manifest(source)` | The manifest — `kind`, `level`, `type`, `rule`, `header`, `entries` (each `{hash, fence, count, bytes}`) — or `None`; a blob holding a value is read no further than its type section |
|
|
44
|
+
|
|
45
|
+
```python
|
|
46
|
+
from east.serialization.beast2 import open_beast2_file, write_beast2_file
|
|
47
|
+
|
|
48
|
+
write_beast2_file(path, rows_t, rows) # any size, one call
|
|
49
|
+
|
|
50
|
+
with open_beast2_file(path) as f: # self-describing: type from the
|
|
51
|
+
row = f[1_234_567] # header (f.wire_type); pass rows_t
|
|
52
|
+
totals = f.group_sum(lambda b, r: r["sku"], # instead to VALIDATE it at open
|
|
53
|
+
lambda b, r: r["qty"]) # whole-file compute: segment folds,
|
|
54
|
+
top = f.maximum(by=lambda b, r: r["qty"]) # never materialized, == load() exactly
|
|
55
|
+
table = f.load() # whole table when you truly need it
|
|
56
|
+
|
|
57
|
+
# The file IS a collection value: bind it into a function and the
|
|
58
|
+
# compiled body's keyed reads answer from the pager — one frame per lookup.
|
|
59
|
+
with open_beast2_file("table.beast2") as t: # Dict<String, Float>
|
|
60
|
+
lookup = East.function([StringType, DictType(StringType, FloatType)], FloatType,
|
|
61
|
+
lambda b, k, d: d.get_or_default(k, 0.0)).bind(t)
|
|
62
|
+
joined = rows.map(lambda b, r: r.v + lookup(r.k)) # loop + callee + pager: all east-c
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
**Buffer level — you hold bytes, not a path. `_for` is NOT `_with_header_for`:**
|
|
66
|
+
|
|
67
|
+
| Signature | Description |
|
|
68
|
+
|-----------|-------------|
|
|
69
|
+
| `encode_beast2_with_header_for(T, *, version=None)` / `decode_beast2_with_header_for(T)` | **The one you want**: the self-describing container — magic, type schema, value. Encode writes the current default container (v5; `version=4` only for a reader that predates it); decode reads v4 and v5, whole (Set/Dict segments must hold the canonical value — sorted, disjoint — or the blob is rejected as corrupt). ❗The header's type must be `T`, or a subtype whose variant cases line up with `T`'s (a `none` reads as any `Option`), else `ValueError: beast2: cannot decode a blob of type … as …` before anything decodes — the words TypeScript and east-c use |
|
|
70
|
+
| `encode_beast2_for(T)` / `decode_beast2_for(T)` | **Headerless** type-directed bytes: the reader must know `T` exactly, and mutable containers (Array/Set/Dict/Ref) are refused. In TypeScript `encodeBeast2For` is the FULL container — do not port a call site by name |
|
|
71
|
+
| `Beast2ElementWriter(T, stream, *, codec="deflate", parallel=False)` | The canonical writer (a context manager): `.add(el)` — an Array/Set element or a Dict `(key, value)` pair — and `.add_all(batch)` (the loop in east-c); Set elements and Dict keys strictly ascending in East order. Segments fall where the cut rule places them — the bytes the managed writer, TypeScript and east-c write; memory is one open segment; `.close()` writes the last segment, the terminator and the index; `.segments` counts the closed ones. An element that does not ascend, or fails to encode, raises and leaves the writer as it was |
|
|
72
|
+
| `encode_beast2_paged_for(T, *, codec="deflate") -> (value) -> bytes` | A whole collection through the canonical writer — the write side of `open_beast2_pages_for` |
|
|
73
|
+
| `Beast2RunSorter(T, open_run, *, merge=None, union=False, codec="deflate", parallel=False)` | A Set's or Dict's elements in ANY order in, sorted canonical runs out: `.add(el)` encodes at once; at `RUN_MAX_COUNT` elements or `RUN_MAX_BYTES` it sorts the run — stably, so a key's values keep the order they came in — folds a repeated key (`merge`, a compiled `(K, V, V) -> V` East function, for a Dict; `union=True` for a Set; neither: it raises) and writes the run to `open_run(run)` (anything with `write(bytes)` and `close()`) as its value's canonical blob. `.finish()` writes the last run; `.runs` counts them. The runs are the ones east-c and TypeScript write; a sink's exception comes back out of the call that wrote the run; a key repeated across runs is the merge's to fold |
|
|
74
|
+
| `Beast2Writer(T, stream, *, codec="deflate", self_contained=True, index=True, parallel=False)` | Segments of your choosing (a context manager): `.write(batch)` appends one segment per non-empty batch; `.close()` writes the terminator and the index; `.segments` counts. Set/Dict batches must ascend strictly across the stream. **Keep both defaults** — `index` and `self_contained` are what `open_beast2_pages_for` needs, and turning either off silently forfeits random access. `codec="none"` for already-compressed payloads or raw write throughput |
|
|
75
|
+
| `encode_beast2_segments_for(T, **opts) -> (batches) -> bytes` | In-memory `Beast2Writer`: one segment per non-empty batch |
|
|
76
|
+
| `encode_beast2_v5_for(T, *, codec="deflate", index=False) -> (value) -> bytes` | A whole value of any root type as v5; decode with `decode_beast2_with_header_for` |
|
|
77
|
+
| `iter_beast2_segments_for(T) -> (source) -> iterator` | One decoded collection per segment, O(segment) memory; `source` is bytes, an `mmap` or a binary stream |
|
|
78
|
+
| `read_beast2_index(T, blob) -> (segments, elements) \| None` | O(1) totals from a v5 blob's trailing index |
|
|
79
|
+
| `open_beast2_pages_for(T) -> (source) -> Beast2Pages` | Random access: `.segment_count` `.element_count` `.self_contained` `.counts`, `.segment(i)`, `.element(row)` (also `len()` / `[]`), each decoding ONE segment. ❗Needs `index=True` and `self_contained=True` (the defaults); `.element()` is for Array roots. ❗Borrows the buffer — keep it (and an mmap) alive for the pages' lifetime, or use `open_beast2_file`, which owns it |
|
|
80
|
+
|
|
81
|
+
**Batch size (`Beast2Writer` only — the canonical writers cut by the rule).** A
|
|
82
|
+
batch is at once your memory ceiling, one segment, one compression window and
|
|
83
|
+
the granularity of random access. ~1000 rows is a good default: one row per
|
|
84
|
+
batch costs about 4× the bytes, and the curve is flat past ~100. `write()` takes
|
|
85
|
+
a batch, never a row — accumulate and flush yourself, or hand rows to
|
|
86
|
+
`Beast2ElementWriter`, which needs no batch at all.
|
|
87
|
+
|
|
88
|
+
```python
|
|
89
|
+
import mmap
|
|
90
|
+
from east.serialization.beast2 import (
|
|
91
|
+
Beast2Writer, iter_beast2_segments_for, open_beast2_pages_for,
|
|
92
|
+
)
|
|
93
|
+
rows_t = ArrayType(row_type)
|
|
94
|
+
|
|
95
|
+
with open(path, "wb") as f, Beast2Writer(rows_t, f) as w:
|
|
96
|
+
for batch in produce_batches(): # each an EastArray(row_type, ...)
|
|
97
|
+
w.write(batch) # O(batch) memory, one segment each
|
|
98
|
+
|
|
99
|
+
# Sequential re-read. mmap, NOT .read() — .read() pulls the whole file in and
|
|
100
|
+
# throws away the bounded-memory property you just paid for.
|
|
101
|
+
with open(path, "rb") as f, mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
|
|
102
|
+
for batch in iter_beast2_segments_for(rows_t)(mm):
|
|
103
|
+
consume(batch) # O(segment) decoded at a time
|
|
104
|
+
|
|
105
|
+
# Or jump straight to a row — decodes only the segment that owns it.
|
|
106
|
+
pages = open_beast2_pages_for(rows_t)(mm)
|
|
107
|
+
row = pages.element(1_234_567)
|
|
108
|
+
```
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# The east-py CLI
|
|
2
|
+
|
|
3
|
+
Part of the `east-py` skill.
|
|
4
|
+
|
|
5
|
+
`east-py` (the `east-py-cli` package) runs East IR with python platform
|
|
6
|
+
functions — the runner e3 drives for `runtime: 'east-py'` — and carries the
|
|
7
|
+
authoring tools:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
east-py run prog.beast2 -p east-py-std -i a.beast2 -i b.json -o out.beast2 # -v: timings, peak memory
|
|
11
|
+
east-py run prog.beast2 -p east-py-std --profile # time per East and platform function
|
|
12
|
+
east-py exec unit.beast2 # one unit: the runner protocol e3 speaks
|
|
13
|
+
east-py convert value.beast2 # a beast2 value as East text (-o value.east to write it)
|
|
14
|
+
east-py transpile prog.beast2 -o prog.py # IR → a python module that rebuilds it
|
|
15
|
+
east-py export-functions pricing -o pricing.functions.beast2 -p east-py-std
|
|
16
|
+
east-py lint src/ ; east-py check src/ ; east-py lsp
|
|
17
|
+
east-py version -p east-py-std # the runner's version, and each package's
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
| Command | Does |
|
|
21
|
+
|---|---|
|
|
22
|
+
| `run <ir> [-p PACKAGE]… [-i FILE]… [-o FILE] [--decode lazy\|whole] [-v] [--profile]` | Compiles an IR file (`.beast2`, `.beast`, `.east` or `.json`) and calls it with one `-i` file per parameter, in order, each decoded by its extension to the parameter's type and FROZEN (a task input is immutable). An indexed beast2 collection input opens lazily, whatever it weighs — mapped, one segment decoded at a time, and decoded whole, once, when an operation the pager cannot serve first needs it — and a manifest file opens as the collection it names; `--decode whole` (east-c's and east-node's flag too) decodes every input before the program runs instead. `-o` writes the result in its extension's format (a collection as canonical indexed beast2); without it the result prints as East text. `-v` adds timings, peak memory and how each input was read: a lazy input's segment reads, or the resident memory a whole decode added |
|
|
23
|
+
| `exec <unit> [-v] [--profile]` | The runner protocol: one unit file says what to run — a program over its inputs, read as its `decode` says (as `run --decode`), or a merge of an output's parts — and where; the output is written by its kind and a result recorded. Exit 0 when it succeeded, 1 when it failed (the message and locations also go to stderr), 2 when the unit cannot be read or the result written. Relative paths in the unit are relative to its directory |
|
|
24
|
+
| `convert <file> [-o FILE.east] [-v]` | Decodes a beast2 value (its header names the type) and prints it as East text, or writes it to a `.east` file |
|
|
25
|
+
| `transpile <ir> [-o FILE] [--name NAME] [-p PACKAGE]…` | IR as a python module rebuilding it (`--name` binds it, default `main`; each `-p` prints its calls as that package's own functions) — [IR ↔ python](ir-and-diagnostics.md#ir--python-east-py-transpile-and-the-east-c-ir-toolbox) |
|
|
26
|
+
| `export-functions <module> -o FILE [-p PACKAGE]… [--name NAME] [--package-version V] [--only NAME]…` | A module's (a dotted name, or a `.py` path) `east_functions` as a function manifest; every platform call must be provided by a `-p` package; `--name` defaults to the module's top-level name, the version to the installed distribution's (else `0.0.0`); `--only` exports just those, building only them and what they call (a sibling that does not build fails nothing). `--batch FILE` in place of `-o`/`-p`/`--only` writes several manifests from one import, FILE a beast2 batch (`FunctionExportBatchType`: each entry's `output`, `packages` and `only`) — what `e3.export` runs per workspace package — [Cross-language functions](cross-language.md#cross-language-functions-east-py-export-functions-and-eastimport_function) |
|
|
27
|
+
| `lint [paths…] [--format text\|json] [--disable RULE]… [--exclude DIR]… [--list-rules]` | The rules (default path `.`); exit 1 on any finding, a warning included, 2 on a usage error — [Diagnostics](ir-and-diagnostics.md#diagnostics-at-edit-time--east-py-lint-east-py-check-flake8-east-py-lsp) |
|
|
28
|
+
| `check <targets…> [--format text\|json] [--only-if-enabled]` | The build's own errors; exit 1 on any |
|
|
29
|
+
| `lsp [--probe]` | The Language Server over stdio (needs `pygls`); `--probe` reports whether it can start here |
|
|
30
|
+
| `version [-p PACKAGE]…` | The `east-py-cli` and `east-py` versions (and the Cython extensions loaded), and each package's version and function count |
|
|
31
|
+
|
|
32
|
+
A `-p` package is a python module name (`east-py-std` imports `east_py_std`)
|
|
33
|
+
exporting a top-level `platform` list; a project's own module is loaded the same
|
|
34
|
+
way (see [Project-owned platform module](patterns.md#project-owned-platform-module-calling-your-python-from-e3)).
|
|
35
|
+
`run` and `exec` take `--exit-with-parent`: exit once stdin reaches end of file,
|
|
36
|
+
for a parent that holds a stdin pipe it never writes to.
|
|
37
|
+
|
|
38
|
+
**Profiling.** `run` and `exec` take `--profile`, east-c's profiler and report:
|
|
39
|
+
after the work (and after a failure's error), every East function and platform
|
|
40
|
+
function it called, by self time, with its calls, its total time and where it
|
|
41
|
+
is — the `file:line:column` it was written at and, for a helper called inside
|
|
42
|
+
another body, where it was called. A platform function (torch, a solver) is an
|
|
43
|
+
entry of its own, so its time is not its caller's, and so is each input read
|
|
44
|
+
lazily, named as `-v` names it (`input 1`): the segments its reads decode, a
|
|
45
|
+
call each — keyed reads at random beyond the segments the runner keeps, or a
|
|
46
|
+
scan repeated, show there, not in the reading function's self time. `EAST_PROFILE=1` profiles
|
|
47
|
+
without the flag — e3 hands its environment to the runner, so a task's report
|
|
48
|
+
lands in its log (`e3 task logs`) — and `EAST_PROFILE_INTERVAL=N` prints the
|
|
49
|
+
profile so far every N seconds while the work runs. A python function linked
|
|
50
|
+
into a TypeScript task carries its locations through the manifest, so the
|
|
51
|
+
report names its python lines.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# Columnar escape hatches, platform functions, and what is not east-c
|
|
2
|
+
|
|
3
|
+
Part of the `east-py` skill.
|
|
4
|
+
|
|
5
|
+
## Columnar escape hatches — when the logic must stay python
|
|
6
|
+
|
|
7
|
+
When per-element logic genuinely needs python (numpy, a model, an external
|
|
8
|
+
library), don't touch rows one at a time — cross the boundary **once per
|
|
9
|
+
column** instead of once per row × field:
|
|
10
|
+
|
|
11
|
+
```python
|
|
12
|
+
cols = rows.to_columns() # {"price": np.float64[...], "qty": np.int64[...],
|
|
13
|
+
# "sku": [str, ...] (interned), ...} — one crossing/column
|
|
14
|
+
amount = cols["price"] * cols["qty"].astype(np.float64) # vectorised numpy
|
|
15
|
+
out = EastArray.from_columns(Out, {"sku": cols["sku"], "amount": amount})
|
|
16
|
+
|
|
17
|
+
result = rows.map_batches(f, out=Out, batch_size=50_000) # f sees columnar chunks;
|
|
18
|
+
# batches may shrink (filter-like)
|
|
19
|
+
|
|
20
|
+
acc = EastDict(StringType, FloatType) # dicts as accumulators:
|
|
21
|
+
acc.update_many(keys, values, combine=lambda b, cur, new: cur + new) # one crossing,
|
|
22
|
+
# combine builds -> collisions in C
|
|
23
|
+
arr.extend(np_array) # bulk push, one crossing
|
|
24
|
+
|
|
25
|
+
solver_input = coerce_to( # struct fields take 1-D numpy
|
|
26
|
+
{"start_nodes": np_i64, "capacities": np_i64, ...}, # columns directly: each
|
|
27
|
+
MinCostFlowInputType) # Array<Int/Float/Bool> fills C-side
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
`Float`/`Integer`/`Boolean` columns move through numpy buffers filled in C
|
|
31
|
+
(`Option<Float>` ↔ float64 with NaN for `none`); `String` columns box once
|
|
32
|
+
through a bounded intern table (repeated categories/ids come back as the
|
|
33
|
+
same python object); other field types fall back to boxed lists. The same
|
|
34
|
+
contract governs the INPUT direction: `from_columns`/`coerce_to` fill C-side
|
|
35
|
+
only when a numeric column arrives as a numpy array of the matching dtype
|
|
36
|
+
(float64/int64/bool; float64-with-NaN for `Option<Float>`) — a plain python
|
|
37
|
+
list converts per cell, so `np.asarray(col)` first when the source is a
|
|
38
|
+
list. Composition rule: **East functions for East-expressible transforms,
|
|
39
|
+
columns/batches for the genuinely-python remainder.**
|
|
40
|
+
|
|
41
|
+
**Put the logic in the platform function, not a pure-python shim.** A
|
|
42
|
+
`@East.platform_function` is just a typed, validated python function; its one
|
|
43
|
+
added cost, validating the declared output, is a feature. A separate helper layer
|
|
44
|
+
over `list`/`dict` that the platform function converts into and out of costs
|
|
45
|
+
**testability** (the typed `inputs`/`output` is the contract you test; untyped
|
|
46
|
+
helpers turn bugs into silent corruption instead of a named `EastTypeError`),
|
|
47
|
+
**portability** (a platform function over East values moves to an e3 task,
|
|
48
|
+
another runtime or a TypeScript mirror unchanged), a **forced sandwich**
|
|
49
|
+
(East→python on the way in, back on the way out) and **purity** (a platform
|
|
50
|
+
function pure in its East inputs is memoisable). Small reusable functions are
|
|
51
|
+
fine — have them take and return East values, pay output validation only at the
|
|
52
|
+
real East↔python edge, and never call a platform function per element in a loop.
|
|
53
|
+
|
|
54
|
+
## Platform functions
|
|
55
|
+
|
|
56
|
+
| Signature | Description |
|
|
57
|
+
|-----------|-------------|
|
|
58
|
+
| `@East.platform_function(*, inputs, output, name=None, is_async=None, validate_output=True, validate_input=False)` | Register a Python fn; infers sync/async from the def (`is_async=` declares it when they differ — a function East calls asynchronously on every runtime whose python is a plain `def`); validates output against `output`. DUAL-MODE: called with values it runs the python (an async def's coroutine, to await); called inside an `East.function` / `East.asyncFunction` body it emits the `Platform` node with this signature — an async one inside a sync body is the build-time error. Paired at compile by name (the def's, or `name=`). Also importable bare: `from east import platform_function` |
|
|
59
|
+
| `@East.generic_platform_function(*, type_parameters, name=None, is_async=False, inputs=None, output=None, type_erased=False)` | Type-parameterized factory: the decorated fn is `fn(platform, *type_params) -> impl`; `is_async` is **explicit** (not inferred). With `inputs=`/`output=` (placeholders allowed anywhere: `output="T"`, `output=ArrayType("T")`) it is dual-mode too: `fs_open_beast(DictType(K, V), path)` in a body — the type arguments first, as TypeScript reads — emits the generic node; without them a body call is a build error naming what to declare. `type_erased=True` when the implementation ignores the type arguments (it reads the values): the decorated fn IS the implementation, python calls it directly (`causal_experiment(rows, config)`) and the factory the runtime binds is derived |
|
|
60
|
+
| a platform function RETURNING a lazy value | A `@platform_function` may return a paged hold — an `open_beast2_file(...)` value, `open_paged_file(T, path)` (a C-owned mapping), `fs_open_beast(...)`'s result — and the compiled body keeps its O(segment) cost model: the C value crosses the return seam by pointer (checked against the declared output type, never re-marshalled through python), and nothing python-side has to outlive it: the mapping is the value's own. The mirror of a paged ARGUMENT crossing in |
|
|
61
|
+
| `East.platform_functions(module) -> list` | Collects every decorated fn in `module` (pass `__name__`). Two consumers: `East.compile()` for in-process use, and a package's top-level `platform` list that `east-py run -p <module>` (and the e3 `{ custom }` runner) loads |
|
|
62
|
+
| `@memoize` / `@memoize(salt="…")` / `memoized = memoize(fn, salt="…")` | Content-addressed memo over ONE platform function. Apply **above** `@East.platform_function` (or inline on an imported one). Key = sha256(name + salts + per-input digests of the with-header BEAST2 encodings via the declared input types); value = with-header BEAST2 of the output, decoded via the declared output type. Inert by default |
|
|
63
|
+
| `configure_memo(directory, salt="")` | Activate (`None` deactivates) memoization for `@memoize` functions; overrides `EAST_MEMO_DIR` / `EAST_MEMO_SALT` env vars. Bump `salt` to invalidate after code edits — input-derived keys can't see them |
|
|
64
|
+
|
|
65
|
+
## What is NOT east-c
|
|
66
|
+
|
|
67
|
+
Nearly everything above delegates to native builtins. What still runs python,
|
|
68
|
+
so you can reason about cost and semantics:
|
|
69
|
+
|
|
70
|
+
- **`EastMatrix.map_elements`** — DEPRECATED: a python callback per element,
|
|
71
|
+
the one shape the strict surface removes; it warns and will go. Use the
|
|
72
|
+
arithmetic / reduction / mask / sparse methods, `map_rows`, or
|
|
73
|
+
`to_numpy()`/`to_torch()`. (`Vector.map`/`reduce` and `Matrix.map_rows` are the
|
|
74
|
+
native VectorMap / VectorFold / MatrixMapRows builtins.)
|
|
75
|
+
- **`Dict.get_or_insert`** composes membership and get python-side so `fn` runs
|
|
76
|
+
only on a miss (lazier than East's strict default). The other singles
|
|
77
|
+
(`insert`/`update`/`swap`/`delete`/`try_delete`/`insert_or_update`) are the
|
|
78
|
+
native builtins, error messages included.
|
|
79
|
+
- **Reduction sugar** — `mean`, `group_mean`, `group_size`, …: several native
|
|
80
|
+
passes, zero python per element, but not one fused builtin (as in TypeScript).
|
|
81
|
+
- **Boundary utilities** — `coerce_to`/`assert_value_of`/`type_of`, `variant`/
|
|
82
|
+
`struct` validation, `match()` dispatch, `compare_for`/`make_east_key`: python
|
|
83
|
+
walkers, by design — they are the python↔East edge.
|
|
84
|
+
- **Iteration** — `for x in arr` / `list(arr)` boxes per element (lazily, under
|
|
85
|
+
East's iteration lock, so mutating during the loop raises `Cannot modify …
|
|
86
|
+
during iteration`, as an East for-loop does); cross once with
|
|
87
|
+
`to_columns`/`to_numpy` instead.
|
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# Control flow, jq, and what a body may reference
|
|
2
|
+
|
|
3
|
+
The expression forms of control flow, `East.jq`, and the captures a body accepts or refuses. Part of the `east-py` skill.
|
|
4
|
+
|
|
5
|
+
## Contents
|
|
6
|
+
|
|
7
|
+
- Control flow — the expression forms (`East.while_`, `East.for_`, …)
|
|
8
|
+
- Queries — `East.jq`
|
|
9
|
+
- What a body may reference
|
|
10
|
+
|
|
11
|
+
## Control flow — the expression forms (`East.while_`, `East.for_`, …)
|
|
12
|
+
|
|
13
|
+
Prefer the collection methods: `map`/`filter`/`reduce`/`group_reduce`
|
|
14
|
+
express most work and are the fastest thing in the runtime. Reach here when
|
|
15
|
+
the next step **depends on the last** — a worklist, a BFS, a fixpoint, a
|
|
16
|
+
topological replay — which no data-parallel method can express. The
|
|
17
|
+
statement forms above (`b.while_`, `b.for_`) build the same nodes; these are
|
|
18
|
+
their EXPRESSION twins, usable inside a callback or a one-expression body.
|
|
19
|
+
|
|
20
|
+
Python cannot overload `=`, and `while`/`if` collapse to a `bool` before any
|
|
21
|
+
build sees them. That is the constraint that produced `East.if_else`, and the
|
|
22
|
+
answer is the same shape: **`while_` is to `while` what `if_else` is to `if`**.
|
|
23
|
+
The body is a pure function of the state that RETURNS the next state, and the
|
|
24
|
+
state struct IS the loop's local variables:
|
|
25
|
+
|
|
26
|
+
```python
|
|
27
|
+
total = East.while_({"i": 0, "acc": 0},
|
|
28
|
+
cond=lambda b, s: s.i < n,
|
|
29
|
+
body=lambda b, s: {"acc": s.acc + s.i, "i": s.i + 1}).acc
|
|
30
|
+
|
|
31
|
+
# `East.if_else` is the `if`; a field left as `s.field` is the empty else;
|
|
32
|
+
# `{**s, …}` changes one field and keeps the rest
|
|
33
|
+
East.for_(rows, {"n": 0, "hi": 0.0},
|
|
34
|
+
lambda b, s, r: {"n": s.n + 1,
|
|
35
|
+
"hi": East.if_else(r.price > s.hi, r.price, s.hi)})
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
This lowers to a `Ref` holding the state, a `While`/`For*` node whose body is
|
|
39
|
+
one `RefUpdate`, and a final read — the whole loop runs inside east-c. Every
|
|
40
|
+
construct is dual-mode like `East.if_else`: outside a build it runs the plain
|
|
41
|
+
python loop, so one body serves both a callback and a direct call on plain
|
|
42
|
+
East values. A worked example is in
|
|
43
|
+
[Key Patterns](patterns.md#sequential-logic-that-stays-in-east-c-worklist--replay).
|
|
44
|
+
|
|
45
|
+
| Call | Emits | Notes |
|
|
46
|
+
|---|---|---|
|
|
47
|
+
| `East.if_else(cond, value, …, otherwise)` | `IfElse` | cond/value pairs then the else — an if/elif/else chain is ONE node; exactly one arm evaluates |
|
|
48
|
+
| `East.while_(state, cond, body, label=…)` | `While` | `cond(b, s) -> Boolean`, `body(b, s) -> next state` |
|
|
49
|
+
| `East.for_(coll, state, body, label=…)` | `ForArray`/`ForSet`/`ForDict` | Array `body(b, s, el[, i])`, Set `body(b, s, el)`, Dict `body(b, s, k, v)` |
|
|
50
|
+
| `East.block(a, b, …)` | `Block` | evaluates in order, yields the last — the sequencing point for mutators (`East.block(lambda b: …)` is the statement form) |
|
|
51
|
+
| `East.let(value, fn)` | `Let` | `fn(b, bound)` — bind once, use many times, inside an expression body (`b.const` is the statement form) |
|
|
52
|
+
| `East.ref(v)` | `NewRef` | a cell — `.get()` / `.update(v)` / `.merge(v, fn(b, current, patch))` |
|
|
53
|
+
| `East.label(name=None)` | — | names a loop, for `break_`/`continue_` from a NESTED one |
|
|
54
|
+
| `East.break_(state=…, label=…)` / `East.continue_(state=…, label=…)` | `Break` / `Continue` | leave / next iteration, optionally committing a last state |
|
|
55
|
+
| `East.try_catch(body, handler, finally_=None)` | `TryCatch` | `body(b)`, `handler(b, message[, stack])`, same East type as `body` |
|
|
56
|
+
| `East.new_array/new_set/new_dict(…)` | `NewArray`/`NewSet`/`NewDict` | a FRESH collection per evaluation — the loop accumulator |
|
|
57
|
+
| `East.new_vector(T, values)` / `East.new_matrix(T, rows, cols, values)` | `NewVector`/`NewMatrix` | a fresh tensor from scalar expressions |
|
|
58
|
+
|
|
59
|
+
**Accumulate in place.** Threading a collection through the state rebuilds it
|
|
60
|
+
every iteration (`order.concat(…)` copies — O(n²) over the loop). The
|
|
61
|
+
mutators extend it in O(1). Each returns what its EAGER twin returns, so
|
|
62
|
+
sequence with `East.block`; a mutation written as a bare statement is
|
|
63
|
+
evaluated and thrown away, and the build says so rather than compiling a
|
|
64
|
+
loop that silently does nothing:
|
|
65
|
+
|
|
66
|
+
```python
|
|
67
|
+
counts = East.for_(rows, {"counts": East.new_dict(StringType, IntegerType)},
|
|
68
|
+
lambda b, s, r: East.block(
|
|
69
|
+
s.counts.insert_or_update(r.sku, 1, lambda b, x, y: x + y),
|
|
70
|
+
s)).counts
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
⚠️ Seed accumulators with `East.new_array/new_set/new_dict`. A captured
|
|
74
|
+
`EastArray(T)` works too — a captured collection is a snapshot of its
|
|
75
|
+
contents, rebuilt wherever the body reads it, so a seed is fresh per call —
|
|
76
|
+
but mutating one anywhere ELSE changes a copy nothing else sees, and the
|
|
77
|
+
build raises rather than compile a mutation that persists nowhere.
|
|
78
|
+
|
|
79
|
+
## Queries — `East.jq`
|
|
80
|
+
|
|
81
|
+
`East.jq(input, program, result_type)` is a jq query as East code: the jq is
|
|
82
|
+
parsed, checked against the input's East type and translated to ordinary East
|
|
83
|
+
IR when the program is built — loops, `Match`, the builtins — so it runs
|
|
84
|
+
wherever East runs, with no interpreter. In a body it is a call of the `Query`
|
|
85
|
+
builtin, which carries the program as written beside its translation and gives
|
|
86
|
+
the translation, so `to_python_source` prints it back as `East.jq(…)`. It is
|
|
87
|
+
the twin of TypeScript's `East.jq` and builds the same IR. Dual-mode, like the
|
|
88
|
+
standard library: in a body it builds the call; on values it compiles the
|
|
89
|
+
translation (once per query and input type) and runs it now.
|
|
90
|
+
|
|
91
|
+
```python
|
|
92
|
+
from east import East, ArrayType, FloatType, IntegerType, StructType, array
|
|
93
|
+
|
|
94
|
+
Order = StructType([("id", IntegerType), ("total", FloatType)])
|
|
95
|
+
orders = array(Order, [{"id": 1, "total": 250.0}, {"id": 2, "total": 1200.0}])
|
|
96
|
+
|
|
97
|
+
# in a body: an expression of `result_type` (here an ArrayExpression), so its methods chain
|
|
98
|
+
count_big = East.function([ArrayType(Order)], IntegerType,
|
|
99
|
+
lambda b, orders: East.jq(orders, "[.[] | select(.total > 1000) | .id]",
|
|
100
|
+
ArrayType(IntegerType)).size())
|
|
101
|
+
count_big(orders) # 1
|
|
102
|
+
|
|
103
|
+
East.jq(orders, "map(.total) | add", FloatType) # 1450.0 — on values, run now
|
|
104
|
+
|
|
105
|
+
# several inputs: a dict, checked as an e3 root — `.orders` reads the `orders` input alone
|
|
106
|
+
East.jq({"orders": orders, "limit": 1000.0},
|
|
107
|
+
".limit as $l | [.orders[] | select(.total > $l) | .id]", ArrayType(IntegerType)) # [2]
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
| | |
|
|
111
|
+
|---|---|
|
|
112
|
+
| ❗ `result_type` | Required: the outputs' type for a query that gives one output (`length`, `[…]`, `reduce`), an `Option` of it for at most one (`first(f)`), an `Array` of it for any number (`.[] \| .id`). The query must check to exactly it, or the build raises `type_mismatch: the query gives Array<Float>, not the Integer it was given.` |
|
|
113
|
+
| A query that does not check | `QueryError` when the program is built (a `QueryBuildError`, an `ExpressionError` too), its diagnostics each `jq <line>:<column>: <message>`: `unknown_field: .totl is not a field of Struct{id: Integer, total: Float}. Did you mean .total?` |
|
|
114
|
+
| An error the query raises | `error(v)`, an integer `% 0`, a date that does not parse: an East runtime error located in the jq text (`jq 1:5`), which `b.try_` catches; on values, a `QueryError` with one `runtime` diagnostic |
|
|
115
|
+
| The input's type | In a body, the expression's; on values, `type_of(value)`. Where that is narrower than the type meant (a variant's is its one case), run `evaluate_jq(program, value, input_type=T)` |
|
|
116
|
+
| The language | jq 1.8 over East's types: Dict keys of any type, DateTime a type (an ISO literal compares with it), Integers exact, a variant read as `{type, value}`. Every departure is listed in `libs/east/devdocs/QUERY.md` §13 |
|
|
117
|
+
| Host tools (`from east.query import …`) | `check_jq(program, input_type, root=False)`, without building: the program as written (`.program`, a `JqType` tree whose constants are typed East values; `None` when it does not check), the type it was checked against (`.input_type`), its `.element_type` and `.multiplicity` (`"one"` / `"maybe"` / `"many"`; `None` when it does not check), the datasets a root query reads (`.reads`) and its `.diagnostics`. A query's types are never stored: check the program against its data wherever they are needed · `evaluate_jq(program, value, input_type=T)`: a query whose input type is data · `translate_jq(checked)` · `parse_jq` / `print_jq` / `lex_jq` · `QueryError` |
|
|
118
|
+
|
|
119
|
+
## What a body may reference
|
|
120
|
+
|
|
121
|
+
- Its own parameters; plain scalar constants (closure floats / ints / strings /
|
|
122
|
+
datetimes bake in — the same value per element either way); East types and
|
|
123
|
+
values (`east_null` included); the `East` namespace, `East.if_else` included;
|
|
124
|
+
the `struct`/`variant`/`some`/`none` constructors (dual-mode: they build IR
|
|
125
|
+
when handed expression fields); **East.function artifacts** (dual-mode: a
|
|
126
|
+
call of their inline Function node at any nesting depth); **compiled East function values**
|
|
127
|
+
(`.bind` results, `compile_from_*` functions — a CALL on one lowers to a native
|
|
128
|
+
IR `Call`); and, two wrapper levels deep — enough for helper lambdas that
|
|
129
|
+
compose a callback — other python functions that pass the same rules.
|
|
130
|
+
- Anything else RAISES an `ExpressionError` **naming the binding**: a module
|
|
131
|
+
(`random.…`, `np.…`), a python builtin (`len`, `str`), a mutable python
|
|
132
|
+
capture, closure mutation (`nonlocal x; x += 1`). A closed-over East
|
|
133
|
+
*collection* raises in an eager callback too — a capture snapshots, `.bind`
|
|
134
|
+
stays live, and which you meant is your choice: an explicit `East.function`
|
|
135
|
+
snapshots a side table, `.bind(table)` keeps it live. For genuine python
|
|
136
|
+
semantics write an explicit python loop outside the body. Side effects are
|
|
137
|
+
therefore never lost or silently doubled: the body runs once, at build time, or
|
|
138
|
+
not at all.
|
|
139
|
+
- Arithmetic follows East types exactly: no implicit Integer↔Float mixing
|
|
140
|
+
(`.to_float()` / `.to_integer()` convert), and `/` is Float division.
|
|
141
|
+
- Captures are CACHED: an eager callback whose code object, captured bindings
|
|
142
|
+
and declared signature match an earlier call reuses its compiled function, so
|
|
143
|
+
the per-group aggregate `group_to_arrays(key).to_array(lambda b, es, k: {…})`
|
|
144
|
+
builds each inner lambda once, not once per group. ⚠️ A lambda whose CAPTURES
|
|
145
|
+
change per call (`lambda b, r: r.v > g` in a python loop over `g`) rebuilds
|
|
146
|
+
every call, since each capture bakes into a different function — hoist an
|
|
147
|
+
`East.function(...)` and pass the varying value as a bound parameter.
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Cross-language functions
|
|
2
|
+
|
|
3
|
+
`east-py export-functions` and `East.import_function`. Part of the `east-py` skill.
|
|
4
|
+
|
|
5
|
+
## Cross-language functions: `east-py export-functions` and `East.import_function`
|
|
6
|
+
|
|
7
|
+
A python `East.function` is *called* from a TypeScript e3 task — and a
|
|
8
|
+
TypeScript one from python — as pure IR: no python at run time, no platform
|
|
9
|
+
bridge. The functions travel as a **function manifest** (each function's IR,
|
|
10
|
+
declared type and platform dependencies with the package providing each);
|
|
11
|
+
the importer names the function and declares its type; linking checks the
|
|
12
|
+
type exactly and embeds the IR. Declare them on the package's root module:
|
|
13
|
+
|
|
14
|
+
```python
|
|
15
|
+
# packages/pricing/src/pricing/__init__.py — the package's root module
|
|
16
|
+
from east import East
|
|
17
|
+
from east.types.types import FloatType, StructType, IntegerType
|
|
18
|
+
|
|
19
|
+
Row = StructType([("qty", IntegerType), ("price", FloatType)])
|
|
20
|
+
score = East.function([Row], FloatType, lambda b, r: r.qty.to_float() * r.price)
|
|
21
|
+
east_functions = {"score": score} # name -> East.function artifact. CLOSED values: the
|
|
22
|
+
# exported IR must BIND every name it reads. A captured
|
|
23
|
+
# lookup table or a stdlib format string is a literal in
|
|
24
|
+
# the body and closes over nothing; a .bind result (no IR
|
|
25
|
+
# of its own) and an unresolved import are refused, and a
|
|
26
|
+
# genuinely free variable is refused BY NAME
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
```typescript
|
|
30
|
+
// the e3 task, in TypeScript (see the e3 skill) — the reference is all there is to write:
|
|
31
|
+
// e3.export finds `pricing` in the uv workspace, exports it (east-py export-functions, in the
|
|
32
|
+
// project's .venv) and links; the providers come from the task's runner
|
|
33
|
+
const score = East.importFunction("pricing", "score", FunctionType([RowType], FloatType));
|
|
34
|
+
await e3.export(pkg, "out.zip");
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
e3 exports each workspace package in one `east-py export-functions --batch`
|
|
38
|
+
process, however many tasks import from it, and keeps the manifests, so a
|
|
39
|
+
redeploy whose python has not changed runs no export (the **e3** skill). A
|
|
40
|
+
function's manifest entry is its own: the same bytes whatever else the process
|
|
41
|
+
built first and whatever directory it ran in, so a task embedding it keeps its
|
|
42
|
+
hash until the function, or a line it was written at, changes.
|
|
43
|
+
|
|
44
|
+
For a package built elsewhere (published, another repo), or to link
|
|
45
|
+
in-process, write the manifest where the package lives and pass it:
|
|
46
|
+
|
|
47
|
+
```bash
|
|
48
|
+
east-py export-functions pricing -o pricing.functions.beast2 -p east-py-std
|
|
49
|
+
# -p names the platform package implementing each platform call the functions make;
|
|
50
|
+
# a call no package provides fails the export (the manifest records the provider)
|
|
51
|
+
# → e3.export(pkg, "out.zip", { functions: ["./pricing.functions.beast2"] })
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
The other direction, in python:
|
|
55
|
+
|
|
56
|
+
```python
|
|
57
|
+
manifest = East.decode_function_manifest(Path("maths.functions.beast2").read_bytes()) # east-node export-functions wrote it
|
|
58
|
+
double = East.import_function("maths", "double", FunctionType([IntegerType], IntegerType))
|
|
59
|
+
user = East.function([IntegerType], IntegerType, lambda b, x: double(x) + 1) # callable in a body
|
|
60
|
+
ir, imports = East.link_imports(user, [manifest]) # exact type check; the IR embedded as a Let
|
|
61
|
+
compile_from_value(ir, [])(20) # 41 — pure IR, runs on any runner
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Unlinked, the reference is a `Platform` node named `east.importFunction`, and
|
|
65
|
+
compiling it raises naming that platform. `East.export_functions(pkg, version,
|
|
66
|
+
{…}, providers)` and `East.encode_function_manifest` are the API behind the CLI,
|
|
67
|
+
and `East.platform_dependencies(fn)` lists the platform functions an artifact
|
|
68
|
+
calls, in first-use order (unresolved imports skipped). Each also answers to its
|
|
69
|
+
TypeScript camelCase name (`East.exportFunctions`, `East.linkImports`, …). The
|
|
70
|
+
contract and the runner rules: `docs/conventions/EAST_CODEGEN.md`.
|