@elaraai/east-codex-plugin 1.0.98 → 1.0.100
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.build/daemon/lsp.js +40 -3
- package/.build/daemon/server.js +40 -3
- package/.build/mcp/server.js +3 -2
- package/.codex-plugin/plugin.json +1 -1
- package/index.json +13340 -482
- package/package.json +1 -1
- package/skills/e3/SKILL.md +108 -677
- package/skills/e3/reference/browser.md +93 -0
- package/skills/e3/reference/cli.md +120 -0
- package/skills/e3/reference/from-code.md +48 -0
- package/skills/e3/reference/rack.md +139 -0
- package/skills/e3/reference/record-indexes.md +326 -0
- package/skills/e3/reference/record-migrations.md +63 -0
- package/skills/e3/reference/repository.md +25 -0
- package/skills/e3-ui/SKILL.md +426 -259
- package/skills/east/SKILL.md +33 -395
- package/skills/east/reference/beast2.md +71 -0
- package/skills/east/reference/cross-language-and-json-schema.md +87 -0
- package/skills/east/reference/queries.md +46 -0
- package/skills/east/reference/runners-and-ir.md +148 -0
- package/skills/east/reference/trees-and-recursion.md +56 -0
- package/skills/east/reference/values-in-typescript.md +62 -0
- package/skills/east-contribute/SKILL.md +0 -1
- package/skills/east-py/SKILL.md +36 -1613
- package/skills/east-py/reference/beast2.md +108 -0
- package/skills/east-py/reference/cli.md +51 -0
- package/skills/east-py/reference/columns-and-platform-functions.md +87 -0
- package/skills/east-py/reference/control-flow-and-queries.md +147 -0
- package/skills/east-py/reference/cross-language.md +70 -0
- package/skills/east-py/reference/ir-and-diagnostics.md +150 -0
- package/skills/east-py/reference/patterns.md +369 -0
- package/skills/east-py/reference/performance.md +70 -0
- package/skills/east-py/reference/value-methods.md +306 -0
- package/skills/east-py/reference/values-construction-and-json.md +88 -0
- package/skills/east-py/reference/writing-east-functions.md +275 -0
- package/skills/east-py-datascience/SKILL.md +11 -28
- package/skills/east-ui/SKILL.md +5 -5
|
@@ -0,0 +1,150 @@
|
|
|
1
|
+
# IR and diagnostics
|
|
2
|
+
|
|
3
|
+
Printing IR as python (`east-py transpile`), the `east-c ir` toolbox, and the edit-time checks (`east-py lint`, `check`, flake8, `lsp`). Part of the `east-py` skill.
|
|
4
|
+
|
|
5
|
+
## Contents
|
|
6
|
+
|
|
7
|
+
- IR ↔ python: `east-py transpile` and the `east-c ir` toolbox
|
|
8
|
+
- Diagnostics at edit time — `east-py lint`, `east-py check`, flake8, `east-py lsp`
|
|
9
|
+
|
|
10
|
+
## IR ↔ python: `east-py transpile` and the `east-c ir` toolbox
|
|
11
|
+
|
|
12
|
+
Any East IR — an `East.function` artifact, or a `.json` / `.beast2` export from
|
|
13
|
+
TypeScript, east-c or east-node — prints as an idiomatic python module that
|
|
14
|
+
REBUILDS it through the surface above:
|
|
15
|
+
|
|
16
|
+
```python
|
|
17
|
+
from east.codegen import to_python_source
|
|
18
|
+
print(to_python_source(classify)) # `@East.function(...)` / `def main(b, …)`
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
east-py transpile program.json -o program.py --name main # the same, from a file
|
|
23
|
+
east-py transpile program.json -p east-py-std -p east-py-io # platform calls print as the packages' own
|
|
24
|
+
# functions and named types, not restated declarations
|
|
25
|
+
east-c ir normalize program.json -o canonical.json # the canonical form
|
|
26
|
+
east-c ir diff a.json b.beast2 # first difference, or "identical"
|
|
27
|
+
east-c ir convert program.json -o program.beast2 # json <-> beast2, source map intact
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
- **The contract:** `build(print(IR))` equals `IR` under `east-c ir normalize`
|
|
31
|
+
(location ids stripped, variables and labels renamed in the TypeScript
|
|
32
|
+
lowering's order, captures recomputed, recursive type ids renumbered — one
|
|
33
|
+
normalizer, in libeast-c). The TypeScript printer (`East.toSource`, `east-node
|
|
34
|
+
transpile`) is its twin: IR → python → IR → TypeScript → IR is equal at every
|
|
35
|
+
leg, over the whole compliance corpus and every example
|
|
36
|
+
(`docs/conventions/EAST_CODEGEN.md`).
|
|
37
|
+
- **Builtins print through one spelling table** (`east.codegen.spellings`):
|
|
38
|
+
operators only where they are exact — `+ - *` on numbers, `/` on Floats, the
|
|
39
|
+
comparisons, `& | ^ ~` on Booleans, `+` on Strings — named `East.<Type>.*` or
|
|
40
|
+
method spellings elsewhere, and `East.builtin(...)` for `RAW_ONLY`.
|
|
41
|
+
- **A function used as a VALUE** (`$.let(East.DateTime.roundDownWeek)`) prints
|
|
42
|
+
as the inline `@East.function` it is — the IR carries the body, not the name.
|
|
43
|
+
- **A platform call** prints as a hoisted `East.platform(...)` declaration — or,
|
|
44
|
+
given the implementing packages
|
|
45
|
+
(`to_python_source(ir, providers=providers_for(["east_py_std"]))`, `east-py
|
|
46
|
+
transpile -p east-py-std`), as `from east_py_std import fs_read_file` and the
|
|
47
|
+
call itself, with a type the package names (`GzipOptionsType`) printed by its
|
|
48
|
+
name; the plugin's example index is rendered this way. A provider is used only
|
|
49
|
+
where its declared signature IS the node's, so a drifted signature keeps the
|
|
50
|
+
declaration rather than printing a wrong import. A package whose function is
|
|
51
|
+
implemented in C exports the DECLARATION under the same name
|
|
52
|
+
(`simulation_run`, `optimization_iterative`), which a body calls identically.
|
|
53
|
+
|
|
54
|
+
## Diagnostics at edit time — `east-py lint`, `east-py check`, flake8, `east-py lsp`
|
|
55
|
+
|
|
56
|
+
Everything the build refuses — a body without the block, `//` on an
|
|
57
|
+
expression, an f-string over one, `if` on one, a callback reaching for `np` — is
|
|
58
|
+
also a **rule** in `east.diagnostics` (the python twin of
|
|
59
|
+
`@elaraai/east-diagnostics`). The rules read a file's `ast`, find its East bodies
|
|
60
|
+
(an `East.function` body and everything nested in it, an eager callback on an
|
|
61
|
+
East value, a `@East.platform_function`'s East inputs), and say at edit time what
|
|
62
|
+
the build would say: every rule's message IS the build's refusal for the same
|
|
63
|
+
code, pinned by building the very source the rules read.
|
|
64
|
+
|
|
65
|
+
Two tiers, at two costs. The **rules** read the `ast` — instant, and useful even
|
|
66
|
+
on a file that does not parse. The **check** runs the build — the only type
|
|
67
|
+
checker a python East body has, since East's type checker IS the builder — at
|
|
68
|
+
the cost of importing the module.
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
east-py lint src/ # file:line:col: category [rule] message — exit 1 on any finding
|
|
72
|
+
east-py lint src/ --format json # the findings as records
|
|
73
|
+
east-py lint src/ --disable no-deprecated-alias --exclude fixtures
|
|
74
|
+
east-py lint --list-rules # every rule, with its EAS code
|
|
75
|
+
east-py check src/ # the BUILD's errors — a file, a directory, or a module; --format json
|
|
76
|
+
flake8 --select EAS src/ # the same rules inside flake8 (east-py-cli registers the plugin)
|
|
77
|
+
east-py lsp # a Language Server over stdio, both tiers (--probe: can it start here?)
|
|
78
|
+
# pylsp # python-lsp-server runs the rules too (east-py-cli registers the plugin)
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
- **`east-py check`** imports each target — a `.py` file, a directory (walked
|
|
82
|
+
as `lint` walks it) or a dotted module name, each executed afresh — builds
|
|
83
|
+
every East function in it, and reports EVERY failure at its authoring line: a
|
|
84
|
+
body whose type differs from the declared `out`, a callback the capture
|
|
85
|
+
refuses, an `IRAnalysisError`. Importing runs the module, so `EAST_CHECK=1` is
|
|
86
|
+
set for the duration — skip import-time work when you see it.
|
|
87
|
+
`--only-if-enabled` honours the project's opt-in (below), as the plugin's read
|
|
88
|
+
hook does, so the hook and the language server agree.
|
|
89
|
+
- **`east-py lsp`** (needs `pygls`) runs the rules on every change and the build
|
|
90
|
+
tier on **open** (debounced) and **save** (at once), on one long-lived worker
|
|
91
|
+
thread, and only when the project opts in: the build reads the module from
|
|
92
|
+
DISK, so on an unsaved buffer it would report the saved version's errors. A
|
|
93
|
+
save evicts the module from the warm process, so its importers are checked
|
|
94
|
+
against the new version. Pyright/Pylance has no plugin API, so flake8 and
|
|
95
|
+
pylsp are the editor paths that need no East-specific server; the Claude Code
|
|
96
|
+
plugin launches the project's own `east-py lsp` for `.py` files, and falls back
|
|
97
|
+
to `east-py lint` per change when that server cannot start.
|
|
98
|
+
|
|
99
|
+
The rules — `east-py lint --list-rules` prints them:
|
|
100
|
+
| Rule | Flags | What the build says |
|
|
101
|
+
|---|---|---|
|
|
102
|
+
| `body-takes-block-first` (EAS001) | `lambda x: …`, a body whose parameter count is not the declared count plus the block, `b.price`, `x + b` | `a body takes the block first` |
|
|
103
|
+
| `no-operator-fork` (EAS002) | `//` `%` `**` on an expression, `xs[-1]` | the operator-fork texts — `East.Integer.divide` / `remainder` / `pow`, `xs.get(xs.size() - 1)` |
|
|
104
|
+
| `no-python-formatting` (EAS003) | `f"{x}"`, `str(x)`, `print(x)`, `format(x)`, `"{}".format(x)`, `"%d" % x` | `f-strings / str() cannot be traced … East.String.print(T, value)` |
|
|
105
|
+
| `no-python-boolean` (EAS004) | `if`/`while`/`assert`/`and`/`or`/`not`/`x if c else y` on an expression, `in`, `for … in xs`, `len`/`int`/`float`/`bool`/`sum`/`sorted`/`max`… over one | `python \`if/and/or/not\` cannot be traced …` (`iteration`, `len()`, …) |
|
|
106
|
+
| `no-python-round` (EAS005) | `round(x)` on a Float expression | `East.Float.round_half(x) rounds half away from zero …` |
|
|
107
|
+
| `no-python-work` (EAS006) | an EAGER callback loading a module (`np`, `math`), a python builtin the capture does not admit (all but `abs`/`bool`/`isinstance`), a name imported from the standard library or an installed package that has no East form (`from math import floor`; a constant like `pi` lifts), a mutable East collection, or a `def` doing any of those (a clean macro `def` is fine) | `the callback cannot be captured automatically: it references np …` |
|
|
108
|
+
| `no-statement-on-outer-block` (EAS007) | `b.if_(p, lambda _b: b.assign(…))` — a statement on an enclosing body's block | `b.assign() was called on an OUTER block …` |
|
|
109
|
+
| `no-deprecated-alias` (EAS008, warning) | `.fold`, `.lower`, `East.Boolean.and_`, … (read off the surface's own deprecation docstrings) | `.fold() is deprecated: the spelling is .reduce() (the TypeScript name)` |
|
|
110
|
+
| `no-discarded-expression` (EAS009) | a bare `acc.push_last(x)` / `East.error(…)` / `xs.size()` line in a body | `.push_last() was evaluated and thrown away … b.do(…)` |
|
|
111
|
+
| `prefer-some-none` (EAS010, warning) | `variant("some", x)` / `variant("none", None)` | use `some(value)` / `none` |
|
|
112
|
+
| `no-handrolled-variant` (EAS011, warning) | a `{"type": …, "value": …}` dict | `variant("Tag", value, Type)` — the encoder needs what it constructs |
|
|
113
|
+
| `prefer-explicit-east-type` (EAS012) | `b.let([])` / `b.let([1, 2])` / `b.const(list())` — a bare python list, set or tuple, empty or not | `cannot lift python value of type list …` — pass the East type: `b.let([], ArrayType(IntegerType))`, or build it with `East.new_array` |
|
|
114
|
+
| `no-let-const-in-expression` (EAS013, warning) | `b.let` / `b.const` buried in a call argument, an element, a chain target (a tuple of declarations is fine) | give the declaration its own statement |
|
|
115
|
+
| `no-untracked-east-data` (EAS014, suggestion) | a plain python list/dict local reaching an expression's method | `b.const(rows, Type)` — a local carries no East type and re-inlines |
|
|
116
|
+
| `no-redundant-east-cast` (EAS015, warning) | `b.let(East.value(x, T), T)` | pass the value and type to `b.let` directly |
|
|
117
|
+
| `prefer-let-const-over-east-value` (EAS016, suggestion) | `East.value(...)` assigned or returned inside a body | `b.const(value, Type)` — `East.value` erases the type at the binding |
|
|
118
|
+
| `no-host-comparison-on-east-values` (EAS017) | `<` / `>` on a decoded VARIANT or OPTION outside a body — it raises `TypeError` (`==` is structural and fine; a decoded Integer is an `int`) | `compare_for(T)` / `less_for(T)`, `make_east_key(T)` for `sorted` |
|
|
119
|
+
| `no-module-scope-east-macro` (EAS018, warning) | a module-scope helper a body calls to build IR, or to assemble a composite `f"{a}|{b}"` key | make it an `East.function`; model typed / nested East data |
|
|
120
|
+
| `no-build-time-clock` (EAS019, warning) | `datetime.now()` / `time.time()` at module scope (a script's `if __name__ == "__main__":` is not module scope) | author the constant, or read the clock at runtime — a platform function, or `east_py_std`'s `time_now()` in a body |
|
|
121
|
+
| `no-inline-credentials` (EAS020, warning) | a literal `password` / `token` / `secret_access_key` (a localhost sibling host, a header NAME like `X-Api-Key`, and a `${TEMPLATE}` are exempt) | read it at runtime: `east_py_std`'s `env_get("VAR")` in a body, `os.environ` in a platform function — IR is content-addressed and replicated |
|
|
122
|
+
| `no-compile-time-data-injection` (EAS021, warning) | `open()` / `json.load` / `Path.read_text` / `os.environ` at module import, when the result REACHES East (a body reads it, or a boundary call takes it) | load at runtime: an e3 input, a dataset, a platform function |
|
|
123
|
+
| `no-python-east-data` (EAS022, warning) | East rows assembled by a module-scope comprehension or loop, then handed to a body | write the rows out, or produce them at runtime |
|
|
124
|
+
| `no-python-string-building` (EAS023, warning) | an f-string assembling an East string CONSTANT — a regex, a template, a key | spell the constant out |
|
|
125
|
+
| `no-derived-struct-fields` (EAS024, warning) | `Derived = StructType([… for f in Other.value])` | a type declaration is a wire format — spell the fields |
|
|
126
|
+
| `no-python-data-work` (EAS025, warning) | a python helper doing parse / strip / null-check / coerce work on an EXPRESSION a body hands it | express it in East, where it runs on every row |
|
|
127
|
+
| `no-reinlined-east-binding` (EAS026, warning) | a python local holding an EXPRESSION (`n = xs.size()`, not a `b.let`/`b.const` binding) read more than once in one body, or read inside a per-element callback, a loop body or a nested function below the body that bound it | each read re-emits — and re-evaluates — the expression: bind it once, `n = b.const(xs.size())`, where it is needed (the TypeScript rule of the same name) |
|
|
128
|
+
|
|
129
|
+
Configure it once in the project's `pyproject.toml`; every surface reads it, and
|
|
130
|
+
`east-py lint`'s own flags add to it:
|
|
131
|
+
|
|
132
|
+
```toml
|
|
133
|
+
[tool.east-py]
|
|
134
|
+
check = true # let an EDITOR run the build tier (off by default)
|
|
135
|
+
disable = ["no-deprecated-alias"] # rules to skip
|
|
136
|
+
exclude = ["fixtures", "vendor"] # extra directory names not to walk
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
`check` is off unless a project asks: the rules READ a file, the build RUNS it,
|
|
140
|
+
and an editor should not import someone's modules on save because a language
|
|
141
|
+
server happened to be installed. `east-py check` on the command line is consent
|
|
142
|
+
in itself and ignores the setting. Unreadable configuration falls back to the
|
|
143
|
+
defaults — a diagnostics tool must never stop a project building.
|
|
144
|
+
|
|
145
|
+
A file that does not import `east` is never diagnosed; a line ending in `# noqa`
|
|
146
|
+
(or `# noqa: EAS002` / `# noqa: no-operator-fork`) is skipped; `.venv`, `venv`,
|
|
147
|
+
`node_modules`, `dist`, `build`, `.git`, `__pycache__`, `tests` and `test` are not
|
|
148
|
+
walked. The rules are syntactic, so a clean `lint` is necessary, not sufficient —
|
|
149
|
+
the type errors live behind `east-py check`. The rules are disjoint (each
|
|
150
|
+
mistake is one rule's to report).
|
|
@@ -0,0 +1,369 @@
|
|
|
1
|
+
# Key patterns
|
|
2
|
+
|
|
3
|
+
Part of the `east-py` skill.
|
|
4
|
+
|
|
5
|
+
## Contents
|
|
6
|
+
|
|
7
|
+
- Python values vs East expressions
|
|
8
|
+
- The canonical platform function
|
|
9
|
+
- Project-owned platform module (calling your Python from e3)
|
|
10
|
+
- Memoize expensive pure stages (dev/test harnesses)
|
|
11
|
+
- Fork-parallel export of a huge table (one file out)
|
|
12
|
+
- Keyed lookups against a file too big to load
|
|
13
|
+
- Sequential logic that stays in east-c (worklist / replay)
|
|
14
|
+
- Trees from flat rows, and recursion
|
|
15
|
+
- Sort uses East's total order
|
|
16
|
+
- Scalars: use the `East.<Type>` utilities for consistency — above all String & DateTime
|
|
17
|
+
- torch interop (in a torch-having package, inline — no helpers)
|
|
18
|
+
|
|
19
|
+
|
|
20
|
+
## Python values vs East expressions
|
|
21
|
+
|
|
22
|
+
The `east` skill's first rule, in python: a body's PARAMETERS are expressions;
|
|
23
|
+
a python value used inside a body must lift. Scalars lift on their own; a
|
|
24
|
+
collection must be an explicit choice.
|
|
25
|
+
|
|
26
|
+
```python
|
|
27
|
+
RATES = EastDict(StringType, FloatType, {"AUD": 0.65}) # python-side East value
|
|
28
|
+
CUTOFF = 100
|
|
29
|
+
|
|
30
|
+
@East.function([Row], FloatType)
|
|
31
|
+
def usd(b, r):
|
|
32
|
+
return East.if_else(r.amount > CUTOFF, # a scalar constant lifts (TS East.value)
|
|
33
|
+
r.amount * RATES.get_or_default(r.ccy, 1.0), # an EXPLICIT build snapshots the dict: a literal, rebuilt per call
|
|
34
|
+
0.0)
|
|
35
|
+
|
|
36
|
+
rows.map(lambda b, r: r.amount * RATES.get_or_default(r.ccy, 1.0)) # ❌ an eager callback REFUSES
|
|
37
|
+
# the collection capture:
|
|
38
|
+
conv = East.function([Row, DictType(StringType, FloatType)], FloatType,
|
|
39
|
+
lambda b, r, t: r.amount * t.get_or_default(r.ccy, 1.0))
|
|
40
|
+
rows.map(conv.bind(RATES)) # ✅ bind it: live, zero-copy
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## The canonical platform function
|
|
44
|
+
|
|
45
|
+
```python
|
|
46
|
+
from east import East, FloatType, StringType, StructType, ArrayType, struct
|
|
47
|
+
|
|
48
|
+
LineItem = StructType([("name", StringType), ("price", FloatType)]) # Struct<String, Float>
|
|
49
|
+
|
|
50
|
+
@East.platform_function(inputs=[FloatType, ArrayType(LineItem)], output=ArrayType(LineItem))
|
|
51
|
+
def convert_prices(fx_rate, items):
|
|
52
|
+
# items: an east-c-backed array with eager methods; row["price"] is a plain float
|
|
53
|
+
return items.map(lambda b, row: struct(
|
|
54
|
+
{"name": row["name"], "price": row["price"] * fx_rate}, # plain f64 * f64
|
|
55
|
+
LineItem, # tag + validate the result
|
|
56
|
+
))
|
|
57
|
+
```
|
|
58
|
+
`.map` builds the lambda into an East function — the loop and the row construction run
|
|
59
|
+
in east-c, with `fx_rate` baked in as a constant; `struct(..., LineItem)` is dual-mode, so
|
|
60
|
+
the same spelling builds the row here and on plain values; the decorator validates the
|
|
61
|
+
`Array<LineItem>` result — a named `EastTypeError` instead of silent corruption.
|
|
62
|
+
|
|
63
|
+
## Project-owned platform module (calling your Python from e3)
|
|
64
|
+
|
|
65
|
+
To call a Python platform function from an **e3 task**, package it so `east-py
|
|
66
|
+
run -p <module>` can load it: each module ends with `<name>_impl =
|
|
67
|
+
East.platform_functions(__name__)`, and the package `__init__.py` aggregates them into
|
|
68
|
+
a top-level `platform` list (the same shape as east-py-std / east-py-datascience).
|
|
69
|
+
|
|
70
|
+
```python
|
|
71
|
+
# platform_module/forecast.py
|
|
72
|
+
@East.platform_function(inputs=[ArrayType(FloatType)], output=FloatType,
|
|
73
|
+
name="my_project.forecast") # dotted "<project>.<fn>"; MUST byte-match
|
|
74
|
+
def forecast(history): # the TS East.platform(...) declaration
|
|
75
|
+
return sum(history) / len(history) if history else 0.0
|
|
76
|
+
forecast_impl = East.platform_functions(__name__)
|
|
77
|
+
|
|
78
|
+
# platform_module/__init__.py
|
|
79
|
+
from .forecast import forecast_impl
|
|
80
|
+
platform = [*forecast_impl] # what `east-py run -p platform_module` loads
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
The e3 task wires it with `{ runtime: "east-py", platforms: [{ custom:
|
|
84
|
+
"platform_module" }, "east-py-std"] }`. East code needs a TS `East.platform(
|
|
85
|
+
"my_project.forecast", [...], ...)` **declaration** with the identical name (no
|
|
86
|
+
codegen). Add native deps (numpy, …) to `pyproject.toml` and run `uv sync`. See
|
|
87
|
+
**east-project** for the full scaffold (`--platform`) and the setuptools
|
|
88
|
+
packaging; **e3** for the runner.
|
|
89
|
+
|
|
90
|
+
## Memoize expensive pure stages (dev/test harnesses)
|
|
91
|
+
|
|
92
|
+
```python
|
|
93
|
+
from east import East, configure_memo, memoize
|
|
94
|
+
|
|
95
|
+
@memoize # eligibility — the author asserts purity
|
|
96
|
+
@East.platform_function(inputs=[ArrayType(RowType), ConfigType], output=ModelType)
|
|
97
|
+
def train_model(rows, config): ...
|
|
98
|
+
|
|
99
|
+
# A test harness flips the whole package on with one call (or EAST_MEMO_DIR):
|
|
100
|
+
configure_memo("/tmp/my_project", salt="v1") # None deactivates
|
|
101
|
+
loaded = memoize(other_pkg.load_csv, salt=file_digest) # inline, per-call salt
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
The platform-function boundary is the cache boundary, e3-style: hit = decode the
|
|
105
|
+
stored BEAST2 blob via the declared output type (skipping the body entirely),
|
|
106
|
+
miss = compute + atomic save. Only mark functions **pure in their East inputs** —
|
|
107
|
+
a hit on a function that writes files/rows/requests silently drops those effects.
|
|
108
|
+
Stochastic fits memoize their first realization. Code edits don't change keys —
|
|
109
|
+
bump the salt. Inert with no directory configured: under e3, the dataflow's
|
|
110
|
+
content-addressed task cache is the real memo and this stays off.
|
|
111
|
+
|
|
112
|
+
## Fork-parallel export of a huge table (one file out)
|
|
113
|
+
|
|
114
|
+
The shape for "N CPUs on one table": build the expensive shared context once,
|
|
115
|
+
let forked workers inherit it copy-on-write, and ship a single indexed file —
|
|
116
|
+
consumers never learn the export was parallel. (Windows runs the same contract
|
|
117
|
+
inline, sequentially; the output is byte-identical either way.)
|
|
118
|
+
|
|
119
|
+
```python
|
|
120
|
+
from east.serialization.beast2 import open_beast2_file, write_beast2_file_parallel
|
|
121
|
+
|
|
122
|
+
ROW = StructType([("order_id", StringType), ("qty", IntegerType), ("total", FloatType)])
|
|
123
|
+
|
|
124
|
+
lookups = load_reference_tables() # multi-GB keyed dicts: built ONCE, pre-fork,
|
|
125
|
+
# inherited COW — never pickled, never rebuilt
|
|
126
|
+
|
|
127
|
+
def produce(span): # runs in the worker process
|
|
128
|
+
start, count = span # yield batches of any size — the managed
|
|
129
|
+
for chunk in chunk_ranges(start, count, 8192): # writer cuts the canonical segments
|
|
130
|
+
yield compute_rows(lookups, chunk) # eager methods + East functions: native
|
|
131
|
+
|
|
132
|
+
write_beast2_file_parallel(
|
|
133
|
+
"orders.beast2", ArrayType(ROW),
|
|
134
|
+
partitions=row_ranges(total_records, shards=13), # partition order = row order
|
|
135
|
+
produce=produce,
|
|
136
|
+
processes=13, # a worker failure kills the rest, cleans up, re-raises
|
|
137
|
+
)
|
|
138
|
+
|
|
139
|
+
# Consumers — e.g. the table's @East.platform_function loader — see ONE file, no
|
|
140
|
+
# catalog, no shard names, and read it at one segment of memory:
|
|
141
|
+
@East.platform_function(inputs=[StringType], output=ArrayType(ROW))
|
|
142
|
+
def load_orders(path):
|
|
143
|
+
with open_beast2_file(path, ArrayType(ROW)) as f:
|
|
144
|
+
return f.load() # or compute on f directly, never holding the table
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Shards produced by your own process topology (or on another machine)? Merge
|
|
148
|
+
them directly — a pure byte copy, nothing re-encoded:
|
|
149
|
+
|
|
150
|
+
```python
|
|
151
|
+
splice_beast2_files("orders.beast2", ArrayType(ROW), sorted(shard_paths), verify=True)
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
## Keyed lookups against a file too big to load
|
|
155
|
+
|
|
156
|
+
A reference table exported as a Dict file answers point reads without ever
|
|
157
|
+
being held in memory: east-c fence-searches the segment index and decodes
|
|
158
|
+
one segment per lookup, so a sparse join against a multi-GB file stays
|
|
159
|
+
cheap. (A *dense* pass over most keys is still better as whole-file compute
|
|
160
|
+
— `f.group_sum(...)`, `f.filter(...)`, any read method directly on the file.)
|
|
161
|
+
|
|
162
|
+
```python
|
|
163
|
+
from east.serialization.beast2 import open_beast2_file
|
|
164
|
+
|
|
165
|
+
with open_beast2_file("orders.beast2") as orders: # Dict<String, Order>; Order = Struct{id, total}
|
|
166
|
+
order = orders[order_id] # fence search → ONE segment decode
|
|
167
|
+
hot = orders.get_keys(wanted_ids, # each owning segment decodes once;
|
|
168
|
+
lambda b, k: {"id": k, "total": 0.0}) # `fill` is a body (a python dict
|
|
169
|
+
if candidate_id in orders: # captured from outside is refused)
|
|
170
|
+
...
|
|
171
|
+
|
|
172
|
+
# A sorted Array file answers range queries the same way — global insertion
|
|
173
|
+
# indices, then slice exactly the covered rows:
|
|
174
|
+
with open_beast2_file("events.beast2") as events: # Array<Event> sorted by timestamp
|
|
175
|
+
span = events.find_sorted_range(cutoff_event)
|
|
176
|
+
window = events.slice(span["start"], span["end"])
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
## Sequential logic that stays in east-c (worklist / replay)
|
|
180
|
+
|
|
181
|
+
When the next step depends on the last — a worklist, a BFS, a fixpoint, a
|
|
182
|
+
topological replay — no `map`/`filter`/`group_reduce` expresses it, and a
|
|
183
|
+
python loop over decoded rows is the whole job's cost. The statement surface
|
|
184
|
+
threads the state through the loop natively, so the whole thing is ONE
|
|
185
|
+
compiled function. Two rules carry most of the weight. **Accumulate in
|
|
186
|
+
place** — a collection threaded through a state is rebuilt every iteration,
|
|
187
|
+
so `order.concat(...)` is O(n²) over the loop where `order.push_last(...)` is
|
|
188
|
+
O(1). And **a mutation is a statement** — `b.do(acc.push_last(x))` in a
|
|
189
|
+
statement body, `East.block(acc.push_last(x), result)` in an expression body;
|
|
190
|
+
a bare `acc.push_last(x)` line is evaluated at build time and thrown away
|
|
191
|
+
(the build raises rather than compile a loop that silently does nothing).
|
|
192
|
+
|
|
193
|
+
```python
|
|
194
|
+
from east import ArrayType, DictType, East, IntegerType, StringType
|
|
195
|
+
|
|
196
|
+
Node, Edges = StringType, DictType(StringType, ArrayType(StringType))
|
|
197
|
+
Indeg = DictType(StringType, IntegerType)
|
|
198
|
+
|
|
199
|
+
# Kahn's algorithm — ONE compiled function, the TypeScript `$` shape: a
|
|
200
|
+
# worklist the loop appends to, a cursor into it, and per-node successors
|
|
201
|
+
# decremented in place.
|
|
202
|
+
@East.function([ArrayType(Node), Edges, Indeg], ArrayType(Node))
|
|
203
|
+
def topo_order(b, roots, succ, indeg):
|
|
204
|
+
ready = b.const(roots.copy()) # task inputs arrive frozen: copy
|
|
205
|
+
remaining = b.const(indeg.copy())
|
|
206
|
+
order = b.const(East.new_array(Node))
|
|
207
|
+
i = b.let(0)
|
|
208
|
+
|
|
209
|
+
def step(b, label):
|
|
210
|
+
node = b.const(ready.get(i))
|
|
211
|
+
b.do(order.push_last(node))
|
|
212
|
+
|
|
213
|
+
def visit(b, v):
|
|
214
|
+
b.do(remaining.insert_or_update(v, -1, lambda _b, old, d: old + d))
|
|
215
|
+
b.if_(remaining.get(v) == 0, lambda b: b.do(ready.push_last(v)))
|
|
216
|
+
|
|
217
|
+
b.for_(succ.get_or_default(node, East.new_array(Node)), visit)
|
|
218
|
+
b.assign(i, i + 1)
|
|
219
|
+
|
|
220
|
+
b.while_(i < ready.size(), step)
|
|
221
|
+
return order
|
|
222
|
+
|
|
223
|
+
@East.platform_function(inputs=[ArrayType(Node), Edges, Indeg], output=ArrayType(Node))
|
|
224
|
+
def replay_order(roots, succ, indeg):
|
|
225
|
+
return topo_order(roots, succ, indeg)
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
The same algorithm as EXPRESSION forms — a state struct threaded through
|
|
229
|
+
`East.while_`/`East.for_`, the shape that fits inside a one-expression
|
|
230
|
+
callback:
|
|
231
|
+
|
|
232
|
+
```python
|
|
233
|
+
topo_order = East.function(
|
|
234
|
+
[ArrayType(Node), Edges, Indeg], ArrayType(Node),
|
|
235
|
+
lambda b, roots, succ, indeg: East.while_(
|
|
236
|
+
{"ready": roots.copy(), "indeg": indeg.copy(),
|
|
237
|
+
"order": East.new_array(Node), "i": 0},
|
|
238
|
+
cond=lambda b, s: s.i < s.ready.size(),
|
|
239
|
+
body=lambda b, s: East.let(s.ready.get(s.i), lambda b, node: East.block(
|
|
240
|
+
s.order.push_last(node),
|
|
241
|
+
East.for_(
|
|
242
|
+
succ.get_or_default(node, East.new_array(Node)),
|
|
243
|
+
{**s, "i": s.i + 1}, # inner loop's state
|
|
244
|
+
lambda b, t, v: East.block(
|
|
245
|
+
t.indeg.insert_or_update(v, -1, lambda b, old, d: old + d),
|
|
246
|
+
East.if_else(t.indeg.get(v) == 0, # newly ready?
|
|
247
|
+
East.block(t.ready.push_last(v), t),
|
|
248
|
+
t))))), # else: unchanged
|
|
249
|
+
).order)
|
|
250
|
+
|
|
251
|
+
# any carried state — a forward fill is a `for_` whose state remembers the last value
|
|
252
|
+
forward_fill = East.function(
|
|
253
|
+
[ArrayType(StringType)], ArrayType(StringType),
|
|
254
|
+
lambda b, cells: East.for_(
|
|
255
|
+
cells, {"last": "", "out": East.new_array(StringType)},
|
|
256
|
+
lambda b, s, cell: East.let(
|
|
257
|
+
East.if_else(cell == "", s.last, cell),
|
|
258
|
+
lambda b, v: East.block(s.out.push_last(v), {"last": v, "out": s.out}))).out)
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
To stop early, put `East.break_(state)` in an `if_else` arm — the state
|
|
262
|
+
argument commits before the jump, so the answer survives; `East.label(...)` on
|
|
263
|
+
an outer loop lets an inner one break all the way out. Reach for all of this
|
|
264
|
+
only when the work is genuinely sequential: a `group_reduce` or a `reduce` is
|
|
265
|
+
both shorter and faster when it fits.
|
|
266
|
+
|
|
267
|
+
## Trees from flat rows, and recursion
|
|
268
|
+
|
|
269
|
+
`to_tree` builds nested data — any depth, your own node type — from rows that
|
|
270
|
+
name their key and their parent's key. `build` runs once per row, children
|
|
271
|
+
first; the result is the roots in source order. The walk is iterative, in
|
|
272
|
+
east-c, on values and in a body alike:
|
|
273
|
+
|
|
274
|
+
```python
|
|
275
|
+
from east import ArrayType, OptionType, StringType, StructType, recursive_type
|
|
276
|
+
|
|
277
|
+
Row = StructType([("id", StringType), ("parent", OptionType(StringType)), ("name", StringType)])
|
|
278
|
+
Node = recursive_type(lambda self: StructType([("name", StringType), ("children", ArrayType(self))]))
|
|
279
|
+
|
|
280
|
+
roots = rows.to_tree(Node,
|
|
281
|
+
lambda b, r: r.id,
|
|
282
|
+
lambda b, r: r.parent,
|
|
283
|
+
lambda b, r, _i, children: {"name": r.name, "children": children})
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
- A row whose parent is `none` is a root, and so is an orphan (its parent key
|
|
287
|
+
is not in the rows). A repeated key ❗ `toTree: duplicate key <key>`; a cycle
|
|
288
|
+
❗ `toTree: cycle through key <key>` — both before any `build`.
|
|
289
|
+
- A non-recursive `Node` folds bottom-up: `IntegerType` with
|
|
290
|
+
`children.sum() + 1` is each root's subtree size. A Set's element and a
|
|
291
|
+
Dict's key are their own keys, so those take no `key`.
|
|
292
|
+
|
|
293
|
+
`flatten_tree` is the way back: every node of nested data, each with its path
|
|
294
|
+
of keys (a Dict's) or indices (an Array's) from its root, depth first — each
|
|
295
|
+
node before its descendants, siblings in key or array order. `children` names a
|
|
296
|
+
node's children, of the collection's own type:
|
|
297
|
+
|
|
298
|
+
```python
|
|
299
|
+
from east import DictType, FloatType, StringType, StructType, recursive_type
|
|
300
|
+
|
|
301
|
+
Part = recursive_type(lambda self: StructType([("name", StringType), ("cost", FloatType), ("parts", DictType(StringType, self))]))
|
|
302
|
+
|
|
303
|
+
nodes = parts.flatten_tree(lambda b, part: part.unwrap().parts)
|
|
304
|
+
# an Array of {path, node}: ["pump"], ["pump", "motor"], ["pump", "motor", "rotor"], ["pump", "seal"]
|
|
305
|
+
```
|
|
306
|
+
|
|
307
|
+
- An Array of `{path, node}`, not a Dict keyed by path: a Dict key is
|
|
308
|
+
immutable, and a path is an Array. `to_tree` over it, keyed by `path` and
|
|
309
|
+
parented by `path` less its last step, rolls the tree up at every root.
|
|
310
|
+
- `children` runs once per node, in the result's order. Each path is an array
|
|
311
|
+
of its own, so a chain n deep holds n(n+1)/2 path elements.
|
|
312
|
+
- **Recursion** — a function calling itself through a captured variable — is
|
|
313
|
+
for shallow depths, in the hundreds. Deeper, a call ❗ `call stack exhausted:
|
|
314
|
+
East calls nested too deeply`, a catchable `EastError` on every runtime. Walk
|
|
315
|
+
deep data with `b.while_`, `to_tree` or `flatten_tree`; a recursive VALUE's depth is still
|
|
316
|
+
bounded, at tens of thousands of levels, by east-c's recursive collect and
|
|
317
|
+
free.
|
|
318
|
+
|
|
319
|
+
## Sort uses East's total order
|
|
320
|
+
|
|
321
|
+
```python
|
|
322
|
+
# WRONG — Python's default ordering (incorrect for floats/NaN, mixed, type-specific)
|
|
323
|
+
sorted(list(arr))
|
|
324
|
+
|
|
325
|
+
# CORRECT — East total order, in east-c (the TypeScript names)
|
|
326
|
+
arr.sort() # new array
|
|
327
|
+
arr.sort_in_place() # in place
|
|
328
|
+
arr.sort(lambda b, r: r["x"]) # by a projection, still East-ordered
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
## Scalars: use the `East.<Type>` utilities for consistency — above all String & DateTime
|
|
332
|
+
|
|
333
|
+
East scalars *are* Python scalars, so Python operators run on them — but the
|
|
334
|
+
`East.<Type>` namespaces (and `East.less`/`compare`/`equal`) are the standardised
|
|
335
|
+
ops that give the **same** answer in Python, C, TS, and cached e3 tasks. Python's
|
|
336
|
+
`str` methods, `re`, `datetime`, `strftime`, `//`, and `<` drift by
|
|
337
|
+
engine / locale / timezone; the East functions do not. Use them for anything with
|
|
338
|
+
divergent semantics — and **always** for strings and dates.
|
|
339
|
+
|
|
340
|
+
```python
|
|
341
|
+
# String — East.String.*, not Python str/re: split edge cases, the regex engine,
|
|
342
|
+
# trim's whitespace set, and JSON encoding all differ from Python
|
|
343
|
+
East.String.split("a,b,c", ",") # East semantics, not str.split
|
|
344
|
+
East.String.regex_replace(s, r"\d+", "#") # East regex engine, not Python re
|
|
345
|
+
East.String.trim(s) # East's whitespace set
|
|
346
|
+
East.String.print_json(IntegerType, 5) # '"5"' — Integer is a JSON *string* in East
|
|
347
|
+
|
|
348
|
+
# DateTime — East.DateTime.* only. East DateTime is UTC; tokens are Day.js, not strftime
|
|
349
|
+
East.DateTime.add_milliseconds(dt, 86_400_000) # +1 day, not dt + timedelta(days=1)
|
|
350
|
+
East.DateTime.duration_milliseconds(a, b) # a − b (ms), standardised
|
|
351
|
+
East.DateTime.get_day_of_week(dt) # Monday == 1, not dt.weekday() (== 0)
|
|
352
|
+
East.DateTime.print_formatted(dt, "YYYY-MM-DD HH:mm:ss") # Day.js tokens, not dt.strftime("%Y-…")
|
|
353
|
+
East.Float.print_currency(total) # "$1,234.57" — the stdlib, same in TS
|
|
354
|
+
|
|
355
|
+
# Ordering / equality — East total order (NaN-correct, mixed types, variants-by-name)
|
|
356
|
+
East.less(FloatType, a, b) # not a < b / sorted(...)
|
|
357
|
+
# Integer is i64 — divide truncates; Python's unbounded int can overflow i64
|
|
358
|
+
East.Integer.divide(7, 2) # 3
|
|
359
|
+
# (you also can't method-call a Python float/str — the ops live on the namespace)
|
|
360
|
+
East.Float.sqrt(2.0) # not (2.0).sqrt()
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
## torch interop (in a torch-having package, inline — no helpers)
|
|
364
|
+
|
|
365
|
+
```python
|
|
366
|
+
import torch, numpy as np
|
|
367
|
+
t = mat.to_torch() # writeable torch copy of the buffer
|
|
368
|
+
out = EastMatrix(FloatType, model(t).detach().cpu().numpy()) # bridge canonicalizes dtype
|
|
369
|
+
```
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Performance
|
|
2
|
+
|
|
3
|
+
Part of the `east-py` skill.
|
|
4
|
+
|
|
5
|
+
## Performance — the levers, ranked
|
|
6
|
+
|
|
7
|
+
The point of the machinery: **data stays in east-c and python never runs per
|
|
8
|
+
element**.
|
|
9
|
+
|
|
10
|
+
1. **Keep values East end-to-end.** `EastArray`/`EastSet`/`EastDict` are
|
|
11
|
+
C-backed, and every eager method runs its loop natively however deeply the
|
|
12
|
+
element type nests. `list(...)`/`dict(...)` or a python loop pays a boxing
|
|
13
|
+
crossing per element — convert at most once, at the very end.
|
|
14
|
+
2. **Let the callbacks build, and chain on the results** — a chain of
|
|
15
|
+
`map`/`filter`/`reduce` stays native between steps. Return a dict literal
|
|
16
|
+
(`lambda b, r: {"a": …, "b": …}`) to derive every column in ONE pass. **Bind
|
|
17
|
+
shared work with `b.const`** — the builder emits exactly what the body
|
|
18
|
+
spells, as TypeScript's does, so a python variable holding an expression is
|
|
19
|
+
re-emitted (and re-evaluated) at every read: `fields = r.data.split("|")`
|
|
20
|
+
read for 30 columns runs the split 30 times per row, and `table =
|
|
21
|
+
derive(rec)` read inside a `.map` runs `derive` once per element. `fields =
|
|
22
|
+
b.const(r.data.split("|"))` and `table = b.const(derive(rec))` run each
|
|
23
|
+
once, as `$.const` does — the `no-reinlined-east-binding` rule (`east-py
|
|
24
|
+
lint`, the LSP) points at every such read. There is no optimisation pass
|
|
25
|
+
(#1312 removed the trace-time CSE: it moved reads across mutations it could
|
|
26
|
+
not see), so what you write is what runs — a callback that pushes onto
|
|
27
|
+
`order` and returns `order.size() - 1` reads the size after its push.
|
|
28
|
+
3. **Side tables: `b.const` a captured one, `bind` big ones** — opposite contracts:
|
|
29
|
+
- **Capture (snapshot).** A collection captured by an `East.function` body
|
|
30
|
+
is snapshot into its IR as a literal of its contents — exactly as a
|
|
31
|
+
TypeScript literal is — and rebuilt wherever the body reads it: per call
|
|
32
|
+
at the function's top level, per element inside a callback. **Bind it
|
|
33
|
+
once per call with `t = b.const(TABLE)`** and read `t`. Right for small
|
|
34
|
+
tables (up to a few thousand entries): the literal's size rides the IR,
|
|
35
|
+
every call rebuilds it, and later mutations are **not** seen. Only an
|
|
36
|
+
EXPLICIT build snapshots: an eager callback refuses a mutable capture.
|
|
37
|
+
- **`East.function(...).bind(table)` (by reference, live).** Declare the
|
|
38
|
+
table as a trailing parameter and pre-bind it: zero copy, O(1) at any
|
|
39
|
+
size, nothing rebuilt per call, and the function **sees later
|
|
40
|
+
mutations**. Rebinding gives independent callables, the unbound function
|
|
41
|
+
stays usable, and a wrong-typed value raises `TypeError`.
|
|
42
|
+
|
|
43
|
+
```python
|
|
44
|
+
fx = EastDict(StringType, FloatType, rates) # small table → capture
|
|
45
|
+
@East.function([Row], FloatType) # (the EXPLICIT build
|
|
46
|
+
def to_usd(b, r): # opts into the snapshot)
|
|
47
|
+
t = b.const(fx) # the literal, once per call
|
|
48
|
+
return r.amount * t.get_or_default(r.ccy, 1.0)
|
|
49
|
+
|
|
50
|
+
TableT = DictType(StringType, FloatType) # huge table → bind
|
|
51
|
+
conv = East.function([Row, TableT], FloatType,
|
|
52
|
+
lambda b, r, t: r.amount * t.get_or_default(r.ccy, 1.0))
|
|
53
|
+
rows.map(conv.bind(big_table)) # loop + lookup stay in east-c
|
|
54
|
+
conv.bind(t1, t2) # multi-table: binds the TRAILING parameters in order
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Function *parameters* always cross by reference (zero copy, any size, any
|
|
58
|
+
depth); `bind` is what lets an eager method use a parameter-taking function
|
|
59
|
+
where the callback signature is fixed.
|
|
60
|
+
4. **Hoist `East.function(...)` out of python loops** and reuse it — building
|
|
61
|
+
is cheap, not free.
|
|
62
|
+
5. **When logic must stay python, go columnar** — one crossing per column or
|
|
63
|
+
batch instead of per row × field, then back with `from_columns`/`extend`
|
|
64
|
+
([Columnar escape hatches](columns-and-platform-functions.md#columnar-escape-hatches--when-the-logic-must-stay-python)).
|
|
65
|
+
6. **See how a hot call ran** — `east.runtime.compiler.eager_stats()`:
|
|
66
|
+
`function_direct` counts callbacks that rode a precompiled function value
|
|
67
|
+
straight in, `c_to_py_decodes` counts values boxed C→python (an eager method
|
|
68
|
+
quietly decoding a whole collection shows here), and the `beast2_*` counters
|
|
69
|
+
report column projection. There is no per-element python counter because
|
|
70
|
+
there is no per-element python path.
|