@elaraai/east-codex-plugin 1.0.99 → 1.0.100

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. package/.build/mcp/server.js +3 -2
  2. package/.codex-plugin/plugin.json +1 -1
  3. package/index.json +12207 -216
  4. package/package.json +1 -1
  5. package/skills/e3/SKILL.md +29 -802
  6. package/skills/e3/reference/browser.md +93 -0
  7. package/skills/e3/reference/cli.md +120 -0
  8. package/skills/e3/reference/from-code.md +48 -0
  9. package/skills/e3/reference/rack.md +139 -0
  10. package/skills/e3/reference/record-indexes.md +326 -0
  11. package/skills/e3/reference/record-migrations.md +63 -0
  12. package/skills/e3/reference/repository.md +25 -0
  13. package/skills/east/SKILL.md +30 -459
  14. package/skills/east/reference/beast2.md +71 -0
  15. package/skills/east/reference/cross-language-and-json-schema.md +87 -0
  16. package/skills/east/reference/queries.md +46 -0
  17. package/skills/east/reference/runners-and-ir.md +148 -0
  18. package/skills/east/reference/trees-and-recursion.md +56 -0
  19. package/skills/east/reference/values-in-typescript.md +62 -0
  20. package/skills/east-py/SKILL.md +28 -1647
  21. package/skills/east-py/reference/beast2.md +108 -0
  22. package/skills/east-py/reference/cli.md +51 -0
  23. package/skills/east-py/reference/columns-and-platform-functions.md +87 -0
  24. package/skills/east-py/reference/control-flow-and-queries.md +147 -0
  25. package/skills/east-py/reference/cross-language.md +70 -0
  26. package/skills/east-py/reference/ir-and-diagnostics.md +150 -0
  27. package/skills/east-py/reference/patterns.md +369 -0
  28. package/skills/east-py/reference/performance.md +70 -0
  29. package/skills/east-py/reference/value-methods.md +306 -0
  30. package/skills/east-py/reference/values-construction-and-json.md +88 -0
  31. package/skills/east-py/reference/writing-east-functions.md +275 -0
  32. package/skills/east-py-datascience/SKILL.md +11 -28
@@ -52,6 +52,17 @@ writing or changing East code:
52
52
  programs, exact and far cheaper, and the signatures omit the runtime rules
53
53
  that make East code correct.
54
54
 
55
+ The index holds two separate sets of python, for two different jobs:
56
+
57
+ - **East functions written in python** (East expressions: `East.function`, the
58
+ block `b`). Search with `language: "python"`; every core example comes back
59
+ printed as python.
60
+ - **Python working with East values directly** (the eager methods on
61
+ `EastArray`, `EastDict`, …, outside any `East.function`, as a platform
62
+ function or script does). Search with `package: "east-values"`. Each of these
63
+ is generated from a core example's IR and checked to give that example's
64
+ result. A search that does not name `east-values` never returns them.
65
+
55
66
  Nothing is injected for you; the search is the step.
56
67
 
57
68
  ## Quick Start
@@ -107,11 +118,11 @@ body reads it — `t = b.const(table)` builds it once per call, and `.bind(table
107
118
  as a trailing parameter keeps it live. The other way, an artifact is a plain
108
119
  callable on values, every eager method accepts one (a VALUE takes no block),
109
120
  and one called inside another body is a call of its inline Function node — see
110
- [Python values vs East expressions](#python-values-vs-east-expressions).
121
+ [Python values vs East expressions](reference/patterns.md#python-values-vs-east-expressions).
111
122
 
112
123
  ## Decision Tree: What Do You Need?
113
124
 
114
- Each leaf names the section holding the whole surface for it.
125
+ Each leaf names the section holding the whole surface for it; [Reference files](#reference-files) says which file holds each section.
115
126
 
116
127
  ```
117
128
  Task → What do you need?
@@ -221,687 +232,6 @@ a narrower numpy dtype (f32, …) is canonicalized to East's width (Float→f64,
221
232
  Integer→i64, Boolean→u8) crossing into east-c. `PatchType(T)` (from
222
233
  `east.types.types`) is the type of a structural patch of a `T`.
223
234
 
224
- ## East expressions — writing East functions in Python
225
-
226
- A python body becomes East IR by being CAPTURED: it runs ONCE against typed
227
- expression proxies, exactly like the TypeScript builder; east-c compiles the
228
- recorded IR, and from then on only the compiled function runs. A body East
229
- cannot express raises at build time, so the same source always costs and means
230
- the same thing.
231
-
232
- ### Declare, implement, build, compile — the four authoring calls
233
-
234
- The TypeScript names, name for name. Two are about **platform functions** —
235
- python the program calls out to, paired **by name** — and two about **East
236
- functions**, the program itself:
237
-
238
- | Step | Call | What it is |
239
- |---|---|---|
240
- | **Implement** a platform function in python | `@East.platform_function(inputs=…, output=…)` (`@East.generic_platform_function(type_parameters=, inputs=, output=)`; `East.platform_functions(__name__)` collects a module's) | the host side AND the body's spelling: a plain python function over East VALUES, its result validated against `output` — and, called inside a body, the `Platform` node with this very signature. Its name — the `def`'s, or `name=` — is what every runtime pairs the call with |
241
- | **Declare** one implemented elsewhere | `East.platform(name, inputs, output)` (`East.asyncPlatform`, `East.genericPlatform`, `East.asyncGenericPlatform`) | a handle a body CALLS — it emits the `Platform` node; nothing runs here. For a function implemented in another package or runtime; a python implementation you hold needs no declaration |
242
- | **Build** an East function | `East.function(inputs, out, body)` / `@East.function(inputs, out)` (`East.asyncFunction`) | runs `body` once over expression proxies and records IR; a pure one is already callable |
243
- | **Compile** with the implementations | `East.compile(fn, platform=[…])` (`East.compileAsync`) | analyzes the IR against the implementations — a declaration no implementation matches by name is `Platform function '<name>' not found` — and returns the native callable |
244
-
245
- An `East.function` has no name of its own: it is a VALUE, called through the
246
- binding that holds it (`score(x)`, `rows.map(score)`), stored in a struct or an
247
- array, exported under a name (`east_functions = {"score": score}`) — its IR
248
- carries its parameters' names, never its own, as in TypeScript. A platform
249
- function is the opposite: the name IS the pairing, on every runtime.
250
-
251
- ```python
252
- from east import East, ArrayType, FloatType, IntegerType, NullType, StringType, StructType
253
- from east_py_std import fs_read_file, platform as std
254
-
255
- Row = StructType([("sku", StringType), ("qty", IntegerType)])
256
-
257
- # BUILD — `out` is REQUIRED and enforced; every body takes the block `b` first
258
- score = East.function([Row], FloatType, lambda b, r: r.qty.to_float() * 1.5)
259
- score({"sku": "a", "qty": 2}) # 3.0 — a pure artifact is a callable on values
260
-
261
- # IMPLEMENT — python over East values; `name=` because "t.log" is not an identifier.
262
- # The decorated function is DUAL-MODE: log_line("x") runs the python;
263
- # inside a body, log_line(name) IS the Platform node 't.log' with this signature.
264
- @East.platform_function(inputs=[StringType], output=NullType, name="t.log")
265
- def log_line(line):
266
- print(line)
267
-
268
- greet = East.function([StringType], NullType,
269
- lambda b, name: log_line(East.String.concat("hello ", name)))
270
- greet("bob") # EastError: Platform function 't.log' is not
271
- # available — compile with East.compile(fn, platform=[...])
272
-
273
- # COMPILE — the implementation, by name (the same record the decorator registered)
274
- run = East.compile(greet, platform=East.platform_functions(__name__))
275
- run("bob") # prints "hello bob"
276
-
277
- # A stock implementation is called the same way — no East.platform line restates it
278
- first = East.function([StringType], StringType, lambda b, path: fs_read_file(path).split("\n").get(0))
279
- East.compile(first, platform=std)("hosts.txt")
280
-
281
- # DECLARE — only for a function implemented elsewhere (another package or runtime):
282
- remote = East.platform("t.remote", [StringType], NullType)
283
- ```
284
-
285
- **An East function inside a platform function.** The implementation is
286
- ordinary python over East values, and an `East.function` is a callable on
287
- values that every eager method accepts — so that is how the implementation
288
- does its East work, with the loop and the body running in east-c:
289
-
290
- ```python
291
- Rows = ArrayType(Row)
292
-
293
- @East.platform_function(inputs=[Rows], output=FloatType) # its name: "total", the def's
294
- def total(rows):
295
- return rows.map(score).sum() # `score` from above — an East function on values
296
-
297
- report = East.function([Rows], StringType, # `total(rows)` in the body IS the
298
- lambda b, rows: East.str("total: ", # Platform node — no declaration
299
- East.Float.print_currency(total(rows))))
300
- East.compile(report, platform=East.platform_functions(__name__))(rows) # "total: $9.00"
301
- ```
302
-
303
- Each call in detail:
304
-
305
- | Call | Builds | Notes |
306
- |---|---|---|
307
- | `East.function(param_types, out, body)` | a `Function` artifact | `param_types` is a LIST (`[]` for none); the body takes the block first — `lambda b, x: …` / `def f(b, x)`, as EVERY body does (`lambda x: …` is refused with the fix-it); `out` is required, and a body whose expression has another type raises naming both; with `body` omitted it is a DECORATOR |
308
- | `East.asyncFunction(param_types, out, body)` | an `AsyncFunction` artifact | for bodies calling async platform declarations; compile with `East.compileAsync` |
309
- | `East.platform(name, inputs, output, optional=False)` | a declaration handle | callable INSIDE a body (emits the `Platform` node); calling one outside raises `expression-level`; `East.genericPlatform(name, ["T"], inputs, output)` is the type-parameterised form (`East.asyncGenericPlatform` the async one). Needed only for a function implemented elsewhere: a `@East.platform_function` you hold emits the same node itself |
310
- | `East.asyncPlatform(name, inputs, output)` | an async declaration | calling it from a SYNC body is a build error naming `East.asyncFunction` |
311
- | `East.compile(fn, platform=[])` / `East.compileAsync(...)` | a native callable | takes an artifact or a raw IR value; the IR is analyzed against `platform` first (`east.ir.analyze`, the TS `analyzeIR`): a signature mismatch or a missing implementation is an `EastError` naming the call — unless the declaration is `optional=True`, which compiles to a stub that raises at the call (TS parity) |
312
-
313
- A **pure** artifact needs no compile step: it is already a native callable,
314
- `.bind(*values)` pre-binds trailing parameters by reference, and calling it
315
- inside another body is a `Call` of its inline `Function` node — TypeScript's
316
- shape — so it runs in ITS OWN frame: a `b.return_` returns from it, an
317
- artifact used as an `East.if_else` arm evaluates only when that arm is taken,
318
- an error inside it names its own lines, and a profile lists it as an entry of
319
- its own, placed where it was written and where it was called. (One called with
320
- arguments narrower than its parameters — a column projection's narrowed rows —
321
- re-runs its source in its own frame and splices in instead.) An artifact that
322
- mutates or raises, called and DISCARDED, is the build's "evaluated and thrown
323
- away" error — spell it `b.do(push(xs))` / `East.block(push(xs), result)`, as
324
- with any mutation. An artifact that calls platform functions stays first-class
325
- (composable, serializable) but raises until `East.compile` pairs it with
326
- implementations.
327
- `East.function(...)` spelled INSIDE a body is the inline `Function` node, a
328
- Function-typed expression: bind it with `b.const`, hand it to a callback slot,
329
- or call it (a `Call`; in an `East.asyncFunction`, calling an async one is a
330
- `CallAsync`).
331
-
332
- **Error locations.** A build records the python frames that built each node,
333
- so a runtime error names its authoring site: `EastError.location` is the stack —
334
- the lambda's `file:line:column` first, then any helper between, then the
335
- `East.function(...)` call — and a platform-signature mismatch at `East.compile`
336
- names the call the same way. The stack stops at that call: the code that
337
- imported or called the module is not recorded. A recorded path is relative to
338
- its file's project root, the nearest directory above it holding a
339
- `pyproject.toml`, so a function's IR is the same whichever module imported it
340
- and whatever directory the process ran in. A file in no project is relative to
341
- the working directory, and `set_location_base_path(dir)` (from `east`) pins one
342
- base for every path, for reproducible fixtures (`None` restores the default);
343
- `east-py check` names a refusal's frames relative to the working directory, as
344
- an editor opens them. The map rides the function's beast2 encoding, so the
345
- error reads the same on east-c or east-node. An error inside a callback a
346
- builtin invokes (`arr.map(...)` and friends) resolves to the builtin's call
347
- site, on every runner.
348
- `set_location_capture(False)` (from `east`) builds without recording any
349
- locations: there is then no stack walk per node, and an error raised by such a
350
- function carries no location. Binding names are read separately and keep
351
- working. The TypeScript twin is `setLocationCapture`.
352
-
353
- ### Every body takes the block first — the TypeScript `$` twin
354
-
355
- A TypeScript body receives `$` and appends statements to it; a python body
356
- receives **`b`**, the block, as its FIRST parameter and does the same. EVERY
357
- body does — an `East.function` body, a builtin's callback, a branch, a loop, a
358
- handler — exactly as every TypeScript body is `($, …) => …`: `lambda x: …` is
359
- refused with the fix-it, a body that uses the block as the element fails on the
360
- block's first use, and each nested body receives ITS OWN block first, so a
361
- statement on any other block is a build-time error. Python `None` returned from a
362
- body is TypeScript's "no return"; the `east_null` sentinel is an explicit `null`:
363
-
364
- ```python
365
- from east import East, IntegerType, StringType
366
-
367
- @East.function([IntegerType], StringType)
368
- def classify(b, n):
369
- acc = b.let(0) # $.let — reassignable
370
- limit = b.const(n * 2) # $.const
371
- def loop(b, label): # $.while body: (b, label)
372
- b.if_(acc >= limit, lambda b: b.break_(label))
373
- b.assign(acc, acc + 1) # $.assign
374
- b.while_(True, loop)
375
- b.if_(acc > 10, lambda b: b.return_("big")) \
376
- .else_if(acc > 5, lambda b: b.return_("mid")) \
377
- .else_(lambda b: b.return_("small")) # every arm returns → Never
378
-
379
- classify(3) # "mid"
380
- ```
381
-
382
- A one-statement body is a `lambda b: …`; a longer one is a `def` written
383
- just before the statement that uses it. Name the block `_b` when the body
384
- does not use it.
385
-
386
- | Body | Spelling |
387
- |---|---|
388
- | `East.function` / `East.asyncFunction` body | `lambda b, x: …`, `def f(b, x)` — the block plus exactly the function's parameters; `lambda x: …` is refused (`a body takes the block first`) |
389
- | a builtin's callback (`xs.map(...)`, `reduce`, …) | `xs.map(lambda b, el, i: …)` — the block, then the builtin's callback signature (map's `(element, index)`; `xs.reduce(lambda b, acc, el, i: …, 0)`; a Dict's `(value, key)`); trailing parameters may be omitted, the block cannot |
390
- | a statement construct's body (`b.if_`/`.else_if`/`.else_`, `b.match_`, `b.while_`, `b.for_`, `b.try_`/`.catch`/`.finally_`) and `East.block(fn)` | `b.for_(xs, lambda b, v, i, label: …)`, `b.match_(v, {"some": lambda b, x: …})`, `East.block(lambda b: …)` |
391
- | an expression form's handler (`.match({...})`, `East.try_catch`, `East.let`, the `East.while_`/`for_` bodies, `.and_`/`.or_`/`.if_else`) | bodies too: `v.match({"some": lambda b, x: …})`, `East.try_catch(lambda b: …, lambda b, msg: …)` — statements inside them go in `East.block(lambda b: …)`; `East.if_else` arms are expressions, not bodies |
392
- | a function VALUE — a compiled `East.function`, a `.bind` result, a Function-typed expression, a platform declaration | takes NO block: a slot invokes what it holds body-style and the value drops the block, so `xs.map(amount)` and `xs.map(lambda b, el: amount(el))` both work (TS: an `Expr<FunctionType>` wherever a `($, …) => …` is accepted); a value declaring fewer parameters than the slot passes takes the prefix |
393
-
394
- | Statement | Emits | Notes |
395
- |---|---|---|
396
- | `b.let(value[, type])` / `b.const(value[, type])` | `Let` (Null) | the variable, mutable / not; a declared `type` widens a narrower literal (a `Variable` of a subtype gets an `As`) |
397
- | `b.assign(var, value)` | `Assign` (Null) | `var` must come from `b.let` — a python `x = …` rebinds the NAME and changes nothing |
398
- | `b.return_([value])` | `Return` (Never) | checked against the declared output; a `do` of the same expression just before is not duplicated |
399
- | `b.if_(pred, fn).else_if(pred, fn).else_(fn)` | `IfElse` (Null; Never when every arm diverges) | each `fn(b)` runs in its own frame; a branch ending in a non-Null value pads with `null`; a body may return the chain itself (`lambda b: b.if_(…)`) |
400
- | `b.match_(variant, {case: fn(b, data)})` | `Match` (Null) | a case without a handler does nothing |
401
- | `b.while_(pred, fn)` / `b.for_(coll, fn)` | `While` / `ForArray`/`ForSet`/`ForDict` (Null) | `fn(b, label)`; Array `fn(b, value, index, label)`, Set `fn(b, key, label)`, Dict `fn(b, value, key, label)` |
402
- | `b.break_(label)` / `b.continue_(label)` | `Break` / `Continue` (Never) | the loop's `label` is what the body received; a bare `East.break_()` (the expression form's jump) inside a statement loop targets it too |
403
- | `b.try_(fn).catch(fn(b, message, stack)).finally_(fn)` | `TryCatch` (Null) | `.catch` at most once; Never when both bodies diverge |
404
- | `b.do(expr)` | the expression as a statement | `$(expr)` — a platform call or mutation evaluated for its effect; a bare `arr.push_last(x)` line is thrown away, and the build says so |
405
- | `b.error(msg)` | `Error` (Never) | `$.error` — raise now; `East.error(msg)` is the expression twin (return it, or use it as an `if_else` arm) |
406
- | `East.block(fn)` | `Block` | the EXPRESSION form: `fn(b)`'s statements, then the value it returns (a block returning nothing must diverge) |
407
-
408
- A statement after one that never completes raises `Unreachable statement
409
- detected`, as in TypeScript; a statement on an OUTER block from inside a nested
410
- body, or on a block whose body has returned, raises naming it (TypeScript's
411
- `no-cross-block-builder` lint is a hard error here). Every other IR node kind has
412
- a spelling too, so any program TypeScript can build, python builds name for
413
- name: `East.value(v, T)` (a typed literal / struct / list / dict), `East.as_(v,
414
- T)` (an explicit widening `As`), `East.wrap_recursive(v, R)` and `expr.unwrap()`
415
- on a recursive-typed expression, and `East.builtin(name, [T…], [args], out)` (a
416
- raw builtin, for the few with no named spelling — `east.codegen.RAW_ONLY`).
417
-
418
- **The analyzer.** Every build and every `East.compile` runs
419
- `east.ir.analyze.analyze_ir`, the twin of TypeScript's `analyzeIR`: scope, exact
420
- types (a `Let`/`Assign`/argument/element/field has exactly its slot's type;
421
- subtyping is spelled with `As`), divergence and node well-formedness, with the
422
- TypeScript messages at the python `file:line:column`. A body that fails it never
423
- compiles.
424
-
425
- ### The expression surface, type by type
426
-
427
- One class per East type in `east.expression.expr`, each the twin of
428
- `libs/east/src/expr/<type>.ts`: the same methods snake_cased, the same builtin
429
- and argument order behind each, and the same aliases where TypeScript has them
430
- (`plus`/`minus`/…, `eq`/`equal`/`equals`). Every TypeScript method exists on the
431
- expression class AND on the eager value class (a scalar's eager twin is the
432
- `East.<Type>` function of the same name, value first: `d.add_days(n)` ↔
433
- `East.DateTime.add_days(d, n)`), and each python-only name is a deprecated
434
- python-idiom alias, a python protocol twin, or a convenience. Python departs from
435
- TypeScript only where the language forces it: operators where they agree (`+ - *
436
- / ** & | ^ ~`, the comparisons), `and_`/`or_`/`not_`, `as_`, and keyword
437
- arguments (`out=`, `key=`, `combine=`) where TypeScript overloads.
438
-
439
- | Type | Operators | Methods (the TypeScript names) |
440
- |---|---|---|
441
- | **Boolean** | `&` `\|` `^` `~` — never `and`/`or`/`not`/`if` (python collapses them to `bool`) · `==` `!=` | `.bit_and(y) .bit_or(y) .bit_xor(y) .not_()` (the builtins — both operands evaluate) · `.and_(fn(b))` `.or_(fn(b))` (TS `and`/`or`: SHORT-CIRCUIT, the other operand is a body) · `.if_else(fn(b), fn(b))` (TS `ifElse`; `East.if_else(cond, value, …, otherwise)` is the value form — pairs then the else, one `IfElse` node) · `.equals/.equal/.eq` `.not_equals/.not_equal/.ne` |
442
- | **Integer** | `+` `-` `*` and unary `-`; `==` `!=` `<` `<=` `>` `>=` (East total order) · ❗ `//` `%` `**` `/` RAISE at build time with the fix-it: python floors/takes the divisor's sign/promotes a negative exponent where East truncates/takes the dividend's sign/yields 0 — spell `.divide(y)` / `East.Integer.divide`, `.remainder(y)` / `East.Integer.remainder`, `.pow(y)` / `East.Integer.pow`, `.to_float() / y` | `.add .subtract .multiply .divide .remainder .pow` (a Float argument widens `self`, like TS; aliases `.plus .sub .minus .mul .times .div .mod .rem .modulo`) · `.negate() .abs() .sign() .log(base)` · `.to_float()` · `.less_than/.less/.lt .greater_than/.greater/.gt .less_than_or_equal/.less_equal/.lte/.le .greater_than_or_equal/.greater_equal/.gte/.ge .equals/.equal/.eq .not_equals/.not_equal/.ne` |
443
- | **Float** | `+` `-` `*` `/` `**` and unary `-`; comparisons · ❗ `//` `%` RAISE (`.remainder(y)` / `East.Float.remainder`) · `math.floor/ceil/trunc(x)` build the stdlib `East.Float.round_floor/round_ceil/round_trunc`; python `round(x)` raises (its tie rule differs — `East.Float.round_half(x)`) | the same named arithmetic and aliases (an Integer argument widens) · `.negate .abs .sign .sqrt .exp .log .sin .cos .tan` · `.to_integer()` ❗ errors at run time on a non-integral/NaN/infinite value · Float → Integer rounding is the stdlib `East.Float.round_floor/round_ceil/round_trunc/round_half(x)` (`round_half` = ties AWAY from zero; `.floor() .ceil() .trunc() .round()` are deprecated aliases) · comparisons as Integer |
444
- | **String** | `+` (concat); comparisons | `.concat(s) .repeat(n) .substring(a, b) .upper_case() .lower_case() .trim() .trim_start() .trim_end()` · `.replace(old, new) .split(sep)` · `.length() .starts_with(p) .ends_with(s) .contains(s) .index_of(s)` · `.parse(T)` ❗ (strict whole-string) `.parse_json(T)` · `.try_parse(T) -> Option<T>` (none on any failure) · `.encode_utf8() .encode_utf16()` · `.regex_contains(pat, flags="") .regex_index_of(pat, flags="") .regex_replace(pat, repl, flags="")` (the builtins TypeScript has no method for) · never an f-string (it would constant-fold the proxy) — `East.str(…)` or `+` |
445
- | **DateTime** | comparisons; a python `datetime` literal lifts as a DateTime | `.get_year() .get_month() .get_day_of_month() .get_day_of_week()` (Monday = 1) `.get_hour() .get_minute() .get_second() .get_millisecond()` · `.add_milliseconds(n) .add_seconds .add_minutes .add_hours .add_days .add_weeks` and `.subtract_milliseconds … .subtract_weeks` (an Integer or Float `n`) · `.duration_milliseconds(other) -> Integer`, `.duration_seconds/minutes/hours/days/weeks(other) -> Float` ❗ `a.duration_days(b)` is `b − a` (positive when `b` is later — the TS method; the namespace `East.DateTime.duration_milliseconds(a, b)` is the raw builtin, `a − b`) · `.to_epoch_milliseconds()` · `.print_formatted(fmt)` (Day.js tokens) |
446
- | **Blob** | comparisons | `.size() .get_uint8(i)` · `.decode_utf8() .decode_utf16()` · `.decode_beast(T, version="v1")` (`"v2"` = the beast2 family) · `.open_beast(T)` (TS `openBeast`: an indexed beast2 `Array`/`Set`/`Dict` blob as a FROZEN paged value — `size`, keyed reads and `for_` decode one segment; an index-less blob or a Ref-/function-bearing element shape decodes whole; the header's type must equal `T`, checked before any decode) · `.decode_csv(RowT, config=None, **options)` |
447
- | **Array / Set / Dict / Vector / Matrix** | `xs[i]` (a negative LITERAL index raises — spell `xs.get(xs.size() - 1)`) · `d[k]` ❗missing · comparisons | The eager surface, name for name and signature for signature — [EastArray](#eastarray--complete-method-surface) · [EastSet](#eastset--complete-method-surface) · [EastDict](#eastdict--complete-method-surface) · [EastVector](#eastvector--complete-method-surface) · [EastMatrix](#eastmatrix--complete-method-surface) — less what only a value has: the python protocol (Array `insert`/`remove`/`count`/`index`, Set `add`/`remove`/`discard`, Dict `values`/`items`), the columnar hatches (`to_columns`, `from_columns`, `map_batches`, `update_many`), the numpy/torch bridges, the classmethod generators (`East.Array.range` and kin are their twins) and the deprecated `map_elements`. A mutator yields Null or Boolean — sequence it with `b.do` or `East.block`. A Dict callback is `(value, key)`: `fn(b, v)` or `fn(b, v, k)`, a fold step `fn(b, acc, v[, k])`, a collision handler `combine(b, existing, incoming[, key])` |
448
- | **Struct** | `r.field` / `r["field"]` (both build IR, both work on real rows) | build a row with a dict literal `{"k": expr, …}` or `struct({…}, T)` (dual-mode) |
449
- | **Variant** | — | `.match({case: fn(b, payload)}, default=fn(b))` (TS's partial match; the arms must agree on one East type — a `some(x)` arm types its `none` sibling) · `.match_tag(tag, fn, default)` · `.unwrap(tag="some", on_other=None)` ❗ · `.has_tag(tag) .get_tag()` · Option: `.is_some() .is_none() .unwrap_or(d)` · build with `some(expr)` / `none` / `variant(case, payload)`, typed from context (the build's declared output, a typed `if_else` sibling, a declared struct field, or an `out=` pin) |
450
- | **Ref** | — | `East.ref(v)` builds one · `.get()` · `.update(v)` (TS `RefUpdate`; `.set` and the read-modify-write `update(fn)` are deprecated — write `r.update(f(r.get()))`) · `.merge(v, fn(b, current, patch))` |
451
- | **Function** | `f(x, …)` — a `Call` node (a `CallAsync` inside an async body) | a Function-typed parameter or a `b.const(East.function(…))` is callable in the body; `FunctionType` parameters bind with `.bind(fn_value)` |
452
-
453
- The expression methods take the eager keywords — `out=` (`map`, `filter_map`,
454
- `map_reduce`, `flat_map`, `flatten_to_set`, `to_array`, `to_set`), `key_out=` /
455
- `value_out=` (`to_dict`), `key_out=` / `acc_out=` (`group_reduce`), `pred=`,
456
- `key=`, `value_fn=` — and an `out=`-family pin also TYPES the callback's build, so
457
- a pinned callback can build a general variant with no other context. Result
458
- types come from the build, never a data sample: `out=` is optional everywhere,
459
- and an EMPTY collection derives what a full one does. `some`/`every`/`first_map`
460
- compile to the native short-circuiting FirstMap scans (`some([])` is False,
461
- `every([])` True). An unsupported method raises `ExpressionError` naming the
462
- supported set.
463
-
464
- ### The standard library — the TypeScript `East.<Type>.*` functions
465
-
466
- Every namespace carries the East standard library — TypeScript's
467
- `libs/east/src/expr/libs/*.ts`, body for body, snake_cased (`printCompact` →
468
- `print_compact`; the TS misspelling `printCommaSeperated` stays
469
- `print_comma_seperated`, with `print_comma_separated` as a python twin). Each is
470
- an `East.function` built on first use: on plain values it runs natively, inside a
471
- body it is a call of its inline Function node like any artifact — as
472
- TypeScript's is:
473
-
474
- ```python
475
- @East.function([IntegerType, FloatType, DateTimeType], StringType)
476
- def label(b, n, x, d):
477
- return East.str(East.Integer.print_compact(n), " / ", # "1.23M"
478
- East.Float.print_currency(x), " @ ", # "$1,234.57"
479
- East.DateTime.print_formatted(East.DateTime.round_down_hour(d, 6), "HH:mm"))
480
- ```
481
-
482
- The outputs are TypeScript's exactly — a Float prints with its point
483
- (`East.Float.print_percentage(1.0, 0)` is `"100.0%"`), the Integer compact forms
484
- carry two decimals below ten units (`print_compact(1500)` is `"1.50K"`), and
485
- `round_nearest`'s Integer half-step truncates (`round_nearest(17, 5)` is `20`).
486
- Every function, with its outputs, is in
487
- [East.<Type> namespaces](#easttype-namespaces--the-builtins-and-the-standard-library);
488
- the dual-mode generators (`East.Array.range`, `East.Set.generate`,
489
- `East.Vector.zeros`, …) are in [Container generators](#container-generators-classmethods).
490
-
491
- ### Control flow — the expression forms (`East.while_`, `East.for_`, …)
492
-
493
- Prefer the collection methods: `map`/`filter`/`reduce`/`group_reduce`
494
- express most work and are the fastest thing in the runtime. Reach here when
495
- the next step **depends on the last** — a worklist, a BFS, a fixpoint, a
496
- topological replay — which no data-parallel method can express. The
497
- statement forms above (`b.while_`, `b.for_`) build the same nodes; these are
498
- their EXPRESSION twins, usable inside a callback or a one-expression body.
499
-
500
- Python cannot overload `=`, and `while`/`if` collapse to a `bool` before any
501
- build sees them. That is the constraint that produced `East.if_else`, and the
502
- answer is the same shape: **`while_` is to `while` what `if_else` is to `if`**.
503
- The body is a pure function of the state that RETURNS the next state, and the
504
- state struct IS the loop's local variables:
505
-
506
- ```python
507
- total = East.while_({"i": 0, "acc": 0},
508
- cond=lambda b, s: s.i < n,
509
- body=lambda b, s: {"acc": s.acc + s.i, "i": s.i + 1}).acc
510
-
511
- # `East.if_else` is the `if`; a field left as `s.field` is the empty else;
512
- # `{**s, …}` changes one field and keeps the rest
513
- East.for_(rows, {"n": 0, "hi": 0.0},
514
- lambda b, s, r: {"n": s.n + 1,
515
- "hi": East.if_else(r.price > s.hi, r.price, s.hi)})
516
- ```
517
-
518
- This lowers to a `Ref` holding the state, a `While`/`For*` node whose body is
519
- one `RefUpdate`, and a final read — the whole loop runs inside east-c. Every
520
- construct is dual-mode like `East.if_else`: outside a build it runs the plain
521
- python loop, so one body serves both a callback and a direct call on plain
522
- East values. A worked example is in
523
- [Key Patterns](#sequential-logic-that-stays-in-east-c-worklist--replay).
524
-
525
- | Call | Emits | Notes |
526
- |---|---|---|
527
- | `East.if_else(cond, value, …, otherwise)` | `IfElse` | cond/value pairs then the else — an if/elif/else chain is ONE node; exactly one arm evaluates |
528
- | `East.while_(state, cond, body, label=…)` | `While` | `cond(b, s) -> Boolean`, `body(b, s) -> next state` |
529
- | `East.for_(coll, state, body, label=…)` | `ForArray`/`ForSet`/`ForDict` | Array `body(b, s, el[, i])`, Set `body(b, s, el)`, Dict `body(b, s, k, v)` |
530
- | `East.block(a, b, …)` | `Block` | evaluates in order, yields the last — the sequencing point for mutators (`East.block(lambda b: …)` is the statement form) |
531
- | `East.let(value, fn)` | `Let` | `fn(b, bound)` — bind once, use many times, inside an expression body (`b.const` is the statement form) |
532
- | `East.ref(v)` | `NewRef` | a cell — `.get()` / `.update(v)` / `.merge(v, fn(b, current, patch))` |
533
- | `East.label(name=None)` | — | names a loop, for `break_`/`continue_` from a NESTED one |
534
- | `East.break_(state=…, label=…)` / `East.continue_(state=…, label=…)` | `Break` / `Continue` | leave / next iteration, optionally committing a last state |
535
- | `East.try_catch(body, handler, finally_=None)` | `TryCatch` | `body(b)`, `handler(b, message[, stack])`, same East type as `body` |
536
- | `East.new_array/new_set/new_dict(…)` | `NewArray`/`NewSet`/`NewDict` | a FRESH collection per evaluation — the loop accumulator |
537
- | `East.new_vector(T, values)` / `East.new_matrix(T, rows, cols, values)` | `NewVector`/`NewMatrix` | a fresh tensor from scalar expressions |
538
-
539
- **Accumulate in place.** Threading a collection through the state rebuilds it
540
- every iteration (`order.concat(…)` copies — O(n²) over the loop). The
541
- mutators extend it in O(1). Each returns what its EAGER twin returns, so
542
- sequence with `East.block`; a mutation written as a bare statement is
543
- evaluated and thrown away, and the build says so rather than compiling a
544
- loop that silently does nothing:
545
-
546
- ```python
547
- counts = East.for_(rows, {"counts": East.new_dict(StringType, IntegerType)},
548
- lambda b, s, r: East.block(
549
- s.counts.insert_or_update(r.sku, 1, lambda b, x, y: x + y),
550
- s)).counts
551
- ```
552
-
553
- ⚠️ Seed accumulators with `East.new_array/new_set/new_dict`. A captured
554
- `EastArray(T)` works too — a captured collection is a snapshot of its
555
- contents, rebuilt wherever the body reads it, so a seed is fresh per call —
556
- but mutating one anywhere ELSE changes a copy nothing else sees, and the
557
- build raises rather than compile a mutation that persists nowhere.
558
-
559
- ### Queries — `East.jq`
560
-
561
- `East.jq(input, program, result_type)` is a jq query as East code: the jq is
562
- parsed, checked against the input's East type and translated to ordinary East
563
- IR when the program is built — loops, `Match`, the builtins — so it runs
564
- wherever East runs, with no interpreter. In a body it is a call of the `Query`
565
- builtin, which carries the program as written beside its translation and gives
566
- the translation, so `to_python_source` prints it back as `East.jq(…)`. It is
567
- the twin of TypeScript's `East.jq` and builds the same IR. Dual-mode, like the
568
- standard library: in a body it builds the call; on values it compiles the
569
- translation (once per query and input type) and runs it now.
570
-
571
- ```python
572
- from east import East, ArrayType, FloatType, IntegerType, StructType, array
573
-
574
- Order = StructType([("id", IntegerType), ("total", FloatType)])
575
- orders = array(Order, [{"id": 1, "total": 250.0}, {"id": 2, "total": 1200.0}])
576
-
577
- # in a body: an expression of `result_type` (here an ArrayExpression), so its methods chain
578
- count_big = East.function([ArrayType(Order)], IntegerType,
579
- lambda b, orders: East.jq(orders, "[.[] | select(.total > 1000) | .id]",
580
- ArrayType(IntegerType)).size())
581
- count_big(orders) # 1
582
-
583
- East.jq(orders, "map(.total) | add", FloatType) # 1450.0 — on values, run now
584
-
585
- # several inputs: a dict, checked as an e3 root — `.orders` reads the `orders` input alone
586
- East.jq({"orders": orders, "limit": 1000.0},
587
- ".limit as $l | [.orders[] | select(.total > $l) | .id]", ArrayType(IntegerType)) # [2]
588
- ```
589
-
590
- | | |
591
- |---|---|
592
- | ❗ `result_type` | Required: the outputs' type for a query that gives one output (`length`, `[…]`, `reduce`), an `Option` of it for at most one (`first(f)`), an `Array` of it for any number (`.[] \| .id`). The query must check to exactly it, or the build raises `type_mismatch: the query gives Array<Float>, not the Integer it was given.` |
593
- | A query that does not check | `QueryError` when the program is built (a `QueryBuildError`, an `ExpressionError` too), its diagnostics each `jq <line>:<column>: <message>`: `unknown_field: .totl is not a field of Struct{id: Integer, total: Float}. Did you mean .total?` |
594
- | An error the query raises | `error(v)`, an integer `% 0`, a date that does not parse: an East runtime error located in the jq text (`jq 1:5`), which `b.try_` catches; on values, a `QueryError` with one `runtime` diagnostic |
595
- | The input's type | In a body, the expression's; on values, `type_of(value)`. Where that is narrower than the type meant (a variant's is its one case), run `evaluate_jq(program, value, input_type=T)` |
596
- | The language | jq 1.8 over East's types: Dict keys of any type, DateTime a type (an ISO literal compares with it), Integers exact, a variant read as `{type, value}`. Every departure is listed in `libs/east/devdocs/QUERY.md` §13 |
597
- | Host tools (`from east.query import …`) | `check_jq(program, input_type, root=False)`, without building: the program as written (`.program`, a `JqType` tree whose constants are typed East values; `None` when it does not check), the type it was checked against (`.input_type`), its `.element_type` and `.multiplicity` (`"one"` / `"maybe"` / `"many"`; `None` when it does not check), the datasets a root query reads (`.reads`) and its `.diagnostics`. A query's types are never stored: check the program against its data wherever they are needed · `evaluate_jq(program, value, input_type=T)`: a query whose input type is data · `translate_jq(checked)` · `parse_jq` / `print_jq` / `lex_jq` · `QueryError` |
598
-
599
- ### What a body may reference
600
-
601
- - Its own parameters; plain scalar constants (closure floats / ints / strings /
602
- datetimes bake in — the same value per element either way); East types and
603
- values (`east_null` included); the `East` namespace, `East.if_else` included;
604
- the `struct`/`variant`/`some`/`none` constructors (dual-mode: they build IR
605
- when handed expression fields); **East.function artifacts** (dual-mode: a
606
- call of their inline Function node at any nesting depth); **compiled East function values**
607
- (`.bind` results, `compile_from_*` functions — a CALL on one lowers to a native
608
- IR `Call`); and, two wrapper levels deep — enough for helper lambdas that
609
- compose a callback — other python functions that pass the same rules.
610
- - Anything else RAISES an `ExpressionError` **naming the binding**: a module
611
- (`random.…`, `np.…`), a python builtin (`len`, `str`), a mutable python
612
- capture, closure mutation (`nonlocal x; x += 1`). A closed-over East
613
- *collection* raises in an eager callback too — a capture snapshots, `.bind`
614
- stays live, and which you meant is your choice: an explicit `East.function`
615
- snapshots a side table, `.bind(table)` keeps it live. For genuine python
616
- semantics write an explicit python loop outside the body. Side effects are
617
- therefore never lost or silently doubled: the body runs once, at build time, or
618
- not at all.
619
- - Arithmetic follows East types exactly: no implicit Integer↔Float mixing
620
- (`.to_float()` / `.to_integer()` convert), and `/` is Float division.
621
- - Captures are CACHED: an eager callback whose code object, captured bindings
622
- and declared signature match an earlier call reuses its compiled function, so
623
- the per-group aggregate `group_to_arrays(key).to_array(lambda b, es, k: {…})`
624
- builds each inner lambda once, not once per group. ⚠️ A lambda whose CAPTURES
625
- change per call (`lambda b, r: r.v > g` in a python loop over `g`) rebuilds
626
- every call, since each capture bakes into a different function — hoist an
627
- `East.function(...)` and pass the varying value as a bound parameter.
628
-
629
- ### Performance — the levers, ranked
630
-
631
- The point of the machinery: **data stays in east-c and python never runs per
632
- element**.
633
-
634
- 1. **Keep values East end-to-end.** `EastArray`/`EastSet`/`EastDict` are
635
- C-backed, and every eager method runs its loop natively however deeply the
636
- element type nests. `list(...)`/`dict(...)` or a python loop pays a boxing
637
- crossing per element — convert at most once, at the very end.
638
- 2. **Let the callbacks build, and chain on the results** — a chain of
639
- `map`/`filter`/`reduce` stays native between steps. Return a dict literal
640
- (`lambda b, r: {"a": …, "b": …}`) to derive every column in ONE pass. **Bind
641
- shared work with `b.const`** — the builder emits exactly what the body
642
- spells, as TypeScript's does, so a python variable holding an expression is
643
- re-emitted (and re-evaluated) at every read: `fields = r.data.split("|")`
644
- read for 30 columns runs the split 30 times per row, and `table =
645
- derive(rec)` read inside a `.map` runs `derive` once per element. `fields =
646
- b.const(r.data.split("|"))` and `table = b.const(derive(rec))` run each
647
- once, as `$.const` does — the `no-reinlined-east-binding` rule (`east-py
648
- lint`, the LSP) points at every such read. There is no optimisation pass
649
- (#1312 removed the trace-time CSE: it moved reads across mutations it could
650
- not see), so what you write is what runs — a callback that pushes onto
651
- `order` and returns `order.size() - 1` reads the size after its push.
652
- 3. **Side tables: `b.const` a captured one, `bind` big ones** — opposite contracts:
653
- - **Capture (snapshot).** A collection captured by an `East.function` body
654
- is snapshot into its IR as a literal of its contents — exactly as a
655
- TypeScript literal is — and rebuilt wherever the body reads it: per call
656
- at the function's top level, per element inside a callback. **Bind it
657
- once per call with `t = b.const(TABLE)`** and read `t`. Right for small
658
- tables (up to a few thousand entries): the literal's size rides the IR,
659
- every call rebuilds it, and later mutations are **not** seen. Only an
660
- EXPLICIT build snapshots: an eager callback refuses a mutable capture.
661
- - **`East.function(...).bind(table)` (by reference, live).** Declare the
662
- table as a trailing parameter and pre-bind it: zero copy, O(1) at any
663
- size, nothing rebuilt per call, and the function **sees later
664
- mutations**. Rebinding gives independent callables, the unbound function
665
- stays usable, and a wrong-typed value raises `TypeError`.
666
-
667
- ```python
668
- fx = EastDict(StringType, FloatType, rates) # small table → capture
669
- @East.function([Row], FloatType) # (the EXPLICIT build
670
- def to_usd(b, r): # opts into the snapshot)
671
- t = b.const(fx) # the literal, once per call
672
- return r.amount * t.get_or_default(r.ccy, 1.0)
673
-
674
- TableT = DictType(StringType, FloatType) # huge table → bind
675
- conv = East.function([Row, TableT], FloatType,
676
- lambda b, r, t: r.amount * t.get_or_default(r.ccy, 1.0))
677
- rows.map(conv.bind(big_table)) # loop + lookup stay in east-c
678
- conv.bind(t1, t2) # multi-table: binds the TRAILING parameters in order
679
- ```
680
-
681
- Function *parameters* always cross by reference (zero copy, any size, any
682
- depth); `bind` is what lets an eager method use a parameter-taking function
683
- where the callback signature is fixed.
684
- 4. **Hoist `East.function(...)` out of python loops** and reuse it — building
685
- is cheap, not free.
686
- 5. **When logic must stay python, go columnar** — one crossing per column or
687
- batch instead of per row × field, then back with `from_columns`/`extend`
688
- ([Columnar escape hatches](#columnar-escape-hatches--when-the-logic-must-stay-python)).
689
- 6. **See how a hot call ran** — `east.runtime.compiler.eager_stats()`:
690
- `function_direct` counts callbacks that rode a precompiled function value
691
- straight in, `c_to_py_decodes` counts values boxed C→python (an eager method
692
- quietly decoding a whole collection shows here), and the `beast2_*` counters
693
- report column projection. There is no per-element python counter because
694
- there is no per-element python path.
695
-
696
- ### IR ↔ python: `east-py transpile` and the `east-c ir` toolbox
697
-
698
- Any East IR — an `East.function` artifact, or a `.json` / `.beast2` export from
699
- TypeScript, east-c or east-node — prints as an idiomatic python module that
700
- REBUILDS it through the surface above:
701
-
702
- ```python
703
- from east.codegen import to_python_source
704
- print(to_python_source(classify)) # `@East.function(...)` / `def main(b, …)`
705
- ```
706
-
707
- ```bash
708
- east-py transpile program.json -o program.py --name main # the same, from a file
709
- east-py transpile program.json -p east-py-std -p east-py-io # platform calls print as the packages' own
710
- # functions and named types, not restated declarations
711
- east-c ir normalize program.json -o canonical.json # the canonical form
712
- east-c ir diff a.json b.beast2 # first difference, or "identical"
713
- east-c ir convert program.json -o program.beast2 # json <-> beast2, source map intact
714
- ```
715
-
716
- - **The contract:** `build(print(IR))` equals `IR` under `east-c ir normalize`
717
- (location ids stripped, variables and labels renamed in the TypeScript
718
- lowering's order, captures recomputed, recursive type ids renumbered — one
719
- normalizer, in libeast-c). The TypeScript printer (`East.toSource`, `east-node
720
- transpile`) is its twin: IR → python → IR → TypeScript → IR is equal at every
721
- leg, over the whole compliance corpus and every example
722
- (`docs/conventions/EAST_CODEGEN.md`).
723
- - **Builtins print through one spelling table** (`east.codegen.spellings`):
724
- operators only where they are exact — `+ - *` on numbers, `/` on Floats, the
725
- comparisons, `& | ^ ~` on Booleans, `+` on Strings — named `East.<Type>.*` or
726
- method spellings elsewhere, and `East.builtin(...)` for `RAW_ONLY`.
727
- - **A function used as a VALUE** (`$.let(East.DateTime.roundDownWeek)`) prints
728
- as the inline `@East.function` it is — the IR carries the body, not the name.
729
- - **A platform call** prints as a hoisted `East.platform(...)` declaration — or,
730
- given the implementing packages
731
- (`to_python_source(ir, providers=providers_for(["east_py_std"]))`, `east-py
732
- transpile -p east-py-std`), as `from east_py_std import fs_read_file` and the
733
- call itself, with a type the package names (`GzipOptionsType`) printed by its
734
- name; the plugin's example index is rendered this way. A provider is used only
735
- where its declared signature IS the node's, so a drifted signature keeps the
736
- declaration rather than printing a wrong import. A package whose function is
737
- implemented in C exports the DECLARATION under the same name
738
- (`simulation_run`, `optimization_iterative`), which a body calls identically.
739
-
740
- ### Diagnostics at edit time — `east-py lint`, `east-py check`, flake8, `east-py lsp`
741
-
742
- Everything the build refuses — a body without the block, `//` on an
743
- expression, an f-string over one, `if` on one, a callback reaching for `np` — is
744
- also a **rule** in `east.diagnostics` (the python twin of
745
- `@elaraai/east-diagnostics`). The rules read a file's `ast`, find its East bodies
746
- (an `East.function` body and everything nested in it, an eager callback on an
747
- East value, a `@East.platform_function`'s East inputs), and say at edit time what
748
- the build would say: every rule's message IS the build's refusal for the same
749
- code, pinned by building the very source the rules read.
750
-
751
- Two tiers, at two costs. The **rules** read the `ast` — instant, and useful even
752
- on a file that does not parse. The **check** runs the build — the only type
753
- checker a python East body has, since East's type checker IS the builder — at
754
- the cost of importing the module.
755
-
756
- ```bash
757
- east-py lint src/ # file:line:col: category [rule] message — exit 1 on any finding
758
- east-py lint src/ --format json # the findings as records
759
- east-py lint src/ --disable no-deprecated-alias --exclude fixtures
760
- east-py lint --list-rules # every rule, with its EAS code
761
- east-py check src/ # the BUILD's errors — a file, a directory, or a module; --format json
762
- flake8 --select EAS src/ # the same rules inside flake8 (east-py-cli registers the plugin)
763
- east-py lsp # a Language Server over stdio, both tiers (--probe: can it start here?)
764
- # pylsp # python-lsp-server runs the rules too (east-py-cli registers the plugin)
765
- ```
766
-
767
- - **`east-py check`** imports each target — a `.py` file, a directory (walked
768
- as `lint` walks it) or a dotted module name, each executed afresh — builds
769
- every East function in it, and reports EVERY failure at its authoring line: a
770
- body whose type differs from the declared `out`, a callback the capture
771
- refuses, an `IRAnalysisError`. Importing runs the module, so `EAST_CHECK=1` is
772
- set for the duration — skip import-time work when you see it.
773
- `--only-if-enabled` honours the project's opt-in (below), as the plugin's read
774
- hook does, so the hook and the language server agree.
775
- - **`east-py lsp`** (needs `pygls`) runs the rules on every change and the build
776
- tier on **open** (debounced) and **save** (at once), on one long-lived worker
777
- thread, and only when the project opts in: the build reads the module from
778
- DISK, so on an unsaved buffer it would report the saved version's errors. A
779
- save evicts the module from the warm process, so its importers are checked
780
- against the new version. Pyright/Pylance has no plugin API, so flake8 and
781
- pylsp are the editor paths that need no East-specific server; the Codex
782
- plugin launches the project's own `east-py lsp` for `.py` files, and falls back
783
- to `east-py lint` per change when that server cannot start.
784
-
785
- The rules — `east-py lint --list-rules` prints them:
786
- | Rule | Flags | What the build says |
787
- |---|---|---|
788
- | `body-takes-block-first` (EAS001) | `lambda x: …`, a body whose parameter count is not the declared count plus the block, `b.price`, `x + b` | `a body takes the block first` |
789
- | `no-operator-fork` (EAS002) | `//` `%` `**` on an expression, `xs[-1]` | the operator-fork texts — `East.Integer.divide` / `remainder` / `pow`, `xs.get(xs.size() - 1)` |
790
- | `no-python-formatting` (EAS003) | `f"{x}"`, `str(x)`, `print(x)`, `format(x)`, `"{}".format(x)`, `"%d" % x` | `f-strings / str() cannot be traced … East.String.print(T, value)` |
791
- | `no-python-boolean` (EAS004) | `if`/`while`/`assert`/`and`/`or`/`not`/`x if c else y` on an expression, `in`, `for … in xs`, `len`/`int`/`float`/`bool`/`sum`/`sorted`/`max`… over one | `python \`if/and/or/not\` cannot be traced …` (`iteration`, `len()`, …) |
792
- | `no-python-round` (EAS005) | `round(x)` on a Float expression | `East.Float.round_half(x) rounds half away from zero …` |
793
- | `no-python-work` (EAS006) | an EAGER callback loading a module (`np`, `math`), a python builtin the capture does not admit (all but `abs`/`bool`/`isinstance`), a name imported from the standard library or an installed package that has no East form (`from math import floor`; a constant like `pi` lifts), a mutable East collection, or a `def` doing any of those (a clean macro `def` is fine) | `the callback cannot be captured automatically: it references np …` |
794
- | `no-statement-on-outer-block` (EAS007) | `b.if_(p, lambda _b: b.assign(…))` — a statement on an enclosing body's block | `b.assign() was called on an OUTER block …` |
795
- | `no-deprecated-alias` (EAS008, warning) | `.fold`, `.lower`, `East.Boolean.and_`, … (read off the surface's own deprecation docstrings) | `.fold() is deprecated: the spelling is .reduce() (the TypeScript name)` |
796
- | `no-discarded-expression` (EAS009) | a bare `acc.push_last(x)` / `East.error(…)` / `xs.size()` line in a body | `.push_last() was evaluated and thrown away … b.do(…)` |
797
- | `prefer-some-none` (EAS010, warning) | `variant("some", x)` / `variant("none", None)` | use `some(value)` / `none` |
798
- | `no-handrolled-variant` (EAS011, warning) | a `{"type": …, "value": …}` dict | `variant("Tag", value, Type)` — the encoder needs what it constructs |
799
- | `prefer-explicit-east-type` (EAS012) | `b.let([])` / `b.let([1, 2])` / `b.const(list())` — a bare python list, set or tuple, empty or not | `cannot lift python value of type list …` — pass the East type: `b.let([], ArrayType(IntegerType))`, or build it with `East.new_array` |
800
- | `no-let-const-in-expression` (EAS013, warning) | `b.let` / `b.const` buried in a call argument, an element, a chain target (a tuple of declarations is fine) | give the declaration its own statement |
801
- | `no-untracked-east-data` (EAS014, suggestion) | a plain python list/dict local reaching an expression's method | `b.const(rows, Type)` — a local carries no East type and re-inlines |
802
- | `no-redundant-east-cast` (EAS015, warning) | `b.let(East.value(x, T), T)` | pass the value and type to `b.let` directly |
803
- | `prefer-let-const-over-east-value` (EAS016, suggestion) | `East.value(...)` assigned or returned inside a body | `b.const(value, Type)` — `East.value` erases the type at the binding |
804
- | `no-host-comparison-on-east-values` (EAS017) | `<` / `>` on a decoded VARIANT or OPTION outside a body — it raises `TypeError` (`==` is structural and fine; a decoded Integer is an `int`) | `compare_for(T)` / `less_for(T)`, `make_east_key(T)` for `sorted` |
805
- | `no-module-scope-east-macro` (EAS018, warning) | a module-scope helper a body calls to build IR, or to assemble a composite `f"{a}|{b}"` key | make it an `East.function`; model typed / nested East data |
806
- | `no-build-time-clock` (EAS019, warning) | `datetime.now()` / `time.time()` at module scope (a script's `if __name__ == "__main__":` is not module scope) | author the constant, or read the clock at runtime — a platform function, or `east_py_std`'s `time_now()` in a body |
807
- | `no-inline-credentials` (EAS020, warning) | a literal `password` / `token` / `secret_access_key` (a localhost sibling host, a header NAME like `X-Api-Key`, and a `${TEMPLATE}` are exempt) | read it at runtime: `east_py_std`'s `env_get("VAR")` in a body, `os.environ` in a platform function — IR is content-addressed and replicated |
808
- | `no-compile-time-data-injection` (EAS021, warning) | `open()` / `json.load` / `Path.read_text` / `os.environ` at module import, when the result REACHES East (a body reads it, or a boundary call takes it) | load at runtime: an e3 input, a dataset, a platform function |
809
- | `no-python-east-data` (EAS022, warning) | East rows assembled by a module-scope comprehension or loop, then handed to a body | write the rows out, or produce them at runtime |
810
- | `no-python-string-building` (EAS023, warning) | an f-string assembling an East string CONSTANT — a regex, a template, a key | spell the constant out |
811
- | `no-derived-struct-fields` (EAS024, warning) | `Derived = StructType([… for f in Other.value])` | a type declaration is a wire format — spell the fields |
812
- | `no-python-data-work` (EAS025, warning) | a python helper doing parse / strip / null-check / coerce work on an EXPRESSION a body hands it | express it in East, where it runs on every row |
813
- | `no-reinlined-east-binding` (EAS026, warning) | a python local holding an EXPRESSION (`n = xs.size()`, not a `b.let`/`b.const` binding) read more than once in one body, or read inside a per-element callback, a loop body or a nested function below the body that bound it | each read re-emits — and re-evaluates — the expression: bind it once, `n = b.const(xs.size())`, where it is needed (the TypeScript rule of the same name) |
814
-
815
- Configure it once in the project's `pyproject.toml`; every surface reads it, and
816
- `east-py lint`'s own flags add to it:
817
-
818
- ```toml
819
- [tool.east-py]
820
- check = true # let an EDITOR run the build tier (off by default)
821
- disable = ["no-deprecated-alias"] # rules to skip
822
- exclude = ["fixtures", "vendor"] # extra directory names not to walk
823
- ```
824
-
825
- `check` is off unless a project asks: the rules READ a file, the build RUNS it,
826
- and an editor should not import someone's modules on save because a language
827
- server happened to be installed. `east-py check` on the command line is consent
828
- in itself and ignores the setting. Unreadable configuration falls back to the
829
- defaults — a diagnostics tool must never stop a project building.
830
-
831
- A file that does not import `east` is never diagnosed; a line ending in `# noqa`
832
- (or `# noqa: EAS002` / `# noqa: no-operator-fork`) is skipped; `.venv`, `venv`,
833
- `node_modules`, `dist`, `build`, `.git`, `__pycache__`, `tests` and `test` are not
834
- walked. The rules are syntactic, so a clean `lint` is necessary, not sufficient —
835
- the type errors live behind `east-py check`. The rules are disjoint (each
836
- mistake is one rule's to report).
837
-
838
- ### Cross-language functions: `east-py export-functions` and `East.import_function`
839
-
840
- A python `East.function` is *called* from a TypeScript e3 task — and a
841
- TypeScript one from python — as pure IR: no python at run time, no platform
842
- bridge. The functions travel as a **function manifest** (each function's IR,
843
- declared type and platform dependencies with the package providing each);
844
- the importer names the function and declares its type; linking checks the
845
- type exactly and embeds the IR. Declare them on the package's root module:
846
-
847
- ```python
848
- # packages/pricing/src/pricing/__init__.py — the package's root module
849
- from east import East
850
- from east.types.types import FloatType, StructType, IntegerType
851
-
852
- Row = StructType([("qty", IntegerType), ("price", FloatType)])
853
- score = East.function([Row], FloatType, lambda b, r: r.qty.to_float() * r.price)
854
- east_functions = {"score": score} # name -> East.function artifact. CLOSED values: the
855
- # exported IR must BIND every name it reads. A captured
856
- # lookup table or a stdlib format string is a literal in
857
- # the body and closes over nothing; a .bind result (no IR
858
- # of its own) and an unresolved import are refused, and a
859
- # genuinely free variable is refused BY NAME
860
- ```
861
-
862
- ```typescript
863
- // the e3 task, in TypeScript (see the e3 skill) — the reference is all there is to write:
864
- // e3.export finds `pricing` in the uv workspace, exports it (east-py export-functions, in the
865
- // project's .venv) and links; the providers come from the task's runner
866
- const score = East.importFunction("pricing", "score", FunctionType([RowType], FloatType));
867
- await e3.export(pkg, "out.zip");
868
- ```
869
-
870
- e3 exports each workspace package in one `east-py export-functions --batch`
871
- process, however many tasks import from it, and keeps the manifests, so a
872
- redeploy whose python has not changed runs no export (the **e3** skill). A
873
- function's manifest entry is its own: the same bytes whatever else the process
874
- built first and whatever directory it ran in, so a task embedding it keeps its
875
- hash until the function, or a line it was written at, changes.
876
-
877
- For a package built elsewhere (published, another repo), or to link
878
- in-process, write the manifest where the package lives and pass it:
879
-
880
- ```bash
881
- east-py export-functions pricing -o pricing.functions.beast2 -p east-py-std
882
- # -p names the platform package implementing each platform call the functions make;
883
- # a call no package provides fails the export (the manifest records the provider)
884
- # → e3.export(pkg, "out.zip", { functions: ["./pricing.functions.beast2"] })
885
- ```
886
-
887
- The other direction, in python:
888
-
889
- ```python
890
- manifest = East.decode_function_manifest(Path("maths.functions.beast2").read_bytes()) # east-node export-functions wrote it
891
- double = East.import_function("maths", "double", FunctionType([IntegerType], IntegerType))
892
- user = East.function([IntegerType], IntegerType, lambda b, x: double(x) + 1) # callable in a body
893
- ir, imports = East.link_imports(user, [manifest]) # exact type check; the IR embedded as a Let
894
- compile_from_value(ir, [])(20) # 41 — pure IR, runs on any runner
895
- ```
896
-
897
- Unlinked, the reference is a `Platform` node named `east.importFunction`, and
898
- compiling it raises naming that platform. `East.export_functions(pkg, version,
899
- {…}, providers)` and `East.encode_function_manifest` are the API behind the CLI,
900
- and `East.platform_dependencies(fn)` lists the platform functions an artifact
901
- calls, in first-use order (unresolved imports skipped). Each also answers to its
902
- TypeScript camelCase name (`East.exportFunctions`, `East.linkImports`, …). The
903
- contract and the runner rules: `docs/conventions/EAST_CODEGEN.md`.
904
-
905
235
  ## East values — the eager runtime
906
236
 
907
237
  ### Work in East values — don't round-trip through Python
@@ -936,7 +266,7 @@ changes both cost and correctness:
936
266
  (`arr.filter(...).to_dict(...).map(...)`) rather than binding intermediates to
937
267
  python names and re-wrapping them. Cross to python only at the edges — a
938
268
  scalar for `East.Float.*` math, a buffer via `to_numpy()`/`to_torch()`.
939
- - **Scalars and dates** — see [Scalars](#scalars-use-the-easttype-utilities-for-consistency--above-all-string--datetime):
269
+ - **Scalars and dates** — see [Scalars](reference/patterns.md#scalars-use-the-easttype-utilities-for-consistency--above-all-string--datetime):
940
270
  the `East.<Type>` utilities for anything whose semantics diverge, and ALWAYS
941
271
  `East.DateTime.*` for dates.
942
272
 
@@ -1031,972 +361,23 @@ callback-free operations (`sort`, `unique`, `union`/`intersection`/`difference`,
1031
361
  loop in east-c. The TypeScript compliance corpus is replayed through these eager
1032
362
  methods, builtin by builtin, and must give the compiled programs' answers.
1033
363
 
1034
- ### Construction & validation (`from east import ...`)
1035
-
1036
- | Signature | Description | Example |
1037
- |-----------|-------------|---------|
1038
- | **Ergonomic constructors** |
1039
- | `array(element_type, items, *, validate=True) -> EastArray` | Each item coerced/validated to `element_type` (dict→struct, int→Float, …); `validate=False` stores as-is | `array(LineItem, [{"name":"a","price":1}])` |
1040
- | `struct(fields: dict, typ: StructType\|None=None) -> EastStruct` | Reorders/coerces keys to `typ` (else infers from fields); dual-mode — a field holding an expression (at any depth) builds Struct IR instead | `struct({"price":1,"name":"a"}, LineItem)` |
1041
- | `variant(case: str, value, typ: VariantType\|None=None) -> EastVariant` | Tagged value; validates `value` against case `case` (matched **by name**); read back via `.type`/`.value`; dual-mode — an expression payload builds Variant IR, typed by `typ` or by context | `variant("named", "red", Color)` |
1042
- | `some(value) -> EastVariant` / `none` | Option `some`; `none` is a **constant**, not a function | `some(5)` / `none` |
1043
- | `match(v, cases: dict, default=None)` | Dispatch on `v.type`; the handler is a body, **always** called `handler(b, v.value)` — the `none` arm is `lambda b, v: …`, not `lambda: …`; `default` is a `default(b)` body for the cases without a handler (a plain value is returned as is) | `match(o, {"some": lambda b, x: x, "none": lambda b, v: -1})` |
1044
- | `east_ref(value) -> EastRef` | Make a mutable ref cell (same as `EastRef(value)`) | `east_ref(0)` |
1045
- | **Validation / coercion** — raise `EastTypeError` (`expected X, got Y (at $.path)`) |
1046
- | `coerce_to(value, typ, *, path="$") -> EastValue` | Canonicalize any Python value to a bridge-ready East value, type-driven; a 1-D numpy array fills `Array<Float/Integer/Boolean>` C-side in one bulk crossing (hand struct fields numpy columns directly) | `coerce_to({"qty": np_i64}, InputType)` |
1047
- | `assert_value_of(value, typ, *, path="$") -> value` ❗ | Validate; return value, or raise path-pinpointed `EastTypeError` on first mismatch | `assert_value_of(s, LineItem)` |
1048
- | `explain_value_of(value, typ) -> list[(path, reason)]` | Every mismatch; `[]` == conforms | `explain_value_of(s, LineItem)` |
1049
- | `is_value_of(value, typ) -> bool` | Boolean conformance check | `is_value_of(items, ArrayType(LineItem))` |
1050
- | `type_of(value) -> EastType` | Infer the East type of a value | `type_of(items)` |
1051
-
1052
- Container constructors are also direct: `EastArray(elem, items=None)`, `EastSet(elem, items=None)`,
1053
- `EastDict(key, value, items=None)`, `EastVector(elem, data=None, length=0)`,
1054
- `EastMatrix(elem, data=None, rows=0, cols=0)`, `EastRef(value)`.
1055
-
1056
- ### A JSON contract for a type — `json_schema_for` / `type_from_json_schema`
1057
-
1058
- `East.String.print_json` writes **East JSON**, lossless and self-describing.
1059
- `json_schema_for(T)` emits the JSON Schema of exactly that encoding, so another
1060
- system can validate a payload before sending it; `type_from_json_schema` goes the
1061
- other way. Host-side functions, like `compare_for` — no expression, no IR:
1062
-
1063
- ```python
1064
- import json
1065
- from pathlib import Path
1066
-
1067
- from east import ArrayType, IntegerType, StringType, StructType
1068
- from east.serialization import json_schema_for, type_from_json_schema
1069
-
1070
- Reading = StructType([("sensor", StringType), ("litres", IntegerType)])
1071
- Path("contract.schema.json").write_text(
1072
- json.dumps(json_schema_for(ArrayType(Reading), draft="draft-07"), indent=2))
1073
-
1074
- T = type_from_json_schema(json.loads(Path("partner.schema.json").read_text()))
1075
- ```
1076
-
1077
- | Signature | Notes |
1078
- |-----------|-------|
1079
- | `json_schema_for(typ, draft="2020-12") -> JsonSchema` | `draft`: `"2020-12"`, `"draft-07"` or `"openapi-3.0"`, as a consumer's validator pins. `Never`, `Function` and `AsyncFunction` raise `TypeError` naming what has no JSON form |
1080
- | `type_from_json_schema(schema) -> EastType` ❗ | Raises `JsonSchemaUnsupportedError` — its `.pointer` the RFC 6901 location, also in the message — on what East cannot express: `allOf`, `not`, `if`/`then`/`else`, `anyOf`, `patternProperties`, `prefixItems`, a union of more than one type, an open record, an optional property, an untagged `oneOf`, a non-local `$ref`, a cycle of definitions no single definition breaks |
1081
- | `EAST_JSON_PATTERNS` | The lexical forms of East JSON's scalars — `.integer` `.blob` `.float_specials` (TypeScript: `floatSpecials`) — so a reader enforces exactly what the schema describes. A DateTime has none: its form is `format: "date-time"` |
1082
-
1083
- - **An `Option<T>` is `null` or `T`'s own encoding** wherever `T` can never
1084
- encode as `null`: `none` → `null`, `some(7)` → `"7"`, a struct field
1085
- `"note": null` or `"note": "x"` (an absent key is still an error). Only
1086
- `Option<Null>` and `Option<Option<T>>` keep the tagged `{"type": …, "value":
1087
- …}` object, which keeps `some(none)` apart from `none`; a recursive payload is
1088
- judged by what it wraps. The rule is a total function of the type, applied by
1089
- every codec and reader and by `json_schema_for` (a flat Option is `oneOf
1090
- [null, T]` annotated `x-east-type: "Option"`), and the tagged object under a
1091
- flat Option is refused by the payload's own decoder (`expected string, got
1092
- {"type":"none","value":null}`).
1093
- - **A `DateTime` is `format: "date-time"`**: every decoder, on every runtime,
1094
- reads any RFC 3339 date-time — `Z` or any offset (the instant is UTC), `t`/`z`
1095
- in either case, any fractional digits (past the millisecond dropped, never
1096
- rounded, so `isoformat()`'s microseconds read to the millisecond), a leap
1097
- second as the Unix time its fields add up to — in years 0001–9999, the range
1098
- every runtime holds. `print_json` writes UTC, three fractional digits, `+00:00`.
1099
- - **Other scalars are described as the encoder emits them**: an `Integer` is a
1100
- quoted i64 decimal, a `Blob` lowercase hex (`parse_json` also takes uppercase,
1101
- which the contract does not), so a producer that validates cannot send what a
1102
- strict reader rejects. Every pattern spells digits `[0-9]` — python's `\d`
1103
- matches any Unicode digit — so validators here and in JavaScript agree.
1104
- - **The document is deterministic and byte-identical to TypeScript's
1105
- `jsonSchemaFor`** (key order, `$defs` names in first-encounter order, variant
1106
- case order). Its `x-east-type` annotations make the inverse exact — `Set` vs
1107
- `Array`, `Dict` vs an array of pairs, a flat Option vs any other `oneOf`. A
1108
- foreign schema without them converts under a structural mapping that does not
1109
- promise to round-trip: `format: "date-time"` reads as a `DateTime` (any other
1110
- `format` is a `String`), and `nullable: true`, `["string", "null"]` and a
1111
- `oneOf` of null and one schema read as an Option.
1112
- - **Recursion binds one `recursive_type` per cycle group**: definitions that
1113
- reference each other convert when every cycle passes through one definition,
1114
- the binder; three that each reference the other two need two binders and are
1115
- refused, naming them.
1116
- - To READ a document larger than memory under this contract: `json_open` /
1117
- `json_next` in **east-py-std**.
1118
-
1119
- ### EastArray — complete method surface
1120
-
1121
- Eager; results are live east-c-backed values that chain. `arr[i]`, `len(arr)`, `for x in arr`
1122
- work via the sequence protocol. Callback methods accept a python body — the block first, then
1123
- the arguments the table lists, so `fn(el)` is written `lambda b, el: …` — (built into a native
1124
- function) or a precompiled `East.function(...)` (a value: it takes no block); anything East cannot
1125
- express raises with the binding named. Result types come from the build, so `out`/`element_type` is optional — pass it
1126
- to PIN a type (a widening map, an Option payload) and a contradicting precompiled function raises
1127
- at the call. `.element_type` is the logical element type.
1128
-
1129
- | Group | Methods |
1130
- |-------|----------------------|
1131
- | Access | `get(i[, default_fn(i)])` (errors when out of bounds — `arr[i]` is the pythonic read; TS `get(index, onMissing?)`) · `at(i)` · `get_or_default(i, default)` · `try_get(i) -> some/none` · `has(i)` · `size()`/`length()` · `get_keys(indices: EastArray)` |
1132
- | Reorder (new array) | `sort(by=None, *, reverse=False)` (TS `sort`; `sorted` is the deprecated spelling) · `reverse()` (`reversed` deprecated) |
1133
- | Slice & combine | `slice(start, end)` · `concat(other)` · `copy()` |
1134
- | Per-element | `map(fn(el), out=None)` · `filter(pred(el))` · `filter_map(fn(el)->some/none, out=None)` · `for_each(fn(el)) -> None` — every element callback may also declare `(el, idx)` to receive the builtin's index |
1135
- | Reduce | `reduce(fn(acc, el), init)` (TS order; `fold(init, fn)` is the deprecated spelling) · `scan(fn(acc, el), init) -> Array` (running fold: one accumulator per element, seed not emitted, `scan(...)[n-1] == reduce(...)`) · `map_reduce(map_fn(el), reduce_fn(acc, m), out=None)` · `sum(fn=None)` · `mean(fn=None) -> float` (NaN when empty) · `maximum(by=None)` ❗empty · `minimum(by=None)` ❗empty (the ELEMENT — of `by(el)` when given — a tie keeps the first, like TS) · `every(pred=None) -> bool` · `some(pred=None) -> bool` (native short-circuit) |
1136
- | Group & index | `group_by(key(el)) -> Dict` · `group_reduce(key, init(gk), fold(acc, el)) -> Dict` · `group_size(key=None)` · `group_sum(key, fn=None)` · `group_mean(key, fn=None)` · `group_maximum/group_minimum(key, by=None)` (Dict of the ELEMENT per group, like TS — `group_find_maximum` is the index finder) · `group_every/group_some(key, pred)` · `group_find_all(key, value, by=None) -> Dict<K, Array<Integer>>` / `group_find_first(key, value, by=None) -> Dict<K, some/none>` (GLOBAL row indices; every group appears, so a group with no match maps to `[]` / `none`) · `group_find_maximum/group_find_minimum(key, by=None) -> Dict<K, Integer>` (the INDEX per group; a tie keeps the earliest) · `group_to_arrays(key, value=None)` · `group_to_sets(key, value=None)` · `group_to_dicts(key, key2, value=None, combine=None)` · `to_dict(key(el), value=None, combine=None) -> Dict` (duplicate key errors without `combine`) · `to_set(key=None) -> Set` · `unique() -> Set` |
1137
- | Search | `find_first(target, key=None) -> some/none` · `find_all(value, by=None) -> Array<Integer>` · `find_maximum/find_minimum(by=None) -> some(index)/none` · `find_sorted_first/last(target, key=None) -> int` · `find_sorted_range(target, key=None) -> {start,end}` · `first_map(fn(el)->some/none, out=None)` · `is_sorted(key=None) -> bool` |
1138
- | Flatten | `flat_map(fn(el)->arr, out=None)` (`flatten_to_array` deprecated) · `flatten_to_set(fn(el)->set, out=None)` · `flatten_to_dict(fn(el)->dict, combine=None)` (a duplicate key errors without `combine`) |
1139
- | Columnar | `to_columns(fields=None) -> dict` (numpy per numeric/bool column, `Option<Float>`→NaN, interned strings) · `EastArray.from_columns(element_type, columns)` *(static)* (C-side fill needs numpy columns — float64/int64/bool, `Option<Float>` as float64+NaN; python lists convert per cell) · `map_batches(fn(cols)->cols, out=None, batch_size=100_000)` |
1140
- | Convert | `to_tree(node, key(el[, i]), parent(el[, i])->some/none, build(el, i, children)) -> Array<node>` (flat parent-keyed rows → nested nodes of any depth, iterative — [Trees from flat rows](#trees-from-flat-rows-and-recursion)) · `flatten_tree(children(el[, i])) -> Array<{path, node}>` (nested nodes back: every node with its path of indices, depth first, iterative) · `string_join(sep) -> str` (String arrays) · `to_vector() -> EastVector` (Float/Integer/Boolean elements, in order — east-c VectorFromArray; the expression method emits the same builtin) · `encode_csv(config=None, **options) -> EastBlob` (Array of structs; east-c ArrayEncodeCsv) |
1141
- | Mutate (in place, the TS names) | `push_last(item)` · `push_first(item)` · `append(array)` (ArrayAppend — a whole array, NOT one element) · `prepend(array)` · `pop_last()` · `pop_first()` · `update(i, item)` ❗bounds · `merge(i, value, update_fn(existing, incoming[, i]))` · `merge_all(array, merge_fn(existing, incoming[, i]))` · `clear()` · `sort_in_place(by=None)` · `reverse_in_place()` — plus the python protocol: `extend(iterable)` (bulk: one crossing; C-to-C for same-type East arrays, raw buffers for numpy) · `insert(i, item)` · `pop(i=-1)` · `remove(item)` · `count(value) -> int` · `index(value) -> int` |
1142
-
1143
- ### EastSet — complete method surface
1144
-
1145
- Mutable, unique, **East-sorted**. `.element_type` is the element type; iteration / `to_array` /
1146
- `reduce` visit in East order. Every callback takes the block first (`lambda b, el: …`); the
1147
- signatures below list the arguments after it. **`map` and `filter_map` return an `EastDict`** keyed by the element
1148
- (Set→Set is `to_set`).
1149
-
1150
- | Group | Methods |
1151
- |-------|---------|
1152
- | Access | `len(s)` · `value in s` · `has(value)` · `for el in s` |
1153
- | Algebra (vs another set) | `union(other)` · `intersection(other)` · `difference(other)` · `symmetric_difference(other)` · `is_subset_of(other) -> bool` · `is_superset_of(other) -> bool` · `is_disjoint_from(other) -> bool` (the TS names; `intersect`/`diff`/`sym_diff`/`is_subset`/`is_disjoint` are deprecated spellings) |
1154
- | Per-element | `map(fn(el)) -> Dict` · `filter(pred(el))` · `filter_map(fn(el)->some/none, out=None) -> Dict` · `first_map(fn(el)->some/none, out=None)` · `to_set(fn(el), out=None)` · `to_array(key=None)` · `to_dict(key(el), value(el), combine=None)` (duplicate key errors without `combine`) · `to_tree(node, parent(el)->some/none, build(el, children)) -> Array<node>` (each element its own key) · `for_each(fn(el)) -> None` |
1155
- | Reduce | `reduce(fn(acc, el), init)` (TS order) · `scan(fn(acc, el), init) -> Array` (running fold in East order) · `map_reduce(fn(el), reduce(a,b))` (raises on empty) · `sum(fn=None)` · `mean(fn=None) -> float` · `every(pred=None)` · `some(pred=None)` (native short-circuit) |
1156
- | Group | `group_reduce(key(el), initial(gk), fold(acc, el)) -> Dict` · `group_size(key)` · `group_sum(key, fn=None)` · `group_mean(key, fn=None)` · `group_every/group_some(key, pred)` · `group_to_arrays/group_to_sets(key, value=None)` · `group_to_dicts(key, key2, value=None, combine=None)` · ⚠️ `group_fold(...)` is the DEPRECATED alias of `group_reduce` |
1157
- | Flatten | `flatten_to_array(fn(el)->arr, out=…)` (the TS Set name; `flat_map` is Array's and a deprecated spelling here) · `flatten_to_set(fn(el)->set, out=…)` · `flatten_to_dict(fn(el)->dict, combine=None)` (duplicate key errors without `combine`) |
1158
- | Mutate (in place) | `add(item)` · `insert(value)` (errors if present) · `try_insert(value) -> bool` (True if newly added) · `remove(item)` · `delete(value)` (errors if absent) · `try_delete(value) -> bool` · `discard(item)` · `union_in_place(other)` (adds all of `other`) · `clear()` · `copy()` |
1159
-
1160
- ### EastDict — complete method surface
1161
-
1162
- Mutable, **East-sorted by key**. `.key_type` / `.value_type`. **Every Dict callback takes
1163
- the builtin's own `(value, key)` order — the TypeScript order** (the block first, as always):
1164
- `fn(b, value)` when the key is not needed, `fn(b, value, key)` when it is — for `map`,
1165
- `filter`, `first_map`, `to_*`, `flatten_to_*`, `map_reduce`, the projections of
1166
- `every`/`some`/`sum`/`mean` and `group_*`'s key/value projections alike; a fold step
1167
- (`reduce`/`scan`/`group_reduce`) is `fn(b, acc, value[, key])`; collision `combine` is
1168
- `combine(existing, incoming, key)` — for `union`/`union_in_place`/`to_dict` a 2-arg
1169
- `combine(existing, incoming)` is also accepted (a 3-arg one still receives the key).
1170
- `union`/`union_in_place` require `other` to have the same key AND value type (a
1171
- mismatch is refused); `merge_all` needs only the same KEYS and is generic
1172
- in `other`'s values; `merge` takes a single differently-typed value.
1173
-
1174
- | Group | Methods |
1175
- |-------|---------|
1176
- | Access | `d[k]` · `k in d` · `has(k)` · `len(d)`/`size()` · `get(k)` ❗missing · `get(k, default)` (a value) / `get(k, fn(k))` (TS `onMissing`) · `get_or_default(k, default)` · `try_get(k) -> some/none` · `keys() -> Set` (TS; east-c DictKeys — `keys_set` deprecated) · `values()`/`items()` (python views) |
1177
- | Combine | `union(other, combine=None) -> Dict` (NEW dict, both inputs untouched; a shared key errors without `combine`) · `union_in_place(other, combine=None) -> None` · `merge_all(other, merge(existing, incoming[, key]), default(key)) -> None` (fold `other` into self in place; `default` seeds absent keys; **generic in `other`'s value type** — only the KEYS must match) · `merge(key, value, update_fn(existing, incoming[, key]), initial_fn=None) -> None` (ONE key, in place; `value` may be a different type — TS `merge`; `merge_key` deprecated) · `get_keys(keys: Set, fill(k)) -> Dict` |
1178
- | Per-entry | `map(fn(value[, key]), out=None)` · `filter(pred(value[, key]))` · `filter_map(fn(value[, key])->some/none, out=None)` · `first_map(fn(value[, key])->some/none, out=None)` · `for_each(fn(value[, key])) -> None` |
1179
- | Reduce | `reduce(fn(acc, value[, key]), init)` (TS order) · `scan(fn(acc, value[, key]), init) -> Array` (running fold in key order) · `map_reduce(map_fn(value[, key]), reduce_fn(a, b), out=None)` (raises on empty) · `sum(fn(value[, key])=None)` · `mean(fn(value[, key])=None) -> float` · `every(pred(value[, key])=None) -> bool` · `some(pred(value[, key])=None) -> bool` (native short-circuit) |
1180
- | Group | `group_reduce(key_fn(value[, key]), init_fn(gk), fold_fn(acc, value[, key]), key_out=None, acc_out=None) -> Dict` · `group_size(key_fn)` · `group_sum(key_fn, fn=None)` · `group_mean(key_fn, fn=None)` · `group_every/group_some(key_fn, pred(value[, key]))` · `group_to_arrays/group_to_sets(key_fn, value_fn=None)` · `group_to_dicts(key_fn, key2_fn, value_fn=None, combine=None)` · ⚠️ `group_fold(...)` is the DEPRECATED alias of `group_reduce` |
1181
- | Flatten | `flatten_to_array(fn(value[, key])->arr, out=None)` (the TS Dict name; `flat_map` is Array's and a deprecated spelling here) · `flatten_to_set(fn(value[, key])->set, out=None)` · `flatten_to_dict(fn(value[, key])->dict, combine=None)` (a duplicate key errors without `combine`) |
1182
- | Convert | `keys() -> Set` · `to_array(fn(value[, key]), out=None)` · `to_set(fn(value[, key]), out=None)` · `to_dict(key_fn, value_fn=None, combine=None, key_out=None, value_out=None)` (the value itself when `value_fn` is omitted; a duplicate key errors without `combine`) · `to_tree(node, parent(value[, key])->some/none, build(value, key, children)) -> Array<node>` (each entry's key its key) · `flatten_tree(children(value[, key])) -> Array<{path, node}>` (nested nodes back: every node with its path of keys, depth first, iterative) · `copy()` |
1183
- | Mutate (in place) | `d[k]=v` · `del d[k]` · `insert(k, v)` (errors if present) · `get_or_insert(k, fn(k))` · `insert_or_update(k, v, combine(existing, incoming, k))` · `update(k, v)` (TS; the read-modify-write `update(k, fn(current))` is deprecated) · `swap(k, v) -> prev` · `delete(k)` · `try_delete(k) -> bool` (`update`/`swap`/`delete` error on a missing key) · `pop(k, *default)` · `clear()` |
1184
- | Bulk (in place) | `update_many(keys, values, combine(existing, incoming)=None)` — the whole batch crosses once; a pure/precompiled `combine` resolves collisions C-to-C (dicts as hot-loop accumulators) |
1185
-
1186
- ### EastVector — complete method surface
1187
-
1188
- Immutable 1-D numeric value: logical element type `Float`/`Integer`/`Boolean`,
1189
- backed by a contiguous NumPy buffer for zero-copy ML interop; `.element_type` is
1190
- fixed, while the storage `.dtype` may be any compatible width (f32, …). The
1191
- **arithmetic surface delegates to the east-c builtins** — reductions fold in
1192
- strict left-to-right index order and comparisons use East's total order (NaN
1193
- greatest, `-0.0 < 0.0`) — bit-identical across the TypeScript, C and python
1194
- runtimes, which numpy's reassociating reductions are not; free-form math goes via
1195
- `to_numpy()`/`to_torch()`. `set` and every transform return a NEW vector. Not
1196
- hashable, but a valid East Set/Dict key (ordered by value). Construct with the
1197
- `EastVector.*` classmethods ([Container generators](#container-generators-classmethods));
1198
- `from_numpy`/`from_torch` infer `element_type` from the dtype when omitted.
1199
- Structural access with an **expression** argument (inside a body) lifts the
1200
- vector as a constant and emits IR, like the eager collections.
1201
-
1202
- | Group | Methods |
1203
- |-------|---------|
1204
- | Access | `get(i)` ❗bounds (promoted Python scalar) · `length() -> int` · `slice(start, end) -> EastVector` (half-open, contiguous copy) |
1205
- | Transform (returns new) | `set(i, v) -> EastVector` ❗bounds (original unchanged) · `concat(other) -> EastVector` (takes this vector's element type) |
1206
- | Arithmetic (east-c; Float/Integer elements) | `scale(alpha)` · `add_scaled(other, alpha)` (`self + alpha*other`) ❗len · `mul(other)` ❗len · `add_scalar(c)` · `abs()` · `clamp(lo, hi)` (East order) · `cum_sum()` (running sum, left to right) |
1207
- | Reduce (east-c; strict left-to-right order) | `sum()` (0 when empty) · `dot(other)` ❗len · `max()`/`min()` ❗empty (TS names; `maximum`/`minimum` deprecated) · `arg_max()`/`arg_min()` ❗empty (ties keep the first index; NaN is greatest) · `mean() -> float` (Integer widens per element; NaN when empty) |
1208
- | Masks & selection (east-c) | `eq(other)`/`lt(other)`/`gt(other)` ❗len `-> Vector<Boolean>` (East equality/order) · `mask.select(a, b)` ❗len (mask receiver) · `v.compress(mask)` ❗len · `mask.count_true() -> int` |
1209
- | Gather / scatter / search (east-c) | `gather(indices)` ❗bounds · `scatter_add(indices, src)` ❗len/bounds (duplicates accumulate in order) · `search_sorted(needles) -> Vector<Integer>` (leftmost insertion index; assumes sorted) |
1210
- | Sparse accumulators (east-c; `East.Vector.*`) | `sparse_axpy(ixA, vA, ixB, vB, alpha)` (union merge `vA + alpha*vB`; absent entries stay absent) ❗ascending/len · `sparse_from_pairs(ix, v)` (sorts + sums duplicates stably, in input order) ❗len · `sparse_filter_gt(ix, v, threshold)` ❗ascending/len — all return `Struct{ix: Vector<Integer>, v: Vector<T>}` with strictly ascending `ix`; every `ix`/`v` input takes a Vector OR an Array |
1211
- | Per-element (east-c; the native callback builtins) | `map(fn(el[, i]), out=None) -> EastVector` (VectorMap; Float/Integer/Boolean results) · `reduce(fn(acc, el[, i]), init)` (VectorFold, index order; `fold(init, fn)` is the deprecated spelling) — a body like every other callback, zero python per element |
1212
- | Convert | `to_array() -> EastArray` (promotes scalars; severs the zero-copy link) · `to_matrix(rows, cols) -> EastMatrix` (row-major reshape; `rows*cols == length`) |
1213
- | NumPy / torch | `to_numpy(dtype=None, copy=False) -> ndarray` (read-only view by default; a cast or `copy=True` is writeable) · `to_torch(dtype=None) -> torch.Tensor` (always a writeable copy) · `np.asarray(v)` via `__array__` · props `.dtype` (storage) / `.element_type` (logical) |
1214
-
1215
- ### EastMatrix — complete method surface
1216
-
1217
- Immutable 2-D row-major numeric value on a contiguous NumPy buffer, with the
1218
- `EastVector` contract: logical `.element_type` apart from storage `.dtype`, the
1219
- arithmetic surface in east-c with the strict reduction order, free-form math via
1220
- `to_numpy()`/`to_torch()`, NEW matrices from `set` and every transform, not
1221
- hashable but a valid Set/Dict key, the `EastMatrix.*` classmethods to construct,
1222
- and an expression argument lifting the matrix as a constant.
1223
-
1224
- | Group | Methods |
1225
- |-------|---------|
1226
- | Access | `get(r, c)` ❗bounds (Python scalar) · `rows() -> int` · `cols() -> int` (TS names; `num_rows`/`num_cols` deprecated) |
1227
- | Transform (returns new) | `set(r, c, v) -> EastMatrix` ❗bounds (original unchanged) · `transpose() -> EastMatrix` (new cols×rows, contiguous) |
1228
- | Arithmetic (east-c; Float/Integer elements) | `scale(alpha)` · `add_scaled(other, alpha)` ❗dims · `mul_elementwise(other)` ❗dims (Hadamard) |
1229
- | Reduce (east-c; ascending index order) | `row_sums() -> EastVector` (ascending column order per row) · `col_sums() -> EastVector` (ascending row order per column) · `vec_mul(v) -> EastVector` ❗cols≠len (row-by-vector dot products) |
1230
- | Rows & cols | `get_row(r) -> EastVector` ❗bounds (contiguous copy) · `get_col(c) -> EastVector` ❗bounds (contiguous copy) |
1231
- | Per-row (east-c) / per-element (DEPRECATED) | `map_rows(fn(row: EastVector[, i]) -> EastVector, out=None)` (the native MatrixMapRows — a body like every callback) · `map_elements(fn(el), out=None)` runs a python callback per element; it warns and will go — use the arithmetic surface or `to_numpy()` |
1232
- | Convert | `to_vector() -> EastVector` (row-major flatten) · `to_array() -> EastArray` (`Array<Array<el>>`, one inner array per row) · `to_rows() -> EastArray` (`Array<Vector<el>>`, one `EastVector` per row) |
1233
- | NumPy / torch | `to_numpy(dtype=None, copy=False) -> ndarray` (2-D; read-only view by default) · `to_torch(dtype=None) -> torch.Tensor` (writeable copy) · `np.asarray(m)` via `__array__` · props `.dtype` / `.element_type` |
1234
-
1235
- ### EastBlob (a `bytes` subclass)
1236
-
1237
- `EastBlob(b"...")` constructs like `bytes` and carries the **full `bytes` API** plus East methods:
1238
-
1239
- | Signature | Description |
1240
- |-----------|-------------|
1241
- | `size() -> int` | Byte length (== `len(blob)`) |
1242
- | `get_uint8(i) -> int` | Unsigned byte at `i` (0–255) |
1243
- | `.data -> bytes` | Raw payload |
1244
- | `decode_utf8() -> str` / `decode_utf16() -> str` | Text decode |
1245
- | `EastBlob.encode_beast2(value) -> EastBlob` *(static)* | Serialize an East value to BEAST2 (type inferred via `type_of`); `East.Blob.encode_beast(value, "v2")` is the builtin twin |
1246
- | `decode_beast(typ, version="v1") -> value` | The east-c BlobDecodeBeast / BlobDecodeBeast2 builtins (TS `decodeBeast`): `"v1"` is the original BEAST format, `"v2"` the beast2 family (any container version) |
1247
- | `decode_beast2(typ) -> value` | Decode BEAST2 as `typ` (the serialization-layer form) |
1248
- | `open_beast(typ) -> value` | The east-c BlobOpenBeast2 builtin (TS `openBeast`): an indexed beast2 `Array`/`Set`/`Dict` blob as a FROZEN paged proxy — the value a task's collection input opens as — whose `len`, keyed reads and iteration decode one segment; an index-less blob (what `encode_beast` writes) or a Ref-/function-bearing element shape decodes whole, frozen; mutation raises, `.copy()` gives a mutable value; a header of another type raises `cannot open a blob of type <wire> as <typ>` before any decode |
1249
- | `decode_csv(element_type, config=None, **options) -> EastArray` | Decode CSV rows into `Array<element_type>` (east-c decoder). Build `config` with `east.serialization.csv.csv_parse_config(...)`: by default **no field text is null** (empty field == empty string); opt in with `null_strings=[""]` (`none` for Option columns, error for required); `defaults={"qty": "0.0"}` gives per-column fallbacks for unparseable fields and constant-fill for absent columns; `skip_short_rows=True` drops ragged rows instead of erroring |
1250
-
1251
- ### EastStruct / EastVariant / EastRef
1252
-
1253
- - **`EastStruct`** — frozen record; read fields by name: `s["price"]` or as an
1254
- attribute, `s.price` (methods shadow same-named fields — item access always
1255
- works). Build/transform with `struct({...}, StructType)`.
1256
- - **`EastVariant`** — frozen tagged value; `.type` is the case name, `.value` the payload.
1257
- Build with `variant(case, value, T)` / `some` / `none`. Dispatch with the
1258
- `.match({case: handler}, default=None)` method (handlers are bodies, `handler(b, payload)`;
1259
- `default` is a `default(b)` body for the other cases — TS's partial match — or a plain
1260
- value; the module-level `match(v, cases, default)` is equivalent) or
1261
- `.match_tag(tag, handler, default)`. Also `get_tag()`, `has_tag(tag)`, and
1262
- `unwrap(tag="some", on_other=None)` ❗ValueError on a different case unless the
1263
- `on_other(b)` body answers — mirroring the TS variant expression surface, and the same
1264
- shapes build inside a body. An option reads with the **same accessors as the
1265
- expression surface** — `is_some()`, `is_none()`, `unwrap_or(default)` — so a
1266
- decoded option needs no `opt.type == "some"` branch.
1267
- - **`EastRef`** — mutable cell: `get()` · `update(value)` (TS `RefUpdate`; `set` is the
1268
- deprecated spelling, and the read-modify-write `update(fn(b, current))` is deprecated too —
1269
- write `ref.update(f(ref.get()))`) · `merge(patch, combine(b, current, patch))`. Inside a
1270
- body the expression twin is `East.ref(v)` with the same `get`/`update`/`merge`.
1271
-
1272
- ### Container generators (classmethods)
1273
-
1274
- Eager factories are **classmethods** on the container classes (snake_case); their dual-mode
1275
- twins — usable inside a body — live on the `East.Array`/`East.Set`/`East.Dict`/
1276
- `East.Vector`/`East.Matrix` namespaces (`East.Array.range`, `East.Vector.zeros(T, n)`,
1277
- `East.Matrix.from_rows(rows)`, …). The namespace `generate`s take the **TypeScript argument
1278
- order** — `East.Array.generate(size, T, fn)`, `East.Set.generate(size, T, fn, on_conflict=None)`,
1279
- `East.Dict.generate(size, K, V, key_fn, value_fn, on_conflict=None)` — and, like TypeScript, a
1280
- key generated twice is a runtime error `Duplicate key <k> in set/dict` unless an `on_conflict`
1281
- handler is given (the pre-TS python order still works with a DeprecationWarning). Do not confuse them with
1282
- `East.new_array`/`new_set`/`new_dict`, which are the CONTROL-FLOW constructors: a fresh
1283
- loop-local collection built per evaluation inside a function.
364
+ ## Reference files
1284
365
 
1285
- | Signature | Example |
1286
- |-----------|---------|
1287
- | `EastArray.range(start, end, step=1)` | `EastArray.range(0, 5, 2)` → `[0, 2, 4]` |
1288
- | `EastArray.linspace(start, end, count)` | `EastArray.linspace(0., 1., 3)` → `[0.0, 0.5, 1.0]` |
1289
- | `EastArray.generate(count, fn(b, i), element_type=None)` | `EastArray.generate(3, lambda b, i: i*i, IntegerType)` |
1290
- | `EastSet.generate(n, fn(b, i), element_type=None, on_conflict=None)` | `EastSet.generate(4, lambda b, i: East.Integer.remainder(i, 2), IntegerType, lambda b, k: None)` — a duplicate ERRORS without `on_conflict` (TS) |
1291
- | `EastDict.generate(n, key_fn(b, i), value_fn(b, i), combine(b, x, y, k), key_type, value_type)` | `EastDict.generate(3, lambda b, i: i, lambda b, i: i*10, lambda b, x, y, k: x + y, IntegerType, IntegerType)` — `combine=None` makes a duplicate key an error (TS) |
1292
- | `EastVector.zeros/ones(element_type, length)` · `fill(element_type, length, value)` · `from_array(element_type, items)` · `from_numpy(array, element_type=None)` · `from_torch(tensor, element_type=None)` | `EastVector.zeros(FloatType, 3)` |
1293
- | `EastMatrix.zeros/ones(element_type, rows, cols)` · `fill(…, value)` · `from_array/from_rows(element_type, rows)` · `from_numpy(array, element_type=None)` · `from_torch(tensor, element_type=None)` | `EastMatrix.from_array(FloatType, [[1.,2.],[3.,4.]])` |
366
+ Read the one a task needs; each holds its sections whole.
1294
367
 
1295
- ### East.<Type> namespaces — the builtins and the standard library
1296
-
1297
- Scalars are plain Python, so their builtins are namespace functions — **complete** lists below.
1298
- Every one delegates to east-c, and every one is dual-mode: on plain values it runs now, on
1299
- expressions it emits IR. The `stdlib:` rows are the TypeScript standard library
1300
- (see [the stdlib](#the-standard-library--the-typescript-easttype-functions)).
1301
-
1302
- **`East.Float`** (f64)
1303
-
1304
- | Signature | Notes |
1305
- |-----------|-------|
1306
- | `add(a,b)` · `subtract(a,b)` · `multiply(a,b)` · `divide(a,b)` · `remainder(a,b)` · `pow(base,exp)` | arithmetic |
1307
- | `negate(x)` · `abs(x)` · `sign(x)` · `sqrt(x)` · `exp(x)` · `log(x)` | unary / powers |
1308
- | `sin(x)` · `cos(x)` · `tan(x)` | trig |
1309
- | `to_integer(x) -> int` | raises on a non-integer float (e.g. `3.9`) |
1310
- | stdlib: `approx_equal(x, y, epsilon) -> bool` · `round_floor/round_ceil/round_half/round_trunc(x) -> int` | `round_half` ties away from zero |
1311
- | stdlib: `round_nearest/round_up/round_down/round_truncate(x, step)` · `round_to_decimals(x, decimals)` | step 0.0 returns `x`; NaN/±Infinity raise `Cannot round …` |
1312
- | stdlib: `print_fixed(x, decimals)` · `print_comma_seperated(x, decimals)` · `print_currency(x)` · `print_compact(x)` · `print_percentage(x, decimals)` | `"3.14"` · `"1,234.57"` · `"-$42.50"` · `"1.5M"` · `"12.34%"`; NaN/±Infinity raise `Cannot format …` |
1313
-
1314
- **`East.Integer`** (i64)
1315
-
1316
- | Signature | Notes |
1317
- |-----------|-------|
1318
- | `add(a,b)` · `subtract(a,b)` · `multiply(a,b)` · `divide(a,b)` · `remainder(a,b)` · `pow(base,exp)` | arithmetic (`divide` truncates) |
1319
- | `negate(x)` · `abs(x)` · `sign(x)` · `log(x, base)` | unary |
1320
- | `to_float(x) -> float` | widen to f64 |
1321
- | stdlib: `print_comma_seperated(x)` · `print_currency(x)` · `print_compact(x)` · `print_compact_si(x)` · `print_compact_computing(x)` · `print_ordinal(x)` · `print_percentage(x)` · `digit_count(x)` | `"1,234,567"` · `"$1,234"` · `"1.23M"` (K/M/B/T/Q) · `"1.23M"` (k/M/G/T/P) · `"1.17Mi"` (base 1024) · `"3rd"` · `"25%"` · `3` |
1322
- | stdlib: `round_nearest/round_up/round_down/round_truncate(x, step)` | step 0 returns `x`; `round_nearest(127, 10)` is `130` |
1323
-
1324
- **`East.String`**
1325
-
1326
- | Signature | Notes |
1327
- |-----------|-------|
1328
- | `concat(a, b)` · `repeat(s, n)` · `substring(s, start, end)` · `length(s) -> int` | build / measure |
1329
- | `upper_case(s)` · `lower_case(s)` · `trim(s)` · `trim_start(s)` · `trim_end(s)` | case / whitespace |
1330
- | `replace(s, find, replacement)` · `split(s, separator) -> Array<String>` | edit / tokenize |
1331
- | `contains(s, substring)` · `starts_with(s, prefix)` · `ends_with(s, suffix)` · `index_of(s, substring) -> int` | search (`-> bool`/`int`) |
1332
- | `regex_contains(s, pattern, flags="")` · `regex_index_of(s, pattern, flags="")` · `regex_replace(s, pattern, replacement, flags="")` | regex |
1333
- | `parse(typ, s)` ❗ · `print(typ, value) -> str` (the root `East.print(value[, typ])` is the same builtin, value first) | East **text** format; `parse` is a **strict whole-string** parser — trailing or leading junk raises (`"598-"`, `"$5"`, `"1.2.3"` all raise; in a body use `.try_parse(T)` for the optional form) |
1334
- | `parse_json(typ, s)` · `print_json(typ, value) -> str` / `print_json(value)` | East **JSON** (`Integer` encodes as a JSON *string*: `print_json(ArrayType(IntegerType), [1,2,3]) == '["1","2","3"]'`; an `Option` is `null` for `none` and the payload itself for `some` wherever the payload cannot be null — `print_json(OptionType(IntegerType), some(7)) == '"7"'` — and only `Option<Null>` / `Option<Option<T>>` keep the `{"type": …, "value": …}` object; a `DateTime` prints as `YYYY-MM-DDTHH:MM:SS.sss+00:00` and parses from any RFC 3339 date-time, as UTC); the one-argument form (TS `printJson(value)`) takes the value's own type |
1335
- | stdlib: `print_error(message, stack) -> str` | `"Error: <message>"` + one `[i] file line:column` per `{filename, line, column}` frame (TS `printError`) |
1336
-
1337
- **`East.Blob`**
1338
-
1339
- | Signature | Notes |
1340
- |-----------|-------|
1341
- | `encode_beast(value, version="v1", *, typ=None) -> EastBlob` | TS `East.Blob.encodeBeast`: `"v1"` the original BEAST format, `"v2"` the beast2 family; the type is the expression's declared type / `type_of(value)`, or `typ` to encode a plain value under a wider type |
1342
-
1343
- **`East`** root: `East.str(*parts)` (TS `East.str` — the parts concatenated, non-String parts printed
1344
- in East text format: `East.str("n=", 5, "!")`), `East.print(value[, typ])` (TS `East.print` — the East
1345
- text format under the value's own type, or `typ`), `East.min(a, b)` / `East.max(a, b)` (`least`/`greatest`
1346
- under East's total order), `East.clamp(value, lo, hi)` and `East.jq(input, program, result_type)`
1347
- ([Queries](#queries--eastjq)) — all dual-mode.
1348
-
1349
- **`East.DateTime`** (see [DateTime format codes](#datetime-format-codes))
1350
-
1351
- | Signature | Notes |
1352
- |-----------|-------|
1353
- | `from_components(year, month=1, day=1, hour=0, minute=0, second=0, millisecond=0)` | construct (TS defaults: the trailing components are the first instant). An out-of-range component NORMALISES into the next one, it never raises: `(2024, 2, 31)` is `2024-03-02`, `(2024, 13, 1)` is `2025-01-01`, `(2023, 2, 29)` is `2023-03-01`. Range-checking the inputs does not catch it (31 February is all in range) — build the date and check `dt.get_month() == month` |
1354
- | `from_epoch_milliseconds(millis)` · `to_epoch_milliseconds(dt) -> int` | epoch round-trip |
1355
- | `get_year/get_month/get_day_of_month/get_day_of_week(dt) -> int` | `get_day_of_week`: Monday == 1 |
1356
- | `get_hour/get_minute/get_second/get_millisecond(dt) -> int` | components |
1357
- | `add_milliseconds(dt, millis)` · `subtract_milliseconds(dt, millis)` · `duration_milliseconds(a, b) -> int` | the raw builtin (`subtract` negates the amount, TS `subtractMilliseconds`): `duration` returns **a − b** (the expression METHOD `a.duration_milliseconds(b)` is the TS method, `b − a`) |
1358
- | `add_/subtract_{seconds,minutes,hours,days,weeks}(dt, n)` | unit sugar over `add_milliseconds` (an int or float `n`; an expression `n` scales inside the body, a Float after scaling) |
1359
- | `duration_{seconds,minutes,hours,days,weeks}(a, b) -> float` | unit sugar over `duration_milliseconds` |
1360
- | `print_formatted(dt, fmt) -> str` · `parse_formatted(s, fmt) -> datetime` | Day.js-style tokens (TS names; `print_format`/`parse_format` are deprecated spellings) |
1361
- | stdlib: `round_down_/round_up_/round_nearest_{millisecond,second,minute,hour,day,week}(dt, step)` | `step` units of the name; weeks align to Mondays (the reference Monday 1969-12-29) |
1362
- | stdlib: `round_down_month(dt, step)` · `round_down_year(dt, step)` | the first instant of the `step`-aligned month / year |
1363
-
1364
- **`East.Boolean`**
1365
-
1366
- | Signature | Notes |
1367
- |-----------|-------|
1368
- | `not_(x)` · `bit_and(a, b)` · `bit_or(a, b)` · `bit_xor(a, b)` | the BooleanNot / BooleanAnd / BooleanOr / BooleanXor builtins under the TypeScript names (`not`, `bitAnd`, `bitOr`, `bitXor`; `and_`/`or_`/`xor` are deprecated spellings) — both operands are values here, so there is nothing to short-circuit; the expression twins are `.not_()`, `.bit_and`, `.bit_or`, `.bit_xor`, and the short-circuit `.and_(fn(b))`/`.or_(fn(b))` take bodies |
1369
-
1370
- **`East`** comparisons (East total order; element type `T` first): `compare(T, a, b) -> int`,
1371
- `equal/not_equal/less/less_equal/greater/greater_equal(T, a, b) -> bool`, and `is_(T, a, b) -> bool` (East `Is`: identity for a mutable container, value otherwise).
1372
-
1373
- **`East`** structural diff/patch (any East type `T`; a patch is a value of `PatchType(T)`; every
1374
- function takes `T` explicitly — a type sampled from one value cannot describe both sides of a
1375
- variant diff):
1376
-
1377
- | Signature | Notes |
1378
- |-----------|-------|
1379
- | `diff(T, before, after) -> patch` | The patch turning `before` into `after` |
1380
- | `apply_patch(T, value, patch) -> value` | Apply; `apply_patch(T, v, diff(T, v, w)) == w` |
1381
- | `compose_patch(T, first, second) -> patch` | One patch equal to applying `first` then `second` |
1382
- | `invert_patch(T, patch) -> patch` | The undo: applying patch then its inverse round-trips |
1383
-
1384
- ### DateTime format codes
1385
-
1386
- `print_formatted(dt, fmt)` / `parse_formatted(s, fmt)` take a Day.js-style string; tokens match
1387
- greedily (longest-first), anything else is a literal, and `\` escapes the next char.
1388
- Examples for `2025-03-05 14:09:07.123` (a Wednesday):
1389
-
1390
- | Code | Meaning | Ex | Code | Meaning | Ex |
1391
- |------|---------|----|------|---------|----|
1392
- | `YYYY` | 4-digit year | `2025` | `YY` | 2-digit year | `25` |
1393
- | `MMMM` | full month | `March` | `MMM` | short month | `Mar` |
1394
- | `MM` | month 01-12 | `03` | `M` | month 1-12 | `3` |
1395
- | `DD` | day 01-31 | `05` | `D` | day 1-31 | `5` |
1396
- | `dddd` | full weekday | `Wednesday` | `ddd` | short weekday | `Wed` |
1397
- | `dd` | min weekday | `We` | `HH` | hour 00-23 | `14` |
1398
- | `H` | hour 0-23 | `14` | `hh` | hour 01-12 | `02` |
1399
- | `h` | hour 1-12 | `2` | `mm` | minute 00-59 | `09` |
1400
- | `m` | minute 0-59 | `9` | `ss` | second 00-59 | `07` |
1401
- | `s` | second 0-59 | `7` | `SSS` | millisecond | `123` |
1402
- | `A` | AM/PM | `PM` | `a` | am/pm | `pm` |
1403
-
1404
- ```python
1405
- East.DateTime.print_formatted(dt, "YYYY-MM-DD HH:mm:ss.SSS") # '2025-03-05 14:09:07.123'
1406
- East.DateTime.print_formatted(dt, "dddd, MMMM D, YYYY h:mm A") # 'Wednesday, March 5, 2025 2:09 PM'
1407
- ```
1408
-
1409
- ### Beast2 streaming — bounded-memory collections (`from east.serialization.beast2 import ...`)
1410
-
1411
- Beast2 v5 encodes a large Array/Set/Dict as an append-only stream of
1412
- independently decodable segments: writer memory is one segment, never the whole
1413
- collection, and the decoders read v4 and v5 through the same entry points. The
1414
- canonical writers — the managed file writer, `Beast2ElementWriter`,
1415
- `encode_beast2_paged_for` — cut segments by one content-defined rule, the same in
1416
- TypeScript and east-c, so a value's bytes are its own whoever writes them;
1417
- `Beast2Writer` makes each batch you give it a segment instead.
1418
-
1419
- **Managed files — start here (`open_beast2_file` / `write_beast2_file`).** Path
1420
- in, East values out. The file is self-describing, so a read needs no declared
1421
- type (a write does; a type declared on a read is validated at open). The file
1422
- object owns the fd and the mmap (closed on `with`-exit), east-c does every byte,
1423
- and the segments are the canonical ones — no buffers, iterators or batch sizes in
1424
- your code. The read flavour mirrors the root collection's read surface name for
1425
- name.
1426
-
1427
- | Signature | Description |
1428
- |-----------|-------------|
1429
- | `write_beast2_file(path, T, value, *, codec="deflate")` | One call writes a collection of any size as one indexed v5 file — the value's canonical blob: segments where the content-defined cut rule places them, bounded in elements and bytes (wide rows never pile into one), key-disjoint for a Dict/Set. The bytes TypeScript and east-c write, and what `encode_beast2_paged_for(T)(value)` returns |
1430
- | `open_beast2_file(path, T, mode="w", *, codec=)` | The streaming writer: `.write()` takes East collections or python `list`/`dict`/`set` batches of any size, which add up to one canonical blob (a Set/Dict batch continues strictly ascending from the last); `.segments` counts the segments closed so far |
1431
- | `open_beast2_file(path, T=None, *, project=None)` | The read: `Beast2ArrayFile` / `Beast2DictFile` / `Beast2SetFile`, a READ-ONLY East collection VALUE subclassing `EastArray`/`EastDict`/`EastSet` — `isinstance`/`type_of` answer, every eager method works (streamed, below), mutation raises, and it binds into functions or passes into compiled calls by reference, where keyed reads answer from the pager, one frame per hit or miss, through a byte-budgeted segment cache (`EAST_PAGED_CACHE_BYTES`); `close()` defers while a bind holds it. `T` is optional (the header supplies it, also as `f.wire_type`); a declared `T` is checked against the header at open, never decoded as garbage |
1432
- | `read_beast2_type(source) -> EastType` | The root type of any beast2 blob (v4 or v5), from a path or a buffer, nothing decoded — for a file you know nothing about |
1433
- | `write_beast2_file_parallel(path, T, partitions, produce, *, processes=, strategy="auto", codec=, keep_shards=False, verify=False)` | N workers, ONE file: `produce(partition)` returns that partition's batches (or one collection); each worker writes a shard cut canonically from its start, and the shards splice **in partition order** as they finish. `strategy="auto"` forks on Linux/macOS — what `produce` closes over is inherited copy-on-write, so build the expensive context before the call (and before starting threads) — and runs inline on Windows, byte-identically. Any worker failure fails the call with the worker's traceback and leaves nothing behind |
1434
- | `splice_beast2_files(path, T, sources, *, verify=False) -> (segments, elements)` | Merges indexed v5 files by **byte copy** — east-c parses the geometry, `os.sendfile` moves the frames, nothing re-encodes. `sources` may be a lazy generator (spliced as they complete, in row order). Each must be v5, indexed and self-contained with an identical type section; a refusal names the path and leaves no destination. `verify=True` re-walks the result with east-c's strict reader |
1435
- | `f.load()` | The whole collection, decoded in east-c off the mmap — input memory stays one segment at any size; also the mutable escape hatch, like `f.copy()` |
1436
- | `f.segments()` | DEPRECATED — the file IS its collection value; still works (with a warning) for per-batch migration code |
1437
- | `len(f)` · `f.segment_count` · `f.self_contained` · `f.indexed` | O(1) from the trailing index, exact for every root kind |
1438
- | Array: `f[i]` / `f[a:b]` · `f.get(i)` ❗bounds · `f.get_or_default(i, d)` · `f.try_get(i)` · `f.has(i)` · `f.slice(a, b)` · `f.get_keys(rows)` | `EastArray`'s names, signatures and errors; a point read decodes only the owning segment, `get_keys` each owning segment once |
1439
- | Dict: `f[k]` ❗KeyError · `f.get(k[, default \| fn(k)])` · `f.get_or_default(k, d)` · `f.try_get(k)` · `f.has(k)` / `k in f` · `f.get_keys(keys, fill)` · `f.items()` / `values()` / iteration (streaming) · `f.keys()` (the Set) · `f.size()` — Set: `x in f` / `f.has(x)` | east-c binary-searches the segment *fences* (each segment's first key, from a bounded probe, cached), then decodes ONLY the owning segment (a small LRU keeps hot ones); `get_keys` merges the sorted keys against the fences and calls `fill` per missing key. The first keyed read verifies the fences: a corrupt or pre-contract blob raises `segments are not disjoint ascending key ranges` instead of reporting false misses |
1440
- | Array sorted search: `f.find_sorted_first/last(target)` → a global index · `f.find_sorted_range(target)` → `{start, end}` | `EastArray`'s contract over the whole file, only the boundary segment decoded; no `key=` projection (the file pages in element order) — pair with `f.slice(start, end)` |
1441
- | Compute: `map` `filter` `filter_map` `first_map` `reduce` `scan` `map_reduce` `sum` `mean` `maximum` `minimum` `every` `some` `find_*` `is_sorted` `to_set` `unique` `to_dict` `to_array` `to_columns` `map_batches` `string_join` `flat_map` (Array) / `flatten_to_array` (Set, Dict) `flatten_to_set` `flatten_to_dict` `for_each`, the whole `group_*` family, Set algebra | Each segment runs the ordinary eager method, and partials combine in east-c in stream order: order-dependent folds thread ONE accumulator and grouped folds seed each segment from the running per-group accumulators, so results equal `load()` exactly, float order included. Array `(el, idx)` callbacks, and the indices `find_*` / `group_find_*` report, are GLOBAL; `first_map`/`some`/`every`/`is_superset_of` stop decoding at the answer. Dict/Set compute streams disjointness-verified segments. A re-keyed collision (`to_dict`, `flatten_to_dict`, `group_to_dicts`) combines left-associatively in stream order — use an associative `combine`. `sort`/`reverse`/`copy`/`concat`/`union` materialize the collection — `load()` first |
1442
- | Column projection — INFERRED on the compute family; EXPLICIT with `open_beast2_file(path, project=NARROW)` | The compute family builds its callbacks FIRST and decodes each segment to just the struct fields their IR reads (the rest are hopped over, never built — building values, not walking bytes, is what decoding costs). Fields subset by name at any depth; a subtree used any other way stays whole, so results never change; Dict keys and Set elements never narrow. Runner-opened task inputs infer the same from the body's loop IR. What cannot be inferred decodes whole and is COUNTED in `eager_stats()` (`beast2_segments_projected/whole`, `beast2_projection_declined_*` by reason) — a projection that silently stopped applying would be an invisible cliff. The explicit form serves the subset to EVERY read (point reads, keyed gets, `load()`); `project` must be a subset of the wire type (a missing field raises `ValueError` naming it), a declared `T` keeps its meaning, and `find_sorted_*` refuse under it. A segment decoded under one mask is never served to a read needing more. No wire change |
1443
- | Degraded blobs | v4 → refused (`decode_beast2_with_header_for` still decodes it whole); index-less v5 → `load()` works, random access refuses; not self-contained → point reads refuse |
1444
- | **Manifest directories** — one object per segment plus a manifest naming them: how e3 stores a collection, and stages a task input |
1445
- | `Beast2ManifestWriter(T, path, *, codec="deflate")` | The canonical writer (a context manager): `.add(el)` / `.add_all(batch)`, as `Beast2ElementWriter` takes them; each segment goes to `<path>.segments/<sha256>.beast2`, a standalone blob under the shared header, and `.close()` writes the manifest to `path` — the directory TypeScript's `Beast2ManifestWriter` and east-c write. A writer left by an exception writes no manifest; `.segments` counts them; east-c does every byte, hashing included |
1446
- | `load_beast2_manifest(path, T=None)` ❗ | The whole collection, segment by segment in east-c — the value its segments spliced into one blob decode to. `ValueError` when `path` holds no manifest, the types differ, or a segment is missing or malformed |
1447
- | `read_beast2_manifest(source)` | The manifest — `kind`, `level`, `type`, `rule`, `header`, `entries` (each `{hash, fence, count, bytes}`) — or `None`; a blob holding a value is read no further than its type section |
1448
-
1449
- ```python
1450
- from east.serialization.beast2 import open_beast2_file, write_beast2_file
1451
-
1452
- write_beast2_file(path, rows_t, rows) # any size, one call
1453
-
1454
- with open_beast2_file(path) as f: # self-describing: type from the
1455
- row = f[1_234_567] # header (f.wire_type); pass rows_t
1456
- totals = f.group_sum(lambda b, r: r["sku"], # instead to VALIDATE it at open
1457
- lambda b, r: r["qty"]) # whole-file compute: segment folds,
1458
- top = f.maximum(by=lambda b, r: r["qty"]) # never materialized, == load() exactly
1459
- table = f.load() # whole table when you truly need it
1460
-
1461
- # The file IS a collection value: bind it into a function and the
1462
- # compiled body's keyed reads answer from the pager — one frame per lookup.
1463
- with open_beast2_file("table.beast2") as t: # Dict<String, Float>
1464
- lookup = East.function([StringType, DictType(StringType, FloatType)], FloatType,
1465
- lambda b, k, d: d.get_or_default(k, 0.0)).bind(t)
1466
- joined = rows.map(lambda b, r: r.v + lookup(r.k)) # loop + callee + pager: all east-c
1467
- ```
1468
-
1469
- **Buffer level — you hold bytes, not a path. `_for` is NOT `_with_header_for`:**
1470
-
1471
- | Signature | Description |
1472
- |-----------|-------------|
1473
- | `encode_beast2_with_header_for(T, *, version=None)` / `decode_beast2_with_header_for(T)` | **The one you want**: the self-describing container — magic, type schema, value. Encode writes the current default container (v5; `version=4` only for a reader that predates it); decode reads v4 and v5, whole (Set/Dict segments must hold the canonical value — sorted, disjoint — or the blob is rejected as corrupt). ❗The header's type must be `T`, or a subtype whose variant cases line up with `T`'s (a `none` reads as any `Option`), else `ValueError: beast2: cannot decode a blob of type … as …` before anything decodes — the words TypeScript and east-c use |
1474
- | `encode_beast2_for(T)` / `decode_beast2_for(T)` | **Headerless** type-directed bytes: the reader must know `T` exactly, and mutable containers (Array/Set/Dict/Ref) are refused. In TypeScript `encodeBeast2For` is the FULL container — do not port a call site by name |
1475
- | `Beast2ElementWriter(T, stream, *, codec="deflate", parallel=False)` | The canonical writer (a context manager): `.add(el)` — an Array/Set element or a Dict `(key, value)` pair — and `.add_all(batch)` (the loop in east-c); Set elements and Dict keys strictly ascending in East order. Segments fall where the cut rule places them — the bytes the managed writer, TypeScript and east-c write; memory is one open segment; `.close()` writes the last segment, the terminator and the index; `.segments` counts the closed ones. An element that does not ascend, or fails to encode, raises and leaves the writer as it was |
1476
- | `encode_beast2_paged_for(T, *, codec="deflate") -> (value) -> bytes` | A whole collection through the canonical writer — the write side of `open_beast2_pages_for` |
1477
- | `Beast2RunSorter(T, open_run, *, merge=None, union=False, codec="deflate", parallel=False)` | A Set's or Dict's elements in ANY order in, sorted canonical runs out: `.add(el)` encodes at once; at `RUN_MAX_COUNT` elements or `RUN_MAX_BYTES` it sorts the run — stably, so a key's values keep the order they came in — folds a repeated key (`merge`, a compiled `(K, V, V) -> V` East function, for a Dict; `union=True` for a Set; neither: it raises) and writes the run to `open_run(run)` (anything with `write(bytes)` and `close()`) as its value's canonical blob. `.finish()` writes the last run; `.runs` counts them. The runs are the ones east-c and TypeScript write; a sink's exception comes back out of the call that wrote the run; a key repeated across runs is the merge's to fold |
1478
- | `Beast2Writer(T, stream, *, codec="deflate", self_contained=True, index=True, parallel=False)` | Segments of your choosing (a context manager): `.write(batch)` appends one segment per non-empty batch; `.close()` writes the terminator and the index; `.segments` counts. Set/Dict batches must ascend strictly across the stream. **Keep both defaults** — `index` and `self_contained` are what `open_beast2_pages_for` needs, and turning either off silently forfeits random access. `codec="none"` for already-compressed payloads or raw write throughput |
1479
- | `encode_beast2_segments_for(T, **opts) -> (batches) -> bytes` | In-memory `Beast2Writer`: one segment per non-empty batch |
1480
- | `encode_beast2_v5_for(T, *, codec="deflate", index=False) -> (value) -> bytes` | A whole value of any root type as v5; decode with `decode_beast2_with_header_for` |
1481
- | `iter_beast2_segments_for(T) -> (source) -> iterator` | One decoded collection per segment, O(segment) memory; `source` is bytes, an `mmap` or a binary stream |
1482
- | `read_beast2_index(T, blob) -> (segments, elements) \| None` | O(1) totals from a v5 blob's trailing index |
1483
- | `open_beast2_pages_for(T) -> (source) -> Beast2Pages` | Random access: `.segment_count` `.element_count` `.self_contained` `.counts`, `.segment(i)`, `.element(row)` (also `len()` / `[]`), each decoding ONE segment. ❗Needs `index=True` and `self_contained=True` (the defaults); `.element()` is for Array roots. ❗Borrows the buffer — keep it (and an mmap) alive for the pages' lifetime, or use `open_beast2_file`, which owns it |
1484
-
1485
- **Batch size (`Beast2Writer` only — the canonical writers cut by the rule).** A
1486
- batch is at once your memory ceiling, one segment, one compression window and
1487
- the granularity of random access. ~1000 rows is a good default: one row per
1488
- batch costs about 4× the bytes, and the curve is flat past ~100. `write()` takes
1489
- a batch, never a row — accumulate and flush yourself, or hand rows to
1490
- `Beast2ElementWriter`, which needs no batch at all.
1491
-
1492
- ```python
1493
- import mmap
1494
- from east.serialization.beast2 import (
1495
- Beast2Writer, iter_beast2_segments_for, open_beast2_pages_for,
1496
- )
1497
- rows_t = ArrayType(row_type)
1498
-
1499
- with open(path, "wb") as f, Beast2Writer(rows_t, f) as w:
1500
- for batch in produce_batches(): # each an EastArray(row_type, ...)
1501
- w.write(batch) # O(batch) memory, one segment each
1502
-
1503
- # Sequential re-read. mmap, NOT .read() — .read() pulls the whole file in and
1504
- # throws away the bounded-memory property you just paid for.
1505
- with open(path, "rb") as f, mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
1506
- for batch in iter_beast2_segments_for(rows_t)(mm):
1507
- consume(batch) # O(segment) decoded at a time
1508
-
1509
- # Or jump straight to a row — decodes only the segment that owns it.
1510
- pages = open_beast2_pages_for(rows_t)(mm)
1511
- row = pages.element(1_234_567)
1512
- ```
1513
-
1514
- ### Columnar escape hatches — when the logic must stay python
1515
-
1516
- When per-element logic genuinely needs python (numpy, a model, an external
1517
- library), don't touch rows one at a time — cross the boundary **once per
1518
- column** instead of once per row × field:
1519
-
1520
- ```python
1521
- cols = rows.to_columns() # {"price": np.float64[...], "qty": np.int64[...],
1522
- # "sku": [str, ...] (interned), ...} — one crossing/column
1523
- amount = cols["price"] * cols["qty"].astype(np.float64) # vectorised numpy
1524
- out = EastArray.from_columns(Out, {"sku": cols["sku"], "amount": amount})
1525
-
1526
- result = rows.map_batches(f, out=Out, batch_size=50_000) # f sees columnar chunks;
1527
- # batches may shrink (filter-like)
1528
-
1529
- acc = EastDict(StringType, FloatType) # dicts as accumulators:
1530
- acc.update_many(keys, values, combine=lambda b, cur, new: cur + new) # one crossing,
1531
- # combine builds -> collisions in C
1532
- arr.extend(np_array) # bulk push, one crossing
1533
-
1534
- solver_input = coerce_to( # struct fields take 1-D numpy
1535
- {"start_nodes": np_i64, "capacities": np_i64, ...}, # columns directly: each
1536
- MinCostFlowInputType) # Array<Int/Float/Bool> fills C-side
1537
- ```
1538
-
1539
- `Float`/`Integer`/`Boolean` columns move through numpy buffers filled in C
1540
- (`Option<Float>` ↔ float64 with NaN for `none`); `String` columns box once
1541
- through a bounded intern table (repeated categories/ids come back as the
1542
- same python object); other field types fall back to boxed lists. The same
1543
- contract governs the INPUT direction: `from_columns`/`coerce_to` fill C-side
1544
- only when a numeric column arrives as a numpy array of the matching dtype
1545
- (float64/int64/bool; float64-with-NaN for `Option<Float>`) — a plain python
1546
- list converts per cell, so `np.asarray(col)` first when the source is a
1547
- list. Composition rule: **East functions for East-expressible transforms,
1548
- columns/batches for the genuinely-python remainder.**
1549
-
1550
- **Put the logic in the platform function, not a pure-python shim.** A
1551
- `@East.platform_function` is just a typed, validated python function; its one
1552
- added cost, validating the declared output, is a feature. A separate helper layer
1553
- over `list`/`dict` that the platform function converts into and out of costs
1554
- **testability** (the typed `inputs`/`output` is the contract you test; untyped
1555
- helpers turn bugs into silent corruption instead of a named `EastTypeError`),
1556
- **portability** (a platform function over East values moves to an e3 task,
1557
- another runtime or a TypeScript mirror unchanged), a **forced sandwich**
1558
- (East→python on the way in, back on the way out) and **purity** (a platform
1559
- function pure in its East inputs is memoisable). Small reusable functions are
1560
- fine — have them take and return East values, pay output validation only at the
1561
- real East↔python edge, and never call a platform function per element in a loop.
1562
-
1563
- ### Platform functions
1564
-
1565
- | Signature | Description |
1566
- |-----------|-------------|
1567
- | `@East.platform_function(*, inputs, output, name=None, is_async=None, validate_output=True, validate_input=False)` | Register a Python fn; infers sync/async from the def (`is_async=` declares it when they differ — a function East calls asynchronously on every runtime whose python is a plain `def`); validates output against `output`. DUAL-MODE: called with values it runs the python (an async def's coroutine, to await); called inside an `East.function` / `East.asyncFunction` body it emits the `Platform` node with this signature — an async one inside a sync body is the build-time error. Paired at compile by name (the def's, or `name=`). Also importable bare: `from east import platform_function` |
1568
- | `@East.generic_platform_function(*, type_parameters, name=None, is_async=False, inputs=None, output=None, type_erased=False)` | Type-parameterized factory: the decorated fn is `fn(platform, *type_params) -> impl`; `is_async` is **explicit** (not inferred). With `inputs=`/`output=` (placeholders allowed anywhere: `output="T"`, `output=ArrayType("T")`) it is dual-mode too: `fs_open_beast(DictType(K, V), path)` in a body — the type arguments first, as TypeScript reads — emits the generic node; without them a body call is a build error naming what to declare. `type_erased=True` when the implementation ignores the type arguments (it reads the values): the decorated fn IS the implementation, python calls it directly (`causal_experiment(rows, config)`) and the factory the runtime binds is derived |
1569
- | a platform function RETURNING a lazy value | A `@platform_function` may return a paged hold — an `open_beast2_file(...)` value, `open_paged_file(T, path)` (a C-owned mapping), `fs_open_beast(...)`'s result — and the compiled body keeps its O(segment) cost model: the C value crosses the return seam by pointer (checked against the declared output type, never re-marshalled through python), and nothing python-side has to outlive it: the mapping is the value's own. The mirror of a paged ARGUMENT crossing in |
1570
- | `East.platform_functions(module) -> list` | Collects every decorated fn in `module` (pass `__name__`). Two consumers: `East.compile()` for in-process use, and a package's top-level `platform` list that `east-py run -p <module>` (and the e3 `{ custom }` runner) loads |
1571
- | `@memoize` / `@memoize(salt="…")` / `memoized = memoize(fn, salt="…")` | Content-addressed memo over ONE platform function. Apply **above** `@East.platform_function` (or inline on an imported one). Key = sha256(name + salts + per-input digests of the with-header BEAST2 encodings via the declared input types); value = with-header BEAST2 of the output, decoded via the declared output type. Inert by default |
1572
- | `configure_memo(directory, salt="")` | Activate (`None` deactivates) memoization for `@memoize` functions; overrides `EAST_MEMO_DIR` / `EAST_MEMO_SALT` env vars. Bump `salt` to invalidate after code edits — input-derived keys can't see them |
1573
-
1574
- ### What is NOT east-c
1575
-
1576
- Nearly everything above delegates to native builtins. What still runs python,
1577
- so you can reason about cost and semantics:
1578
-
1579
- - **`EastMatrix.map_elements`** — DEPRECATED: a python callback per element,
1580
- the one shape the strict surface removes; it warns and will go. Use the
1581
- arithmetic / reduction / mask / sparse methods, `map_rows`, or
1582
- `to_numpy()`/`to_torch()`. (`Vector.map`/`reduce` and `Matrix.map_rows` are the
1583
- native VectorMap / VectorFold / MatrixMapRows builtins.)
1584
- - **`Dict.get_or_insert`** composes membership and get python-side so `fn` runs
1585
- only on a miss (lazier than East's strict default). The other singles
1586
- (`insert`/`update`/`swap`/`delete`/`try_delete`/`insert_or_update`) are the
1587
- native builtins, error messages included.
1588
- - **Reduction sugar** — `mean`, `group_mean`, `group_size`, …: several native
1589
- passes, zero python per element, but not one fused builtin (as in TypeScript).
1590
- - **Boundary utilities** — `coerce_to`/`assert_value_of`/`type_of`, `variant`/
1591
- `struct` validation, `match()` dispatch, `compare_for`/`make_east_key`: python
1592
- walkers, by design — they are the python↔East edge.
1593
- - **Iteration** — `for x in arr` / `list(arr)` boxes per element (lazily, under
1594
- East's iteration lock, so mutating during the loop raises `Cannot modify …
1595
- during iteration`, as an East for-loop does); cross once with
1596
- `to_columns`/`to_numpy` instead.
1597
-
1598
- ## The east-py CLI
1599
-
1600
- `east-py` (the `east-py-cli` package) runs East IR with python platform
1601
- functions — the runner e3 drives for `runtime: 'east-py'` — and carries the
1602
- authoring tools:
1603
-
1604
- ```bash
1605
- east-py run prog.beast2 -p east-py-std -i a.beast2 -i b.json -o out.beast2 # -v: timings, peak memory
1606
- east-py run prog.beast2 -p east-py-std --profile # time per East and platform function
1607
- east-py exec unit.beast2 # one unit: the runner protocol e3 speaks
1608
- east-py convert value.beast2 # a beast2 value as East text (-o value.east to write it)
1609
- east-py transpile prog.beast2 -o prog.py # IR → a python module that rebuilds it
1610
- east-py export-functions pricing -o pricing.functions.beast2 -p east-py-std
1611
- east-py lint src/ ; east-py check src/ ; east-py lsp
1612
- east-py version -p east-py-std # the runner's version, and each package's
1613
- ```
1614
-
1615
- | Command | Does |
368
+ | File | Sections |
1616
369
  |---|---|
1617
- | `run <ir> [-p PACKAGE]… [-i FILE]… [-o FILE] [--decode lazy\|whole] [-v] [--profile]` | Compiles an IR file (`.beast2`, `.beast`, `.east` or `.json`) and calls it with one `-i` file per parameter, in order, each decoded by its extension to the parameter's type and FROZEN (a task input is immutable). An indexed beast2 collection input opens lazily, whatever it weighs — mapped, one segment decoded at a time, and decoded whole, once, when an operation the pager cannot serve first needs it — and a manifest file opens as the collection it names; `--decode whole` (east-c's and east-node's flag too) decodes every input before the program runs instead. `-o` writes the result in its extension's format (a collection as canonical indexed beast2); without it the result prints as East text. `-v` adds timings, peak memory and how each input was read: a lazy input's segment reads, or the resident memory a whole decode added |
1618
- | `exec <unit> [-v] [--profile]` | The runner protocol: one unit file says what to run — a program over its inputs, read as its `decode` says (as `run --decode`), or a merge of an output's parts — and where; the output is written by its kind and a result recorded. Exit 0 when it succeeded, 1 when it failed (the message and locations also go to stderr), 2 when the unit cannot be read or the result written. Relative paths in the unit are relative to its directory |
1619
- | `convert <file> [-o FILE.east] [-v]` | Decodes a beast2 value (its header names the type) and prints it as East text, or writes it to a `.east` file |
1620
- | `transpile <ir> [-o FILE] [--name NAME] [-p PACKAGE]…` | IR as a python module rebuilding it (`--name` binds it, default `main`; each `-p` prints its calls as that package's own functions) — [IR ↔ python](#ir--python-east-py-transpile-and-the-east-c-ir-toolbox) |
1621
- | `export-functions <module> -o FILE [-p PACKAGE]… [--name NAME] [--package-version V] [--only NAME]…` | A module's (a dotted name, or a `.py` path) `east_functions` as a function manifest; every platform call must be provided by a `-p` package; `--name` defaults to the module's top-level name, the version to the installed distribution's (else `0.0.0`); `--only` exports just those, building only them and what they call (a sibling that does not build fails nothing). `--batch FILE` in place of `-o`/`-p`/`--only` writes several manifests from one import, FILE a beast2 batch (`FunctionExportBatchType`: each entry's `output`, `packages` and `only`) — what `e3.export` runs per workspace package — [Cross-language functions](#cross-language-functions-east-py-export-functions-and-eastimport_function) |
1622
- | `lint [paths…] [--format text\|json] [--disable RULE]… [--exclude DIR]… [--list-rules]` | The rules (default path `.`); exit 1 on any finding, a warning included, 2 on a usage error — [Diagnostics](#diagnostics-at-edit-time--east-py-lint-east-py-check-flake8-east-py-lsp) |
1623
- | `check <targets…> [--format text\|json] [--only-if-enabled]` | The build's own errors; exit 1 on any |
1624
- | `lsp [--probe]` | The Language Server over stdio (needs `pygls`); `--probe` reports whether it can start here |
1625
- | `version [-p PACKAGE]…` | The `east-py-cli` and `east-py` versions (and the Cython extensions loaded), and each package's version and function count |
1626
-
1627
- A `-p` package is a python module name (`east-py-std` imports `east_py_std`)
1628
- exporting a top-level `platform` list; a project's own module is loaded the same
1629
- way (see [Project-owned platform module](#project-owned-platform-module-calling-your-python-from-e3)).
1630
- `run` and `exec` take `--exit-with-parent`: exit once stdin reaches end of file,
1631
- for a parent that holds a stdin pipe it never writes to.
1632
-
1633
- **Profiling.** `run` and `exec` take `--profile`, east-c's profiler and report:
1634
- after the work (and after a failure's error), every East function and platform
1635
- function it called, by self time, with its calls, its total time and where it
1636
- is — the `file:line:column` it was written at and, for a helper called inside
1637
- another body, where it was called. A platform function (torch, a solver) is an
1638
- entry of its own, so its time is not its caller's, and so is each input read
1639
- lazily, named as `-v` names it (`input 1`): the segments its reads decode, a
1640
- call each — keyed reads at random beyond the segments the runner keeps, or a
1641
- scan repeated, show there, not in the reading function's self time. `EAST_PROFILE=1` profiles
1642
- without the flag — e3 hands its environment to the runner, so a task's report
1643
- lands in its log (`e3 task logs`) — and `EAST_PROFILE_INTERVAL=N` prints the
1644
- profile so far every N seconds while the work runs. A python function linked
1645
- into a TypeScript task carries its locations through the manifest, so the
1646
- report names its python lines.
1647
-
1648
- ## Key Patterns
1649
-
1650
- ### Python values vs East expressions
1651
-
1652
- The `east` skill's first rule, in python: a body's PARAMETERS are expressions;
1653
- a python value used inside a body must lift. Scalars lift on their own; a
1654
- collection must be an explicit choice.
1655
-
1656
- ```python
1657
- RATES = EastDict(StringType, FloatType, {"AUD": 0.65}) # python-side East value
1658
- CUTOFF = 100
1659
-
1660
- @East.function([Row], FloatType)
1661
- def usd(b, r):
1662
- return East.if_else(r.amount > CUTOFF, # a scalar constant lifts (TS East.value)
1663
- r.amount * RATES.get_or_default(r.ccy, 1.0), # an EXPLICIT build snapshots the dict: a literal, rebuilt per call
1664
- 0.0)
1665
-
1666
- rows.map(lambda b, r: r.amount * RATES.get_or_default(r.ccy, 1.0)) # ❌ an eager callback REFUSES
1667
- # the collection capture:
1668
- conv = East.function([Row, DictType(StringType, FloatType)], FloatType,
1669
- lambda b, r, t: r.amount * t.get_or_default(r.ccy, 1.0))
1670
- rows.map(conv.bind(RATES)) # ✅ bind it: live, zero-copy
1671
- ```
1672
-
1673
- ### The canonical platform function
1674
-
1675
- ```python
1676
- from east import East, FloatType, StringType, StructType, ArrayType, struct
1677
-
1678
- LineItem = StructType([("name", StringType), ("price", FloatType)]) # Struct<String, Float>
1679
-
1680
- @East.platform_function(inputs=[FloatType, ArrayType(LineItem)], output=ArrayType(LineItem))
1681
- def convert_prices(fx_rate, items):
1682
- # items: an east-c-backed array with eager methods; row["price"] is a plain float
1683
- return items.map(lambda b, row: struct(
1684
- {"name": row["name"], "price": row["price"] * fx_rate}, # plain f64 * f64
1685
- LineItem, # tag + validate the result
1686
- ))
1687
- ```
1688
- `.map` builds the lambda into an East function — the loop and the row construction run
1689
- in east-c, with `fx_rate` baked in as a constant; `struct(..., LineItem)` is dual-mode, so
1690
- the same spelling builds the row here and on plain values; the decorator validates the
1691
- `Array<LineItem>` result — a named `EastTypeError` instead of silent corruption.
1692
-
1693
- ### Project-owned platform module (calling your Python from e3)
1694
-
1695
- To call a Python platform function from an **e3 task**, package it so `east-py
1696
- run -p <module>` can load it: each module ends with `<name>_impl =
1697
- East.platform_functions(__name__)`, and the package `__init__.py` aggregates them into
1698
- a top-level `platform` list (the same shape as east-py-std / east-py-datascience).
1699
-
1700
- ```python
1701
- # platform_module/forecast.py
1702
- @East.platform_function(inputs=[ArrayType(FloatType)], output=FloatType,
1703
- name="my_project.forecast") # dotted "<project>.<fn>"; MUST byte-match
1704
- def forecast(history): # the TS East.platform(...) declaration
1705
- return sum(history) / len(history) if history else 0.0
1706
- forecast_impl = East.platform_functions(__name__)
1707
-
1708
- # platform_module/__init__.py
1709
- from .forecast import forecast_impl
1710
- platform = [*forecast_impl] # what `east-py run -p platform_module` loads
1711
- ```
1712
-
1713
- The e3 task wires it with `{ runtime: "east-py", platforms: [{ custom:
1714
- "platform_module" }, "east-py-std"] }`. East code needs a TS `East.platform(
1715
- "my_project.forecast", [...], ...)` **declaration** with the identical name (no
1716
- codegen). Add native deps (numpy, …) to `pyproject.toml` and run `uv sync`. See
1717
- **east-project** for the full scaffold (`--platform`) and the setuptools
1718
- packaging; **e3** for the runner.
1719
-
1720
- ### Memoize expensive pure stages (dev/test harnesses)
1721
-
1722
- ```python
1723
- from east import East, configure_memo, memoize
1724
-
1725
- @memoize # eligibility — the author asserts purity
1726
- @East.platform_function(inputs=[ArrayType(RowType), ConfigType], output=ModelType)
1727
- def train_model(rows, config): ...
1728
-
1729
- # A test harness flips the whole package on with one call (or EAST_MEMO_DIR):
1730
- configure_memo("/tmp/my_project", salt="v1") # None deactivates
1731
- loaded = memoize(other_pkg.load_csv, salt=file_digest) # inline, per-call salt
1732
- ```
1733
-
1734
- The platform-function boundary is the cache boundary, e3-style: hit = decode the
1735
- stored BEAST2 blob via the declared output type (skipping the body entirely),
1736
- miss = compute + atomic save. Only mark functions **pure in their East inputs** —
1737
- a hit on a function that writes files/rows/requests silently drops those effects.
1738
- Stochastic fits memoize their first realization. Code edits don't change keys —
1739
- bump the salt. Inert with no directory configured: under e3, the dataflow's
1740
- content-addressed task cache is the real memo and this stays off.
1741
-
1742
- ### Fork-parallel export of a huge table (one file out)
1743
-
1744
- The shape for "N CPUs on one table": build the expensive shared context once,
1745
- let forked workers inherit it copy-on-write, and ship a single indexed file —
1746
- consumers never learn the export was parallel. (Windows runs the same contract
1747
- inline, sequentially; the output is byte-identical either way.)
1748
-
1749
- ```python
1750
- from east.serialization.beast2 import open_beast2_file, write_beast2_file_parallel
1751
-
1752
- ROW = StructType([("order_id", StringType), ("qty", IntegerType), ("total", FloatType)])
1753
-
1754
- lookups = load_reference_tables() # multi-GB keyed dicts: built ONCE, pre-fork,
1755
- # inherited COW — never pickled, never rebuilt
1756
-
1757
- def produce(span): # runs in the worker process
1758
- start, count = span # yield batches of any size — the managed
1759
- for chunk in chunk_ranges(start, count, 8192): # writer cuts the canonical segments
1760
- yield compute_rows(lookups, chunk) # eager methods + East functions: native
1761
-
1762
- write_beast2_file_parallel(
1763
- "orders.beast2", ArrayType(ROW),
1764
- partitions=row_ranges(total_records, shards=13), # partition order = row order
1765
- produce=produce,
1766
- processes=13, # a worker failure kills the rest, cleans up, re-raises
1767
- )
1768
-
1769
- # Consumers — e.g. the table's @East.platform_function loader — see ONE file, no
1770
- # catalog, no shard names, and read it at one segment of memory:
1771
- @East.platform_function(inputs=[StringType], output=ArrayType(ROW))
1772
- def load_orders(path):
1773
- with open_beast2_file(path, ArrayType(ROW)) as f:
1774
- return f.load() # or compute on f directly, never holding the table
1775
- ```
1776
-
1777
- Shards produced by your own process topology (or on another machine)? Merge
1778
- them directly — a pure byte copy, nothing re-encoded:
1779
-
1780
- ```python
1781
- splice_beast2_files("orders.beast2", ArrayType(ROW), sorted(shard_paths), verify=True)
1782
- ```
1783
-
1784
- ### Keyed lookups against a file too big to load
1785
-
1786
- A reference table exported as a Dict file answers point reads without ever
1787
- being held in memory: east-c fence-searches the segment index and decodes
1788
- one segment per lookup, so a sparse join against a multi-GB file stays
1789
- cheap. (A *dense* pass over most keys is still better as whole-file compute
1790
- — `f.group_sum(...)`, `f.filter(...)`, any read method directly on the file.)
1791
-
1792
- ```python
1793
- from east.serialization.beast2 import open_beast2_file
1794
-
1795
- with open_beast2_file("orders.beast2") as orders: # Dict<String, Order>; Order = Struct{id, total}
1796
- order = orders[order_id] # fence search → ONE segment decode
1797
- hot = orders.get_keys(wanted_ids, # each owning segment decodes once;
1798
- lambda b, k: {"id": k, "total": 0.0}) # `fill` is a body (a python dict
1799
- if candidate_id in orders: # captured from outside is refused)
1800
- ...
1801
-
1802
- # A sorted Array file answers range queries the same way — global insertion
1803
- # indices, then slice exactly the covered rows:
1804
- with open_beast2_file("events.beast2") as events: # Array<Event> sorted by timestamp
1805
- span = events.find_sorted_range(cutoff_event)
1806
- window = events.slice(span["start"], span["end"])
1807
- ```
1808
-
1809
- ### Sequential logic that stays in east-c (worklist / replay)
1810
-
1811
- When the next step depends on the last — a worklist, a BFS, a fixpoint, a
1812
- topological replay — no `map`/`filter`/`group_reduce` expresses it, and a
1813
- python loop over decoded rows is the whole job's cost. The statement surface
1814
- threads the state through the loop natively, so the whole thing is ONE
1815
- compiled function. Two rules carry most of the weight. **Accumulate in
1816
- place** — a collection threaded through a state is rebuilt every iteration,
1817
- so `order.concat(...)` is O(n²) over the loop where `order.push_last(...)` is
1818
- O(1). And **a mutation is a statement** — `b.do(acc.push_last(x))` in a
1819
- statement body, `East.block(acc.push_last(x), result)` in an expression body;
1820
- a bare `acc.push_last(x)` line is evaluated at build time and thrown away
1821
- (the build raises rather than compile a loop that silently does nothing).
1822
-
1823
- ```python
1824
- from east import ArrayType, DictType, East, IntegerType, StringType
1825
-
1826
- Node, Edges = StringType, DictType(StringType, ArrayType(StringType))
1827
- Indeg = DictType(StringType, IntegerType)
1828
-
1829
- # Kahn's algorithm — ONE compiled function, the TypeScript `$` shape: a
1830
- # worklist the loop appends to, a cursor into it, and per-node successors
1831
- # decremented in place.
1832
- @East.function([ArrayType(Node), Edges, Indeg], ArrayType(Node))
1833
- def topo_order(b, roots, succ, indeg):
1834
- ready = b.const(roots.copy()) # task inputs arrive frozen: copy
1835
- remaining = b.const(indeg.copy())
1836
- order = b.const(East.new_array(Node))
1837
- i = b.let(0)
1838
-
1839
- def step(b, label):
1840
- node = b.const(ready.get(i))
1841
- b.do(order.push_last(node))
1842
-
1843
- def visit(b, v):
1844
- b.do(remaining.insert_or_update(v, -1, lambda _b, old, d: old + d))
1845
- b.if_(remaining.get(v) == 0, lambda b: b.do(ready.push_last(v)))
1846
-
1847
- b.for_(succ.get_or_default(node, East.new_array(Node)), visit)
1848
- b.assign(i, i + 1)
1849
-
1850
- b.while_(i < ready.size(), step)
1851
- return order
1852
-
1853
- @East.platform_function(inputs=[ArrayType(Node), Edges, Indeg], output=ArrayType(Node))
1854
- def replay_order(roots, succ, indeg):
1855
- return topo_order(roots, succ, indeg)
1856
- ```
1857
-
1858
- The same algorithm as EXPRESSION forms — a state struct threaded through
1859
- `East.while_`/`East.for_`, the shape that fits inside a one-expression
1860
- callback:
1861
-
1862
- ```python
1863
- topo_order = East.function(
1864
- [ArrayType(Node), Edges, Indeg], ArrayType(Node),
1865
- lambda b, roots, succ, indeg: East.while_(
1866
- {"ready": roots.copy(), "indeg": indeg.copy(),
1867
- "order": East.new_array(Node), "i": 0},
1868
- cond=lambda b, s: s.i < s.ready.size(),
1869
- body=lambda b, s: East.let(s.ready.get(s.i), lambda b, node: East.block(
1870
- s.order.push_last(node),
1871
- East.for_(
1872
- succ.get_or_default(node, East.new_array(Node)),
1873
- {**s, "i": s.i + 1}, # inner loop's state
1874
- lambda b, t, v: East.block(
1875
- t.indeg.insert_or_update(v, -1, lambda b, old, d: old + d),
1876
- East.if_else(t.indeg.get(v) == 0, # newly ready?
1877
- East.block(t.ready.push_last(v), t),
1878
- t))))), # else: unchanged
1879
- ).order)
1880
-
1881
- # any carried state — a forward fill is a `for_` whose state remembers the last value
1882
- forward_fill = East.function(
1883
- [ArrayType(StringType)], ArrayType(StringType),
1884
- lambda b, cells: East.for_(
1885
- cells, {"last": "", "out": East.new_array(StringType)},
1886
- lambda b, s, cell: East.let(
1887
- East.if_else(cell == "", s.last, cell),
1888
- lambda b, v: East.block(s.out.push_last(v), {"last": v, "out": s.out}))).out)
1889
- ```
1890
-
1891
- To stop early, put `East.break_(state)` in an `if_else` arm — the state
1892
- argument commits before the jump, so the answer survives; `East.label(...)` on
1893
- an outer loop lets an inner one break all the way out. Reach for all of this
1894
- only when the work is genuinely sequential: a `group_reduce` or a `reduce` is
1895
- both shorter and faster when it fits.
1896
-
1897
- ### Trees from flat rows, and recursion
1898
-
1899
- `to_tree` builds nested data — any depth, your own node type — from rows that
1900
- name their key and their parent's key. `build` runs once per row, children
1901
- first; the result is the roots in source order. The walk is iterative, in
1902
- east-c, on values and in a body alike:
1903
-
1904
- ```python
1905
- from east import ArrayType, OptionType, StringType, StructType, recursive_type
1906
-
1907
- Row = StructType([("id", StringType), ("parent", OptionType(StringType)), ("name", StringType)])
1908
- Node = recursive_type(lambda self: StructType([("name", StringType), ("children", ArrayType(self))]))
1909
-
1910
- roots = rows.to_tree(Node,
1911
- lambda b, r: r.id,
1912
- lambda b, r: r.parent,
1913
- lambda b, r, _i, children: {"name": r.name, "children": children})
1914
- ```
1915
-
1916
- - A row whose parent is `none` is a root, and so is an orphan (its parent key
1917
- is not in the rows). A repeated key ❗ `toTree: duplicate key <key>`; a cycle
1918
- ❗ `toTree: cycle through key <key>` — both before any `build`.
1919
- - A non-recursive `Node` folds bottom-up: `IntegerType` with
1920
- `children.sum() + 1` is each root's subtree size. A Set's element and a
1921
- Dict's key are their own keys, so those take no `key`.
1922
-
1923
- `flatten_tree` is the way back: every node of nested data, each with its path
1924
- of keys (a Dict's) or indices (an Array's) from its root, depth first — each
1925
- node before its descendants, siblings in key or array order. `children` names a
1926
- node's children, of the collection's own type:
1927
-
1928
- ```python
1929
- from east import DictType, FloatType, StringType, StructType, recursive_type
1930
-
1931
- Part = recursive_type(lambda self: StructType([("name", StringType), ("cost", FloatType), ("parts", DictType(StringType, self))]))
1932
-
1933
- nodes = parts.flatten_tree(lambda b, part: part.unwrap().parts)
1934
- # an Array of {path, node}: ["pump"], ["pump", "motor"], ["pump", "motor", "rotor"], ["pump", "seal"]
1935
- ```
1936
-
1937
- - An Array of `{path, node}`, not a Dict keyed by path: a Dict key is
1938
- immutable, and a path is an Array. `to_tree` over it, keyed by `path` and
1939
- parented by `path` less its last step, rolls the tree up at every root.
1940
- - `children` runs once per node, in the result's order. Each path is an array
1941
- of its own, so a chain n deep holds n(n+1)/2 path elements.
1942
- - **Recursion** — a function calling itself through a captured variable — is
1943
- for shallow depths, in the hundreds. Deeper, a call ❗ `call stack exhausted:
1944
- East calls nested too deeply`, a catchable `EastError` on every runtime. Walk
1945
- deep data with `b.while_`, `to_tree` or `flatten_tree`; a recursive VALUE's depth is still
1946
- bounded, at tens of thousands of levels, by east-c's recursive collect and
1947
- free.
1948
-
1949
- ### Sort uses East's total order
1950
-
1951
- ```python
1952
- # WRONG — Python's default ordering (incorrect for floats/NaN, mixed, type-specific)
1953
- sorted(list(arr))
1954
-
1955
- # CORRECT — East total order, in east-c (the TypeScript names)
1956
- arr.sort() # new array
1957
- arr.sort_in_place() # in place
1958
- arr.sort(lambda b, r: r["x"]) # by a projection, still East-ordered
1959
- ```
1960
-
1961
- ### Scalars: use the `East.<Type>` utilities for consistency — above all String & DateTime
1962
-
1963
- East scalars *are* Python scalars, so Python operators run on them — but the
1964
- `East.<Type>` namespaces (and `East.less`/`compare`/`equal`) are the standardised
1965
- ops that give the **same** answer in Python, C, TS, and cached e3 tasks. Python's
1966
- `str` methods, `re`, `datetime`, `strftime`, `//`, and `<` drift by
1967
- engine / locale / timezone; the East functions do not. Use them for anything with
1968
- divergent semantics — and **always** for strings and dates.
1969
-
1970
- ```python
1971
- # String — East.String.*, not Python str/re: split edge cases, the regex engine,
1972
- # trim's whitespace set, and JSON encoding all differ from Python
1973
- East.String.split("a,b,c", ",") # East semantics, not str.split
1974
- East.String.regex_replace(s, r"\d+", "#") # East regex engine, not Python re
1975
- East.String.trim(s) # East's whitespace set
1976
- East.String.print_json(IntegerType, 5) # '"5"' — Integer is a JSON *string* in East
1977
-
1978
- # DateTime — East.DateTime.* only. East DateTime is UTC; tokens are Day.js, not strftime
1979
- East.DateTime.add_milliseconds(dt, 86_400_000) # +1 day, not dt + timedelta(days=1)
1980
- East.DateTime.duration_milliseconds(a, b) # a − b (ms), standardised
1981
- East.DateTime.get_day_of_week(dt) # Monday == 1, not dt.weekday() (== 0)
1982
- East.DateTime.print_formatted(dt, "YYYY-MM-DD HH:mm:ss") # Day.js tokens, not dt.strftime("%Y-…")
1983
- East.Float.print_currency(total) # "$1,234.57" — the stdlib, same in TS
1984
-
1985
- # Ordering / equality — East total order (NaN-correct, mixed types, variants-by-name)
1986
- East.less(FloatType, a, b) # not a < b / sorted(...)
1987
- # Integer is i64 — divide truncates; Python's unbounded int can overflow i64
1988
- East.Integer.divide(7, 2) # 3
1989
- # (you also can't method-call a Python float/str — the ops live on the namespace)
1990
- East.Float.sqrt(2.0) # not (2.0).sqrt()
1991
- ```
1992
-
1993
- ### torch interop (in a torch-having package, inline — no helpers)
1994
-
1995
- ```python
1996
- import torch, numpy as np
1997
- t = mat.to_torch() # writeable torch copy of the buffer
1998
- out = EastMatrix(FloatType, model(t).detach().cpu().numpy()) # bridge canonicalizes dtype
1999
- ```
370
+ | [reference/writing-east-functions.md](reference/writing-east-functions.md) | Declare, implement, build, compile · Every body takes the block first · The expression surface, type by type · The standard library |
371
+ | [reference/control-flow-and-queries.md](reference/control-flow-and-queries.md) | Control flow — the expression forms · Queries — `East.jq` · What a body may reference |
372
+ | [reference/performance.md](reference/performance.md) | Performance — the levers, ranked |
373
+ | [reference/ir-and-diagnostics.md](reference/ir-and-diagnostics.md) | IR ↔ python · Diagnostics at edit time |
374
+ | [reference/cross-language.md](reference/cross-language.md) | Cross-language functions |
375
+ | [reference/values-construction-and-json.md](reference/values-construction-and-json.md) | Construction & validation · A JSON contract for a type |
376
+ | [reference/value-methods.md](reference/value-methods.md) | EastArray / EastSet / EastDict / EastVector / EastMatrix / EastBlob / EastStruct, EastVariant, EastRef method surfaces · Container generators · East.&lt;Type&gt; namespaces · DateTime format codes |
377
+ | [reference/beast2.md](reference/beast2.md) | Beast2 streaming |
378
+ | [reference/columns-and-platform-functions.md](reference/columns-and-platform-functions.md) | Columnar escape hatches · Platform functions · What is NOT east-c |
379
+ | [reference/cli.md](reference/cli.md) | The east-py CLI |
380
+ | [reference/patterns.md](reference/patterns.md) | Python values vs East expressions · The canonical platform function · Project-owned platform module · Memoize · Fork-parallel export · Keyed lookups in a big file · Sequential logic that stays in east-c · Trees from flat rows, and recursion · Sort uses East's total order · Scalars: use the `East.<Type>` utilities · torch interop |
2000
381
 
2001
382
  ## Sharp edges
2002
383