@avytheone/efmesh 0.1.0-beta.1 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +139 -89
- package/README.md +61 -9
- package/README.ru.md +57 -5
- package/SPEC.md +542 -459
- package/package.json +4 -1
- package/src/cli.ts +352 -12
- package/src/efmesh.ts +2 -0
- package/src/index.ts +32 -2
- package/src/plan/categorize.ts +4 -1
- package/src/plan/diff.ts +221 -1
- package/src/plan/executor.ts +20 -0
- package/src/plan/explain.ts +114 -0
- package/src/plan/fingerprint.ts +60 -11
- package/src/plan/planner.ts +175 -6
- package/src/plan/run.ts +50 -12
- package/src/plan/schedule.ts +212 -0
- package/src/plan/status.ts +95 -0
- package/src/state/postgres.ts +49 -0
- package/src/state/sqlite.ts +47 -0
- package/src/state/store.ts +33 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,96 +1,146 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
## [
|
|
3
|
+
Format — [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
4
|
+
versioning — [SemVer](https://semver.org/).
|
|
5
|
+
Internal development history was tracked in phases F0–F6 (SPEC.md §13);
|
|
6
|
+
the first version gathers them in full.
|
|
7
|
+
|
|
8
|
+
## [0.2.0] — 2026-07-16
|
|
9
|
+
|
|
10
|
+
Theme: "operator and team" — efmesh in the hands of a non-author: the
|
|
11
|
+
operator of the nightly cron and a team with CI.
|
|
12
|
+
|
|
13
|
+
- `efmesh status <env>` — what is going on in one command: the last applied
|
|
14
|
+
plan, interval lag per incremental model, failed intervals, recent run
|
|
15
|
+
ticks (#1).
|
|
16
|
+
- Run tick journal in the state store (`runs`, schema v4 — `efmesh migrate`
|
|
17
|
+
with a store backup): every tick records its outcome, including
|
|
18
|
+
unsuccessful ones (#2).
|
|
19
|
+
- `--json` on `plan`, `audit` and `status`: a stable machine-readable shape
|
|
20
|
+
for CI and bots; exit codes unchanged (#3).
|
|
21
|
+
- Canonicalization cache in the state store (schema v5): a no-op plan on
|
|
22
|
+
2000 models drops from ~0.6 s to ~50 ms; the cache key includes the
|
|
23
|
+
dialect and `FINGERPRINT_VERSION`, so canon drift can never be masked (#8).
|
|
24
|
+
- Integration test: stale lock reclaim under a real `kill -9` (#7).
|
|
25
|
+
- `plan --explain`: for every change — which canonical-AST nodes diverged
|
|
26
|
+
and why the category followed (cascade sources for `indirect`, inherited
|
|
27
|
+
physics for `forward-only`); also shipped in `--json` as `explain`
|
|
28
|
+
(`PlanAction.explain` in the library API). AST paths are a debugging
|
|
29
|
+
hint, not a contract (#4).
|
|
30
|
+
- Indirect physics reuse — the sqlmesh "indirect non-breaking" class: a
|
|
31
|
+
descendant whose own AST did not change and whose changed parents are all
|
|
32
|
+
non-breaking/forward-only inherits the previous version's physical table
|
|
33
|
+
and interval accounting instead of a full rebuild (scdType2 keeps its row
|
|
34
|
+
history). Safety is proven, not assumed: the fingerprint recomputed with
|
|
35
|
+
the parents' old fingerprints must reproduce the old version, so
|
|
36
|
+
simultaneous metadata drift disables reuse (#5).
|
|
37
|
+
- `--reclassify model=breaking|non-breaking` on `plan`/`apply`: the
|
|
38
|
+
operator's verdict as a flag on top of `--explain` (no interactive
|
|
39
|
+
dialog — works in CI), journaled with `applied_by`
|
|
40
|
+
(`PlanAction.reclassifiedFrom`); governs descendants' physics reuse.
|
|
41
|
+
Guard rail: dropped columns declared non-breaking are refused (#5).
|
|
42
|
+
- `diff --data` — compare the DATA of two environments: full row counts,
|
|
43
|
+
key overlap (grain or the kind's key), per-column mismatch rates among
|
|
44
|
+
matched keys, schema drift between sides. `--sample P` compares a
|
|
45
|
+
deterministic share of keys (md5 buckets aligned across both sides — no
|
|
46
|
+
false only-in rows); `--json` for CI (`dataDiffEnvironments` in the
|
|
47
|
+
library API) (#6).
|
|
48
|
+
- `efmesh schedule <env>` — register the `run` tick in the OS scheduler via
|
|
49
|
+
`Bun.cron` (crontab/launchd/Task Scheduler; `engines.bun >= 1.3.11`).
|
|
50
|
+
Idempotent by title; `--remove`, `--list`; on cron-less Linux
|
|
51
|
+
(Arch family) it detects the missing daemon and `--print-systemd` emits
|
|
52
|
+
user units with `Persistent=true` instead (#10).
|
|
53
|
+
|
|
54
|
+
## [0.1.0-beta.2] — 2026-07-16
|
|
55
|
+
|
|
56
|
+
- Release pipeline: publishing to npmjs.org from GitHub Actions on a `v*`
|
|
57
|
+
tag via Trusted Publishing (OIDC, no tokens) with provenance.
|
|
58
|
+
The package contents did not change — the release verifies the pipeline itself.
|
|
9
59
|
|
|
10
60
|
## [0.1.0-beta.1] — 2026-07-16
|
|
11
61
|
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
###
|
|
15
|
-
|
|
16
|
-
- `defineModel`/`defineExternal`:
|
|
17
|
-
|
|
18
|
-
- Fingerprint
|
|
19
|
-
|
|
20
|
-
-
|
|
21
|
-
|
|
22
|
-
- `kind.incrementalByTimeRange`:
|
|
23
|
-
|
|
24
|
-
-
|
|
25
|
-
- `target: "parquet"`:
|
|
26
|
-
|
|
27
|
-
###
|
|
28
|
-
|
|
29
|
-
-
|
|
30
|
-
- `testModel` (`efmesh/testing`):
|
|
31
|
-
|
|
32
|
-
-
|
|
33
|
-
- `efmesh run`:
|
|
34
|
-
`Runner.daemon`
|
|
62
|
+
First public beta. Everything below is new.
|
|
63
|
+
|
|
64
|
+
### Models and plans (F0–F1)
|
|
65
|
+
|
|
66
|
+
- `defineModel`/`defineExternal`: models are TypeScript modules, references
|
|
67
|
+
are imports, data shape is Effect Schema; the DAG is built from values.
|
|
68
|
+
- Fingerprint over the canonical AST (`json_serialize_sql` in DuckDB):
|
|
69
|
+
reformatting SQL does not trigger a rebuild.
|
|
70
|
+
- Version snapshots, virtual environments (views over physical storage),
|
|
71
|
+
a plan as a diff, promotion without recomputation.
|
|
72
|
+
- `kind.incrementalByTimeRange`: interval ledger in the state store, backfill
|
|
73
|
+
in DELETE+INSERT batches within a transaction, resume from where it stopped, `lookback`.
|
|
74
|
+
- A schema contract before a build (`DESCRIBE` against the declared Schema).
|
|
75
|
+
- `target: "parquet"`: a lake locally or on S3 (httpfs), interval = partition.
|
|
76
|
+
|
|
77
|
+
### Quality and operations (F2)
|
|
78
|
+
|
|
79
|
+
- `notNull`/`unique`/`accepted` audits, blocking/warn.
|
|
80
|
+
- `testModel` (`efmesh/testing`): unit tests for models on fixtures
|
|
81
|
+
in in-memory DuckDB.
|
|
82
|
+
- Change categorization breaking / non-breaking / indirect by AST.
|
|
83
|
+
- `efmesh run`: an idempotent scheduler tick with a cross-process lock;
|
|
84
|
+
`Runner.daemon` for embedding.
|
|
35
85
|
- `efmesh janitor`, `efmesh diff`, `defineSeed`,
|
|
36
|
-
`kind.incrementalByUniqueKey` (upsert),
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
###
|
|
40
|
-
|
|
41
|
-
- Postgres:
|
|
42
|
-
canonicalize
|
|
43
|
-
- `--forward-only`:
|
|
44
|
-
|
|
45
|
-
- `kind.scdType2` (
|
|
46
|
-
|
|
47
|
-
-
|
|
48
|
-
|
|
49
|
-
###
|
|
50
|
-
|
|
51
|
-
-
|
|
52
|
-
|
|
53
|
-
- `target: "ducklake"`:
|
|
54
|
-
|
|
55
|
-
-
|
|
56
|
-
- `efmesh init` (
|
|
57
|
-
-
|
|
58
|
-
|
|
59
|
-
###
|
|
60
|
-
|
|
61
|
-
-
|
|
62
|
-
|
|
63
|
-
(`LockHeldError`);
|
|
64
|
-
-
|
|
65
|
-
-
|
|
66
|
-
|
|
67
|
-
- `applied_by`
|
|
68
|
-
-
|
|
69
|
-
|
|
70
|
-
###
|
|
71
|
-
|
|
72
|
-
- effect
|
|
73
|
-
|
|
74
|
-
- Fingerprint
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
- Parquet
|
|
81
|
-
|
|
82
|
-
-
|
|
83
|
-
exit
|
|
86
|
+
`kind.incrementalByUniqueKey` (upsert), export of marts to ATTACH databases,
|
|
87
|
+
metrics and spans.
|
|
88
|
+
|
|
89
|
+
### Breadth (F3)
|
|
90
|
+
|
|
91
|
+
- Postgres: an engine on `Bun.SQL` (pool, parallel backfill batches),
|
|
92
|
+
canonicalize via libpg_query, state store in the `efmesh_state` schema.
|
|
93
|
+
- `--forward-only`: a change without replaying history — the new version
|
|
94
|
+
inherits physical storage and done-intervals, new columns via `ALTER`.
|
|
95
|
+
- `kind.scdType2` (row history), `kind.embedded` (subquery without
|
|
96
|
+
materialization), `defineSqlModel` (raw `.sql` with `@ref`/`@start`/`@end`).
|
|
97
|
+
- Column-level `efmesh lineage`, `efmesh graph --html`.
|
|
98
|
+
|
|
99
|
+
### Operational maturity (F4)
|
|
100
|
+
|
|
101
|
+
- Cross-model DAG concurrency for apply (`--jobs`): a model starts
|
|
102
|
+
as soon as its parents are ready; on DuckDB it is honestly 1 (a single connection).
|
|
103
|
+
- `target: "ducklake"`: physical storage in a DuckLake catalog, catalog
|
|
104
|
+
snapshots and time travel come as a bonus.
|
|
105
|
+
- Standalone `efmesh audit <env>` over an environment's view layer.
|
|
106
|
+
- `efmesh init` (scaffold), state-store schema version + `efmesh migrate`.
|
|
107
|
+
- Plan confirmation in a TTY (`--yes` skips it, a non-TTY proceeds without asking).
|
|
108
|
+
|
|
109
|
+
### Beta gate (F5)
|
|
110
|
+
|
|
111
|
+
- Cross-process lock on `apply` — a shared env lock with `run`: parallel
|
|
112
|
+
mutations of an environment from different processes are mutually exclusive
|
|
113
|
+
(`LockHeldError`); the janitor has its own global lock.
|
|
114
|
+
- Model discovery by glob: `discovery: "models/**/*.ts"` in the config.
|
|
115
|
+
- Backfill batch retries: `--retries N`, `Schedule.exponential`;
|
|
116
|
+
audits are not retried.
|
|
117
|
+
- `applied_by` in the plans journal (store schema version 2).
|
|
118
|
+
- MIT license.
|
|
119
|
+
|
|
120
|
+
### Beta gate, part 2 (F6)
|
|
121
|
+
|
|
122
|
+
- effect is a peerDependency with an exact beta pin; a weekly CI job against
|
|
123
|
+
a fresh beta (an early signal of v4 drift).
|
|
124
|
+
- Fingerprint as a contract: `FINGERPRINT_VERSION` in snapshots (store
|
|
125
|
+
schema 3), golden tests freeze DuckDB and libpg_query canonicalization;
|
|
126
|
+
a snapshot of a foreign version is a loud `FingerprintVersionError`.
|
|
127
|
+
- The janitor↔apply race is closed transactionally: claiming an orphan is
|
|
128
|
+
atomic with the checks, resurrection clears the orphan status, promotion
|
|
129
|
+
checks snapshot liveness — a view never switches to removed physical storage.
|
|
130
|
+
- Parquet partitions are written atomically (temp + rename); `migrate` takes
|
|
131
|
+
a copy of the SQLite store before upgrading the schema.
|
|
132
|
+
- **Breaking:** `apply` in a non-TTY with changes requires `--yes`;
|
|
133
|
+
exit code `2` = "awaiting a human" (confirmation refused,
|
|
84
134
|
`RunBlockedByChangesError`).
|
|
85
|
-
-
|
|
86
|
-
- README
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
###
|
|
90
|
-
|
|
91
|
-
- Effect v4
|
|
92
|
-
|
|
93
|
-
-
|
|
94
|
-
|
|
95
|
-
- Nullability
|
|
96
|
-
|
|
135
|
+
- The public API is narrowed to a deliberate whitelist (`index.ts`).
|
|
136
|
+
- README in English (the Russian mirror is README.ru.md), a
|
|
137
|
+
"who this is for (and who it isn't)" section.
|
|
138
|
+
|
|
139
|
+
### Known limitations
|
|
140
|
+
|
|
141
|
+
- Effect v4 is a beta dependency; the efmesh API sticks to a stable
|
|
142
|
+
subset.
|
|
143
|
+
- A single `bun build --compile` binary builds, but standalone Bun
|
|
144
|
+
cannot resolve the `"efmesh"` import from a runtime config — distribution is via the package.
|
|
145
|
+
- Nullability is not part of the schema contract (DuckDB `DESCRIBE` does not report it) —
|
|
146
|
+
it is expressed with the `notNull` audit.
|
package/README.md
CHANGED
|
@@ -2,12 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
> Data transformation in the spirit of [sqlmesh](https://sqlmesh.com) — on TypeScript, [Bun](https://bun.sh) and [Effect](https://effect.website).
|
|
4
4
|
|
|
5
|
-
[](https://github.com/avytheone/efmesh/actions/workflows/ci.yml)  ](https://github.com/avytheone/efmesh/actions/workflows/ci.yml)     
|
|
6
6
|
|
|
7
7
|
*Русская версия: [README.ru.md](./README.ru.md).*
|
|
8
8
|
|
|
9
9
|
Models are plain TypeScript modules: SQL bodies, imports as dependencies, Effect Schema as the data shape. efmesh fingerprints every model by its canonical AST, keeps versions as snapshots, computes a plan as the diff between your project and an environment, and applies exactly that plan: physical tables are rebuilt only where something actually changed, while environments (dev/prod/…) are virtual views over shared physical storage — promoting to prod costs zero recomputation.
|
|
10
10
|
|
|
11
|
+
<p align="center"><img src="docs/demo.svg" alt="efmesh demo: a ref typo is a compile error; a plan rebuilds exactly the changed branch; promotion is a view swap" width="840"></p>
|
|
12
|
+
|
|
11
13
|
```ts
|
|
12
14
|
import { Schema } from "effect"
|
|
13
15
|
import { defineModel, kind } from "@avytheone/efmesh"
|
|
@@ -46,9 +48,15 @@ export const moves = defineModel(
|
|
|
46
48
|
|
|
47
49
|
A typo in a `ref` is a compile error, not an empty run; renaming a parent's column breaks the child's build before any SQL reaches the database.
|
|
48
50
|
|
|
51
|
+
## Why this exists: small data lakes
|
|
52
|
+
|
|
53
|
+
Most analytics in the world is not a cloud warehouse — it is DuckDB-class data: gigabytes to a terabyte on one machine. Product analytics of a startup, the marts of one department, on-prem and edge deployments, a pipeline living inside a SaaS app. dbt and sqlmesh were born in the cloud-DWH world and carry that weight with them (Python, adapters, infrastructure). efmesh is the sqlmesh approach that is `bun add` and go — and the lake is a folder of parquet files.
|
|
54
|
+
|
|
55
|
+
Honestly placed: the **core** is the same class of system as sqlmesh — snapshots, AST fingerprints, plans as diffs, virtual environments. The **breadth** is not: no cloud engines, no multi-dialect, no web UI, no ecosystem of packages — and no ambition to catch up on all of it. Compared to dbt-core the trade is reversed: dbt has an industry around it, but no state-based plans, no virtual environments, no fingerprint versioning — every team reinvents "how not to rebuild everything". Our four honest advantages: **types as the DAG contract** (a typo breaks the build, not the nightly run), **library before CLI** (embed `Efmesh.apply(...)` in your app), **one language for the whole stack** (models, app, tests), and a **codebase you can read in an evening** (~5.5k lines).
|
|
56
|
+
|
|
49
57
|
## Who this is for (and who it isn't)
|
|
50
58
|
|
|
51
|
-
**For you**, if you are a TypeScript team on Bun, want a typed dbt/sqlmesh-style workflow on top of DuckDB or Postgres, and are fine living on a beta (efmesh is 0.
|
|
59
|
+
**For you**, if you are a TypeScript team on Bun, want a typed dbt/sqlmesh-style workflow on top of DuckDB or Postgres, and are fine living on a beta (efmesh is 0.2.x; Effect v4 is beta, pinned exactly as a peer dependency).
|
|
52
60
|
|
|
53
61
|
**Not for you**, if you need: a Node runtime (Bun-only for now), multi-dialect or cloud DWHs (Snowflake/BigQuery are out of scope), 1.0-grade stability, or the Python ecosystem — take sqlmesh instead, honestly.
|
|
54
62
|
|
|
@@ -58,7 +66,7 @@ A typo in a `ref` is a compile error, not an empty run; renaming a parent's colu
|
|
|
58
66
|
|
|
59
67
|
**Materialization targets.** Native engine tables, `parquet` (a lake, local or s3://, interval = partition, views over `read_parquet`), `ducklake` (table-per-fingerprint in a [DuckLake](https://ducklake.select) catalog — catalog snapshots and time travel come as a bonus).
|
|
60
68
|
|
|
61
|
-
**Plans and versions.** Fingerprints over canonical ASTs (reformatting SQL never triggers a rebuild — frozen by golden tests), change categorization breaking / non-breaking / indirect / forward-only, `--forward-only` applies a change without replaying history (the new version inherits physical storage and done-intervals; new columns via `ALTER`), plan confirmation in a TTY, an applied-plans journal with `applied_by`.
|
|
69
|
+
**Plans and versions.** Fingerprints over canonical ASTs (reformatting SQL never triggers a rebuild — frozen by golden tests), change categorization breaking / non-breaking / indirect / forward-only with `plan --explain` reasoning and a `--reclassify` operator override, indirect physics reuse (descendants of a non-breaking change are not rebuilt — scdType2 keeps its history), `--forward-only` applies a change without replaying history (the new version inherits physical storage and done-intervals; new columns via `ALTER`), plan confirmation in a TTY, an applied-plans journal with `applied_by`.
|
|
62
70
|
|
|
63
71
|
**Data quality.** A schema contract before every build (`DESCRIBE` of the query against the declared Schema), `notNull` / `unique` / `accepted` audits (blocking fails the apply, `warn` logs), a standalone `efmesh audit` over an environment's view layer, and `testModel` — unit tests for models on fixtures in in-memory DuckDB.
|
|
64
72
|
|
|
@@ -69,7 +77,7 @@ A typo in a `ref` is a compile error, not an empty run; renaming a parent's colu
|
|
|
69
77
|
## Quickstart
|
|
70
78
|
|
|
71
79
|
```sh
|
|
72
|
-
bun add -d @avytheone/efmesh
|
|
80
|
+
bun add -d @avytheone/efmesh
|
|
73
81
|
bunx efmesh init my-warehouse && cd my-warehouse
|
|
74
82
|
bunx efmesh plan dev # what would be done
|
|
75
83
|
bunx efmesh apply dev # physical tables, backfill, view layer
|
|
@@ -102,7 +110,7 @@ models (TS modules) ──► DAG + fingerprints over canonical ASTs
|
|
|
102
110
|
- **Virtual layer** — views `<env>__<schema>.<table>` (prod is just `<schema>.<table>`) pointing at physical storage. An environment is a set of pointers; promotion and rollback are view swaps.
|
|
103
111
|
- **The interval ledger** is the single source of truth about what has been computed: an interrupted backfill resumes where it stopped; recomputing an interval is a transactional DELETE+INSERT of the range — no duplicates.
|
|
104
112
|
|
|
105
|
-
Full architecture, invariants and decisions: [SPEC.md](./SPEC.md)
|
|
113
|
+
Full architecture, invariants and decisions: [SPEC.md](./SPEC.md).
|
|
106
114
|
|
|
107
115
|
## Data quality
|
|
108
116
|
|
|
@@ -157,18 +165,62 @@ export default defineConfig({
|
|
|
157
165
|
| `efmesh plan <env>` | diff the project against an environment + missing intervals; changes nothing |
|
|
158
166
|
| `efmesh apply <env>` | plan → confirmation (TTY) → physical tables, backfill, view layer |
|
|
159
167
|
| `efmesh run <env>` | scheduler tick: new intervals only, under the lock; for cron |
|
|
168
|
+
| `efmesh status <env>` | what is going on: last plan, interval lag, recent run ticks |
|
|
160
169
|
| `efmesh audit <env>` | audit the environment's view layer — catches after-the-fact degradation |
|
|
161
|
-
| `efmesh diff <envA> <envB>` | how two environments differ |
|
|
170
|
+
| `efmesh diff <envA> <envB>` | how two environments differ; `--data` compares the actual data |
|
|
162
171
|
| `efmesh render <model> [--env]` | the final SQL of a model |
|
|
163
172
|
| `efmesh lineage <model[.col]>` | column lineage down to the raw sources |
|
|
164
173
|
| `efmesh graph [--html]` | the model DAG as text or a page |
|
|
165
174
|
| `efmesh janitor [--ttl 7]` | remove orphaned physical storage older than ttl |
|
|
166
175
|
| `efmesh migrate` | bring the state-store schema up to the current version |
|
|
176
|
+
| `efmesh schedule <env>` | register `run <env>` in the OS scheduler via `Bun.cron` |
|
|
167
177
|
|
|
168
178
|
`apply`/`run` flags: `--jobs N` — DAG concurrency (always 1 on DuckDB — single connection), `--retries N` — retries for transient batch failures (exponential backoff), `--yes`/`-y` — skip confirmation, `--forward-only <model>,…` — reuse physical storage and history.
|
|
169
179
|
|
|
180
|
+
`plan`/`apply` take `--reclassify <model>=breaking|non-breaking[,…]` — the
|
|
181
|
+
operator's verdict on top of `--explain`, journaled with `applied_by`. A
|
|
182
|
+
non-breaking parent lets unchanged descendants reuse their previous physical
|
|
183
|
+
tables instead of rebuilding (scdType2 keeps its row history); an override
|
|
184
|
+
that plainly contradicts the AST (dropped columns) is refused.
|
|
185
|
+
|
|
186
|
+
`plan`, `audit` and `status` take `--json` — a stable machine-readable shape
|
|
187
|
+
(a contract under semver) for CI and bots; exit codes are unchanged, stdout
|
|
188
|
+
stays pure JSON.
|
|
189
|
+
|
|
190
|
+
`plan --explain` adds the reasoning to every change: which canonical-AST
|
|
191
|
+
nodes diverged (`where_clause`, `select_list[2] (added)`, …) and why the
|
|
192
|
+
category followed — including cascade sources for `indirect`. The same
|
|
193
|
+
data ships in `--json` as `explain`; the AST paths are a debugging hint,
|
|
194
|
+
not part of the contract.
|
|
195
|
+
|
|
196
|
+
`diff <envA> <envB> --data` compares the actual data of two environments:
|
|
197
|
+
row counts, key overlap (grain or the kind's key), per-column mismatch
|
|
198
|
+
rates among matched keys, schema drift between sides. `--sample P` (1–99)
|
|
199
|
+
compares a deterministic share of keys — md5 buckets aligned across both
|
|
200
|
+
sides, so sampling never fabricates only-in rows. `--model a,b` narrows,
|
|
201
|
+
`--json` for CI.
|
|
202
|
+
|
|
203
|
+
`schedule <env> [--cron '@hourly']` registers the `run` tick in the OS
|
|
204
|
+
scheduler (crontab / launchd / Task Scheduler) via `Bun.cron` — idempotent
|
|
205
|
+
by title, `--remove` unregisters, `--list` shows what's there. Honest
|
|
206
|
+
caveats: OS cron runs in the local timezone and does not catch up on missed
|
|
207
|
+
runs, and Arch-family Linux ships no cron daemon at all — `--print-systemd`
|
|
208
|
+
emits user-unit files instead (`Persistent=true` catches up). Overlapping
|
|
209
|
+
ticks are safe by construction: `run` takes the env lock and exits `2` when
|
|
210
|
+
changes await a human.
|
|
211
|
+
|
|
170
212
|
Exit codes: `0` — success, `1` — error, `2` — "awaiting a human": the plan needs confirmation in a non-TTY (add `--yes`), or `run` hit unapplied changes. In a non-TTY, `apply` with changes and no `--yes` refuses — efmesh will not silently roll out a plan nobody has seen.
|
|
171
213
|
|
|
214
|
+
## Performance
|
|
215
|
+
|
|
216
|
+
The framework overhead is negligible for any realistic project (in-memory DuckDB, `bun bench/plan-bench.ts N`):
|
|
217
|
+
|
|
218
|
+
| models | cold plan | apply (all physical) | no-op plan | promote to prod |
|
|
219
|
+
|---|---|---|---|---|
|
|
220
|
+
| 100 | 54 ms | 158 ms | 3 ms | 51 ms |
|
|
221
|
+
| 500 | 228 ms | 759 ms | 11 ms | 197 ms |
|
|
222
|
+
| 2000 | 0.9 s | 2.9 s | 50 ms | 1.3 s |
|
|
223
|
+
|
|
172
224
|
## Postgres
|
|
173
225
|
|
|
174
226
|
```ts
|
|
@@ -186,10 +238,10 @@ Next up: categorization override in the plan dialog, non-time-based intervals. K
|
|
|
186
238
|
|
|
187
239
|
## Documentation
|
|
188
240
|
|
|
189
|
-
- [SPEC.md](./SPEC.md) — the architecture spec: decisions, invariants, open questions
|
|
190
|
-
- [CHANGELOG.md](./CHANGELOG.md) — release history
|
|
241
|
+
- [SPEC.md](./SPEC.md) — the architecture spec: decisions, invariants, open questions;
|
|
242
|
+
- [CHANGELOG.md](./CHANGELOG.md) — release history;
|
|
191
243
|
- [examples/hospital](./examples/hospital/) — a live example with every model kind;
|
|
192
|
-
- [CONTRIBUTING.md](./CONTRIBUTING.md) — build, test and PR guide
|
|
244
|
+
- [CONTRIBUTING.md](./CONTRIBUTING.md) — build, test and PR guide.
|
|
193
245
|
|
|
194
246
|
## License
|
|
195
247
|
|
package/README.ru.md
CHANGED
|
@@ -4,10 +4,12 @@
|
|
|
4
4
|
|
|
5
5
|
*Русское зеркало; основной README — [английский](./README.md).*
|
|
6
6
|
|
|
7
|
-
[](https://github.com/avytheone/efmesh/actions/workflows/ci.yml)  ](https://github.com/avytheone/efmesh/actions/workflows/ci.yml)     
|
|
8
8
|
|
|
9
9
|
Модели — обычные TypeScript-модули: тело на SQL, ссылки между моделями — импорты, форма данных — Effect Schema. efmesh считает fingerprint каждой модели по каноническому AST, хранит версии снапшотами, строит план как diff «проект против окружения» и применяет ровно его: физика пересчитывается только там, где что-то реально изменилось, а окружения (dev/prod/…) — виртуальные view поверх общей физики, промоушен в prod не стоит ни одного пересчёта.
|
|
10
10
|
|
|
11
|
+
<p align="center"><img src="docs/demo.svg" alt="efmesh demo: a ref typo is a compile error; a plan rebuilds exactly the changed branch; promotion is a view swap" width="840"></p>
|
|
12
|
+
|
|
11
13
|
```ts
|
|
12
14
|
import { Schema } from "effect"
|
|
13
15
|
import { defineModel, kind } from "@avytheone/efmesh"
|
|
@@ -46,11 +48,17 @@ export const moves = defineModel(
|
|
|
46
48
|
|
|
47
49
|
Опечатка в `ref` у нас — ошибка компиляции, а не пустой прогон; переименованная колонка родителя ломает сборку потомка до того, как SQL уедет в базу.
|
|
48
50
|
|
|
51
|
+
## Зачем это существует: маленькие озёра данных
|
|
52
|
+
|
|
53
|
+
Большинство аналитики в мире — не облачный warehouse, а данные DuckDB-класса: от гигабайт до терабайта на одной машине. Продуктовая аналитика стартапа, витрины одного отдела, on-prem и edge, pipeline внутри SaaS-приложения. dbt и sqlmesh родом из мира облачных DWH и тащат этот вес с собой (Python, адаптеры, инфраструктура). efmesh — это sqlmesh-подход, который `bun add` и поехал, а озеро — это папка parquet-файлов.
|
|
54
|
+
|
|
55
|
+
Честная позиция: **ядро** — тот же класс систем, что sqlmesh (снапшоты, AST-fingerprint, план как diff, виртуальные окружения), но **широты** нет: ни облачных движков, ни мульти-диалекта, ни веб-UI, ни экосистемы пакетов — и нет амбиции всё это догонять. С dbt-core размен обратный: у dbt индустрия вокруг, но нет state-based планов, виртуальных окружений и версионирования по fingerprint — каждая команда сама изобретает «как не пересчитывать всё». Наши четыре честных преимущества: **типы как контракт DAG** (опечатка ломает сборку, а не ночной прогон), **библиотека прежде CLI** (`Efmesh.apply(...)` встраивается в приложение), **один язык на весь стек** (модели, приложение, тесты) и **кодовая база, читаемая за вечер** (~5.5k строк).
|
|
56
|
+
|
|
49
57
|
## Для кого это (и для кого нет)
|
|
50
58
|
|
|
51
59
|
**Для вас**, если вы TypeScript-команда на Bun, хотите типизированный
|
|
52
60
|
dbt/sqlmesh-подход поверх DuckDB или Postgres и готовы жить на бете
|
|
53
|
-
(efmesh — 0.
|
|
61
|
+
(efmesh — 0.2.x, Effect v4 — beta, версия effect пинована peer-зависимостью).
|
|
54
62
|
|
|
55
63
|
**Не для вас**, если нужны: Node-рантайм (пока только Bun), мульти-диалект
|
|
56
64
|
или облачные DWH (Snowflake/BigQuery — вне scope), стабильность уровня 1.0
|
|
@@ -62,7 +70,7 @@ dbt/sqlmesh-подход поверх DuckDB или Postgres и готовы ж
|
|
|
62
70
|
|
|
63
71
|
**Цели материализации.** Нативная таблица движка, `parquet` (озеро: локально или s3://, интервал = партиция, view поверх `read_parquet`), `ducklake` (таблица-на-fingerprint в [DuckLake](https://ducklake.select)-каталоге — снапшоты и time travel каталога бонусом).
|
|
64
72
|
|
|
65
|
-
**План и версии.** Fingerprint по каноническому AST (переформатирование SQL не триггерит пересборку), категоризация изменений breaking / non-breaking / indirect / forward-only, `--forward-only` — правка без переигрывания истории (новая версия наследует физику и done-интервалы, новые колонки — `ALTER`), подтверждение плана в TTY, журнал применений с `applied_by`.
|
|
73
|
+
**План и версии.** Fingerprint по каноническому AST (переформатирование SQL не триггерит пересборку), категоризация изменений breaking / non-breaking / indirect / forward-only с обоснованием `plan --explain` и оператором-override `--reclassify`, indirect-реюз физики (потомки non-breaking-правки не пересобираются — scdType2 не теряет историю), `--forward-only` — правка без переигрывания истории (новая версия наследует физику и done-интервалы, новые колонки — `ALTER`), подтверждение плана в TTY, журнал применений с `applied_by`.
|
|
66
74
|
|
|
67
75
|
**Качество.** Контракт схемы перед сборкой (`DESCRIBE` запроса против объявленной Schema), аудиты `notNull` / `unique` / `accepted` (blocking роняет apply, `warn` — логирует), автономный `efmesh audit` по view-слою окружения, `testModel` — юнит-тесты моделей на фикстурах в in-memory DuckDB.
|
|
68
76
|
|
|
@@ -73,7 +81,7 @@ dbt/sqlmesh-подход поверх DuckDB или Postgres и готовы ж
|
|
|
73
81
|
## Быстрый старт
|
|
74
82
|
|
|
75
83
|
```sh
|
|
76
|
-
bun add -d @avytheone/efmesh
|
|
84
|
+
bun add -d @avytheone/efmesh
|
|
77
85
|
bunx efmesh init my-warehouse && cd my-warehouse
|
|
78
86
|
bunx efmesh plan dev # что будет сделано
|
|
79
87
|
bunx efmesh apply dev # физика, бэкфилл, view-слой
|
|
@@ -161,21 +169,65 @@ export default defineConfig({
|
|
|
161
169
|
| `efmesh plan <env>` | diff проекта против окружения + недостающие интервалы, ничего не меняет |
|
|
162
170
|
| `efmesh apply <env>` | план → подтверждение (TTY) → физика, бэкфилл, view-слой |
|
|
163
171
|
| `efmesh run <env>` | тик планировщика: только новые интервалы, под локом; для cron |
|
|
172
|
+
| `efmesh status <env>` | что происходит: последний план, отставание интервалов, тики run |
|
|
164
173
|
| `efmesh audit <env>` | аудиты view-слоя окружения — ловит деградацию задним числом |
|
|
165
|
-
| `efmesh diff <envA> <envB>` | чем окружения
|
|
174
|
+
| `efmesh diff <envA> <envB>` | чем окружения отличаются; `--data` сравнивает сами данные |
|
|
166
175
|
| `efmesh render <model> [--env]` | итоговый SQL модели |
|
|
167
176
|
| `efmesh lineage <model[.col]>` | колоночный lineage до сырья |
|
|
168
177
|
| `efmesh graph [--html]` | DAG моделей текстом или страницей |
|
|
169
178
|
| `efmesh janitor [--ttl 7]` | снести осиротевшую физику старше ttl |
|
|
170
179
|
| `efmesh migrate` | догнать схему state store до текущей версии |
|
|
180
|
+
| `efmesh schedule <env>` | зарегистрировать `run <env>` в OS-шедулере через `Bun.cron` |
|
|
171
181
|
|
|
172
182
|
Флаги `apply`/`run`: `--jobs N` — DAG-конкурентность (на DuckDB всегда 1 — одно соединение), `--retries N` — ретраи транзиентных сбоев батча (экспоненциальная пауза), `--yes`/`-y` — без подтверждения, `--forward-only <model>,…` — реюз физики и истории.
|
|
173
183
|
|
|
184
|
+
`plan`/`apply` принимают `--reclassify <model>=breaking|non-breaking[,…]` —
|
|
185
|
+
вердикт оператора поверх `--explain`, журналируется с `applied_by`.
|
|
186
|
+
Non-breaking-родитель разрешает нетронутым потомкам реюзать физику вместо
|
|
187
|
+
пересборки (scdType2 не теряет историю строк); override, очевидно
|
|
188
|
+
противоречащий AST (удалённые колонки), не принимается.
|
|
189
|
+
|
|
190
|
+
`plan`, `audit` и `status` принимают `--json` — стабильная машиночитаемая
|
|
191
|
+
форма (контракт в рамках semver) для CI и ботов; exit-коды не меняются,
|
|
192
|
+
stdout — чистый JSON.
|
|
193
|
+
|
|
194
|
+
`plan --explain` добавляет к каждому изменению обоснование: какие узлы
|
|
195
|
+
канонического AST разошлись (`where_clause`, `select_list[2] (добавлен)`, …)
|
|
196
|
+
и почему категория именно такая — включая источники каскада у `indirect`.
|
|
197
|
+
Те же данные попадают в `--json` полем `explain`; пути по AST — отладочная
|
|
198
|
+
подсказка, не часть контракта.
|
|
199
|
+
|
|
200
|
+
`diff <envA> <envB> --data` сравнивает сами данные двух окружений: счётчики
|
|
201
|
+
строк, пересечение по ключу (grain или ключ вида), доли расхождений по
|
|
202
|
+
колонкам среди сопоставленных ключей, дрейф схемы между сторонами.
|
|
203
|
+
`--sample P` (1–99) сравнивает детерминированную долю ключей — md5-бакеты
|
|
204
|
+
выровнены между сторонами, выборка не рождает ложных only-in. `--model a,b`
|
|
205
|
+
сужает, `--json` для CI.
|
|
206
|
+
|
|
207
|
+
`schedule <env> [--cron '@hourly']` регистрирует тик `run` в OS-шедулере
|
|
208
|
+
(crontab / launchd / Task Scheduler) через `Bun.cron` — идемпотентно по
|
|
209
|
+
заголовку, `--remove` снимает, `--list` показывает записи. Честные
|
|
210
|
+
оговорки: OS-cron живёт в локальной таймзоне и не догоняет пропущенные
|
|
211
|
+
запуски, а Arch-семейство Linux вовсе не ставит cron-демона —
|
|
212
|
+
`--print-systemd` печатает user-юниты (`Persistent=true` догоняет).
|
|
213
|
+
Наложение тиков безопасно по построению: `run` берёт env-лок и выходит
|
|
214
|
+
с кодом `2`, когда изменения ждут человека.
|
|
215
|
+
|
|
174
216
|
Exit-коды: `0` — успех, `1` — ошибка, `2` — «ждёт человека»: план требует
|
|
175
217
|
подтверждения в не-TTY (нужен `--yes`) или `run` упёрся в неприменённые
|
|
176
218
|
изменения. В не-TTY `apply` с изменениями БЕЗ `--yes` не применяется —
|
|
177
219
|
молча катить план, который никто не видел, efmesh отказывается.
|
|
178
220
|
|
|
221
|
+
## Перфоманс
|
|
222
|
+
|
|
223
|
+
Оверхед фреймворка пренебрежим для любого реалистичного проекта (in-memory DuckDB, `bun bench/plan-bench.ts N`):
|
|
224
|
+
|
|
225
|
+
| моделей | plan холодный | apply (вся физика) | plan пустой | промоушен prod |
|
|
226
|
+
|---|---|---|---|---|
|
|
227
|
+
| 100 | 54 мс | 158 мс | 3 мс | 51 мс |
|
|
228
|
+
| 500 | 228 мс | 759 мс | 11 мс | 197 мс |
|
|
229
|
+
| 2000 | 0.9 с | 2.9 с | 50 мс | 1.3 с |
|
|
230
|
+
|
|
179
231
|
## Postgres
|
|
180
232
|
|
|
181
233
|
```ts
|