@niscorp/vex 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/DESIGN.md +881 -0
- package/DOCS.md +1295 -0
- package/LICENSE +202 -0
- package/README.md +97 -0
- package/dist/adapters/express/index.cjs +1406 -0
- package/dist/adapters/express/index.cjs.map +1 -0
- package/dist/adapters/express/index.d.cts +25 -0
- package/dist/adapters/express/index.d.ts +25 -0
- package/dist/adapters/express/index.js +60 -0
- package/dist/adapters/express/index.js.map +1 -0
- package/dist/adapters/hono/index.cjs +1426 -0
- package/dist/adapters/hono/index.cjs.map +1 -0
- package/dist/adapters/hono/index.d.cts +25 -0
- package/dist/adapters/hono/index.d.ts +25 -0
- package/dist/adapters/hono/index.js +78 -0
- package/dist/adapters/hono/index.js.map +1 -0
- package/dist/adapters/pglite/index.cjs +17 -0
- package/dist/adapters/pglite/index.cjs.map +1 -0
- package/dist/adapters/pglite/index.d.cts +21 -0
- package/dist/adapters/pglite/index.d.ts +21 -0
- package/dist/adapters/pglite/index.js +14 -0
- package/dist/adapters/pglite/index.js.map +1 -0
- package/dist/agent/index.cjs +478 -0
- package/dist/agent/index.cjs.map +1 -0
- package/dist/agent/index.d.cts +29 -0
- package/dist/agent/index.d.ts +29 -0
- package/dist/agent/index.js +313 -0
- package/dist/agent/index.js.map +1 -0
- package/dist/chunk-FPKW4TRX.js +1528 -0
- package/dist/chunk-FPKW4TRX.js.map +1 -0
- package/dist/chunk-HF25Q53I.js +165 -0
- package/dist/chunk-HF25Q53I.js.map +1 -0
- package/dist/handler-Cql8SJ3F.d.cts +133 -0
- package/dist/handler-mICf9ins.d.ts +133 -0
- package/dist/index.cjs +4078 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +199 -0
- package/dist/index.d.ts +199 -0
- package/dist/index.js +2340 -0
- package/dist/index.js.map +1 -0
- package/dist/introspect-D3OaXHBc.d.cts +16 -0
- package/dist/introspect-D3OaXHBc.d.ts +16 -0
- package/dist/types-BRvMB6IM.d.cts +1491 -0
- package/dist/types-BRvMB6IM.d.ts +1491 -0
- package/package.json +125 -0
package/DESIGN.md
ADDED
|
@@ -0,0 +1,881 @@
|
|
|
1
|
+
# Vex — Design Document
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
|
|
5
|
+
Intent-based data access. A caller describes *what* data they need and the
|
|
6
|
+
*shape* they want it in. Vex synthesizes a query, executes it, and reshapes the
|
|
7
|
+
result to match. Because it caches each generation under a fingerprint, replaying that
|
|
8
|
+
fingerprint costs zero LLM calls.
|
|
9
|
+
|
|
10
|
+
**One sentence:** ask for data in English, get it in any shape, from any database.
|
|
11
|
+
|
|
12
|
+
The thesis: most applications have a small, stable set of data-access patterns.
|
|
13
|
+
Each pattern has a consistent output shape. Once the DSL (and any result
|
|
14
|
+
mapping) is generated for a shape, it is reused for every later request with
|
|
15
|
+
that shape — the LLM cost amortizes to zero, and the hot path is pure compile +
|
|
16
|
+
execute.
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## Core principles
|
|
21
|
+
|
|
22
|
+
1. **The DSL is the source of truth.** The LLM never emits SQL. It fills a
|
|
23
|
+
constrained JSON DSL that is Zod-validated before anything else touches it.
|
|
24
|
+
Scope, discovery, analysis, and compilation all operate on the DSL or on a
|
|
25
|
+
value derived from it.
|
|
26
|
+
2. **Deterministic core, injected intelligence.** The engine itself contains no
|
|
27
|
+
LLM code. Natural-language→DSL and rows→shape are two optional function
|
|
28
|
+
hooks (`generateDsl`, `mapToShape`). With them, Vex answers free-form
|
|
29
|
+
requests. Without them, it is a deterministic DSL compiler + executor + cache
|
|
30
|
+
that still serves any already-cached shape. Cortex/Signal/Prism are optional
|
|
31
|
+
peer dependencies, not hard ones.
|
|
32
|
+
3. **Shape-only caching.** The positive cache key is a hash of the request's
|
|
33
|
+
*shape* with all values replaced by type tokens. Two requests with the same
|
|
34
|
+
structure but different values share a cache entry. Intent is deliberately
|
|
35
|
+
excluded from this key.
|
|
36
|
+
4. **Scope is server-side and LLM-invisible.** Access control is injected into
|
|
37
|
+
the DSL *after* generation, as ordinary filter clauses bound to server
|
|
38
|
+
`$scope` values. The model cannot see or forge them, so prompt injection
|
|
39
|
+
cannot bypass them.
|
|
40
|
+
5. **One-level field paths.** Always `entity.field`, never
|
|
41
|
+
`entity.relation.field`. Every entity appears explicitly in `from`. This
|
|
42
|
+
keeps the resolver simple and entity discovery a pure string walk.
|
|
43
|
+
6. **Postgres first, adapter interface from day one.** SQL generation is the
|
|
44
|
+
only database-specific step. It lives entirely behind the `DatabaseAdapter`
|
|
45
|
+
interface so other backends can follow without changing the core.
|
|
46
|
+
7. **Schemas teach the model.** Every DSL schema field carries a Zod
|
|
47
|
+
`.describe()`. The JSON Schema generated from it — not prose — is what the
|
|
48
|
+
query agent reads to learn the DSL.
|
|
49
|
+
|
|
50
|
+
---
|
|
51
|
+
|
|
52
|
+
## Architecture
|
|
53
|
+
|
|
54
|
+
```
|
|
55
|
+
┌──────────────────────────────────────┐
|
|
56
|
+
QueryRequest │ { intent?, shape, context } │
|
|
57
|
+
+ ExecuteOptions │ options: { scope?, cache?, entities?} │
|
|
58
|
+
└────────────────┬─────────────────────┘
|
|
59
|
+
↓
|
|
60
|
+
┌──────────────────────────────────────┐
|
|
61
|
+
│ Validate (Zod) + hash │
|
|
62
|
+
│ shapeHash (positive key) │
|
|
63
|
+
│ requestHash (negative key / single- │
|
|
64
|
+
│ flight) │
|
|
65
|
+
└────────────────┬─────────────────────┘
|
|
66
|
+
↓
|
|
67
|
+
┌──────────────────────────────────────┐
|
|
68
|
+
│ Cache read │
|
|
69
|
+
│ positive HIT → cached DSL (+ Prism IR)│
|
|
70
|
+
│ negative HIT → throw unsatisfiable │
|
|
71
|
+
│ miss + mode 'only' → throw cache_miss │
|
|
72
|
+
└────────────────┬─────────────────────┘
|
|
73
|
+
miss ↓ ↑ hit (skip generation)
|
|
74
|
+
┌──────────────────────────────────────┐
|
|
75
|
+
│ generateDsl hook (LLM, optional) │
|
|
76
|
+
│ single-flighted; negative-caches a │
|
|
77
|
+
│ "cannot satisfy" result │
|
|
78
|
+
└────────────────┬─────────────────────┘
|
|
79
|
+
↓
|
|
80
|
+
── deterministic pipeline (runPipeline) ───────────────────────────
|
|
81
|
+
┌──────────────────────────────────────┐
|
|
82
|
+
│ clamp limit → discover entities → │
|
|
83
|
+
│ apply scope → resolve → analyze → │
|
|
84
|
+
│ adapter.compile → CompiledQuery │
|
|
85
|
+
└────────────────┬─────────────────────┘
|
|
86
|
+
↓
|
|
87
|
+
┌──────────────────────────────────────┐
|
|
88
|
+
│ Missing-context check │
|
|
89
|
+
│ → return empty result + meta. │
|
|
90
|
+
│ missingContext (valid response) │
|
|
91
|
+
└────────────────┬─────────────────────┘
|
|
92
|
+
↓
|
|
93
|
+
┌──────────────────────────────────────┐
|
|
94
|
+
│ Bind params (context/scope/embed) → │
|
|
95
|
+
│ adapter.execute → raw rows │
|
|
96
|
+
└────────────────┬─────────────────────┘
|
|
97
|
+
↓
|
|
98
|
+
┌──────────────────────────────────────┐
|
|
99
|
+
│ Map to shape (optional) │
|
|
100
|
+
│ cached Prism IR → apply; else │
|
|
101
|
+
│ mapToShape hook (LLM) → IR + rows │
|
|
102
|
+
└────────────────┬─────────────────────┘
|
|
103
|
+
↓
|
|
104
|
+
┌──────────────────────────────────────┐
|
|
105
|
+
│ Cache write (by fingerprint) │
|
|
106
|
+
│ DSL + Prism IR + schema fingerprint │
|
|
107
|
+
└────────────────┬─────────────────────┘
|
|
108
|
+
↓
|
|
109
|
+
┌──────────────────────────────────────┐
|
|
110
|
+
│ { result, meta: { cache, context, │
|
|
111
|
+
│ timing, warnings? } } │
|
|
112
|
+
└──────────────────────────────────────┘
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
On a cache hit, generation and mapping are skipped: the pipeline re-runs
|
|
116
|
+
deterministically against the cached DSL and applies the cached Prism IR. No
|
|
117
|
+
LLM calls. That re-run (resolve → analyze → compile → bind → execute) is what
|
|
118
|
+
makes scope and context values current on every request, even cached ones.
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
## Subsystems
|
|
123
|
+
|
|
124
|
+
### The DSL
|
|
125
|
+
|
|
126
|
+
A constrained JSON structure (`schemas/`) that can only express safe, valid
|
|
127
|
+
queries. Top level (`QuerySchema`):
|
|
128
|
+
|
|
129
|
+
- `from` — array of entity names or `{ as, query }` subqueries (min 1).
|
|
130
|
+
- `fields` — array of `entity.field` paths (min 1; **required** — no `SELECT *`).
|
|
131
|
+
- `filter` — a recursive `FilterSchema` union.
|
|
132
|
+
- `compute` — record of `alias → ComputeExpression` (arithmetic, `concat`,
|
|
133
|
+
`coalesce`, `case`).
|
|
134
|
+
- `aggregate` — record of `alias → AggregateExpression` (`count`/`sum`/`avg`/
|
|
135
|
+
`min`/`max`).
|
|
136
|
+
- `groupBy`, `sort`, `limit`, `distinct`.
|
|
137
|
+
|
|
138
|
+
Filter operators: `eq`/`neq`/`gt`/`gte`/`lt`/`lte`, `in`/`notIn`, `like`/
|
|
139
|
+
`ilike`, `isNull`/`isNotNull`, `and`/`or`/`not`, plus `semantic` (vector
|
|
140
|
+
similarity) and `fuzzy` (string distance). Values are literals, `{ $context }`
|
|
141
|
+
(caller-provided) or `{ $scope }` (server-injected).
|
|
142
|
+
|
|
143
|
+
Two refinements over a naive design, both for the model's benefit: there is no
|
|
144
|
+
`offset` (cursor pagination via a `gt` filter is safer and cacheable) and no
|
|
145
|
+
`between` (it is sugar for `and(gte, lte)`). Fewer operators means a smaller
|
|
146
|
+
JSON Schema and better model adherence.
|
|
147
|
+
|
|
148
|
+
### Scope
|
|
149
|
+
|
|
150
|
+
Two pure functions over the DSL:
|
|
151
|
+
|
|
152
|
+
- `discoverEntities(dsl)` walks every place an entity name can appear — `from`,
|
|
153
|
+
field paths, filter paths (recursively through `and`/`or`/`not`), compute and
|
|
154
|
+
aggregate references, `groupBy`, `sort`, and into subqueries — and returns the
|
|
155
|
+
set of referenced entities. No schema needed; pure string extraction.
|
|
156
|
+
- `applyScope(dsl, entities, policy)` looks up each discovered entity in the
|
|
157
|
+
policy and produces a new DSL. `public` rules pass through; `deny` (or no rule
|
|
158
|
+
under a `default: 'deny'` policy) throws `VexScopeError`; filter rules become
|
|
159
|
+
`{ eq|neq|in: ["entity.field", { $scope: "source" }] }` clauses AND-merged
|
|
160
|
+
into the existing filter. It recurses into subqueries.
|
|
161
|
+
|
|
162
|
+
Scope runs after generation and before resolution. The resolver, analyzer, and
|
|
163
|
+
compiler never know scope exists — they just process a DSL that happens to carry
|
|
164
|
+
extra filter clauses.
|
|
165
|
+
|
|
166
|
+
### Engine (deterministic pipeline)
|
|
167
|
+
|
|
168
|
+
- **resolver** (`resolve`) turns the scoped DSL into a `ResolvedQuery`: field
|
|
169
|
+
paths → table aliases + columns, FK-based join discovery between entity
|
|
170
|
+
sources (a nullable FK resolves to a LEFT join so a null FK never drops the
|
|
171
|
+
referencing row; subquery sources are not FK-joined — the compiler cross-joins them,
|
|
172
|
+
and the outer query can reference their field/compute/aggregate outputs),
|
|
173
|
+
subquery recursion.
|
|
174
|
+
- **analyzer** (`analyze`) validates the resolved query and returns
|
|
175
|
+
`{ warnings, errors }`. Errors halt the pipeline; warnings ride along in the
|
|
176
|
+
response. Checks: nesting depth vs `maxNestingDepth`; cartesian products
|
|
177
|
+
(multiple entity sources with no connecting join); unindexed filter columns
|
|
178
|
+
(warn or reject); aggregates without `groupBy` (warn); vector field without an
|
|
179
|
+
HNSW/IVFFlat index for semantic search (warn).
|
|
180
|
+
- **executor** (`executeQuery`, `buildContextContract`, `findMissingContext`)
|
|
181
|
+
binds parameters and calls `adapter.execute`. Binding resolves each
|
|
182
|
+
`ParamSlot` from context (`context`), scope (`scope`), or by embedding text
|
|
183
|
+
(`semantic`). `findMissingContext` reports which required keys the caller
|
|
184
|
+
omitted so the runtime can return a valid-but-empty response instead of
|
|
185
|
+
failing.
|
|
186
|
+
|
|
187
|
+
### Adapters
|
|
188
|
+
|
|
189
|
+
`DatabaseAdapter` is the whole database boundary: `introspect`, `compile`
|
|
190
|
+
(`ResolvedQuery → CompiledQuery`), `execute`, and a `capabilities` flag set.
|
|
191
|
+
Compilation is adapter-internal — there is no shared compiler in `engine/`; each
|
|
192
|
+
adapter owns its full SQL dialect (the Postgres adapter has its own
|
|
193
|
+
filter/compute/aggregate SQL generation under `adapters/postgres/`).
|
|
194
|
+
|
|
195
|
+
A `CompiledQuery` is `{ sql, paramSlots, contextContract }`. The ordered
|
|
196
|
+
`paramSlots` array drives positional binding (`$1`, `$2`, …); each slot knows
|
|
197
|
+
its `kind` (context/scope/semantic), `type`, and (for semantic) `dimensions`.
|
|
198
|
+
The text→vector `embed` function is an engine-level injected provider
|
|
199
|
+
(`createQueryEngine({ embed })`), used by the executor during binding for
|
|
200
|
+
`semantic` slots — the adapter only emits the vector-distance SQL, never embeds.
|
|
201
|
+
|
|
202
|
+
The Postgres adapter advertises vector search, JSON fields, full-text, CTEs,
|
|
203
|
+
window functions, RETURNING, and statement timeout; it coerces numeric column
|
|
204
|
+
strings (int8/numeric/float) back to JS numbers on the way out.
|
|
205
|
+
|
|
206
|
+
### Cache — mechanics
|
|
207
|
+
|
|
208
|
+
Identity is the **fingerprint** (see Cache v2 below). The supporting hashes:
|
|
209
|
+
|
|
210
|
+
- **Request hash** (`computeRequestHash`) hashes intent + normalized shape +
|
|
211
|
+
*context key names* (not values). It identifies the *request* stored with a
|
|
212
|
+
named slot (match → hit, differ → regenerate-and-replace), and keys the
|
|
213
|
+
negative cache and single-flight de-duplication.
|
|
214
|
+
- **Schema fingerprint** (`computeSchemaFingerprint`) is a stable hash of the
|
|
215
|
+
structural schema (names, types, nullability, PKs, relations, indexes —
|
|
216
|
+
excluding volatile rowCount and cosmetic fields). It is stored on each entry;
|
|
217
|
+
a mismatch means the cached DSL may reference changed columns, so the entry is
|
|
218
|
+
treated as a miss and evicted. Stable across restarts so warm-up entries
|
|
219
|
+
aren't wrongly discarded.
|
|
220
|
+
- **Entry kinds.** A discriminated union: `ok` (DSL + optional Prism IR) and
|
|
221
|
+
`unsatisfiable` (a TTL'd negative result — a request the agent declared
|
|
222
|
+
impossible, so it isn't re-run until the TTL lapses or the schema changes).
|
|
223
|
+
- **Backends.** In-memory (`createMemoryCache`), durable Postgres
|
|
224
|
+
(`createPostgresCache`, jsonb table, server-side TTL, `protected`/
|
|
225
|
+
`last_used_at`/`request_hash` columns with idempotent ALTER migrations,
|
|
226
|
+
validate-on-write and evict-on-read), and tiered (`createTieredCache`, L1 +
|
|
227
|
+
L2 read-through/write-through with `full`/`lazy`/`partial` warm-up). Every
|
|
228
|
+
backend implements the same `CacheBackend` contract; `init`/`entries` are
|
|
229
|
+
optional capabilities used for warm-up and the discovery overview.
|
|
230
|
+
- **Validation** (`validateEntry`) guards every write and every promotion into
|
|
231
|
+
memory, so a corrupt or schema-drifted row can never poison the cache.
|
|
232
|
+
- **GC** (`sweepCache`) evicts entries idle past `maxIdleMs`
|
|
233
|
+
(`lastUsedAt ?? createdAt`), skipping protected ones. Hygiene, not necessity.
|
|
234
|
+
|
|
235
|
+
### Cache v2 — fingerprints
|
|
236
|
+
|
|
237
|
+
Agreed and implemented 2026-07-10; replaced shape-keyed caching everywhere
|
|
238
|
+
(engine, HTTP layer, and all consumers — relay, fable, mythos, showroom).
|
|
239
|
+
|
|
240
|
+
**Why v1 has to go.** Shape-hash identity collides: two different intents with
|
|
241
|
+
the same normalized shape share one slot, so the second silently returns the
|
|
242
|
+
first's data stamped `hit: true`. Entries are anonymous — no way to pin,
|
|
243
|
+
promote, iterate, or roll back a generation. And running a query writes the
|
|
244
|
+
cache as a side effect, so *testing* and *publishing* are the same physical
|
|
245
|
+
act: callers either bypass the cache and lose the artifact, or hit it and can
|
|
246
|
+
never change it.
|
|
247
|
+
|
|
248
|
+
**The model: one identity, `fingerprint`.**
|
|
249
|
+
|
|
250
|
+
- A request WITHOUT a fingerprint generates and caches; the server mints a
|
|
251
|
+
fingerprint (`fp_…`) and returns it in `meta.cache.fingerprint`.
|
|
252
|
+
- A minted fingerprint is an **immutable pin** — embed it and replay the exact
|
|
253
|
+
generation forever. Changing the query means proving again and embedding the
|
|
254
|
+
new fingerprint. (git: a commit hash.)
|
|
255
|
+
- A caller-chosen fingerprint (any string) is a **mutable named slot** — the
|
|
256
|
+
name is yours, its content follows your request. (git: a branch.)
|
|
257
|
+
- Intent NEVER keys anything: natural language is fake identity in both
|
|
258
|
+
directions. It remains generator input + human metadata.
|
|
259
|
+
|
|
260
|
+
**Three request postures** — the body replaces every cache mode:
|
|
261
|
+
|
|
262
|
+
```jsonc
|
|
263
|
+
// 1. Explore: generate, cache, mint
|
|
264
|
+
{ "intent": "top 10 open deals by value…", "shape": [{ "id": "", "value_display": "" }] }
|
|
265
|
+
// → meta.cache: { "fingerprint": "fp_9k27ax", "hit": false }
|
|
266
|
+
|
|
267
|
+
// 2. Replay: fingerprint alone (+ context VALUES welcome) — never generates
|
|
268
|
+
{ "fingerprint": "fp_9k27ax", "context": { "sortBy": "deals.value" } }
|
|
269
|
+
// → meta.cache: { "fingerprint": "fp_9k27ax", "hit": true }; unknown fp → 404
|
|
270
|
+
|
|
271
|
+
// 3. Own a slot: fingerprint + request
|
|
272
|
+
{ "fingerprint": "deals/table", "intent": "…", "shape": [ … ] }
|
|
273
|
+
// stored request matches → hit; differs → regenerate + REPLACE the slot
|
|
274
|
+
// (meta.cache.replaced: true) — unless protected → 409
|
|
275
|
+
```
|
|
276
|
+
|
|
277
|
+
Request equality = intent string + normalized shape + context KEY NAMES
|
|
278
|
+
(`normalizeShape` survives for this comparison; context values are runtime
|
|
279
|
+
data and never identity — which is what makes replays parameterizable:
|
|
280
|
+
`sortBy`/search terms may vary per call against one cached DSL).
|
|
281
|
+
|
|
282
|
+
**Lifetime = usage.** Every replay stamps `lastUsedAt`; a GC sweep evicts
|
|
283
|
+
entries unused for N days. An embedded pin is touched on every mount and lives
|
|
284
|
+
forever by being used; an abandoned exploration ages out. There are NO
|
|
285
|
+
provisional/permanent tiers and NO promote API — the distinction is replaced
|
|
286
|
+
by the convention *alive = used*. (Entries are single-digit KB; the sweep is
|
|
287
|
+
hygiene, not necessity.)
|
|
288
|
+
|
|
289
|
+
**Protection: one stored bit, two writers, one knob.**
|
|
290
|
+
|
|
291
|
+
- `protected: true` on an entry means a differing request 409s instead of
|
|
292
|
+
replacing. Written ONLY by (i) the seed path and (ii) an explicit `PATCH`
|
|
293
|
+
(which is what "save/lock this view" calls). Runtime writes are never
|
|
294
|
+
protected by default.
|
|
295
|
+
- One endpoint knob: `locked: true` = replay-only posture (no generation at
|
|
296
|
+
all — the production stance; v1's `cache: 'only'` promoted from caller flag
|
|
297
|
+
to endpoint config).
|
|
298
|
+
- No governance fields ever ride in the query body.
|
|
299
|
+
|
|
300
|
+
**The REST surface.** Vex is an API endpoint over REST and is designed as one.
|
|
301
|
+
Scoped mounts (`/api/deals/vex`) ride the existing `options.entities` filter —
|
|
302
|
+
scope constrains *generation* (smaller schema, no out-of-scope joins);
|
|
303
|
+
identity stays global.
|
|
304
|
+
|
|
305
|
+
- `POST /api/<scope>/vex` — execute (the three postures above). Errors:
|
|
306
|
+
unknown fingerprint replay → 404 `cache_miss`; protected slot + differing
|
|
307
|
+
request → 409 `fingerprint_protected`; locked endpoint asked to generate →
|
|
308
|
+
403 `locked`.
|
|
309
|
+
- `GET /api/<scope>/vex` — self-description: `{ protection:
|
|
310
|
+
"all"|"some"|"none", locked, fingerprints: [{ fingerprint, protected,
|
|
311
|
+
schemaFresh, intent, lastUsedAt }] }` plus the schema/DSL/body docs. Doubles
|
|
312
|
+
as agent-facing discovery: "does a named query already cover this?" is one
|
|
313
|
+
call.
|
|
314
|
+
- `PATCH /…/vex` with `{ fingerprint, protected }` — flip the bit. `DELETE`
|
|
315
|
+
with `{ fingerprint }` — evict (unprotected only). Fingerprints ride the
|
|
316
|
+
body, not the path — names contain `/`.
|
|
317
|
+
|
|
318
|
+
**What dies with v1:**
|
|
319
|
+
|
|
320
|
+
- `CacheMode` (`use`/`refresh`/`bypass`/`only`), `ExecuteOptions.cache`, the
|
|
321
|
+
`?cache=` query param — the request body says everything now.
|
|
322
|
+
- The shape hash as a cache key (and with it the entire collision class).
|
|
323
|
+
- The embed-VERBATIM requirement for generated screens: a pin has no wording
|
|
324
|
+
to drift, and context may vary per replay — sorting and parameterized
|
|
325
|
+
reloads become legal on cached queries again.
|
|
326
|
+
|
|
327
|
+
**Unchanged:** the negative cache (request-hash keyed, TTL'd), schema-
|
|
328
|
+
fingerprint freshness/eviction, backends and tiering, `validateEntry`.
|
|
329
|
+
|
|
330
|
+
**Where it landed:**
|
|
331
|
+
|
|
332
|
+
- Engine (`engine/runtime.ts`): `resolveFingerprint` — no fp → mint; fp alone
|
|
333
|
+
→ replay or `cache_miss`; fp + request → hit on matching request hash,
|
|
334
|
+
regenerate-and-replace on mismatch (409 when protected). Fingerprint-only
|
|
335
|
+
replays use the entry's STORED shape for the array-vs-single envelope.
|
|
336
|
+
Replays touch `lastUsedAt`.
|
|
337
|
+
- Request schema: `fingerprint` in; `intent`/`shape` optional. Meta:
|
|
338
|
+
`cache: { hit, fingerprint, replaced?, intent? }`.
|
|
339
|
+
- Backends: keys are fingerprint strings; entries carry `protected`,
|
|
340
|
+
`lastUsedAt`, `requestHash`; `sweepCache` GC; postgres migrates in place.
|
|
341
|
+
- HTTP: `handleQuery` (no cache param), async `handleDiscovery` overview,
|
|
342
|
+
`handleFingerprintPatch`/`handleFingerprintDelete`; express + hono adapters.
|
|
343
|
+
- Consumers: seeds re-keyed to protected names (`deals/list`, `todos/open`,
|
|
344
|
+
…); every hand-authored read replays `{ fingerprint, context }`; relay's
|
|
345
|
+
architect embeds `{ fingerprint: meta.cache.fingerprint, context }`; mythos'
|
|
346
|
+
`/api/query` is `locked` (replay-only); showroom's demos own
|
|
347
|
+
`vex-demo/<id>` slots (canned = seed + replay, live = delete + regenerate).
|
|
348
|
+
|
|
349
|
+
### Mutations (write pipeline)
|
|
350
|
+
|
|
351
|
+
Writes are first-class but deliberately NARROWER than reads. The asymmetry is
|
|
352
|
+
the design: a bad generated query shows wrong data, a bad generated write
|
|
353
|
+
destroys it — so mutations have **no generation path at all**. A mutation is
|
|
354
|
+
a dev-authored artifact that enters through the seed path, lives in the cache
|
|
355
|
+
as a `kind: 'mutation'` entry (the def in the same jsonb slot a query's DSL
|
|
356
|
+
uses), and is invoked by fingerprint. The def never travels: an inline
|
|
357
|
+
`{ mutation }` body is not a request shape, and the wire is ONE shape for
|
|
358
|
+
both kinds — `POST { fingerprint, context }`, dispatched by the entry's kind.
|
|
359
|
+
Replay-only holds under every posture (`locked` only ever governed query
|
|
360
|
+
generation); the query engine refuses a mutation fingerprint explicitly.
|
|
361
|
+
|
|
362
|
+
The pipeline: validate (closed grammar — `$scope` unauthorable, update/delete
|
|
363
|
+
require a WHERE) → desugar (upsert → insert/update by key presence) →
|
|
364
|
+
require context (a write never executes with holes; the `missing_context`
|
|
365
|
+
error carries the FULL derived signature) → scope (the same `ScopePolicy`
|
|
366
|
+
reads use: `set` writes identity on insert and update, `match` pins rows) → column
|
|
367
|
+
check against the introspected schema → compile (through the read
|
|
368
|
+
pipeline's own `compileFilter`/`compileFieldOrValue`/`resolveParams`, so
|
|
369
|
+
values bind as SQL parameters exactly like reads) → execute. Batches run in
|
|
370
|
+
one transaction; the client is structural (`MutationClient` — PGlite, a pg
|
|
371
|
+
wrapper, a test double), so the core imports no driver.
|
|
372
|
+
|
|
373
|
+
**Derived context signatures** close the discoverability loop for both
|
|
374
|
+
kinds: a `$context` ref sits at a position whose column the schema types, so
|
|
375
|
+
"what do I pass this fingerprint" is computed from the stored def — never
|
|
376
|
+
authored, never stale. Discovery lists every entry with kind, intent, the
|
|
377
|
+
typed context contract, and shape (reads) or effect (`{op, table, columns}`,
|
|
378
|
+
writes). An authoring lint (`lintMutation`) refuses un-keyed update/delete
|
|
379
|
+
defs at seed time.
|
|
380
|
+
|
|
381
|
+
### Reactive reads (`refresh`)
|
|
382
|
+
|
|
383
|
+
Agreed and built 2026-09-26. A read can keep answering.
|
|
384
|
+
|
|
385
|
+
**The declaration is on the entry**, because freshness is a property of the
|
|
386
|
+
question — "the current standings" is live, "the answer as asked" is not:
|
|
387
|
+
|
|
388
|
+
```ts
|
|
389
|
+
{ fingerprint: 'members/roster', refresh: 'reactive', dsl: { … } }
|
|
390
|
+
```
|
|
391
|
+
|
|
392
|
+
`refresh` is a mode, not a flag: `'snapshot'` (the default, and what every read
|
|
393
|
+
was before this existed — answered when asked, never again) or `'reactive'`. It
|
|
394
|
+
will grow; `'clock'` — time as a source of change, for a read that depends on
|
|
395
|
+
`today` — is the next state, and is not accepted until it is built, because a
|
|
396
|
+
value the schema takes and the engine ignores would read as live and behave as
|
|
397
|
+
a snapshot. Only a stored entry can be reactive (the mode is authored, never
|
|
398
|
+
generated), and only one whose mapping is compiled, because a later answer must
|
|
399
|
+
never wait on a model. It persists like `reach`: a column in the Postgres
|
|
400
|
+
cache, validated on write, compared by the seed path.
|
|
401
|
+
|
|
402
|
+
**The caller follows by passing `{ signal, onChange }`** to `execute` — the
|
|
403
|
+
first answer is the returned promise as always; every later answer that
|
|
404
|
+
differs goes to `onChange` until `signal` aborts. Over HTTP there is no
|
|
405
|
+
callback, so `handleQuery` takes an optional in-process `VexLive`, and the
|
|
406
|
+
hono adapter reads one from the request's env — the third argument of hono's
|
|
407
|
+
`app.request`, which only an in-process caller can set. A read nobody follows
|
|
408
|
+
is simply a read answered from the rows cache.
|
|
409
|
+
|
|
410
|
+
**The rows cache** (`engine/live.ts`) holds two things apart:
|
|
411
|
+
|
|
412
|
+
- **rows** — key → the rows a reactive query returned. The only large thing,
|
|
413
|
+
bounded by age (`ttlMs`, default 60 s), count (`maxEntries`) and text
|
|
414
|
+
(`maxBytes`, measured as each result's JSON length), least-recently-used
|
|
415
|
+
first out; a result over `maxEntryBytes` is never stored. Evicting rows never
|
|
416
|
+
breaks anything; it costs one query. It lives in process memory, per engine.
|
|
417
|
+
- **follows** — key → the tables the query reads, how to run it again, and who
|
|
418
|
+
to tell. Bounded by what is followed, not by this module: a follow ends when
|
|
419
|
+
its last follower's signal aborts.
|
|
420
|
+
|
|
421
|
+
**The key is the compiled SQL and its bound values.** Scope compiles into both
|
|
422
|
+
— row rules into the SQL, scope values into the parameters — so two callers
|
|
423
|
+
share rows exactly when they would provably get the same rows. The cache never
|
|
424
|
+
reasons about policy. The MAPPING, which reads `$.scope` (a reader's
|
|
425
|
+
language), runs per caller over the shared rows: one query for twenty
|
|
426
|
+
viewers, twenty cheap mappings.
|
|
427
|
+
|
|
428
|
+
**A write bumps its tables' versions.** Stored rows remember the versions they
|
|
429
|
+
were read at, and are stale when those moved — invalidation is a counter bump,
|
|
430
|
+
not a walk over the cache. The same stamp closes the race that would otherwise
|
|
431
|
+
poison it: rows fetched by a read that started before a write and finished
|
|
432
|
+
after it carry the versions from before, so they are stale the moment they
|
|
433
|
+
land, and a new read does not join that fetch.
|
|
434
|
+
|
|
435
|
+
**Who invalidates: the handler, after every committed mutation** —
|
|
436
|
+
`engine.invalidate(tablesChangedBy(writes, schema))`. It sits there, and not in
|
|
437
|
+
any host's observer, because this is the one door every vex write passes: an
|
|
438
|
+
HTTP request, a host's in-process machinery write, a reflex's effect. A read
|
|
439
|
+
that heard about only some writes would be live only sometimes, which is worse
|
|
440
|
+
than a snapshot that says what it is. `tablesChangedBy` adds, for a delete,
|
|
441
|
+
every table that references a deleted one, transitively — a cascade changes
|
|
442
|
+
rows vex never wrote, and the introspected schema carries the reference but
|
|
443
|
+
not its rule, so a delete is assumed to reach them all (over-telling costs a
|
|
444
|
+
refetch; under-telling leaves a screen stale). Writes vex never sees — raw SQL,
|
|
445
|
+
a database trigger — are not invalidations; a host that makes them calls
|
|
446
|
+
`engine.invalidate` itself, and the TTL heals the rest.
|
|
447
|
+
|
|
448
|
+
**Followed reads refetch once per burst** (`debounceMs`, trailing) — thirty
|
|
449
|
+
writes are one query — and are numbered, so a slow refetch landing after a
|
|
450
|
+
newer one is dropped. Change is decided twice: once per key on the rows (the
|
|
451
|
+
same rows → nobody is mapped), then per follower on its own mapped answer
|
|
452
|
+
(canonical JSON, hashed), so a follower hears only when *its* answer changed.
|
|
453
|
+
Every followed read is also re-checked once per TTL, bypassing the cache — a
|
|
454
|
+
lost invalidation heals within one TTL. A refetch that fails keeps the last
|
|
455
|
+
good answer and says so (`rows.error`).
|
|
456
|
+
|
|
457
|
+
**Visibility.** `engine.rows.stats()` — entries, bytes, follows, followers,
|
|
458
|
+
hits, misses, evictions, oversized, refreshes — and `rows.evict` /
|
|
459
|
+
`rows.error` events on the engine's `onEvent`. An engine evicting faster than
|
|
460
|
+
it answers warns once a minute: its `rows` limits are too small for what it
|
|
461
|
+
holds.
|
|
462
|
+
|
|
463
|
+
**Rules for authors.** A reactive entry should `sort` — an unordered read can
|
|
464
|
+
come back in another order after a write and count as changed (harmless: one
|
|
465
|
+
extra answer). It reads time from `$scope`, never context: the values bound
|
|
466
|
+
when the read was made are the ones it refetches with.
|
|
467
|
+
|
|
468
|
+
### Who "me" is (`behaviors`, `GenerationCaller.bindings`)
|
|
469
|
+
|
|
470
|
+
Built 2026-09-27. A generated query can mean the caller — "what's my name?",
|
|
471
|
+
"my orders", "after me" — without the model ever seeing who that is.
|
|
472
|
+
|
|
473
|
+
**The meaning comes from what the host declared, never from a name.** The
|
|
474
|
+
host's behaviors (the `ScopeBehaviors` its policies are compiled from) bind
|
|
475
|
+
columns to scope keys: `{ set: 'member_id', to: 'userId' }`,
|
|
476
|
+
`{ match: 'studio_id', to: 'tenantId' }`. Scope values are always the caller's
|
|
477
|
+
own, so a column bound to a key the caller carries holds that value on the
|
|
478
|
+
caller's own rows — whether it is called `user_id`, `guest_id` or `id`.
|
|
479
|
+
|
|
480
|
+
**Why the behaviors and not the policy.** A compiled policy keeps only the
|
|
481
|
+
phases a caller is granted. A member who may only read `members` has
|
|
482
|
+
`members: { read: [] }`; the rule that says `member_id` is them lives on an
|
|
483
|
+
insert they do not hold. So the engine takes the behaviors themselves
|
|
484
|
+
(`QueryEngineConfig.behaviors`) and reads every reach and every phase.
|
|
485
|
+
|
|
486
|
+
**What a generation is handed**: `GenerationCaller.bindings` — every column
|
|
487
|
+
bound to a scope key this caller actually carries, on a table their policy lets
|
|
488
|
+
them read. Keys and columns, never values. A caller carrying none of the keys —
|
|
489
|
+
machinery, an `executeAs` job — has no self, and is told so; nothing else about
|
|
490
|
+
it is special. The reference agent turns the bindings into one context line
|
|
491
|
+
(`describeCaller`): filter such a column with `{ $scope: key }` to mean the
|
|
492
|
+
caller, and read the caller's row as an aliased subquery source to compare other
|
|
493
|
+
rows against it.
|
|
494
|
+
|
|
495
|
+
**Safety is unchanged.** The bindings are guidance for the model; access is
|
|
496
|
+
still the policy, and the value is still bound by the engine. A query written
|
|
497
|
+
this way is stored like any other and answers for whoever replays it — one
|
|
498
|
+
"what's my name?" serves everybody.
|
|
499
|
+
|
|
500
|
+
**Checked at `introspect`**: every column a behavior names must exist, or the
|
|
501
|
+
engine refuses to start.
|
|
502
|
+
|
|
503
|
+
### LLM integration (decoupled)
|
|
504
|
+
|
|
505
|
+
The engine takes two optional hooks:
|
|
506
|
+
|
|
507
|
+
- `generateDsl(request, schema, caller) → Query` — natural language + shape →
|
|
508
|
+
DSL. `schema` is only the tables the caller's policy can read; `caller.read`
|
|
509
|
+
runs a DSL through the engine's pipeline as that caller. A generation sees
|
|
510
|
+
what the person asking could see — and it is keyed (single-flight, negative
|
|
511
|
+
cache) by request AND policy, because two callers who see different tables
|
|
512
|
+
are asking different agents.
|
|
513
|
+
- `mapToShape(rows, shape) → { ir, transformed }` — raw rows → requested shape,
|
|
514
|
+
returning a compiled Prism IR that is cached for reuse.
|
|
515
|
+
|
|
516
|
+
This is the seam that keeps the core deterministic. Vex ships a reference
|
|
517
|
+
implementation of both, built on `@niscorp/cortex` and `@niscorp/prism` and
|
|
518
|
+
exported from the `@niscorp/vex/agent` subpath — `createQueryDsl` (fills
|
|
519
|
+
`generateDsl`) and `createShapeMapper` (fills `mapToShape`):
|
|
520
|
+
|
|
521
|
+
- The **query agent** (`vexQueryDslAgent`, id `vex.query`) is a structured-output
|
|
522
|
+
Cortex agent over `QuerySchema` with six tools — `getSchema`, `getSampleRows`,
|
|
523
|
+
`getDistinctValues`, `describeField`, `testQuery`, `cannotSatisfy` — and two
|
|
524
|
+
rules: a tool-call limiter (nudge at 8, abort at 10) and an unsatisfiable
|
|
525
|
+
abort. Context producers inject the live schema and the DSL JSON Schema so the
|
|
526
|
+
static instructions stay stable. Every data-touching tool is a DSL run through
|
|
527
|
+
`caller.read`; the tool set holds no adapter and no SQL.
|
|
528
|
+
`createQueryDsl` wraps it into the `generateDsl` hook.
|
|
529
|
+
- The **mapping agent** is Prism's exported `mappingAgent`; `createShapeMapper`
|
|
530
|
+
builds the same envelope the runtime replays against — `{ result: rows }` for an
|
|
531
|
+
array shape, `{ result: rows[0] ?? null }` for an object shape — runs the agent
|
|
532
|
+
once to get a transform config, compiles it to IR, and applies the IR once over
|
|
533
|
+
that envelope (the identity, with no model call, when the rows already fit).
|
|
534
|
+
|
|
535
|
+
> The reference agents are a separate concern from the deterministic core: they
|
|
536
|
+
> live behind the `@niscorp/vex/agent` subpath, so `@niscorp/cortex`,
|
|
537
|
+
> `@niscorp/signal`, and `@niscorp/prism` are pulled in only when you import it.
|
|
538
|
+
> The engine itself depends on none of them — provide your own
|
|
539
|
+
> `generateDsl`/`mapToShape`, or use the reference factories.
|
|
540
|
+
|
|
541
|
+
### HTTP layer (optional)
|
|
542
|
+
|
|
543
|
+
Vex is a library; it does not own a server. But it ships two thin framework
|
|
544
|
+
adapters and a framework-agnostic handler so a resource can be exposed with a
|
|
545
|
+
few lines:
|
|
546
|
+
|
|
547
|
+
- `handler.ts` — `handleDiscovery` returns a self-describing JSON document
|
|
548
|
+
(entities, fields, relations, the request contract, and the DSL JSON Schema);
|
|
549
|
+
`handleQuery` parses a body, runs `engine.execute`, and maps `VexError`s to
|
|
550
|
+
HTTP status codes.
|
|
551
|
+
- `@niscorp/vex/hono` and `@niscorp/vex/express` wrap that handler: `GET`
|
|
552
|
+
returns discovery, `POST` runs a query, scope is resolved per-request via a
|
|
553
|
+
`getScope` callback, and the `cache` query param selects the cache mode.
|
|
554
|
+
|
|
555
|
+
### Events
|
|
556
|
+
|
|
557
|
+
An optional `onEvent` handler receives a typed `VexEvent` stream across the
|
|
558
|
+
pipeline — `query.start`, `query.cache`, `query.dsl`, `query.sql`,
|
|
559
|
+
`query.rows`, `query.mapped`, `query.done`, `query.error`, plus `llm.request`/
|
|
560
|
+
`llm.response` for callers that wrap their LLM client. This is how the dev
|
|
561
|
+
server renders per-stage timing and a full LLM transcript log.
|
|
562
|
+
|
|
563
|
+
---
|
|
564
|
+
|
|
565
|
+
## File structure
|
|
566
|
+
|
|
567
|
+
```
|
|
568
|
+
src/
|
|
569
|
+
index.ts Public API barrel
|
|
570
|
+
types.ts QueryEngine, QueryEngineConfig, ExecuteOptions, hook types
|
|
571
|
+
errors.ts VexError (code + details)
|
|
572
|
+
events.ts VexEvent union, VexEventHandler
|
|
573
|
+
handler.ts Framework-agnostic discovery + query handlers
|
|
574
|
+
|
|
575
|
+
schemas/
|
|
576
|
+
query.schema.ts QuerySchema (top-level DSL) + Source, SortEntry
|
|
577
|
+
filter.schema.ts FilterSchema (recursive union)
|
|
578
|
+
compute.schema.ts ComputeExpressionSchema
|
|
579
|
+
aggregate.schema.ts AggregateExpressionSchema
|
|
580
|
+
value.schema.ts FieldOrValue, ContextRef ($context), ScopeRef ($scope)
|
|
581
|
+
database.schema.ts DatabaseSchema, EntitySchema, FieldSchema, ...
|
|
582
|
+
request.schema.ts QueryRequestSchema, QueryResponse, error codes
|
|
583
|
+
index.ts Barrel
|
|
584
|
+
|
|
585
|
+
scope/
|
|
586
|
+
discover.ts discoverEntities(dsl)
|
|
587
|
+
apply.ts applyScope(dsl, entities, policy), VexScopeError
|
|
588
|
+
scope.types.ts ScopePolicy, ScopeEntityRule, ScopeFilterRule
|
|
589
|
+
|
|
590
|
+
engine/
|
|
591
|
+
runtime.ts createQueryEngine — orchestration + cache + single-flight
|
|
592
|
+
resolver.ts resolve(dsl, schema) → ResolvedQuery
|
|
593
|
+
analyzer.ts analyze(resolved, config) → { warnings, errors }
|
|
594
|
+
executor.ts executeQuery, buildContextContract, findMissingContext
|
|
595
|
+
live.ts createLiveRows — the rows cache and follows behind refresh: 'reactive'
|
|
596
|
+
engine.types.ts ResolvedQuery and friends, AnalysisConfig, TestResult
|
|
597
|
+
|
|
598
|
+
adapters/
|
|
599
|
+
adapter.types.ts DatabaseAdapter, CompiledQuery, ParamSlot, capabilities
|
|
600
|
+
postgres/
|
|
601
|
+
postgres.adapter.ts createPostgresAdapter
|
|
602
|
+
introspect.ts schema discovery from pg_catalog
|
|
603
|
+
compile.ts ResolvedQuery → parameterized SQL
|
|
604
|
+
operators.ts per-operator SQL generation
|
|
605
|
+
hono/index.ts vex() Hono app (@niscorp/vex/hono)
|
|
606
|
+
express/index.ts vex() Express handler (@niscorp/vex/express)
|
|
607
|
+
index.ts adapter barrel
|
|
608
|
+
|
|
609
|
+
mutations/
|
|
610
|
+
schema.ts MutationSchema (closed grammar), MutationDefinitionSchema
|
|
611
|
+
engine.ts executeMutation — desugar/scope/validate/compile/execute
|
|
612
|
+
signature.ts collectMutationContext, collectQueryContext, mutationEffect, lintMutation
|
|
613
|
+
index.ts Barrel
|
|
614
|
+
|
|
615
|
+
cache/
|
|
616
|
+
cache.types.ts CacheBackend, CacheEntry (ok | mutation | unsatisfiable)
|
|
617
|
+
hash.ts computeShapeHash, computeRequestHash, computeSchemaFingerprint
|
|
618
|
+
memory.ts createMemoryCache (L1)
|
|
619
|
+
postgres.ts createPostgresCache (L2, durable)
|
|
620
|
+
tiered.ts createTieredCache (L1 + L2, warm-up)
|
|
621
|
+
validate.ts validateEntry
|
|
622
|
+
util.ts isEntryFresh, fireAndForget
|
|
623
|
+
index.ts Barrel
|
|
624
|
+
|
|
625
|
+
agent/ Reference Cortex agents (exported via ./agent)
|
|
626
|
+
query.agent.ts defineAgent: vexQueryDslAgent (vex.query → Query)
|
|
627
|
+
tools.ts the six query tools
|
|
628
|
+
rules.ts tool-limit + unsatisfiable rules
|
|
629
|
+
producers.ts schema + DSL-spec context producers
|
|
630
|
+
index.ts createQueryDsl, createShapeMapper, vexQueryDslAgent
|
|
631
|
+
|
|
632
|
+
utils/
|
|
633
|
+
context.ts buildValidationContext, resolveParams
|
|
634
|
+
|
|
635
|
+
scripts/ Dev-only: docker, seed, dev server, fixtures
|
|
636
|
+
```
|
|
637
|
+
|
|
638
|
+
---
|
|
639
|
+
|
|
640
|
+
## Dependencies
|
|
641
|
+
|
|
642
|
+
| Package | Relationship | Purpose |
|
|
643
|
+
|---------|--------------|---------|
|
|
644
|
+
| `zod` | peer (required) | Schema validation + JSON Schema generation |
|
|
645
|
+
| `pg` | peer (optional) | Postgres adapter and Postgres cache backend |
|
|
646
|
+
| `hono` | peer (optional) | Hono framework adapter |
|
|
647
|
+
| `express` | peer (optional) | Express framework adapter |
|
|
648
|
+
| `@niscorp/cortex` | peer (optional) | Reference query agent (`defineAgent`/`defineTool`/`defineRule`, standalone runtime) |
|
|
649
|
+
| `@niscorp/signal` | peer (optional) | LLM calls + embeddings, via Cortex |
|
|
650
|
+
| `@niscorp/prism` | peer (optional) | Result mapping (`mappingAgent`, IR compile/execute) |
|
|
651
|
+
|
|
652
|
+
Only `zod` is mandatory. Everything else is pulled in only by the path you use.
|
|
653
|
+
|
|
654
|
+
---
|
|
655
|
+
|
|
656
|
+
## Key design decisions
|
|
657
|
+
|
|
658
|
+
1. **Constrained DSL, not text-to-SQL.** Security (the model can't emit
|
|
659
|
+
injection through a validated JSON union), cacheability (a structure hashes
|
|
660
|
+
more reliably than free text), and safety (the analyzer rejects cartesian
|
|
661
|
+
products and over-deep nesting *before* SQL exists).
|
|
662
|
+
|
|
663
|
+
A validated JSON union is only injection-proof if every string in it reaches
|
|
664
|
+
SQL as something the union can vouch for. Authored text reaches a statement
|
|
665
|
+
exactly three ways: a **column** that resolved against the introspected
|
|
666
|
+
schema, a **literal** quoted the way Postgres's `quote_literal` does (safe
|
|
667
|
+
under either `standard_conforming_strings`), or an **output name** held to
|
|
668
|
+
`ident` and quoted. Values from a request bind as parameters and never reach
|
|
669
|
+
text at all. The name grammar (`identifier.schema.ts`) is checked three
|
|
670
|
+
times — at parse, in the resolver for a DSL that skipped parsing, and by a
|
|
671
|
+
compiler that refuses any field position that did not resolve.
|
|
672
|
+
|
|
673
|
+
⟲ It was not, until 2026-09-24. Field positions (`isNull`, `in`, `like`,
|
|
674
|
+
aggregate arguments) were typed `z.string()`; the resolver skipped any
|
|
675
|
+
string without a dot and the compiler pasted it into the statement, so
|
|
676
|
+
`{ isNotNull: "(SELECT secret FROM …)" }` was a boolean oracle over a table
|
|
677
|
+
the policy denied. Output names went into `AS "…"` unescaped, and a literal
|
|
678
|
+
containing a backslash could close its own string on a server with
|
|
679
|
+
`standard_conforming_strings` off. `test/security/injection.test.ts` is the
|
|
680
|
+
attack list, against a real Postgres; it failed 35 of 41 before the fix.
|
|
681
|
+
|
|
682
|
+
2. **The LLM is injected, not embedded.** Making `generateDsl`/`mapToShape`
|
|
683
|
+
plain hooks keeps the engine deterministic and testable without a model,
|
|
684
|
+
lets the heavy Cortex/Signal/Prism stack stay optional, and lets a consumer
|
|
685
|
+
swap in their own generator. The bundled Cortex agents are a reference, not a
|
|
686
|
+
requirement.
|
|
687
|
+
|
|
688
|
+
3. **Fingerprint identity, minted or named.** The output shape still
|
|
689
|
+
determines the DSL and the mapping — but the cache *key* is a fingerprint,
|
|
690
|
+
not the shape. A first request mints one (`fp_…`); a caller may instead own
|
|
691
|
+
a named slot. Shape-hashing collided (two intents, one shape, one slot) and
|
|
692
|
+
left every entry anonymous; a fingerprint makes a generation an addressable,
|
|
693
|
+
replayable artifact — pin it, replay it, roll it back. (See *Cache v2*.)
|
|
694
|
+
|
|
695
|
+
4. **Scope injected server-side, after RESOLUTION.** If the model could see
|
|
696
|
+
scope filters it could be talked out of them. Binding them to `$scope`
|
|
697
|
+
values after the DSL is fixed makes them unforgeable.
|
|
698
|
+
|
|
699
|
+
The policy decides whether scope applies; the values only fill it. An
|
|
700
|
+
engine with a policy enforces it on every run, and a `$scope` slot nobody
|
|
701
|
+
filled refuses (`missing_scope`) rather than binding NULL — which matches no
|
|
702
|
+
row on a read and stamps NULL on a write.
|
|
703
|
+
|
|
704
|
+
⟲ Until 2026-09-24 scoping switched on only when a policy AND values were
|
|
705
|
+
both present. Omitting the values turned the policy off entirely, table
|
|
706
|
+
access included — a call that forgot to say who it was for read as
|
|
707
|
+
everybody, and one did (relay's record lookup). Writes had always required
|
|
708
|
+
both; reads now agree. The generating agent's tools had the same hole by a
|
|
709
|
+
different route: they ran their own SQL, unscoped, beside the pipeline.
|
|
710
|
+
They now reach the database only through `caller.read`.
|
|
711
|
+
|
|
712
|
+
Scope answers two questions and they run at different stages, which is the
|
|
713
|
+
correction to an earlier version that ran both at once:
|
|
714
|
+
|
|
715
|
+
- **May this caller touch this table?** — `checkScope`, over entity names,
|
|
716
|
+
before resolution. Needs no schema and no aliases.
|
|
717
|
+
- **Where does the row rule go?** — `scopeResolved`, after resolution,
|
|
718
|
+
because the answer depends on HOW the table is reached.
|
|
719
|
+
|
|
720
|
+
That second stage exists because merging row rules into the DSL's `filter`
|
|
721
|
+
compiles them to WHERE, and **a WHERE predicate on a left-joined table is a
|
|
722
|
+
LEFT JOIN demoted to an INNER one.** A driving row whose optional FK is null
|
|
723
|
+
gets null-padded columns, `null = $tenant` is null rather than true, and the
|
|
724
|
+
row disappears. No error, no warning, a shorter list — and invisible until
|
|
725
|
+
the data actually contains a null FK, which in one consumer took months.
|
|
726
|
+
The resolver had already gone to the trouble of inferring `left` for exactly
|
|
727
|
+
this reason; the scope layer was undoing it two steps later.
|
|
728
|
+
|
|
729
|
+
Row rules for left-joined tables now go in that join's `ON` clause. The
|
|
730
|
+
boundary is unchanged — another tenant's row still cannot contribute
|
|
731
|
+
columns — it simply no longer annihilates the row that had none.
|
|
732
|
+
|
|
733
|
+
A consequence worth having on its own: scope writes nothing into the
|
|
734
|
+
authored `Query`. The document integrations ship over the wire carries no
|
|
735
|
+
engine-trusted field, so there is no key to strip at intake and no forgery
|
|
736
|
+
surface to remember.
|
|
737
|
+
|
|
738
|
+
10. **`exists` shadows SQL rather than inventing a vocabulary.** Correlated
|
|
739
|
+
subqueries were the one thing the filter grammar could not express, so
|
|
740
|
+
"issues nobody was sent to" and "rooms nobody is in" became two round trips
|
|
741
|
+
each — a read returning a flat array of ids, then a `notIn` against it.
|
|
742
|
+
That is fine at three dozen rows, wrong at three thousand, and racy in
|
|
743
|
+
between.
|
|
744
|
+
|
|
745
|
+
`{ exists: { from, filter } }` is the same `{ from, filter }` a query
|
|
746
|
+
already is, and nothing more: no fields, no sort, no limit. EXISTS asks
|
|
747
|
+
whether a row is there, so anything shaping output would be noise — and
|
|
748
|
+
refusing it is what keeps this an operator rather than an invitation to
|
|
749
|
+
nest arbitrary SQL. The correlation needs no new syntax at all, because a
|
|
750
|
+
dotted string on either side of a comparison is already a field path:
|
|
751
|
+
|
|
752
|
+
```json
|
|
753
|
+
{ "exists": { "from": ["tasks"],
|
|
754
|
+
"filter": { "eq": ["tasks.issue_id", "issues.id"] } } }
|
|
755
|
+
```
|
|
756
|
+
|
|
757
|
+
`not` composes for NOT EXISTS. Whoever knows SQL writes this on the first
|
|
758
|
+
try, which was the entire design constraint.
|
|
759
|
+
|
|
760
|
+
Two implementation notes that are load-bearing. It compiles **inline**,
|
|
761
|
+
into the parent's own parameter counter — the other subquery path compiles
|
|
762
|
+
independently and renumbers afterwards, which is how two parameterised
|
|
763
|
+
subqueries came to collide on `$1`; there is nothing to renumber if nothing
|
|
764
|
+
was numbered apart. And its inner aliases come from the **shared** alias
|
|
765
|
+
counter, so an `exists` over the same table as the outer query cannot take
|
|
766
|
+
the same alias and silently correlate a table with itself.
|
|
767
|
+
|
|
768
|
+
Scope recurses into it. Access-checking the inner table is not enough: an
|
|
769
|
+
EXISTS returns a boolean, but a boolean about somebody else's rows is still
|
|
770
|
+
an answer about somebody else's rows, and an uncorrelated `exists` over a
|
|
771
|
+
scoped table would otherwise report whether ANY tenant had one.
|
|
772
|
+
|
|
773
|
+
5. **Negative caching with a TTL.** An "impossible" request is expensive to
|
|
774
|
+
re-discover every time, so a `cannotSatisfy` result is cached — but TTL'd and
|
|
775
|
+
fingerprinted, because a schema change can make yesterday's impossible
|
|
776
|
+
request possible.
|
|
777
|
+
|
|
778
|
+
6. **Single-flight on the default path.** A burst of identical cache misses
|
|
779
|
+
collapses to one generation; the rest await it. `refresh`/`bypass` opt out,
|
|
780
|
+
since they explicitly mean "don't share cache state."
|
|
781
|
+
|
|
782
|
+
7. **Adapters own compilation.** The DSL and agent loop are database-agnostic;
|
|
783
|
+
only SQL generation is not. Keeping the entire compile step inside the
|
|
784
|
+
adapter means a new backend is a self-contained unit and the core never grows
|
|
785
|
+
dialect knowledge.
|
|
786
|
+
|
|
787
|
+
8. **A discovery endpoint, not just a query endpoint.** A self-describing
|
|
788
|
+
resource (`GET` → entities + contract + DSL JSON Schema) lets a client — or
|
|
789
|
+
another agent — learn how to query it without out-of-band documentation.
|
|
790
|
+
|
|
791
|
+
9. **Writes are replay-only, one wire shape with reads.** Mutations are
|
|
792
|
+
dev-authored cache entries (`kind: 'mutation'`) invoked by fingerprint;
|
|
793
|
+
the def never travels and no generation path exists. The client-facing
|
|
794
|
+
API is uniformly `{ fingerprint, context }` — the entry's kind picks the
|
|
795
|
+
pipeline. Context signatures are DERIVED from the stored def (typed from
|
|
796
|
+
the schema by position), so the input contract can't drift.
|
|
797
|
+
|
|
798
|
+
---
|
|
799
|
+
|
|
800
|
+
## Deferred
|
|
801
|
+
|
|
802
|
+
- Dynamic mutation generation — icebox, on purpose. If it ever returns, it
|
|
803
|
+
feeds the existing pipeline (author → validate → seed) behind an effect
|
|
804
|
+
contract and an approval step; nothing shipped today gets replaced.
|
|
805
|
+
- Mutation transactions with cross-statement refs (`$returned.id` — the
|
|
806
|
+
create-parent-then-children case). Batches are atomic today but cannot
|
|
807
|
+
reference earlier statements' results.
|
|
808
|
+
- MySQL / SQLite adapters — the interface is ready; implementation on demand.
|
|
809
|
+
- Multi-level field paths — addable in the resolver without DSL changes.
|
|
810
|
+
- ~~Intent-aware positive caching~~ — superseded: collisions did bite, and the
|
|
811
|
+
answer is identity, not intent-matching. See **Cache v2 — fingerprints**.
|
|
812
|
+
- Cross-process cache coherence for the tiered backend (an L1 TTL refresh).
|
|
813
|
+
- `refresh: 'clock'` — time as a source of change: a reactive read whose bound
|
|
814
|
+
scope includes a clock value (`today`) refetches when that value next moves.
|
|
815
|
+
- Cross-process invalidation for reactive reads — a host carries
|
|
816
|
+
`engine.invalidate` between processes (moss's fabric); until then each
|
|
817
|
+
process's TTL bounds the staleness.
|
|
818
|
+
- Row-level pruning — for a single-table reactive read, test a written row
|
|
819
|
+
against the scoped filter and skip the refetch. An optimisation, not a design.
|
|
820
|
+
- Adapter-owned vector serialization — the pgvector `[...]` literal is currently
|
|
821
|
+
formed in the generic binding step; a second adapter would move it behind the
|
|
822
|
+
adapter.
|
|
823
|
+
|
|
824
|
+
## Reach is the caller's, not the table's
|
|
825
|
+
|
|
826
|
+
A `ScopePolicy` fuses two questions: which phases an entity has, and what a
|
|
827
|
+
granted phase does. The second used to be answered per TABLE — one `match` rule
|
|
828
|
+
per entity, applied to every caller that held any grant on it.
|
|
829
|
+
|
|
830
|
+
That is right until two callers need the same grant at different distances. "The
|
|
831
|
+
desk reads every booking; a member reads their own" cannot be said with one rule
|
|
832
|
+
per table, and the workaround is always the same: a SECOND TABLE carrying the
|
|
833
|
+
tighter rule, kept level with the first by a trigger. One fact in two places is
|
|
834
|
+
a drift bug waiting for the first ordering nobody thought about.
|
|
835
|
+
|
|
836
|
+
So a table may declare NAMED rule sets, and `createScopePolicy` takes the
|
|
837
|
+
caller's profile:
|
|
838
|
+
|
|
839
|
+
bookings: { default: { read: [tenant] },
|
|
840
|
+
personal: { read: [tenant, own] } }
|
|
841
|
+
|
|
842
|
+
createScopePolicy(grants, behaviors) // studio-wide
|
|
843
|
+
createScopePolicy(grants, behaviors, 'personal') // + their own rows
|
|
844
|
+
|
|
845
|
+
The profile is chosen ONCE per caller rather than per grant, because "acts for
|
|
846
|
+
themselves" is a property of the caller and not of each thing they read — and
|
|
847
|
+
saying it once is one place to get it wrong instead of eight. A table declaring
|
|
848
|
+
no variant under a profile falls back to its `default`, which is what keeps
|
|
849
|
+
shared reads (a timetable everyone sees the same way) free of per-profile
|
|
850
|
+
entries.
|
|
851
|
+
|
|
852
|
+
It fails closed in the direction that matters. An unknown profile denies
|
|
853
|
+
everything rather than falling back: a mistyped name that quietly meant "the
|
|
854
|
+
default" would widen a caller to every row of every table they hold a grant on,
|
|
855
|
+
which is the one failure mode a policy layer may not have. A table that declares
|
|
856
|
+
named variants and no `default` likewise refuses an unprofiled caller instead of
|
|
857
|
+
guessing that "no rule" is safe.
|
|
858
|
+
|
|
859
|
+
AND A CALLER IS NOT ONE ROLE. Reach belongs to a role, and people hold several —
|
|
860
|
+
the instructor who trains at the studio they teach at is the ordinary case, not
|
|
861
|
+
the edge. So the unit of compilation is the ROLE: one policy each, merged
|
|
862
|
+
(`mergeScopePolicies`), broadest rule set winning per entity and phase. A
|
|
863
|
+
principal may do anything any of their roles permits.
|
|
864
|
+
|
|
865
|
+
That merge has a consequence worth stating plainly: for somebody holding both, a
|
|
866
|
+
staff role's studio-wide reach WINS over a member role's personal one on the
|
|
867
|
+
tables they share. Correct for the roster they are paid to read, wrong for their
|
|
868
|
+
own membership card.
|
|
869
|
+
|
|
870
|
+
SO REACH IS SAID TWICE, and by two different parties. The CALLER carries a floor
|
|
871
|
+
— how far this principal may see, from their role. The ENTRY may carry a ceiling
|
|
872
|
+
— how far this READ may see, whoever asks (`reach`). "Their own card" is a
|
|
873
|
+
property of the question, not of whoever happens to be asking it, and an app that
|
|
874
|
+
could only say it about people was an app that got it wrong for anybody holding
|
|
875
|
+
two roles.
|
|
876
|
+
|
|
877
|
+
The two compose in one direction only: an entry's reach recompiles the caller's
|
|
878
|
+
OWN grants under a different profile, so it narrows rows and never widens verbs.
|
|
879
|
+
A principal without the verb is refused either way. And a declared reach that
|
|
880
|
+
cannot be compiled refuses the read rather than falling back — the fallback would
|
|
881
|
+
be an answer with too much in it, and nothing would say so.
|