@niscorp/vex 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/DESIGN.md +881 -0
- package/DOCS.md +1295 -0
- package/LICENSE +202 -0
- package/README.md +97 -0
- package/dist/adapters/express/index.cjs +1406 -0
- package/dist/adapters/express/index.cjs.map +1 -0
- package/dist/adapters/express/index.d.cts +25 -0
- package/dist/adapters/express/index.d.ts +25 -0
- package/dist/adapters/express/index.js +60 -0
- package/dist/adapters/express/index.js.map +1 -0
- package/dist/adapters/hono/index.cjs +1426 -0
- package/dist/adapters/hono/index.cjs.map +1 -0
- package/dist/adapters/hono/index.d.cts +25 -0
- package/dist/adapters/hono/index.d.ts +25 -0
- package/dist/adapters/hono/index.js +78 -0
- package/dist/adapters/hono/index.js.map +1 -0
- package/dist/adapters/pglite/index.cjs +17 -0
- package/dist/adapters/pglite/index.cjs.map +1 -0
- package/dist/adapters/pglite/index.d.cts +21 -0
- package/dist/adapters/pglite/index.d.ts +21 -0
- package/dist/adapters/pglite/index.js +14 -0
- package/dist/adapters/pglite/index.js.map +1 -0
- package/dist/agent/index.cjs +478 -0
- package/dist/agent/index.cjs.map +1 -0
- package/dist/agent/index.d.cts +29 -0
- package/dist/agent/index.d.ts +29 -0
- package/dist/agent/index.js +313 -0
- package/dist/agent/index.js.map +1 -0
- package/dist/chunk-FPKW4TRX.js +1528 -0
- package/dist/chunk-FPKW4TRX.js.map +1 -0
- package/dist/chunk-HF25Q53I.js +165 -0
- package/dist/chunk-HF25Q53I.js.map +1 -0
- package/dist/handler-Cql8SJ3F.d.cts +133 -0
- package/dist/handler-mICf9ins.d.ts +133 -0
- package/dist/index.cjs +4078 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +199 -0
- package/dist/index.d.ts +199 -0
- package/dist/index.js +2340 -0
- package/dist/index.js.map +1 -0
- package/dist/introspect-D3OaXHBc.d.cts +16 -0
- package/dist/introspect-D3OaXHBc.d.ts +16 -0
- package/dist/types-BRvMB6IM.d.cts +1491 -0
- package/dist/types-BRvMB6IM.d.ts +1491 -0
- package/package.json +125 -0
package/DOCS.md
ADDED
|
@@ -0,0 +1,1295 @@
|
|
|
1
|
+
# Vex Documentation
|
|
2
|
+
|
|
3
|
+
In-depth usage guide for `@niscorp/vex`. For the high-level pitch see
|
|
4
|
+
[README.md](./README.md); for architecture and rationale see
|
|
5
|
+
[DESIGN.md](./DESIGN.md).
|
|
6
|
+
|
|
7
|
+
## Contents
|
|
8
|
+
|
|
9
|
+
- [Concepts](#concepts)
|
|
10
|
+
- [Installation](#installation)
|
|
11
|
+
- [Creating an engine](#creating-an-engine)
|
|
12
|
+
- [Configuration reference](#configuration-reference)
|
|
13
|
+
- [The request/response contract](#the-requestresponse-contract)
|
|
14
|
+
- [Engine methods](#engine-methods)
|
|
15
|
+
- [The DSL](#the-dsl)
|
|
16
|
+
- [Scope policies](#scope-policies)
|
|
17
|
+
- [Grants and behaviors](#grants-and-behaviors)
|
|
18
|
+
- [Mutations](#mutations)
|
|
19
|
+
- [Caching](#caching)
|
|
20
|
+
- [Wiring the LLM agents](#wiring-the-llm-agents)
|
|
21
|
+
- [Framework adapters](#framework-adapters)
|
|
22
|
+
- [Events](#events)
|
|
23
|
+
- [Error handling](#error-handling)
|
|
24
|
+
- [Local development](#local-development)
|
|
25
|
+
|
|
26
|
+
---
|
|
27
|
+
|
|
28
|
+
## Concepts
|
|
29
|
+
|
|
30
|
+
**Request → shape, not query.** A caller sends an `intent` (optional natural
|
|
31
|
+
language), a `shape` (an example of the structure they want back), and a
|
|
32
|
+
`context` object (runtime values). They do not write SQL or even the DSL.
|
|
33
|
+
|
|
34
|
+
**The DSL is the intermediate.** Either you hand-write a DSL `Query` and call
|
|
35
|
+
`compile`/`test`/`execute`, or an LLM hook generates one from the request. The
|
|
36
|
+
DSL is validated by Zod before anything compiles it.
|
|
37
|
+
|
|
38
|
+
**Deterministic core.** Everything except DSL generation and result mapping is
|
|
39
|
+
pure and LLM-free: discover entities → apply scope → resolve to columns/joins →
|
|
40
|
+
analyze → compile to parameterized SQL → bind → execute. You can run this whole
|
|
41
|
+
path with no model and no network beyond the database.
|
|
42
|
+
|
|
43
|
+
**Fingerprint caching.** Every generation is cached under a **fingerprint** —
|
|
44
|
+
server-minted (`fp_…`, returned in `meta.cache.fingerprint`) or caller-chosen
|
|
45
|
+
(a name like `deals/list`). Replaying a fingerprint skips generation entirely
|
|
46
|
+
and re-runs only the deterministic path — so scope and context stay current
|
|
47
|
+
while LLM cost drops to zero. A minted fingerprint is an immutable pin; a
|
|
48
|
+
named one is a mutable slot whose content follows your request. Design and
|
|
49
|
+
rationale: [DESIGN.md](./DESIGN.md) **Cache v2 — fingerprints**.
|
|
50
|
+
|
|
51
|
+
**Scope is separate from context.** `context` is caller data, visible to the
|
|
52
|
+
agent and referenced via `{ $context: "key" }`. `scope` is server-side access
|
|
53
|
+
control, invisible to the agent, injected as filters via `{ $scope: "key" }`.
|
|
54
|
+
They are passed through different channels and must never be conflated.
|
|
55
|
+
|
|
56
|
+
**Writes are replay-only, same wire.** A mutation is a dev-authored cache
|
|
57
|
+
entry (`kind: 'mutation'`) invoked exactly like a read replay —
|
|
58
|
+
`{ fingerprint, context }` — and the entry's kind picks the pipeline. The
|
|
59
|
+
def never travels and no generation path exists for writes. See
|
|
60
|
+
[Mutations](#mutations).
|
|
61
|
+
|
|
62
|
+
**Reserved sort keys.** `sortBy` (an `entity.field`) and `sortDir` (`asc`/`desc`)
|
|
63
|
+
are reserved keys in `context`: the engine reads them straight into the `ORDER BY`
|
|
64
|
+
(the column is schema-validated) and they are never bound as parameters. The
|
|
65
|
+
caller's column **leads**; the query's own `sort` keys stay behind it as
|
|
66
|
+
tiebreakers, minus a duplicate of the chosen column. That matters for a keyset
|
|
67
|
+
page: an entry sorted `(name, id)` keeps `id` behind whatever the caller asks
|
|
68
|
+
for, so the order stays total and no row can straddle a page boundary
|
|
69
|
+
unreachably. An empty `sortBy` leaves the authored order alone.
|
|
70
|
+
|
|
71
|
+
**Optional conditions.** `{ optional: { key, then } }` lets a caller switch a
|
|
72
|
+
condition ON by supplying its key — omit it and the condition is removed from
|
|
73
|
+
the query before it compiles. It is *not there*, as opposed to there and
|
|
74
|
+
matching everything, so one entry answers "everyone", "everyone matching a
|
|
75
|
+
search", and "one by id" without sentinel values like `'%'`. Absent means
|
|
76
|
+
missing, `undefined` or `null` (`''` is a value); a condition reading several
|
|
77
|
+
keys names them all, and the direction of the change follows position — under
|
|
78
|
+
`and` supplying the key narrows, under `or` it widens. Reads only: the seed lint
|
|
79
|
+
refuses one inside a mutation, and the correlation inside an `exists` must hold
|
|
80
|
+
unconditionally. `meta.context` marks such keys `optional: true`, plus
|
|
81
|
+
`absent: true` on a run that did not supply them, so a caller can learn what a
|
|
82
|
+
fingerprint accepts without failing a request first.
|
|
83
|
+
|
|
84
|
+
---
|
|
85
|
+
|
|
86
|
+
## Installation
|
|
87
|
+
|
|
88
|
+
```bash
|
|
89
|
+
pnpm add @niscorp/vex zod
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
`zod` is the only required peer. Add the others for the paths you use:
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
pnpm add pg # Postgres adapter + Postgres cache
|
|
96
|
+
pnpm add hono # or: express # framework adapter
|
|
97
|
+
pnpm add @niscorp/cortex @niscorp/signal @niscorp/prism # reference LLM agents
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
For semantic (vector) search you need a Postgres with `pgvector` (the dev setup
|
|
101
|
+
uses the `pgvector/pgvector:pg16` image) and an embedding function wired onto the
|
|
102
|
+
engine — see [Semantic search](#semantic-search).
|
|
103
|
+
|
|
104
|
+
---
|
|
105
|
+
|
|
106
|
+
## Creating an engine
|
|
107
|
+
|
|
108
|
+
### Minimal (deterministic, no LLM)
|
|
109
|
+
|
|
110
|
+
```typescript
|
|
111
|
+
import pg from 'pg';
|
|
112
|
+
import { createQueryEngine, createPostgresAdapter } from '@niscorp/vex';
|
|
113
|
+
|
|
114
|
+
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });
|
|
115
|
+
const adapter = createPostgresAdapter({ pool });
|
|
116
|
+
|
|
117
|
+
const engine = createQueryEngine({ adapter });
|
|
118
|
+
|
|
119
|
+
await engine.introspect(); // REQUIRED before compile/test/execute
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
`introspect()` discovers tables, columns, normalized types, relations, and
|
|
123
|
+
indexes, and caches the result plus a schema fingerprint. The engine throws
|
|
124
|
+
`execution_error` ("schema not loaded") if you call `compile`/`execute`/`test`
|
|
125
|
+
before introspecting.
|
|
126
|
+
|
|
127
|
+
With no `generateDsl` hook, `execute` serves only requests whose shape is
|
|
128
|
+
already cached (and otherwise throws `agent_failed`). `compile` and `test` work
|
|
129
|
+
fully because you pass them a DSL directly.
|
|
130
|
+
|
|
131
|
+
### With LLM-backed generation and mapping
|
|
132
|
+
|
|
133
|
+
```typescript
|
|
134
|
+
const engine = createQueryEngine({
|
|
135
|
+
adapter,
|
|
136
|
+
cache, // optional; defaults to in-memory
|
|
137
|
+
scope: scopePolicy, // optional
|
|
138
|
+
onEvent: handler, // optional
|
|
139
|
+
generateDsl, // natural language → DSL (see "Wiring the LLM agents")
|
|
140
|
+
mapToShape, // raw rows → requested shape
|
|
141
|
+
});
|
|
142
|
+
await engine.introspect();
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## Configuration reference
|
|
148
|
+
|
|
149
|
+
### `QueryEngineConfig`
|
|
150
|
+
|
|
151
|
+
```typescript
|
|
152
|
+
type QueryEngineConfig = {
|
|
153
|
+
adapter: DatabaseAdapter; // required
|
|
154
|
+
scope?: ScopePolicy; // server-side access control
|
|
155
|
+
cache?: CacheBackend; // default: createMemoryCache()
|
|
156
|
+
onEvent?: VexEventHandler; // pipeline event stream
|
|
157
|
+
generateDsl?: (request, schema) => Promise<Query>;
|
|
158
|
+
mapToShape?: (rows, shape) => Promise<{ ir: CompiledIr; transformed: JsonValue }>;
|
|
159
|
+
embed?: (text, dimensions?) => Promise<number[]>; // text→vector for semantic filters
|
|
160
|
+
config?: {
|
|
161
|
+
maxNestingDepth?: number; // default: 2
|
|
162
|
+
defaultLimit?: number; // default: 100
|
|
163
|
+
maxLimit?: number; // default: 1000
|
|
164
|
+
rejectCartesianProducts?: boolean; // default: true
|
|
165
|
+
warnUnindexedFilters?: boolean; // default: true
|
|
166
|
+
rejectUnindexedFilters?: boolean; // default: false
|
|
167
|
+
entities?: string[]; // whitelist for introspection
|
|
168
|
+
unsatisfiableTtlMs?: number; // negative-cache TTL, default 300_000 (5 min)
|
|
169
|
+
};
|
|
170
|
+
};
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
- `config.entities`, when set, restricts `introspect()` to those tables.
|
|
174
|
+
- `defaultLimit` is applied when a DSL omits `limit`; `maxLimit` clamps any
|
|
175
|
+
larger value.
|
|
176
|
+
- **`embed`** turns text into a vector for `semantic` filters, at parameter-
|
|
177
|
+
binding time. It's an injected provider (wire it to Signal) — *not* a database
|
|
178
|
+
concern, so it lives on the engine, not the adapter (the adapter only emits the
|
|
179
|
+
vector-distance SQL). Required only if your schema has vector columns you
|
|
180
|
+
search over.
|
|
181
|
+
|
|
182
|
+
### `ExecuteOptions`
|
|
183
|
+
|
|
184
|
+
```typescript
|
|
185
|
+
type ExecuteOptions = {
|
|
186
|
+
scope?: ScopeValues; // { [key]: value } for $scope references
|
|
187
|
+
entities?: string[]; // restrict generation to this entity subset for this call
|
|
188
|
+
locked?: boolean; // replay-only: unknown/changed fingerprints throw `locked` instead of generating
|
|
189
|
+
};
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
There is no cache mode — the request body itself expresses
|
|
193
|
+
replay/generate/replace semantics via its `fingerprint` field (see the
|
|
194
|
+
request contract below).
|
|
195
|
+
|
|
196
|
+
An engine with a policy (configured, or `options.scopePolicy` per request)
|
|
197
|
+
enforces it on every run. `options.scope` supplies the values its row rules
|
|
198
|
+
compare against; it never decides whether the policy applies. An engine with no
|
|
199
|
+
policy is open and needs no values; a policy with no row rules needs none
|
|
200
|
+
either. A statement whose `$scope` slot the caller left empty is refused with
|
|
201
|
+
`missing_scope` — it is never read as "everybody".
|
|
202
|
+
|
|
203
|
+
---
|
|
204
|
+
|
|
205
|
+
## The request/response contract
|
|
206
|
+
|
|
207
|
+
### Request
|
|
208
|
+
|
|
209
|
+
```typescript
|
|
210
|
+
type QueryRequest = {
|
|
211
|
+
fingerprint?: string; // the cache identity (see postures below)
|
|
212
|
+
intent?: string; // natural language (drives the agent)
|
|
213
|
+
shape?: unknown; // example of the desired output structure
|
|
214
|
+
context?: Record<string, unknown>; // runtime values for { $context } refs; reserved `sortBy`/`sortDir` drive ORDER BY (default {})
|
|
215
|
+
};
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
The request is Zod-validated; `execute` throws `invalid_request` on failure.
|
|
219
|
+
It must carry a `fingerprint`, a `shape`, or both — the three postures:
|
|
220
|
+
|
|
221
|
+
1. **Explore** — `{ intent, shape, context }`: generate, cache, mint a
|
|
222
|
+
fingerprint (`fp_…`, returned in `meta.cache.fingerprint`).
|
|
223
|
+
2. **Replay** — `{ fingerprint, context }`: serve the stored entry; never
|
|
224
|
+
generates. Unknown fingerprint → `cache_miss`. Context values may vary per
|
|
225
|
+
call (they are runtime data, not identity).
|
|
226
|
+
3. **Own a slot** — `{ fingerprint, intent, shape, context }`: hit when the
|
|
227
|
+
stored request matches (intent + normalized shape + context key names),
|
|
228
|
+
regenerate and REPLACE the slot when it differs
|
|
229
|
+
(`meta.cache.replaced: true`) — unless the entry is `protected` → throws
|
|
230
|
+
`fingerprint_protected`.
|
|
231
|
+
|
|
232
|
+
The **shape** is an example, not a schema: use empty strings, zeros, and
|
|
233
|
+
booleans as type markers. `[{ id: '', total: 0 }]` means "an array of objects
|
|
234
|
+
with a string `id` and a number `total`." A bare object means a single record.
|
|
235
|
+
|
|
236
|
+
The shape also picks the result envelope: an **array** shape maps over the whole
|
|
237
|
+
row set (the mapping sees `$.result` as the rows array — a `$map`/identity); a
|
|
238
|
+
**non-array** shape maps the single (first) row (the mapping sees `$.result` as
|
|
239
|
+
that row, so a detail reads `$.result.field`). So the mapping — not Vex — owns
|
|
240
|
+
the output: array, single object, or scalar. On a fingerprint-only replay the
|
|
241
|
+
entry's STORED shape drives this — the caller sends none.
|
|
242
|
+
|
|
243
|
+
Two more roots sit beside `$.result` for the mapping to read:
|
|
244
|
+
|
|
245
|
+
| root | what it holds |
|
|
246
|
+
|---|---|
|
|
247
|
+
| `$.result` | the rows (or the single row) — what every mapping is written against |
|
|
248
|
+
| `$.context` | the request's own context values |
|
|
249
|
+
| `$.scope` | the engine-side scope values (`studioId`, `today`, `locale`, …) |
|
|
250
|
+
|
|
251
|
+
`$.scope` is what lets a mapping produce words in the **reader's** language:
|
|
252
|
+
a mapping is where a row becomes `"Active"`, `"Fri 14 Mar"`, `"€45"`, and words
|
|
253
|
+
have a language. Reading it here widens nothing a caller can reach — scope
|
|
254
|
+
values are injected by the host per session and are unauthorable by a request.
|
|
255
|
+
|
|
256
|
+
None of this touches the cache. Vex caches the query **plan** (`dsl` +
|
|
257
|
+
`prismIr`), never rows, and a compiled mapping holds the *lookup*, not the
|
|
258
|
+
looked-up value — so a locale-parameterised mapping shares one cache entry
|
|
259
|
+
across every language.
|
|
260
|
+
|
|
261
|
+
### Response
|
|
262
|
+
|
|
263
|
+
```typescript
|
|
264
|
+
type QueryResponse = {
|
|
265
|
+
result: JsonValue; // the mapping's output (array / object / scalar); raw Row[] if no mapping
|
|
266
|
+
meta: {
|
|
267
|
+
cache: {
|
|
268
|
+
hit: boolean;
|
|
269
|
+
fingerprint?: string; // the entry's identity — minted or your name
|
|
270
|
+
replaced?: boolean; // true when a named slot was regenerated + replaced
|
|
271
|
+
intent?: string; // the stored intent (descriptive only)
|
|
272
|
+
};
|
|
273
|
+
context: Record<string, { // the resolved parameter contract
|
|
274
|
+
type?: 'string' | 'number' | 'boolean' | 'string[]' | 'number[]';
|
|
275
|
+
kind: 'context' | 'scope' | 'semantic';
|
|
276
|
+
optional?: true; // supplying it switches a condition on
|
|
277
|
+
absent?: true; // optional, and not supplied this run (so: no type)
|
|
278
|
+
}>;
|
|
279
|
+
timing?: { agentMs?: number; executionMs: number; mappingMs?: number };
|
|
280
|
+
warnings?: string[]; // analyzer warnings, if any
|
|
281
|
+
missingContext?: string[]; // present when REQUIRED keys were not supplied;
|
|
282
|
+
// optional keys never appear here
|
|
283
|
+
};
|
|
284
|
+
};
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
If the compiled query needs context/scope values the caller did not provide,
|
|
288
|
+
`execute` returns a **valid** response with an empty `result` and
|
|
289
|
+
`meta.missingContext` listing the missing keys — it does not throw. Retry with
|
|
290
|
+
the values filled in. `meta.context` always describes the full parameter
|
|
291
|
+
contract so a caller knows what to provide.
|
|
292
|
+
|
|
293
|
+
---
|
|
294
|
+
|
|
295
|
+
## Engine methods
|
|
296
|
+
|
|
297
|
+
```typescript
|
|
298
|
+
type QueryEngine = {
|
|
299
|
+
introspect: () => Promise<DatabaseSchema>;
|
|
300
|
+
execute: (request: QueryRequest, options?: ExecuteOptions) => Promise<QueryResponse>;
|
|
301
|
+
compile: (dsl: Query, scope?: ScopeValues) => CompiledQuery;
|
|
302
|
+
test: (dsl: Query, scope?: ScopeValues) => Promise<TestResult>;
|
|
303
|
+
getDslSchema: () => object;
|
|
304
|
+
getSchema: () => DatabaseSchema | undefined;
|
|
305
|
+
cache: CacheBackend;
|
|
306
|
+
};
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
- **`introspect()`** — discover and cache the schema + fingerprint. Call once at
|
|
310
|
+
startup (and again after a schema migration).
|
|
311
|
+
- **`execute(request, options)`** — the full pipeline described in DESIGN.md.
|
|
312
|
+
- **`compile(dsl)`** — run the deterministic pipeline (clamp limit →
|
|
313
|
+
discover → scope → resolve → analyze → adapter compile) and return the
|
|
314
|
+
`CompiledQuery` (`{ sql, paramSlots, contextContract }`). No execution, no
|
|
315
|
+
LLM. Throws `invalid_dsl` if the analyzer finds errors.
|
|
316
|
+
- **`test(dsl, scope?)`** — compile with `limit` forced to 5, build synthetic
|
|
317
|
+
context values, execute, and return `{ rows, warnings, errors }`. Errors are
|
|
318
|
+
caught and returned in `errors` rather than thrown. Useful for validating a
|
|
319
|
+
hand-written DSL against real data.
|
|
320
|
+
- **`getDslSchema()`** — the DSL as a JSON Schema (draft-7). This is what you
|
|
321
|
+
feed an LLM so it learns the DSL.
|
|
322
|
+
- **`getSchema()`** — the last introspected `DatabaseSchema`, or `undefined`.
|
|
323
|
+
- **`cache`** — the live `CacheBackend` for inspection or manual eviction.
|
|
324
|
+
|
|
325
|
+
---
|
|
326
|
+
|
|
327
|
+
## The DSL
|
|
328
|
+
|
|
329
|
+
A `Query` is a constrained JSON object. The model never produces SQL — it fills
|
|
330
|
+
this in, and the schema's `.describe()` annotations teach it how.
|
|
331
|
+
|
|
332
|
+
```typescript
|
|
333
|
+
type Query = {
|
|
334
|
+
from: Source[]; // min 1
|
|
335
|
+
fields?: FieldRef[]; // optional with `aggregate`; "entity.field" or { field, as }
|
|
336
|
+
filter?: Filter;
|
|
337
|
+
compute?: Record<string, ComputeExpression>; // alias → expression
|
|
338
|
+
aggregate?: Record<string, AggregateExpression>; // alias → function
|
|
339
|
+
groupBy?: string[];
|
|
340
|
+
sort?: { field: string; dir?: 'asc' | 'desc' }[];
|
|
341
|
+
limit?: number;
|
|
342
|
+
distinct?: boolean;
|
|
343
|
+
};
|
|
344
|
+
|
|
345
|
+
type Source = string | { as: string; query: Query }; // entity name or subquery
|
|
346
|
+
type FieldRef = string | { field: string; as: string }; // bare column, or aliased output key
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
### Fields
|
|
350
|
+
|
|
351
|
+
Each entry is `entity.field`, or `{ field, as }` to set a distinct output key
|
|
352
|
+
(so three joined `name` columns can become `company` / `stage` / `owner`
|
|
353
|
+
without a compute). There is no `SELECT *`. `fields` is **optional** — omit it
|
|
354
|
+
for an aggregate-only query (e.g. a bare `COUNT(*)`); a query must have at least
|
|
355
|
+
one of `fields` / `aggregate`. Every entity referenced anywhere must also appear
|
|
356
|
+
in `from`.
|
|
357
|
+
|
|
358
|
+
```json
|
|
359
|
+
{ "from": ["order", "customer"],
|
|
360
|
+
"fields": ["order.id", "order.total", { "field": "customer.name", "as": "customer" }] }
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
### Filters
|
|
364
|
+
|
|
365
|
+
```typescript
|
|
366
|
+
// Comparison — both sides are field paths, literals, or { $context }/{ $scope }
|
|
367
|
+
{ eq: [a, b] } { neq: [a, b] }
|
|
368
|
+
{ gt: [a, b] } { gte: [a, b] }
|
|
369
|
+
{ lt: [a, b] } { lte: [a, b] }
|
|
370
|
+
|
|
371
|
+
// Collection — first arg is a field path; second is an array or a ref
|
|
372
|
+
{ in: ["entity.field", [v1, v2]] }
|
|
373
|
+
{ notIn: ["entity.field", { $context: "ids" }] }
|
|
374
|
+
|
|
375
|
+
// String pattern (% wildcard)
|
|
376
|
+
{ like: ["entity.field", "%abc%"] } // case-sensitive
|
|
377
|
+
{ ilike: ["entity.field", "%abc%"] } // case-insensitive
|
|
378
|
+
|
|
379
|
+
// Null
|
|
380
|
+
{ isNull: "entity.field" }
|
|
381
|
+
{ isNotNull: "entity.field" }
|
|
382
|
+
|
|
383
|
+
// Logic ('and'/'or' require ≥ 2 conditions)
|
|
384
|
+
{ and: [f1, f2, ...] }
|
|
385
|
+
{ or: [f1, f2, ...] }
|
|
386
|
+
{ not: f }
|
|
387
|
+
|
|
388
|
+
// OPTIONAL — a condition the CALLER turns on, by supplying its key(s).
|
|
389
|
+
// Absent, the condition is removed before the query compiles: it is not
|
|
390
|
+
// there, rather than there and matching everything.
|
|
391
|
+
{ optional: { key: "q", then: { ilike: ["people.name", { $context: "q" }] } } }
|
|
392
|
+
{ optional: { key: ["after", "afterId"], then: SEEK } } // needs both, so it names both
|
|
393
|
+
|
|
394
|
+
// EXISTS — is there a row over there that points back at this one?
|
|
395
|
+
// The subquery is the same { from, filter } a query is, and NOTHING else:
|
|
396
|
+
// no fields, no sort, no limit. EXISTS asks whether, not what.
|
|
397
|
+
// The correlation is an ordinary field-to-field comparison against the
|
|
398
|
+
// OUTER entity — no special syntax, because a dotted string is already a
|
|
399
|
+
// field path.
|
|
400
|
+
{ exists: { from: ["task"], filter: { eq: ["task.issue_id", "issue.id"] } } }
|
|
401
|
+
|
|
402
|
+
// NOT EXISTS is `not` around it — "issues nobody has been sent to":
|
|
403
|
+
{ not: { exists: { from: ["task"], filter: { eq: ["task.issue_id", "issue.id"] } } } }
|
|
404
|
+
|
|
405
|
+
// Prefer this over fetching a list of ids and passing them back in a
|
|
406
|
+
// `notIn`. Two round trips do not scale, and the set can change between them.
|
|
407
|
+
|
|
408
|
+
// Semantic vector similarity (query MUST be a $context/$scope ref)
|
|
409
|
+
{ semantic: { field: "product.embedding", query: { $context: "search" }, minScore?: 0.7 } }
|
|
410
|
+
|
|
411
|
+
// Fuzzy string match (Levenshtein)
|
|
412
|
+
{ fuzzy: { field: "customer.name", query: { $context: "q" }, maxDistance?: 2 } }
|
|
413
|
+
```
|
|
414
|
+
|
|
415
|
+
There is intentionally **no** `offset` (use a cursor: `{ gt: ["entity.id",
|
|
416
|
+
{ $context: "cursor" }] }`) and **no** `between` (use `{ and: [{ gte }, { lte }] }`).
|
|
417
|
+
|
|
418
|
+
Wrap the cursor in an `optional` so the first page is one that sends no cursor
|
|
419
|
+
at all rather than one that sends a sentinel — and name **every** key the seek
|
|
420
|
+
reads, because a surviving clause with a missing key returns nothing, which on a
|
|
421
|
+
paging loop reads as the end of the list.
|
|
422
|
+
|
|
423
|
+
### Values
|
|
424
|
+
|
|
425
|
+
```typescript
|
|
426
|
+
"order.status" // a field path — any string in "entity.field" form (a dot, not at either end)
|
|
427
|
+
"published" // a literal string — any string without a usable dot
|
|
428
|
+
42, true, null // literals
|
|
429
|
+
{ $context: "userId" } // caller-provided value, from request.context
|
|
430
|
+
{ $scope: "accountId" } // server-injected value, from options.scope
|
|
431
|
+
```
|
|
432
|
+
|
|
433
|
+
The resolver distinguishes a field path from a literal string purely by the
|
|
434
|
+
`entity.field` shape (a dot that is neither the first nor last character). So
|
|
435
|
+
`"order.status"` resolves to a column, while `"published"` is a literal value.
|
|
436
|
+
|
|
437
|
+
### Compute
|
|
438
|
+
|
|
439
|
+
Record of `alias → expression`. The alias is the output key.
|
|
440
|
+
|
|
441
|
+
```json
|
|
442
|
+
{
|
|
443
|
+
"compute": {
|
|
444
|
+
"total": { "add": ["order.price", "order.tax"] },
|
|
445
|
+
"displayName": { "coalesce": ["user.nickname", "user.name"] },
|
|
446
|
+
"tier": { "case": {
|
|
447
|
+
"when": [{ "condition": { "gte": ["order.total", 100] }, "then": "premium" }],
|
|
448
|
+
"else": "standard"
|
|
449
|
+
} }
|
|
450
|
+
}
|
|
451
|
+
}
|
|
452
|
+
```
|
|
453
|
+
|
|
454
|
+
Operators: `add`, `subtract`, `multiply`, `divide` (binary), `concat`,
|
|
455
|
+
`coalesce` (≥ 2 args), `case` (`when[]` of `{ condition, then }` + `else`).
|
|
456
|
+
|
|
457
|
+
### Aggregates
|
|
458
|
+
|
|
459
|
+
Record of `alias → function`. Pair with `groupBy` for grouped results; without
|
|
460
|
+
it the analyzer warns and you get a single aggregated row. Omit `fields` for an
|
|
461
|
+
aggregate-only query (a bare count):
|
|
462
|
+
|
|
463
|
+
```json
|
|
464
|
+
{ "from": ["order"], "aggregate": { "orders": { "count": "*" } } }
|
|
465
|
+
```
|
|
466
|
+
|
|
467
|
+
`sum` / `avg` / `min` / `max` take a field path **or a compute expression**, so
|
|
468
|
+
you can aggregate a derived value (no extra column needed):
|
|
469
|
+
|
|
470
|
+
```json
|
|
471
|
+
{
|
|
472
|
+
"from": ["order", "stage"],
|
|
473
|
+
"aggregate": { "weighted": { "sum": { "multiply": ["order.value", "stage.win_probability"] } } },
|
|
474
|
+
"filter": { "eq": ["order.status", "open"] }
|
|
475
|
+
}
|
|
476
|
+
```
|
|
477
|
+
|
|
478
|
+
Grouped, with a renamed field:
|
|
479
|
+
|
|
480
|
+
```json
|
|
481
|
+
{
|
|
482
|
+
"from": ["order", "customer"],
|
|
483
|
+
"fields": [{ "field": "customer.name", "as": "customer" }],
|
|
484
|
+
"aggregate": { "revenue": { "sum": "order.total" }, "orders": { "count": "*" } },
|
|
485
|
+
"groupBy": ["customer.name"]
|
|
486
|
+
}
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
Functions: `count` (field path or `"*"`); `sum` / `avg` / `min` / `max` (a field
|
|
490
|
+
path or a compute expression).
|
|
491
|
+
|
|
492
|
+
### Subqueries
|
|
493
|
+
|
|
494
|
+
A `from` entry can be `{ as, query }`. Nesting is bounded by
|
|
495
|
+
`config.maxNestingDepth` (default 2). Scope, discovery, and resolution all
|
|
496
|
+
recurse into subqueries. A subquery's output columns are referenceable from the
|
|
497
|
+
outer query as `alias.field` — every selected field (under its `as`, if
|
|
498
|
+
aliased) plus every `compute`/`aggregate` alias.
|
|
499
|
+
|
|
500
|
+
**Joins.** String entity sources are joined automatically by foreign key; a
|
|
501
|
+
source with no FK path to the others is an error. A **nullable** FK compiles to
|
|
502
|
+
a `LEFT JOIN` — a null FK never silently drops the referencing row from the
|
|
503
|
+
read (the joined fields come back null); a non-nullable FK stays an inner
|
|
504
|
+
`JOIN` (equivalent — the column can't be null). Reverse one-to-many joins are
|
|
505
|
+
inner by design (rows multiply per child). Subquery sources are **never**
|
|
506
|
+
FK-joined — any beyond the first are `CROSS JOIN`ed. That makes a row of
|
|
507
|
+
independent aggregates expressible: cross-join N single-row `COUNT(*)`
|
|
508
|
+
subqueries and select each one's count, and you get one row with N counts in a
|
|
509
|
+
single read (use an object shape so Vex maps that single row).
|
|
510
|
+
|
|
511
|
+
```json
|
|
512
|
+
{ "from": [{ "as": "c", "query": { "from": ["contact"], "aggregate": { "n": { "count": "*" } } } },
|
|
513
|
+
{ "as": "co", "query": { "from": ["company"], "aggregate": { "n": { "count": "*" } } } }],
|
|
514
|
+
"fields": [{ "field": "c.n", "as": "contacts" }, { "field": "co.n", "as": "companies" }] }
|
|
515
|
+
```
|
|
516
|
+
|
|
517
|
+
(Limitation: subquery SQL params are embedded verbatim, so at most one bound
|
|
518
|
+
`$context`/`$scope` param across all subqueries is supported today.)
|
|
519
|
+
|
|
520
|
+
### Semantic search
|
|
521
|
+
|
|
522
|
+
`{ semantic: { field, query } }` needs three things wired up: a `vector` column
|
|
523
|
+
in the schema, an HNSW or IVFFlat index on it (else the analyzer warns it will
|
|
524
|
+
be slow), and an `embed` function on the engine:
|
|
525
|
+
|
|
526
|
+
```typescript
|
|
527
|
+
import { createSignal } from '@niscorp/signal';
|
|
528
|
+
|
|
529
|
+
const embedder = createSignal('openai').model('text-embedding-3-small');
|
|
530
|
+
const engine = createQueryEngine({
|
|
531
|
+
adapter,
|
|
532
|
+
embed: (text, dimensions) =>
|
|
533
|
+
embedder.embed(text, dimensions ? { dimensions } : undefined),
|
|
534
|
+
});
|
|
535
|
+
```
|
|
536
|
+
|
|
537
|
+
At execution, the `query` ref's text is read from context, embedded, and bound
|
|
538
|
+
as the vector parameter. (Never select a vector column directly in `fields` — it
|
|
539
|
+
is for `semantic` filters only.)
|
|
540
|
+
|
|
541
|
+
---
|
|
542
|
+
|
|
543
|
+
## Scope policies
|
|
544
|
+
|
|
545
|
+
Server-side, LLM-invisible access control, applied to the resolved query/mutation
|
|
546
|
+
*after* it is authored — so a generated or injected DSL can never reference,
|
|
547
|
+
dodge, or forge it. Two roles, named for the SQL they produce:
|
|
548
|
+
|
|
549
|
+
- **`match`** (RLS) — the row's column must equal a scope value. A `WHERE` filter
|
|
550
|
+
on read/update/delete; on insert the column is pinned to that value.
|
|
551
|
+
- **`set`** (identity) — the engine writes the column from a scope value on every
|
|
552
|
+
column-writing mutation: insert values and update set alike (`write` means every
|
|
553
|
+
write; delete writes no columns, so it is structurally exempt).
|
|
554
|
+
|
|
555
|
+
```typescript
|
|
556
|
+
type ScopePolicy = {
|
|
557
|
+
default: 'allow' | 'deny'; // fallback for unlisted entities / absent phases
|
|
558
|
+
entities: Record<string, ScopeEntityRule>;
|
|
559
|
+
};
|
|
560
|
+
|
|
561
|
+
type ScopeEntityRule =
|
|
562
|
+
| { public: true } // fully open
|
|
563
|
+
| { deny: true } // fully closed
|
|
564
|
+
| {
|
|
565
|
+
read?: ScopeMatch[]; // SELECT WHERE (match only)
|
|
566
|
+
write?: (ScopeMatch | ScopeSet)[]; // UMBRELLA — grants + rules for insert/update/delete
|
|
567
|
+
insert?: (ScopeMatch | ScopeSet)[]; // specific: just INSERT, rules stack on write's
|
|
568
|
+
update?: (ScopeMatch | ScopeSet)[]; // specific: just UPDATE, rules stack on write's
|
|
569
|
+
delete?: ScopeMatch[]; // specific: just DELETE (match only — nothing to set)
|
|
570
|
+
};
|
|
571
|
+
|
|
572
|
+
type ScopeMatch = { match: string; to: string }; // row.<match> = scope[to]
|
|
573
|
+
type ScopeSet = { set: string; to: string }; // INSERT/UPDATE row.<set> := scope[to]
|
|
574
|
+
```
|
|
575
|
+
|
|
576
|
+
Example:
|
|
577
|
+
|
|
578
|
+
```typescript
|
|
579
|
+
const scope: ScopePolicy = {
|
|
580
|
+
default: 'deny',
|
|
581
|
+
entities: {
|
|
582
|
+
order: { read: [{ match: 'accountId', to: 'accountId' }],
|
|
583
|
+
write: [{ match: 'accountId', to: 'accountId' }, { set: 'owner_id', to: 'userId' }] },
|
|
584
|
+
country: { public: true },
|
|
585
|
+
secret: { deny: true },
|
|
586
|
+
},
|
|
587
|
+
};
|
|
588
|
+
|
|
589
|
+
const engine = createQueryEngine({ adapter, scope });
|
|
590
|
+
await engine.execute(request, { scope: { accountId: 'acc-99', userId: 'u-1' } });
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
Behavior:
|
|
594
|
+
|
|
595
|
+
- The query engine enforces the **`read`** rules: each `match` becomes
|
|
596
|
+
`{ eq: ["order.accountId", { $scope: "accountId" }] }`, AND-merged into the DSL.
|
|
597
|
+
- `default` covers a discovered entity with no rule *and* a listed entity with no
|
|
598
|
+
matching phase: `'deny'` throws `VexScopeError` (`scope_denied`), `'allow'`
|
|
599
|
+
passes through unfiltered. `{ public: true }` passes; `{ deny: true }` throws.
|
|
600
|
+
- The write phases are the contract for a mutation executor (the query engine
|
|
601
|
+
is read-only). **`write`** is the umbrella: its presence grants insert, update
|
|
602
|
+
and delete, and its rules apply to all of them. **`insert`**/**`update`**/**`delete`**
|
|
603
|
+
are specific phases: each grants just its op, its rules stacking on the
|
|
604
|
+
umbrella's (`delete` is match-only — nothing to set). Rule mechanics: `match`
|
|
605
|
+
filters update/delete and pins the column on insert; `set` writes the column
|
|
606
|
+
on insert AND update. Applied server-side, never authored in the mutation
|
|
607
|
+
— so identity/ownership can't be forged.
|
|
608
|
+
- A missing `$scope` value throws `missing_scope` — on a read, and on a write,
|
|
609
|
+
where binding it would stamp NULL. A missing `$context` value is different: it
|
|
610
|
+
is the caller's, and a read surfaces it via `meta.missingContext`
|
|
611
|
+
(empty-but-valid).
|
|
612
|
+
- `match` is equality today; an `op` for `in` / multi-valued boundaries is additive
|
|
613
|
+
when a real one appears.
|
|
614
|
+
|
|
615
|
+
The policy applies whenever one is configured; the values fill it. Omitting
|
|
616
|
+
the values means nobody, never everybody.
|
|
617
|
+
|
|
618
|
+
---
|
|
619
|
+
|
|
620
|
+
|
|
621
|
+
## Grants and behaviors
|
|
622
|
+
|
|
623
|
+
A `ScopePolicy` written by hand is fine for one app. A policy *layer* — a
|
|
624
|
+
charter, a role system, a config file — wants to hand vex a flat set of strings
|
|
625
|
+
and get vex's native contract back. That is the grant dialect.
|
|
626
|
+
|
|
627
|
+
A grant is `<table>.<verb>` where the verb is one of `SCOPE_VERBS`:
|
|
628
|
+
`read`, `write.insert`, `write.update`, `write.delete`. The strings are opaque to
|
|
629
|
+
the layer that resolves them and native to vex, so neither imports the other.
|
|
630
|
+
|
|
631
|
+
```typescript
|
|
632
|
+
scopeGrants(['orders', 'users']) // every grant those tables can carry
|
|
633
|
+
createScopePolicy(grants, behaviors, scoping?) // → ScopePolicy
|
|
634
|
+
```
|
|
635
|
+
|
|
636
|
+
**Grants decide whether a phase exists. Behaviors decide what a granted phase
|
|
637
|
+
does.** Listing a table in `behaviors` grants nothing.
|
|
638
|
+
|
|
639
|
+
```typescript
|
|
640
|
+
type ScopeRules = { read?: ScopeMatch[]; write?: ScopeRule[]; insert?: ScopeRule[]; update?: ScopeRule[]; delete?: ScopeMatch[] };
|
|
641
|
+
|
|
642
|
+
type ScopeBehaviors = Record<string, ScopeRules | Record<string, ScopeRules>>;
|
|
643
|
+
```
|
|
644
|
+
|
|
645
|
+
### Scoping profiles
|
|
646
|
+
|
|
647
|
+
A table carries **either** one rule set — the shape above — **or several named
|
|
648
|
+
ones**, of which `default` is what an unprofiled principal gets.
|
|
649
|
+
|
|
650
|
+
The problem they solve: scoping used to be a property of the *table*, so every
|
|
651
|
+
role holding any grant on it got the same reach. *"The desk reads every booking;
|
|
652
|
+
a member reads their own"* was unsayable, and the workaround was a second table
|
|
653
|
+
carrying the tighter rule — one fact in two places, kept level by a trigger.
|
|
654
|
+
|
|
655
|
+
Reach is a property of the **principal**, so it is chosen once and applies to
|
|
656
|
+
every table they touch:
|
|
657
|
+
|
|
658
|
+
```typescript
|
|
659
|
+
const behaviors: ScopeBehaviors = {
|
|
660
|
+
bookings: {
|
|
661
|
+
default: { read: [{ match: 'studio_id', to: 'studioId' }] },
|
|
662
|
+
personal: { read: [{ match: 'studio_id', to: 'studioId' },
|
|
663
|
+
{ match: 'membership_id', to: 'membershipId' }] },
|
|
664
|
+
},
|
|
665
|
+
// No variant: everybody reads it the same way, and a profiled principal
|
|
666
|
+
// falls back to this without needing an entry per profile.
|
|
667
|
+
class_sessions: { default: { read: [{ match: 'studio_id', to: 'studioId' }] } },
|
|
668
|
+
};
|
|
669
|
+
|
|
670
|
+
createScopePolicy(new Set(['bookings.read']), behaviors) // studio-wide
|
|
671
|
+
createScopePolicy(new Set(['bookings.read']), behaviors, 'personal') // + their own rows
|
|
672
|
+
```
|
|
673
|
+
|
|
674
|
+
The two shapes are told apart by their keys: a rule set has at least one of
|
|
675
|
+
`read`/`write`/`insert`/`update`/`delete`; a named map has none of them. The plain
|
|
676
|
+
shape is unchanged and still means "one reach for everyone".
|
|
677
|
+
|
|
678
|
+
**It fails closed.** An unknown profile denies *everything* — not just the table
|
|
679
|
+
it could not find. A mistyped profile that quietly meant "the default" would
|
|
680
|
+
widen a principal to every row of every table they hold a grant on, which is the
|
|
681
|
+
one failure mode a policy layer may not have. A table declaring named variants
|
|
682
|
+
and no `default` likewise refuses an unprofiled principal rather than guessing
|
|
683
|
+
that "no rule" is safe.
|
|
684
|
+
|
|
685
|
+
`scopeProfiles(behaviors)` lists every profile name a map declares — what a
|
|
686
|
+
policy layer validates its own names against.
|
|
687
|
+
|
|
688
|
+
### A principal with several roles
|
|
689
|
+
|
|
690
|
+
Reach belongs to the role, and a principal may hold more than one — somebody who
|
|
691
|
+
teaches at a studio and trains there is staff on the roster and a member in the
|
|
692
|
+
class. No single profile describes them: studio-wide on the read they do as
|
|
693
|
+
staff, their own rows on the write they do as a member.
|
|
694
|
+
|
|
695
|
+
So compile **one policy per role**, each with that role's own profile, and merge:
|
|
696
|
+
|
|
697
|
+
```typescript
|
|
698
|
+
mergeScopePolicies(
|
|
699
|
+
roles.map((role) => createScopePolicy(grantsFor(role), behaviors, profileFor(role))),
|
|
700
|
+
)
|
|
701
|
+
```
|
|
702
|
+
|
|
703
|
+
A principal may do anything **any** of their roles permits, which is what makes
|
|
704
|
+
holding two additive rather than a conflict to arbitrate.
|
|
705
|
+
|
|
706
|
+
**Merge rule:** per entity and phase, the *broadest* rule set wins — fewest
|
|
707
|
+
`match` rules, since every match narrows. Profiles are refinements of a default
|
|
708
|
+
(`personal` is the tenant rule plus one more), so "fewest" is "widest" in
|
|
709
|
+
practice. A phase only one role grants keeps that role's rules intact, which is
|
|
710
|
+
what carries a member's identity stamp onto a write no other role grants.
|
|
711
|
+
|
|
712
|
+
### A read that must stay narrow
|
|
713
|
+
|
|
714
|
+
The merge above widens. For a read whose answer legitimately widens with the
|
|
715
|
+
reader — a roster, a schedule, a members list — that is right. For one that means
|
|
716
|
+
*mine* it is a leak: an instructor who also trains would open "the classes you
|
|
717
|
+
have booked" and get the studio's.
|
|
718
|
+
|
|
719
|
+
So the ENTRY names the reach it must be served at, and the host recompiles the
|
|
720
|
+
same principal under that profile:
|
|
721
|
+
|
|
722
|
+
```typescript
|
|
723
|
+
{ kind: "ok", dsl: { from: ["bookings"], ... }, reach: "personal" }
|
|
724
|
+
```
|
|
725
|
+
|
|
726
|
+
Grants are unchanged — the union of every role the caller holds — so this can
|
|
727
|
+
only narrow rows, never widen phases. A caller with no verb for the table is
|
|
728
|
+
still refused.
|
|
729
|
+
|
|
730
|
+
The host supplies `policyForReach(reach)` (hono: `getPolicyForReach(c, reach)`),
|
|
731
|
+
because vex holds neither grants nor behaviors. **It fails closed both ways:** an
|
|
732
|
+
entry declaring a reach no host can resolve is a 500, and a host returning
|
|
733
|
+
`undefined` is a 403. Falling back to the caller's own reach would be silent,
|
|
734
|
+
and silence is the whole risk — the read would answer, with too much.
|
|
735
|
+
|
|
736
|
+
## Mutations
|
|
737
|
+
|
|
738
|
+
Writes are first-class and deliberately narrower than reads: **no generation
|
|
739
|
+
path exists.** A mutation is authored by a developer, seeded into the cache
|
|
740
|
+
as a `kind: 'mutation'` entry under a named fingerprint (normally
|
|
741
|
+
`protected`), and replayed over the same wire shape reads use. An inline
|
|
742
|
+
`{ mutation }` body is not a request shape at all.
|
|
743
|
+
|
|
744
|
+
### The grammar
|
|
745
|
+
|
|
746
|
+
Five ops, closed and strict. Values are literals, `{ $context: "key" }`, or a
|
|
747
|
+
`{ $lookup }` scalar subquery (below) — **never `$scope`** (identity is
|
|
748
|
+
engine-injected, unforgeable) and never a field path. `update`/`delete`
|
|
749
|
+
REQUIRE a `where` (a real vex `Filter`).
|
|
750
|
+
|
|
751
|
+
```ts
|
|
752
|
+
{ op: 'insert', table: 'tasks', values: { title: { $context: 'title' } } }
|
|
753
|
+
{ op: 'update', table: 'tasks', set: { done: { $context: 'done' } },
|
|
754
|
+
where: { eq: ['tasks.id', { $context: 'id' }] } }
|
|
755
|
+
{ op: 'delete', table: 'tasks', where: { eq: ['tasks.id', { $context: 'id' }] } }
|
|
756
|
+
// one INSERT per element of a caller-sized array (see insertEach below)
|
|
757
|
+
{ op: 'insertEach', table: 'slots', items: { $context: 'days' },
|
|
758
|
+
values: { weekday: { $item: 'weekday' }, name: { $context: 'name' } } }
|
|
759
|
+
// sugar — insert-or-update keyed on `key`; `insert` columns apply on create only
|
|
760
|
+
{ op: 'upsert', table: 'tasks', key: 'id',
|
|
761
|
+
columns: { title: { $context: 'title' } },
|
|
762
|
+
insert: { deal_id: { $context: 'deal_id' } } }
|
|
763
|
+
```
|
|
764
|
+
|
|
765
|
+
An array of ops is a batch and runs in ONE transaction (the client must
|
|
766
|
+
expose `transaction`; a batch on a non-transactional client is refused).
|
|
767
|
+
|
|
768
|
+
### `onConflict` — the database arbitrates
|
|
769
|
+
|
|
770
|
+
`insert` and `insertEach` take an optional `onConflict`. Where the `upsert`
|
|
771
|
+
sugar branches on what the CALLER sent (key present → update that row),
|
|
772
|
+
`onConflict` branches on what the DATABASE holds — atomically, in one
|
|
773
|
+
statement, the answer to every "look up, then insert if absent" race:
|
|
774
|
+
|
|
775
|
+
```ts
|
|
776
|
+
// create-or-fetch: the no-op "touch" keeps RETURNING * yielding the row on
|
|
777
|
+
// BOTH paths — this is the idiom for "ensure this person exists, give me their id"
|
|
778
|
+
{ op: 'insert', table: 'people',
|
|
779
|
+
values: { email: { $context: 'email' }, name: { $context: 'name' } },
|
|
780
|
+
onConflict: { target: ['email'], set: { email: { $context: 'email' } } } }
|
|
781
|
+
|
|
782
|
+
// insert-if-absent: DO NOTHING returns NO row on conflict — callers read the
|
|
783
|
+
// absence as "already existed" (a batch reply's row count tells the story)
|
|
784
|
+
{ op: 'insert', table: 'memberships',
|
|
785
|
+
values: { person_id: { $context: 'personId' } },
|
|
786
|
+
onConflict: { target: ['studio_id', 'person_id'] } }
|
|
787
|
+
```
|
|
788
|
+
|
|
789
|
+
Two guarantees, both enforced before any SQL runs:
|
|
790
|
+
|
|
791
|
+
- **`target` must name a real unique constraint.** It is validated against
|
|
792
|
+
the introspected schema (primary key + unique indexes, order-insensitive);
|
|
793
|
+
an ON CONFLICT that arrests nothing is an authoring error with the actual
|
|
794
|
+
unique column sets in the message — never a runtime surprise.
|
|
795
|
+
- **DO UPDATE is an update and is governed like one.** Declaring `set`
|
|
796
|
+
requires the update (or `write` umbrella) phase — an insert-only grant
|
|
797
|
+
refuses it. Update-phase `set` rules stamp identity into the DO UPDATE SET;
|
|
798
|
+
`match` rules whose column the conflict target already pins are inherently
|
|
799
|
+
satisfied (the inserted, scope-pinned value must equal the existing row's
|
|
800
|
+
for a conflict to arise), and any other `match` compiles to a WHERE on the
|
|
801
|
+
DO UPDATE half — the RLS boundary, again. `mutationEffect` declares the
|
|
802
|
+
entry as the insert-plus-update it really is, so discovery visibility
|
|
803
|
+
follows the same rule.
|
|
804
|
+
|
|
805
|
+
### `$lookup` — a value read from another table
|
|
806
|
+
|
|
807
|
+
Any value position (insert `values`, update `set`, `onConflict.set`) may be a
|
|
808
|
+
scalar subquery — how a statement references a row it cannot otherwise name:
|
|
809
|
+
|
|
810
|
+
```ts
|
|
811
|
+
{ op: 'insert', table: 'memberships',
|
|
812
|
+
values: {
|
|
813
|
+
person_id: { $lookup: { from: 'people', field: 'id',
|
|
814
|
+
where: { eq: ['people.email', { $context: 'email' }] } } },
|
|
815
|
+
} }
|
|
816
|
+
// → INSERT INTO memberships (person_id, …) VALUES ((SELECT id FROM people WHERE …), …)
|
|
817
|
+
```
|
|
818
|
+
|
|
819
|
+
The lookup READS, so the read-phase scope rules of `from` are ANDed into its
|
|
820
|
+
WHERE by the engine — a mutation entry is never a read-scope bypass, and a
|
|
821
|
+
table the principal cannot read refuses the whole write. The WHERE should hit
|
|
822
|
+
a unique key: more than one matching row is a runtime error (deliberately —
|
|
823
|
+
silently picking an arbitrary row would be worse), and zero rows write NULL
|
|
824
|
+
(the column's constraints decide whether that lands).
|
|
825
|
+
|
|
826
|
+
### `insertEach` — one statement for a caller-sized list
|
|
827
|
+
|
|
828
|
+
`items` names a context key holding an array of objects; `{ $item: "key" }`
|
|
829
|
+
values read from the current element (cast to the column's type from the
|
|
830
|
+
schema); every other value — literal, `$context`, `$lookup`, engine-injected
|
|
831
|
+
`$scope` — is constant across rows. Compiles to a single
|
|
832
|
+
`INSERT … SELECT … FROM jsonb_array_elements($items)`, so "one row per ticked
|
|
833
|
+
weekday" is one authored statement, not a code loop:
|
|
834
|
+
|
|
835
|
+
```ts
|
|
836
|
+
{ op: 'insertEach', table: 'class_templates', items: { $context: 'slots' },
|
|
837
|
+
values: { weekday: { $item: 'weekday' }, starts_at: { $item: 'startsAt' },
|
|
838
|
+
name: { $context: 'name' } },
|
|
839
|
+
onConflict: { target: ['studio_id', 'weekday', 'starts_at'] } } // composes
|
|
840
|
+
```
|
|
841
|
+
|
|
842
|
+
Scope-wise it is an insert that happens N times: the insert phase gates it,
|
|
843
|
+
and `set`/`match` rules pin their columns identically on every row. A
|
|
844
|
+
non-array `items` value is refused loudly before any SQL runs; an empty array
|
|
845
|
+
inserts nothing and returns no rows.
|
|
846
|
+
|
|
847
|
+
### Replay
|
|
848
|
+
|
|
849
|
+
The handler dispatches by the entry's kind — one wire shape:
|
|
850
|
+
|
|
851
|
+
```
|
|
852
|
+
POST { "fingerprint": "tasks/setDone", "context": { "id": "task_1", "done": true } }
|
|
853
|
+
→ 200 { "result": <affected row — or rows for a batch/bulk write> }
|
|
854
|
+
```
|
|
855
|
+
|
|
856
|
+
The pipeline per statement: parse → desugar → **require context** (a write
|
|
857
|
+
never executes with holes — missing keys are a hard 400 `missing_context`
|
|
858
|
+
whose `details.expected` carries the FULL derived signature) → scope (the
|
|
859
|
+
same `ScopePolicy` reads use: `write` `set` rules write identity on insert
|
|
860
|
+
and update, `match` rules pin rows) → column check against the introspected schema →
|
|
861
|
+
compile (parameterized, via the read pipeline's own operators) → execute
|
|
862
|
+
with `RETURNING *`.
|
|
863
|
+
|
|
864
|
+
Enable replay on a handler by configuring the client and policy:
|
|
865
|
+
|
|
866
|
+
```ts
|
|
867
|
+
handleQuery({ engine, locked: true, mutations: { client: db, policy } }, body, scope);
|
|
868
|
+
```
|
|
869
|
+
|
|
870
|
+
`client` is structural (`MutationClient` — PGlite, a pg wrapper, a test
|
|
871
|
+
double). `locked` does not affect writes: they are replay-only under every
|
|
872
|
+
posture. Direct library use: `executeMutation(client, def, { context, scope, policy, schema })`.
|
|
873
|
+
|
|
874
|
+
A host that resolves a policy **per principal** (e.g. compiled from an ACL
|
|
875
|
+
layer) passes it per request: `VexHandlerConfig.scopePolicy` overrides the
|
|
876
|
+
engine's configured read policy for that request (`ExecuteOptions.scopePolicy`
|
|
877
|
+
at the library level), and the same policy goes in `mutations.policy` — one
|
|
878
|
+
policy, both sides of the wire. Omitted, the engine default applies.
|
|
879
|
+
|
|
880
|
+
### Derived context signatures
|
|
881
|
+
|
|
882
|
+
The input contract of ANY entry — read or write — is computed from its
|
|
883
|
+
stored def: each `{ $context }` ref sits at a position whose column the
|
|
884
|
+
schema types. Discovery lists it per fingerprint (`context`, plus `shape`
|
|
885
|
+
for reads / `effect` = `{ op, table, columns }` for writes), and the
|
|
886
|
+
`missing_context` error teaches it. Nothing is authored; nothing drifts.
|
|
887
|
+
|
|
888
|
+
```ts
|
|
889
|
+
collectMutationContext(def, schema); // { id: { type: 'string', column: 'tasks.id', note: 'upsert key — …' }, … }
|
|
890
|
+
collectQueryContext(dsl, schema); // same idea over a query's filter/compute/subqueries
|
|
891
|
+
mutationEffect(def); // [{ op: 'update', table: 'tasks', columns: ['done'] }]
|
|
892
|
+
```
|
|
893
|
+
|
|
894
|
+
### Authoring lint
|
|
895
|
+
|
|
896
|
+
`lintMutation(def)` flags an `update`/`delete` whose WHERE binds no
|
|
897
|
+
`$context` — such a write is not caller-bounded (its only row limit is the
|
|
898
|
+
scope policy). Run it in your seed path so an unkeyed write never ships:
|
|
899
|
+
loud at boot, never at runtime.
|
|
900
|
+
|
|
901
|
+
### The write observer
|
|
902
|
+
|
|
903
|
+
Vex is the choke point every application write passes through, which makes
|
|
904
|
+
it the one place a host can hear about all of them. `mutations.onWrite`
|
|
905
|
+
(handler config; the hono adapter passes it through verbatim) fires once per
|
|
906
|
+
successful mutation replay, AFTER the commit:
|
|
907
|
+
|
|
908
|
+
```ts
|
|
909
|
+
mutations: {
|
|
910
|
+
client, policy,
|
|
911
|
+
onWrite: ({ fingerprint, writes, scope }) => { … },
|
|
912
|
+
// writes: [{ table, op, rows }] — per statement, resolved post-desugar
|
|
913
|
+
// (an upsert reports the branch it took; insertEach reports 'insert'),
|
|
914
|
+
// rows as the database returned them. scope: what the request could
|
|
915
|
+
// not forge.
|
|
916
|
+
}
|
|
917
|
+
```
|
|
918
|
+
|
|
919
|
+
The grain is the point: a two-statement mutation is two writes, each with
|
|
920
|
+
its own `RETURNING` rows, and an UPDATE the scope narrowed to nothing
|
|
921
|
+
carries zero rows — the observer is never told "a write landed" about a
|
|
922
|
+
write that changed nothing. Reads never fire it; a refused or failed
|
|
923
|
+
mutation never reaches it; a listener that throws is logged and contained,
|
|
924
|
+
because the response must tell the truth about a commit that already
|
|
925
|
+
happened.
|
|
926
|
+
|
|
927
|
+
What the host does with the news is not the engine's business — moss mints
|
|
928
|
+
tide write facts from it and fans row-less pings to the app. If you hand
|
|
929
|
+
`rows` onward, know who is on the other end: a row given to code is a row
|
|
930
|
+
outside every scope policy this engine compiles.
|
|
931
|
+
|
|
932
|
+
`executeWrites(client, def, mctx)` is the same per-statement shape at the
|
|
933
|
+
library level; `executeMutation` remains the flattened `Row[]` every
|
|
934
|
+
existing caller expects.
|
|
935
|
+
|
|
936
|
+
---
|
|
937
|
+
|
|
938
|
+
## Caching
|
|
939
|
+
|
|
940
|
+
### Fingerprints — the one identity
|
|
941
|
+
|
|
942
|
+
The positive cache is keyed by the request's `fingerprint` (see the request
|
|
943
|
+
contract above). There are no cache modes — the body's posture says
|
|
944
|
+
everything:
|
|
945
|
+
|
|
946
|
+
- **No fingerprint** → generate, cache, mint (`fp_…`). The mint is an
|
|
947
|
+
immutable pin: embed it and replay that exact generation forever.
|
|
948
|
+
- **Fingerprint alone** → replay-or-`cache_miss`. Never generates. Context
|
|
949
|
+
values may vary per call — sorting and parameterized reloads are legal
|
|
950
|
+
against one cached DSL.
|
|
951
|
+
- **Fingerprint + request** → a named slot: hit when the stored request
|
|
952
|
+
matches, regenerate-and-replace when it differs; `protected` entries throw
|
|
953
|
+
`fingerprint_protected` instead of being replaced.
|
|
954
|
+
|
|
955
|
+
Concurrent identical generations collapse to one (single-flight, keyed by the
|
|
956
|
+
request hash).
|
|
957
|
+
|
|
958
|
+
### Lifetime and protection
|
|
959
|
+
|
|
960
|
+
- Every replay stamps `lastUsedAt`; `sweepCache(cache, { maxIdleMs })` evicts
|
|
961
|
+
entries idle past the limit, skipping protected ones. Alive = used.
|
|
962
|
+
- `protected: true` is written only by a seed path or an explicit
|
|
963
|
+
`handleFingerprintPatch` — never by a runtime query.
|
|
964
|
+
- `ExecuteOptions.locked` (and `VexHandlerConfig.locked`) makes an engine call
|
|
965
|
+
or endpoint replay-only: asking it to generate throws `locked`.
|
|
966
|
+
|
|
967
|
+
### Supporting hashes
|
|
968
|
+
|
|
969
|
+
- **Request hash** — intent + normalized shape + the *names* of context keys
|
|
970
|
+
(values excluded). Stored with named slots (the match/replace test) and the
|
|
971
|
+
key of the **negative cache**: a `cannotSatisfy` result is cached with a TTL
|
|
972
|
+
(`config.unsatisfiableTtlMs`, default 5 min) so an impossible request isn't
|
|
973
|
+
re-run until it expires.
|
|
974
|
+
- Every entry stores a **schema fingerprint**. After a schema change it no
|
|
975
|
+
longer matches and the entry is treated as a miss and evicted — its cached
|
|
976
|
+
DSL might reference columns that changed.
|
|
977
|
+
|
|
978
|
+
### Backends
|
|
979
|
+
|
|
980
|
+
```typescript
|
|
981
|
+
import {
|
|
982
|
+
createMemoryCache,
|
|
983
|
+
createPostgresCache,
|
|
984
|
+
createTieredCache,
|
|
985
|
+
} from '@niscorp/vex';
|
|
986
|
+
```
|
|
987
|
+
|
|
988
|
+
**In-memory** (default) — a `Map`. Fast, volatile, single-process.
|
|
989
|
+
|
|
990
|
+
```typescript
|
|
991
|
+
const cache = createMemoryCache();
|
|
992
|
+
```
|
|
993
|
+
|
|
994
|
+
**Postgres** — durable L2. Stores entries as `jsonb`, enforces TTL server-side,
|
|
995
|
+
validates on write, and evicts corrupt/stale rows on read. Used directly it
|
|
996
|
+
reads from Postgres on every `get`, so it is meant to sit behind an L1.
|
|
997
|
+
|
|
998
|
+
```typescript
|
|
999
|
+
const l2 = createPostgresCache({
|
|
1000
|
+
pool,
|
|
1001
|
+
schema: 'public', // default
|
|
1002
|
+
table: 'vex_cache', // default
|
|
1003
|
+
onError: (err) => console.error(err),
|
|
1004
|
+
});
|
|
1005
|
+
await l2.init(); // applies l2.sequence through strata's ledger — once, recorded
|
|
1006
|
+
```
|
|
1007
|
+
|
|
1008
|
+
**Tiered** — L1 (memory) + L2 (durable), read-through/write-through. Reads hit
|
|
1009
|
+
L1; on an L1 miss it promotes a validated L2 entry into L1. Writes go to L1
|
|
1010
|
+
synchronously and L2 fire-and-forget.
|
|
1011
|
+
|
|
1012
|
+
```typescript
|
|
1013
|
+
const cache = createTieredCache({
|
|
1014
|
+
l1: createMemoryCache(),
|
|
1015
|
+
l2: createPostgresCache({ pool }),
|
|
1016
|
+
warmup: 'full', // 'full' (default) | 'lazy' | { mode: 'partial', fingerprints: [...] }
|
|
1017
|
+
onError: (err) => console.error(err),
|
|
1018
|
+
});
|
|
1019
|
+
await cache.init(); // runs L2/L1 init + warm-up
|
|
1020
|
+
|
|
1021
|
+
const engine = createQueryEngine({ adapter, cache });
|
|
1022
|
+
```
|
|
1023
|
+
|
|
1024
|
+
Warm-up modes: `full` loads all of L2 into L1 at startup; `lazy` fills L1 on
|
|
1025
|
+
demand; `partial` preloads only the listed fingerprints and lazy-fills the
|
|
1026
|
+
rest. The tiered cache is single-instance: separate processes keep separate
|
|
1027
|
+
L1s.
|
|
1028
|
+
|
|
1029
|
+
---
|
|
1030
|
+
|
|
1031
|
+
## Wiring the LLM agents
|
|
1032
|
+
|
|
1033
|
+
The engine consumes two hooks. Both are optional; supply them to answer
|
|
1034
|
+
free-form requests.
|
|
1035
|
+
|
|
1036
|
+
```typescript
|
|
1037
|
+
type GenerateDsl = (request: QueryRequest, schema: DatabaseSchema) => Promise<Query>;
|
|
1038
|
+
type MapToShape = (rows: Row[], shape: unknown) =>
|
|
1039
|
+
Promise<{ ir: CompiledIr; transformed: unknown[] }>;
|
|
1040
|
+
```
|
|
1041
|
+
|
|
1042
|
+
- `generateDsl` runs on a cache miss; it must return a DSL that passes
|
|
1043
|
+
`QuerySchema`. To signal an impossible request, throw
|
|
1044
|
+
`new VexError('unsatisfiable', reason)` — the runtime negative-caches it.
|
|
1045
|
+
- `mapToShape` runs when the requested shape differs from the raw rows; it
|
|
1046
|
+
returns a compiled Prism `ir` (cached for reuse) plus the transformed rows.
|
|
1047
|
+
|
|
1048
|
+
You can implement these however you like. Vex also ships a **reference
|
|
1049
|
+
implementation** built on `@niscorp/cortex` and `@niscorp/prism`, exported from
|
|
1050
|
+
the `@niscorp/vex/agent` subpath (`@niscorp/cortex`, `@niscorp/signal`, and
|
|
1051
|
+
`@niscorp/prism` become required peers only if you import it):
|
|
1052
|
+
|
|
1053
|
+
```typescript
|
|
1054
|
+
import { createQueryDsl, createShapeMapper } from '@niscorp/vex/agent';
|
|
1055
|
+
import { createSignal } from '@niscorp/signal';
|
|
1056
|
+
|
|
1057
|
+
const schema = await engine.introspect();
|
|
1058
|
+
const queryJsonSchema = engine.getDslSchema();
|
|
1059
|
+
|
|
1060
|
+
const llm = createSignal('openrouter', { apiKey, model: 'qwen/qwen3.8-27b' });
|
|
1061
|
+
|
|
1062
|
+
const generateDsl = createQueryDsl({ llm, queryJsonSchema });
|
|
1063
|
+
const mapToShape = createShapeMapper(llm);
|
|
1064
|
+
|
|
1065
|
+
const engine = createQueryEngine({ adapter, generateDsl, mapToShape, cache });
|
|
1066
|
+
await engine.introspect();
|
|
1067
|
+
```
|
|
1068
|
+
|
|
1069
|
+
- `createQueryDsl(config)` builds the `generateDsl` hook by running
|
|
1070
|
+
`vexQueryDslAgent`. It is built from a model and the DSL spec only — who it
|
|
1071
|
+
runs for arrives with every call. The engine calls the hook as
|
|
1072
|
+
`generateDsl(request, schema, caller)`: `schema` holds only the tables the
|
|
1073
|
+
caller's policy can read, and `caller.read(dsl)` runs a DSL through the
|
|
1074
|
+
engine's own pipeline under the caller's policy and scope values (`$context`
|
|
1075
|
+
binds NULL). That capability is the tools' only way to the database.
|
|
1076
|
+
- `createShapeMapper(llm)` builds the `mapToShape` hook by running Prism's
|
|
1077
|
+
`mappingAgent`. The `@niscorp/vex/agent` subpath also exports `vexQueryDslAgent`
|
|
1078
|
+
itself plus `createQueryTools` for customizing the tool set.
|
|
1079
|
+
|
|
1080
|
+
The reference query agent (`vexQueryDslAgent`, id `vex.query`) is a Cortex
|
|
1081
|
+
agent whose envelope payload is a `QuerySchema` query, with six tools:
|
|
1082
|
+
|
|
1083
|
+
| Tool | Input | Purpose |
|
|
1084
|
+
|------|-------|---------|
|
|
1085
|
+
| `getSchema` | `{ entities? }` | Read the schema (optionally a subset) |
|
|
1086
|
+
| `getSampleRows` | `{ entity, limit? }` | See real rows (default 5) |
|
|
1087
|
+
| `getDistinctValues` | `{ entity, field, limit? }` | Field cardinality (default 20) |
|
|
1088
|
+
| `describeField` | `{ entity, field }` | type, nullable, cardinality, null count, min/max |
|
|
1089
|
+
| `testQuery` | `{ dsl }` | Validate + compile + execute with synthetic params (LIMIT 5) |
|
|
1090
|
+
| `cannotSatisfy` | `{ reason }` | Declare the request impossible (`createQueryDsl` aborts the run and throws `unsatisfiable`) |
|
|
1091
|
+
|
|
1092
|
+
Bounds: `stepCount(20)` + `outputRetries(3)` stop conditions on the agent. Every
|
|
1093
|
+
data-touching tool is a DSL run through `caller.read` — there is no SQL in the
|
|
1094
|
+
tool set — so a sample row, a distinct value and a statistic are all things the
|
|
1095
|
+
caller could have read. (If you build your own generator,
|
|
1096
|
+
none of this applies — only the `Query` you return matters.)
|
|
1097
|
+
|
|
1098
|
+
`testQuery` executes drafts with **NULL synthetic params** — valid SQL against
|
|
1099
|
+
a column of any type (Postgres infers from the column side), so a draft is
|
|
1100
|
+
judged on executability, never on a guessed value's cast.
|
|
1101
|
+
|
|
1102
|
+
### `vexGuide()` — the exported agent-facing contract
|
|
1103
|
+
|
|
1104
|
+
Any agent that reads or embeds Vex endpoints should inject `vexGuide()` (a
|
|
1105
|
+
string): the three request postures, fingerprint semantics, per-call context
|
|
1106
|
+
values, reserved `sortBy`/`sortDir`, the response envelope, and discovery.
|
|
1107
|
+
Attach it as a cortex tool `guide` or context producer. Apps never hand-write
|
|
1108
|
+
their own Vex explanation — that is how docs drift.
|
|
1109
|
+
|
|
1110
|
+
---
|
|
1111
|
+
|
|
1112
|
+
## Framework adapters
|
|
1113
|
+
|
|
1114
|
+
Vex does not own a server, but ships thin adapters that expose a resource with a
|
|
1115
|
+
discovery endpoint (`GET`) and a query endpoint (`POST`). Both are optional
|
|
1116
|
+
sub-exports.
|
|
1117
|
+
|
|
1118
|
+
### Hono
|
|
1119
|
+
|
|
1120
|
+
```typescript
|
|
1121
|
+
import { Hono } from 'hono';
|
|
1122
|
+
import { vex } from '@niscorp/vex/hono';
|
|
1123
|
+
|
|
1124
|
+
const app = new Hono();
|
|
1125
|
+
|
|
1126
|
+
app.route('/api/orders/vex', vex({
|
|
1127
|
+
engine,
|
|
1128
|
+
entities: ['orders', 'order_items'], // optional: restrict this resource
|
|
1129
|
+
getScope: async (c) => ({ accountId: await resolveAccount(c) }),
|
|
1130
|
+
}));
|
|
1131
|
+
```
|
|
1132
|
+
|
|
1133
|
+
### Express
|
|
1134
|
+
|
|
1135
|
+
```typescript
|
|
1136
|
+
import express from 'express';
|
|
1137
|
+
import { vex } from '@niscorp/vex/express';
|
|
1138
|
+
|
|
1139
|
+
const app = express();
|
|
1140
|
+
app.use(express.json());
|
|
1141
|
+
|
|
1142
|
+
app.all('/api/orders/vex', vex({
|
|
1143
|
+
engine,
|
|
1144
|
+
entities: ['orders', 'order_items'],
|
|
1145
|
+
getScope: (req) => ({ accountId: req.user.accountId }),
|
|
1146
|
+
}));
|
|
1147
|
+
```
|
|
1148
|
+
|
|
1149
|
+
### The endpoints
|
|
1150
|
+
|
|
1151
|
+
- **`GET`** → discovery JSON: `vex` protocol version, description, the filtered
|
|
1152
|
+
entities (fields, relations, row counts), the request body contract (the
|
|
1153
|
+
three fingerprint postures), the protection summary (`protection:
|
|
1154
|
+
"all"|"some"|"none"`, `locked`), the fingerprint overview (`[{ fingerprint,
|
|
1155
|
+
kind, protected, schemaFresh, intent, context, shape|effect, lastUsedAt }]`
|
|
1156
|
+
— `context` is the derived, typed input signature; `effect` summarizes a
|
|
1157
|
+
write), and the full DSL JSON Schema. A client or agent can read this to
|
|
1158
|
+
learn how to call the resource — and exactly what each fingerprint takes.
|
|
1159
|
+
- **`POST`** → runs a query OR replays a write, one body shape. A
|
|
1160
|
+
`fingerprint` naming a `kind: 'mutation'` entry dispatches to the write
|
|
1161
|
+
pipeline (when `mutations` is configured); anything else is the read path
|
|
1162
|
+
(fingerprint replay, generation, or named slot); scope comes from
|
|
1163
|
+
`getScope`. Returns `{ status, body }` mapped to the HTTP response —
|
|
1164
|
+
unknown fingerprint → 404 `cache_miss`, protected mismatch → 409
|
|
1165
|
+
`fingerprint_protected`, locked → 403, a write with missing context → 400
|
|
1166
|
+
`missing_context` with the full derived signature in `details.expected`,
|
|
1167
|
+
other `VexError`s → 400 with `{ error, message, details }`, unexpected
|
|
1168
|
+
errors → 500.
|
|
1169
|
+
- **`PATCH`** with `{ fingerprint, protected }` → flip the protection bit.
|
|
1170
|
+
**`DELETE`** with `{ fingerprint }` → evict (unprotected only). Fingerprints
|
|
1171
|
+
ride the body — names contain `/`.
|
|
1172
|
+
|
|
1173
|
+
Under the hood all of these call the framework-agnostic `handleDiscovery` /
|
|
1174
|
+
`handleQuery` / `handleFingerprintPatch` / `handleFingerprintDelete` from
|
|
1175
|
+
`handler.ts`, which you can use directly with any framework. A handler config
|
|
1176
|
+
with `locked: true` makes the whole endpoint replay-only.
|
|
1177
|
+
|
|
1178
|
+
---
|
|
1179
|
+
|
|
1180
|
+
## Events
|
|
1181
|
+
|
|
1182
|
+
Pass `onEvent` to observe the pipeline. Every event is a discriminated union
|
|
1183
|
+
member on `type`:
|
|
1184
|
+
|
|
1185
|
+
```typescript
|
|
1186
|
+
import type { VexEvent } from '@niscorp/vex';
|
|
1187
|
+
|
|
1188
|
+
const onEvent = (e: VexEvent) => {
|
|
1189
|
+
switch (e.type) {
|
|
1190
|
+
case 'query.start': /* intent, shape, fingerprint, entities */ break;
|
|
1191
|
+
case 'query.cache': /* hit, fingerprint, replaced */ break;
|
|
1192
|
+
case 'query.dsl': /* dsl, agentMs */ break;
|
|
1193
|
+
case 'query.sql': /* sql, warnings */ break;
|
|
1194
|
+
case 'query.rows': /* count, executionMs */ break;
|
|
1195
|
+
case 'query.mapped': /* mappingMs */ break;
|
|
1196
|
+
case 'query.done': /* totalMs */ break;
|
|
1197
|
+
case 'query.error': /* code, message */ break;
|
|
1198
|
+
// llm.request / llm.response — only if you emit them from your LLM wrapper
|
|
1199
|
+
}
|
|
1200
|
+
};
|
|
1201
|
+
|
|
1202
|
+
const engine = createQueryEngine({ adapter, onEvent });
|
|
1203
|
+
```
|
|
1204
|
+
|
|
1205
|
+
The `llm.request`/`llm.response` variants exist for callers that wrap their LLM
|
|
1206
|
+
client to record a transcript (the dev server does this to write a full debug
|
|
1207
|
+
log); the engine itself emits the `query.*` events.
|
|
1208
|
+
|
|
1209
|
+
---
|
|
1210
|
+
|
|
1211
|
+
## Error handling
|
|
1212
|
+
|
|
1213
|
+
`execute` throws `VexError` on failure (the framework handlers convert these to
|
|
1214
|
+
HTTP responses). Missing context is **not** an error — it returns an empty
|
|
1215
|
+
response with `meta.missingContext`.
|
|
1216
|
+
|
|
1217
|
+
```typescript
|
|
1218
|
+
import { VexError } from '@niscorp/vex';
|
|
1219
|
+
|
|
1220
|
+
try {
|
|
1221
|
+
const res = await engine.execute(request, { scope });
|
|
1222
|
+
} catch (err) {
|
|
1223
|
+
if (err instanceof VexError) {
|
|
1224
|
+
// err.code, err.message, err.details
|
|
1225
|
+
}
|
|
1226
|
+
}
|
|
1227
|
+
```
|
|
1228
|
+
|
|
1229
|
+
`VexError.code` is one of:
|
|
1230
|
+
|
|
1231
|
+
| Code | Meaning |
|
|
1232
|
+
|------|---------|
|
|
1233
|
+
| `invalid_request` | The request failed Zod validation |
|
|
1234
|
+
| `invalid_dsl` | The analyzer rejected the resolved query (cartesian product, nesting, etc.) |
|
|
1235
|
+
| `missing_scope` | A `$scope` slot the host did not fill. The host's fault, not the request's: the HTTP handler answers 500 and keeps the key names in its log |
|
|
1236
|
+
| `scope_denied` | An entity is denied by the scope policy (`VexScopeError`) |
|
|
1237
|
+
| `missing_context` | A required context value was absent. Reads surface it softly via `meta.missingContext`; a WRITE hard-400s with the full derived signature in `details.expected` |
|
|
1238
|
+
| `execution_error` | Schema not loaded, or a database/runtime failure |
|
|
1239
|
+
| `agent_failed` | No `generateDsl` hook and no cached DSL for the shape |
|
|
1240
|
+
| `cache_miss` | Cache mode `only` and nothing cached |
|
|
1241
|
+
| `unsatisfiable` | The agent declared the request impossible (negative-cached) |
|
|
1242
|
+
|
|
1243
|
+
---
|
|
1244
|
+
|
|
1245
|
+
## Local development
|
|
1246
|
+
|
|
1247
|
+
The package ships a Postgres-backed dev environment and a runnable HTTP server.
|
|
1248
|
+
|
|
1249
|
+
### Database
|
|
1250
|
+
|
|
1251
|
+
```bash
|
|
1252
|
+
pnpm --filter @niscorp/vex docker:up # pgvector/pgvector:pg16 on host port 5433
|
|
1253
|
+
pnpm --filter @niscorp/vex seed # idempotent schema + sample data
|
|
1254
|
+
pnpm --filter @niscorp/vex docker:down
|
|
1255
|
+
```
|
|
1256
|
+
|
|
1257
|
+
The dev database is `vex_dev` (user/pass `vex`/`vex`), exposed on **5433** to
|
|
1258
|
+
avoid clashing with a local 5432. `scripts/init.sql` runs on first boot; `seed`
|
|
1259
|
+
loads a realistic schema (customers, orders, products, categories, reviews, …)
|
|
1260
|
+
including a vector column for semantic search.
|
|
1261
|
+
|
|
1262
|
+
### Environment
|
|
1263
|
+
|
|
1264
|
+
`.env` (see `.env.example`):
|
|
1265
|
+
|
|
1266
|
+
```bash
|
|
1267
|
+
OPENROUTER_API_KEY= # enables the query + mapping agents
|
|
1268
|
+
OPENAI_API_KEY= # enables embeddings (text-embedding-3-small) for semantic search
|
|
1269
|
+
```
|
|
1270
|
+
|
|
1271
|
+
The dev server also reads `DATABASE_URL` (default
|
|
1272
|
+
`postgresql://vex:vex@localhost:5433/vex_dev`), `PORT` (default 3456), and
|
|
1273
|
+
`VEX_CACHE` (`memory` | `postgres`) with `VEX_CACHE_SCHEMA` / `VEX_CACHE_TABLE`.
|
|
1274
|
+
|
|
1275
|
+
### Run the server
|
|
1276
|
+
|
|
1277
|
+
```bash
|
|
1278
|
+
pnpm --filter @niscorp/vex dev:server
|
|
1279
|
+
```
|
|
1280
|
+
|
|
1281
|
+
It seeds if needed, introspects, wires the reference agents when
|
|
1282
|
+
`OPENROUTER_API_KEY` is set (falling back to deterministic-only otherwise),
|
|
1283
|
+
mounts several scoped resources via the Hono adapter, and logs per-stage timing
|
|
1284
|
+
to the console and a full LLM transcript to `logs/vex-debug.log`. `GET` any
|
|
1285
|
+
endpoint for discovery; `POST` to query.
|
|
1286
|
+
|
|
1287
|
+
### Tests
|
|
1288
|
+
|
|
1289
|
+
```bash
|
|
1290
|
+
pnpm --filter @niscorp/vex test
|
|
1291
|
+
```
|
|
1292
|
+
|
|
1293
|
+
Unit tests (schemas, discovery, scope, hashing, resolver, analyzer) need no
|
|
1294
|
+
database or LLM. Integration tests (introspection, compilation, execution, cache
|
|
1295
|
+
backends, handlers) run against the seeded Postgres.
|