@niscorp/vex 0.2.2 → 0.2.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (36) hide show
  1. package/CHANGELOG.md +121 -0
  2. package/DESIGN.md +66 -3
  3. package/DOCS.md +156 -12
  4. package/dist/adapters/express/index.cjs +102 -17
  5. package/dist/adapters/express/index.cjs.map +1 -1
  6. package/dist/adapters/express/index.d.cts +2 -2
  7. package/dist/adapters/express/index.d.ts +2 -2
  8. package/dist/adapters/express/index.js +2 -2
  9. package/dist/adapters/hono/index.cjs +121 -22
  10. package/dist/adapters/hono/index.cjs.map +1 -1
  11. package/dist/adapters/hono/index.d.cts +2 -2
  12. package/dist/adapters/hono/index.d.ts +2 -2
  13. package/dist/adapters/hono/index.js +21 -7
  14. package/dist/adapters/hono/index.js.map +1 -1
  15. package/dist/agent/index.cjs +2 -0
  16. package/dist/agent/index.cjs.map +1 -1
  17. package/dist/agent/index.d.cts +1 -1
  18. package/dist/agent/index.d.ts +1 -1
  19. package/dist/agent/index.js +1 -1
  20. package/dist/{chunk-HF25Q53I.js → chunk-H4BHW76Q.js} +6 -3
  21. package/dist/chunk-H4BHW76Q.js.map +1 -0
  22. package/dist/{chunk-HCEH55PP.js → chunk-IQM4SQE2.js} +123 -21
  23. package/dist/chunk-IQM4SQE2.js.map +1 -0
  24. package/dist/{handler-Cql8SJ3F.d.cts → handler-BGCL6_Si.d.ts} +3 -2
  25. package/dist/{handler-mICf9ins.d.ts → handler-CMrjTXNx.d.cts} +3 -2
  26. package/dist/index.cjs +163 -35
  27. package/dist/index.cjs.map +1 -1
  28. package/dist/index.d.cts +3 -3
  29. package/dist/index.d.ts +3 -3
  30. package/dist/index.js +44 -22
  31. package/dist/index.js.map +1 -1
  32. package/dist/{types-BRvMB6IM.d.cts → types-Bi-Lg86b.d.cts} +48 -2
  33. package/dist/{types-BRvMB6IM.d.ts → types-Bi-Lg86b.d.ts} +48 -2
  34. package/package.json +5 -5
  35. package/dist/chunk-HCEH55PP.js.map +0 -1
  36. package/dist/chunk-HF25Q53I.js.map +0 -1
package/CHANGELOG.md CHANGED
@@ -1,5 +1,126 @@
1
1
  # @niscorp/vex
2
2
 
3
+ ## 0.2.3
4
+
5
+ ### Patch Changes
6
+
7
+ - 1e5ad2d: An insert requires every key its `values` bind, whatever `onConflict.set` puts in the same column.
8
+
9
+ An `insert` or `insertEach` may set the same column in `values` and in `onConflict.set`. Vex read the two by column name, so where they put different things in a column it saw only what `onConflict.set` put there. A `$context` key that `values` used for that column, and that nothing else in the statement used, was not in the entry's derived signature and was not required. A call that left it out was not refused. The statement ran with that parameter unbound, and a new row was written with NULL in the column — or, where the column is NOT NULL, the database's own error came back in place of vex's. DOCS.md says a write never executes with holes; now this one does not.
10
+
11
+ ```ts
12
+ { op: 'insert', table: 'people',
13
+ values: { email: { $context: 'email' }, name: { $context: 'name' } },
14
+ onConflict: { target: ['email'], set: { name: { $context: 'newName' } } } }
15
+ ```
16
+
17
+ - A call without `name` is refused before any SQL runs: 400 `missing_context`, `Mutation is missing context: name.` (was 200, and a row with `name` NULL).
18
+ - Discovery, `collectMutationContext` and the refusal's `details.expected` list both `name` and `newName` (listed `newName` only). `requiredContextKeys` returns both.
19
+ - The same where `onConflict.set` puts a literal or a `$lookup` in the column rather than another key, and for the keys a `$lookup` in `values` binds.
20
+
21
+ An entry that puts different things in one column in the two places has its signature read `values` first, then `onConflict.set`. Its keys can come in another order than before, and a key used in both is listed with the column and type of its place in `values`.
22
+
23
+ Nothing else moves. An entry whose `values` and `onConflict.set` share no column, or put the same thing in every column they share (the create-or-fetch touch, `set: { email: { $context: 'email' } }`), lists and requires what it did, in the same order. `update`, `delete` and `mutationEffect` are unchanged.
24
+
25
+ **What to change:** nothing, unless an entry sets a column in `values` from a `$context` key, sets the same column in `onConflict.set` from anything else, and some caller leaves that key out. That call wrote a NULL and is now answered 400. Send the key — `null` to write NULL on purpose; only an absent key is refused — or, if the column was never meant to be set on insert, take it out of `values`.
26
+
27
+ - 2011951: A mutation batch can use what an earlier statement wrote: `{ $returned: 'table.column' }`.
28
+
29
+ A batch ran in one transaction but no statement could see another's result. A row whose id the database generates could not be referenced by the rows written after it in the same batch.
30
+
31
+ A later statement of a batch can now read a column of the one row an earlier statement wrote:
32
+
33
+ ```ts
34
+ [
35
+ { op: 'insert', table: 'orders', values: { note: { $context: 'note' } } },
36
+ {
37
+ op: 'insertEach',
38
+ table: 'order_lines',
39
+ items: { $context: 'lines' },
40
+ values: { order_id: { $returned: 'orders.id' }, sku: { $item: 'sku' } },
41
+ },
42
+ ];
43
+ ```
44
+
45
+ The reference names the earlier statement — an insert, update, upsert or delete — by the table it writes. It goes in any value position (`values`, `set`, `onConflict.set`, an `upsert`'s `columns` and `insert`), and not in a `where`, a `$lookup`, or an `insertEach`'s `items`. It is for keys: the value passes through the driver, and a timestamp read back that way keeps milliseconds only.
46
+
47
+ The statement it names must write exactly one row. When it wrote none (its WHERE matched nothing, or the scope boundary kept the row back) or several, the batch fails with `execution_error` and nothing is written. `lintMutation` — so `seedCache` — refuses a reference that no earlier statement can answer, that two earlier statements could answer, that names an `insertEach`, or that names an `insert` whose `onConflict` has no `set`. Scope is unchanged: each statement is scoped on its own, as before.
48
+
49
+ Every existing entry, request and reply behaves as it did. `MutationValue` gains one union member, `ReturnedRef`.
50
+
51
+ **What to change:** nothing. To use it, upgrade every process that reads the cache first: a vex older than this one deletes a stored entry whose grammar it does not know, as it does for any form added since it was built.
52
+
53
+ - 2c57a38: Under `require`, an engine error on a read keeps its HTTP status.
54
+
55
+ The CommonJS build gives each entry point its own copy of `VexError`, and the HTTP handler recognised an engine's error by `instanceof`. With the engine from `require('@niscorp/vex')` and the endpoint from `require('@niscorp/vex/hono')` or `require('@niscorp/vex/express')`, every error the engine threw on a read was therefore answered as an unexpected one: 500, without its `details`, and with a `[vex] unhandled error` line in the log. The ESM build has one copy of the class and answered as DOCS.md says. Now both do:
56
+ - an unknown fingerprint is 404 `cache_miss` (was 500);
57
+ - a protected entry asked to change is 409 `fingerprint_protected` (was 500);
58
+ - `locked`, `agent_failed`, `unsatisfiable`, `scope_denied`, `invalid_dsl` and an `execution_error` thrown by the engine are 400, with `details` where the error has them (were 500, without);
59
+ - `missing_scope` is still 500, and its reply no longer names the scope keys the host left out — they go to the log, as they always did under ESM.
60
+
61
+ The same was true of an app that loads vex in both formats at once, and of moss loaded with `require` (its `/api/…/vex` endpoints). Writes were never affected: the handler runs them itself.
62
+
63
+ The engine has the same check for a generation hook's `unsatisfiable`. With the hook from `require('@niscorp/vex/agent')` it never matched, so nothing was negative-cached and the model was asked again on every repeat of a request it had already called impossible. The refusal is cached now, for `unsatisfiableTtlMs` (five minutes unless set), as it is under ESM.
64
+
65
+ Vex now recognises its own errors by a mark every copy of the class carries. Nothing is added to the exports, and nothing an error prints or serializes changes. An app's own `err instanceof VexError` is as it was: it holds for an error the engine throws, and under `require` it still does not hold for one made in `@niscorp/vex/agent` — read `err.code` there (DOCS.md, "Error handling").
66
+
67
+ **What to change:** nothing in an ESM app, and nothing in a CommonJS app that calls `handleQuery` from `@niscorp/vex` itself. A CommonJS app that had come to depend on those 500s now sees the documented status: a client that reads 500 as "unknown fingerprint", a retry or an alert on 5xx, a log search for `[vex] unhandled error`.
68
+
69
+ - 2d20daf: The Hono adapter answers a body that is not JSON with 400 `invalid_request`, not 500.
70
+
71
+ `POST`, `PATCH` and `DELETE` parsed the body unguarded, so a parse failure threw out of the route: text, a multipart or urlencoded form, truncated JSON, or no body at all was answered `500 Internal Server Error` in plain text, with the `SyntaxError` on the server's log — a server fault for what is the request's mistake, and the one failure on this surface without the `{ error, message }` shape. It is now `400 { "error": "invalid_request", "message": "Body must be JSON" }`, on a locked endpoint as well. A moss app's vex surfaces (`/api/vex`, `/api/<resource>/vex`) are this adapter, so they answer the same.
72
+
73
+ Nothing else moves. Every request that was answered before is answered the same, byte for byte: JSON of the wrong shape keeps its own 400; JSON sent as `text/plain` or with no content type is still read (the content type is never looked at); `getScope` still runs before the body is read; `onExecute` hears nothing for a body that was never a request, as before. The body is still read through hono's own `c.req.json()`, so a middleware that read it first shares hono's cache with the adapter exactly as before, on every hono 4. Only the parse failing is caught — a body that cannot be read at all, or a stream that broke, still throws. The Express adapter was not affected: it is handed `req.body` and already answered 400.
74
+
75
+ **What to change:** nothing, unless something of yours was built on the 500. A host's own `onError` (or a middleware reading `c.error`) no longer sees a `SyntaxError` for these requests — the response is the 400 above, not the one the handler made. A client that retried on 5xx stops retrying them. A nova endpoint aimed at a vex url with no `request` declared now fails with `@error.status` 400 and the message `Body must be JSON`, where it had 500 and `HTTP 500`.
76
+
77
+ - b711c38: The reply says when the engine's own limit may have cut a list.
78
+
79
+ A list query that states no `limit` is given `defaultLimit` (100), and a generated one asking for more than `maxLimit` is clamped. Either way the reply carried the rows and nothing else: a list of 100 looked like a list that had 100 things in it.
80
+
81
+ When the `LIMIT` a query ran with was the engine's number — the default, or the maximum — and exactly that many rows came back for a list, `meta.warnings` now carries one line saying the list may have been cut and which number did it. For a request that named a fingerprint the same line is written once per fingerprint with `console.warn`, because a host that hands callers `result` without `meta` would show it to no one.
82
+
83
+ It says "may": no extra row is fetched to find out, so a list of exactly that many rows gets the line too. A list that ended on a `limit` its author stated gets none, and neither does a single-row answer. No `result` changes, and no query reads a different number of rows.
84
+
85
+ **What to change:** nothing. If the line appears for one of your entries, state the `limit` that list should have.
86
+
87
+ - 3d8518c: `mutationMisfits(def, schema)`: why a stored write does not fit a schema, without running it.
88
+
89
+ Every write passes two column gates before it runs: its table, the columns it writes, its WHERE, its `$lookup`s and its conflict target must be the schema's, and so must a column it reads from an earlier statement (`$returned`). Those gates could only be met by running the write.
90
+
91
+ `mutationMisfits` asks them and runs nothing: one sentence for each refusal, none when the write fits. An upsert is asked as both statements it can become. Scope is not applied — what a policy stamps or bounds is checked where the principal is known.
92
+
93
+ **What to change:** nothing.
94
+
95
+ - b711c38: A seeded entry runs with the `limit` its author wrote, past `maxLimit`.
96
+
97
+ `maxLimit` (1000 by default) clamped every query, including one a developer wrote and seeded: a seeded entry that stated a larger `limit` ran with `maxLimit` in its place, and the reply did not say so.
98
+
99
+ `maxLimit` now clamps only a query nobody reviewed — one generated for a request. An entry seeded with `seedCache` runs with the `limit` it states. Which kind an entry is comes from the stored row and not from the request: protected, with no request hash. That is what `seedCache` writes, and what a host writes when it stores an entry through the cache backend itself; no request can produce it. A generated entry stays clamped when it is replayed, when it is stored under a name the caller chose, and when it is protected afterwards. `engine.compile` takes a bare query and clamps as before; `engine.test` still runs with a limit of 5.
100
+
101
+ One edge: a generated entry that was protected and is then seeded with the identical definition keeps its request hash, because `seedCache` leaves a row that already matches alone. It stays clamped, and its reply says so, until it is unprotected or deleted and seeded again.
102
+
103
+ **This changes what an existing app receives in one setup:** a read entry the host stored itself — seeded, or written protected through the cache backend — whose `limit` is above the engine's `maxLimit` returned `maxLimit` rows and now returns up to its stated `limit`. Nothing changes for an entry that states no limit, one that states `maxLimit` or less, a generated query, or a mutation.
104
+
105
+ **What to change:** nothing, if the larger answer is what the entry was written for. To keep the clamp on seeded entries, set `config.capAuthored: true` on the engine (under moss: `runtime.vexConfig: { capAuthored: true }`).
106
+
107
+ - 1903342: An upsert's signature lists the key its `columns` bind, also for a column its `insert` sets too.
108
+
109
+ An `upsert` may set the same column in `columns` and in `insert`: the first is written when it updates, the second when it creates. Its derived signature read the two merged by column name, so where they used different `$context` keys only the one in `insert` was listed. The key the update binds was missing from discovery and from `collectMutationContext` — and from `details.expected` of the very `missing_context` refusal that named it. What an upsert requires was right all along, on both branches; only the listing was short.
110
+
111
+ ```ts
112
+ { op: 'upsert', table: 'people', key: 'id',
113
+ columns: { name: { $context: 'name' } },
114
+ insert: { name: { $context: 'firstName' } } }
115
+ ```
116
+
117
+ - The signature lists `name` beside `firstName` (listed `firstName` only). `firstName` is still marked `insert only`.
118
+ - Where both halves use the same key for the column, that key is no longer marked `insert only`: the update binds it too.
119
+
120
+ Nothing else moves. An upsert whose `columns` and `insert` share no column is listed as it was, and no write runs, is refused or lands differently. Inside, the signature, the required keys and the `$returned` check now read what a statement sets through one function.
121
+
122
+ **What to change:** nothing.
123
+
3
124
  ## 0.2.2
4
125
 
5
126
  ### Patch Changes
package/DESIGN.md CHANGED
@@ -188,6 +188,31 @@ extra filter clauses.
188
188
  omitted so the runtime can return a valid-but-empty response instead of
189
189
  failing.
190
190
 
191
+ **The limit, and whose number it is.** Every query compiles with a `LIMIT`. A
192
+ query that states none is given `defaultLimit`; one that states more than
193
+ `maxLimit` is clamped to it — unless its author seeded it. `maxLimit` exists
194
+ for the query nobody reviewed: a generated one must not be able to ask for the
195
+ table. Applied to a number a developer wrote into a seeded entry it did
196
+ something else: it replaced that number with a smaller one, and the reply did
197
+ not say so. So the clamp reads provenance off the stored entry: `protected`
198
+ with no `requestHash` is what `seedCache` writes, and what a host writes when
199
+ it stores an entry through the cache backend itself. No request can produce it
200
+ (a generated entry always carries the hash of the request it was born from,
201
+ and the one field a wire call can change on a stored entry is `protected`),
202
+ and an entry that has it runs with its stated limit. It records who put the
203
+ entry there, not whether a person read it: a host that stores a query it did
204
+ not write has vouched for it. `config.capAuthored` puts the
205
+ clamp back for a host that wants one ceiling over everything.
206
+
207
+ What is left is the case nothing can decide for the author: a list that states
208
+ no limit. Returning every row is not offered: nothing would bound the read,
209
+ and an unbounded read of a large table is the failure `defaultLimit` is there
210
+ to stop. So the default
211
+ stays, and the engine stops being silent about it: when the `LIMIT` was its own
212
+ number and exactly that many rows came back for a list, the reply's
213
+ `meta.warnings` says the list may have been cut (and the console, once per
214
+ named entry). No extra row is fetched to make it certain; the line says "may".
215
+
191
216
  ### Adapters
192
217
 
193
218
  `DatabaseAdapter` is the whole database boundary: `introspect`, `compile`
@@ -376,6 +401,37 @@ values bind as SQL parameters exactly like reads) → execute. Batches run in
376
401
  one transaction; the client is structural (`MutationClient` — PGlite, a pg
377
402
  wrapper, a test double), so the core imports no driver.
378
403
 
404
+ **A batch that uses what it wrote (`$returned`).** `{ $returned: 'table.column' }`
405
+ in a later statement reads the one row an earlier statement of the same batch
406
+ wrote — the create-a-parent-then-its-children case, where the parent's id is
407
+ the database's to generate and no natural key exists to `$lookup` it by. Three
408
+ decisions shape it:
409
+
410
+ - *Named by table, nothing else.* A statement has no name of its own, and
411
+ giving it one (`as`) would be a second vocabulary inside the batch. A
412
+ reference names the table an earlier statement writes; if two earlier
413
+ statements write that table the lint refuses the reference, because nothing
414
+ could say which was meant.
415
+ - *Bound between statements, not compiled into one.* The statements stay the
416
+ separate, sequential statements they were: each compiles to an ordinary
417
+ parameter where the reference sits, and the engine fills that parameter from
418
+ the earlier statement's `RETURNING` row before running it. Folding the batch
419
+ into one statement of CTEs was tried and dropped — every CTE in a statement
420
+ sees the same snapshot, so a `$lookup` could no longer read a row an earlier
421
+ statement of the batch had inserted, which a batch can do today.
422
+ - *Exactly one row, or the batch fails.* A `$lookup` that finds nothing writes
423
+ NULL, because the row it reads may honestly not exist. A reference reads a
424
+ row the batch claims to have written; when the scope boundary or a WHERE
425
+ left that statement with no row, writing NULL (or skipping the child) would
426
+ commit half of what the author described. So it throws inside the
427
+ transaction and everything rolls back. What can be known without running —
428
+ no earlier writer, two of them, an `insertEach`, a do-nothing `onConflict` —
429
+ is refused at seed by `lintMutation`.
430
+
431
+ Scope is untouched by it: each statement is scoped on its own exactly as
432
+ before, and the value a reference carries comes from a row that statement was
433
+ already allowed to write.
434
+
379
435
  **Derived context signatures** close the discoverability loop for both
380
436
  kinds: a `$context` ref sits at a position whose column the schema types, so
381
437
  "what do I pass this fingerprint" is computed from the stored def — never
@@ -831,9 +887,16 @@ byte for byte, so nothing a cache already holds under one has moved.
831
887
  - Dynamic mutation generation — icebox, on purpose. If it ever returns, it
832
888
  feeds the existing pipeline (author → validate → seed) behind an effect
833
889
  contract and an approval step; nothing shipped today gets replaced.
834
- - Mutation transactions with cross-statement refs (`$returned.id` — the
835
- create-parent-then-children case). Batches are atomic today but cannot
836
- reference earlier statements' results.
890
+ - ~~Mutation transactions with cross-statement refs~~ — built: see
891
+ **A batch that uses what it wrote (`$returned`)** under Mutations.
892
+ - A reply shape for writes. A mutation still answers with the rows its
893
+ statements returned (one row as that row, several as an array); choosing
894
+ what of a batch the caller is handed — a mapping over the writes — is not
895
+ built.
896
+ - Engine-derived paging (a cursor the engine mints from a query's sort).
897
+ Considered with the limit work and left out: an author can already page
898
+ with a `$context` bound in the filter, and a second way to do it was not
899
+ earned.
837
900
  - MySQL / SQLite adapters — the interface is ready; implementation on demand.
838
901
  - Multi-level field paths — addable in the resolver without DSL changes.
839
902
  - ~~Intent-aware positive caching~~ — superseded: collisions did bite, and the
package/DOCS.md CHANGED
@@ -165,6 +165,7 @@ type QueryEngineConfig = {
165
165
  maxNestingDepth?: number; // default: 2
166
166
  defaultLimit?: number; // default: 100
167
167
  maxLimit?: number; // default: 1000
168
+ capAuthored?: boolean; // default: false — true holds seeded entries to maxLimit too
168
169
  rejectCartesianProducts?: boolean; // default: true
169
170
  warnUnindexedFilters?: boolean; // default: true
170
171
  rejectUnindexedFilters?: boolean; // default: false
@@ -190,8 +191,10 @@ type QueryEngineConfig = {
190
191
  throws `execution_error` if a rule names a column the database does not have.
191
192
  - **`rows`** sizes the reactive rows cache — see
192
193
  [Reactive reads](#reactive-reads-refresh).
193
- - `defaultLimit` is applied when a DSL omits `limit`; `maxLimit` clamps any
194
- larger value.
194
+ - `defaultLimit` is applied when a DSL omits `limit`. `maxLimit` clamps a larger
195
+ `limit` on a query nobody reviewed — one generated for a request. An entry
196
+ the host stored itself — **seeded** — runs with the `limit` it states, past
197
+ `maxLimit`; `capAuthored: true` clamps those as well. See [Limit](#limit).
195
198
  - **`embed`** turns text into a vector for `semantic` filters, at parameter-
196
199
  binding time. It's an injected provider (wire it to Signal) — *not* a database
197
200
  concern, so it lives on the engine, not the adapter (the adapter only emits the
@@ -299,7 +302,8 @@ type QueryResponse = {
299
302
  absent?: true; // optional, and not supplied this run (so: no type)
300
303
  }>;
301
304
  timing?: { agentMs?: number; executionMs: number; mappingMs?: number };
302
- warnings?: string[]; // analyzer warnings, if any
305
+ warnings?: string[]; // analyzer warnings, and the line that says the
306
+ // engine's own limit may have cut a list (see Limit)
303
307
  missingContext?: string[]; // present when REQUIRED keys were not supplied;
304
308
  // optional keys never appear here
305
309
  };
@@ -380,6 +384,53 @@ type Source = string | { as: string; query: Query }; // entity name or subque
380
384
  type FieldRef = string | { field: string; as: string }; // bare column, or aliased output key
381
385
  ```
382
386
 
387
+ ### Limit
388
+
389
+ Every query runs with a `LIMIT`. Whose number it is depends on who wrote the
390
+ query:
391
+
392
+ | The query | States no `limit` | States one |
393
+ |---|---|---|
394
+ | **seeded** — stored by the host itself | `defaultLimit` (100) | that number, whatever `maxLimit` is |
395
+ | **generated** for a request | `defaultLimit` (100) | that number, at most `maxLimit` (1000) |
396
+
397
+ `maxLimit` guards against a query nobody reviewed asking for the table. A
398
+ number a developer wrote into a seeded entry is a reviewed decision, and it is
399
+ honoured: a seeded entry that states `limit: 5000` reads up to 5000 rows. Set
400
+ `config.capAuthored: true` to hold seeded entries to `maxLimit` as well.
401
+
402
+ "Seeded" is not something a request can claim. It is read off the stored
403
+ entry: protected, and carrying no request hash. That is what `seedCache`
404
+ writes, and what a host writes when it stores an entry through the cache
405
+ backend itself (`cache.set(key, { …, protected: true })`). No request can
406
+ produce it: a generated entry always carries the hash of the request it was
407
+ born from, and protecting one afterwards does not remove it. One consequence:
408
+ a generated entry that was protected and is then seeded with the *identical*
409
+ definition keeps its hash — `seedCache` leaves a row that already matches
410
+ alone — and stays clamped, with the warning below, until it is unprotected or
411
+ deleted and seeded again.
412
+
413
+ `compile(dsl)` takes a bare query with no stored entry behind it, and clamps.
414
+ `test(dsl)` runs with a limit of 5, as it always has.
415
+
416
+ **When the engine's number may have ended a list, the reply says so.** If the
417
+ `LIMIT` was the engine's — the default a query was given, or the maximum it was
418
+ clamped to — and exactly that many rows came back for a list, `meta.warnings`
419
+ carries a line:
420
+
421
+ ```
422
+ This list states no limit, so it ran with the engine's defaultLimit of 100, and exactly 100 rows came back — it may have been cut. State a `limit`.
423
+ This query asks for 5000 rows, the engine's maxLimit is 1000, and exactly 1000 came back — the list may have been cut. Raise `maxLimit` to read more.
424
+ ```
425
+
426
+ It is "may": no extra row is fetched to find out, so a list of exactly that
427
+ many rows is told the same. A list that ended on a `limit` its author stated
428
+ is never told anything, and neither is a single-row answer (a shape that is
429
+ not an array). For a request that named a fingerprint the same line is also
430
+ written once per fingerprint to `console.warn` — a host that hands callers
431
+ `result` without `meta` would otherwise show it to no one. The fix is in the
432
+ entry: state the `limit` the list should have.
433
+
383
434
  ### Fields
384
435
 
385
436
  Each entry is `entity.field`, or `{ field, as }` to set a distinct output key
@@ -789,8 +840,9 @@ as a `kind: 'mutation'` entry under a named fingerprint (normally
789
840
 
790
841
  ### The grammar
791
842
 
792
- Five ops, closed and strict. Values are literals, `{ $context: "key" }`, or a
793
- `{ $lookup }` scalar subquery (below) — **never `$scope`** (identity is
843
+ Five ops, closed and strict. Values are literals, `{ $context: "key" }`, a
844
+ `{ $lookup }` scalar subquery, or — in a batch — a `{ $returned }` read of the
845
+ row an earlier statement wrote (both below) — **never `$scope`** (identity is
794
846
  engine-injected, unforgeable) and never a field path. `update`/`delete`
795
847
  REQUIRE a `where` (a real vex `Filter`).
796
848
 
@@ -848,6 +900,11 @@ Two guarantees, both enforced before any SQL runs:
848
900
  entry as the insert-plus-update it really is, so discovery visibility
849
901
  follows the same rule.
850
902
 
903
+ `values` and `set` are separate positions, each binding its own parameter —
904
+ also where they set the same column. Every `$context` key in either is in the
905
+ entry's signature and must be sent on every call: the caller cannot know
906
+ which half will run.
907
+
851
908
  ### `$lookup` — a value read from another table
852
909
 
853
910
  Any value position (insert `values`, update `set`, `onConflict.set`) may be a
@@ -869,12 +926,72 @@ a unique key: more than one matching row is a runtime error (deliberately —
869
926
  silently picking an arbitrary row would be worse), and zero rows write NULL
870
927
  (the column's constraints decide whether that lands).
871
928
 
929
+ ### `$returned` — a value from the row an earlier statement wrote
930
+
931
+ In a batch, a later statement may read a column of the one row an earlier
932
+ statement wrote — how a child row gets the id the database generated for its
933
+ parent, in the same transaction:
934
+
935
+ ```ts
936
+ [
937
+ { op: 'insert', table: 'orders', values: { note: { $context: 'note' } } },
938
+ { op: 'insertEach', table: 'order_lines', items: { $context: 'lines' },
939
+ values: { order_id: { $returned: 'orders.id' }, sku: { $item: 'sku' }, qty: { $item: 'qty' } } },
940
+ ]
941
+ ```
942
+
943
+ `{ $returned: 'table.column' }` names the earlier statement by the **table** it
944
+ writes — an `insert`, `update`, `upsert` or `delete` (a delete returns the row
945
+ it removed); `column` is a column of that table, including one the database
946
+ generated. It goes in any value position, the same ones `$lookup` does (the
947
+ `values`, `set` and `onConflict.set` of every op, an `upsert`'s
948
+ `columns`/`insert` among them), and nowhere else: not in a `where`, not inside
949
+ a `$lookup`, not as an `insertEach`'s `items`.
950
+
951
+ Where `$lookup` reads a row that may honestly not exist (and writes NULL when
952
+ it does not), `$returned` reads a row this batch claims to have written. So:
953
+
954
+ - **Exactly one row, or nothing is written.** If the statement it names wrote
955
+ no row — its WHERE matched nothing, the scope boundary kept the row back, an
956
+ `upsert` updated a row that is not there — or wrote several, the batch fails
957
+ with `execution_error` (`details: { returned, rows }`) and the transaction
958
+ rolls back, earlier statements included.
959
+ - **Refused before any request** by `lintMutation` (so by `seedCache`), and
960
+ again by the engine for a definition that was never seeded:
961
+ - no **earlier** statement of the batch writes that table (this covers a
962
+ reference pointing forwards, and one in a statement that is not in a batch);
963
+ - **two** earlier statements write it — a reference could not tell them apart;
964
+ - the statement it names is an `insertEach` (it writes a row per item);
965
+ - the statement it names is an `insert` whose `onConflict` has no `set` — on a
966
+ conflict that returns no row. Give it a `set` (the create-or-fetch idiom
967
+ above), and the reference holds on both paths.
968
+ - **An unknown column** is `invalid_dsl`, checked against the introspected
969
+ schema before any SQL runs.
970
+
971
+ The value never passes through the request: the engine binds it from the
972
+ returned row between statements, and a context key spelled `orders.id` is not
973
+ it. Scope is unchanged — every statement is scoped on its own, and a `set` rule
974
+ on a column still wins over anything the author put there, a reference
975
+ included.
976
+
977
+ Two things to know when using it:
978
+
979
+ - **It is for keys.** The value passes through the database driver on its way
980
+ from one statement to the next. An id, a number, text and json arrive as
981
+ they were; a timestamp read back as a JS `Date` keeps its milliseconds and
982
+ no more.
983
+ - **The reply is the rows of every statement, in order, as for any batch** —
984
+ and a reply of exactly one row is that row, not an array of one. A parent
985
+ with two children answers `[parent, child, child]`; the same batch with an
986
+ empty `items` answers `parent`. A caller that needs the new id reads the
987
+ first row in both forms.
988
+
872
989
  ### `insertEach` — one statement for a caller-sized list
873
990
 
874
991
  `items` names a context key holding an array of objects; `{ $item: "key" }`
875
992
  values read from the current element (cast to the column's type from the
876
- schema); every other value — literal, `$context`, `$lookup`, engine-injected
877
- `$scope` — is constant across rows. Compiles to a single
993
+ schema); every other value — literal, `$context`, `$lookup`, `$returned`,
994
+ engine-injected `$scope` — is constant across rows. Compiles to a single
878
995
  `INSERT … SELECT … FROM jsonb_array_elements($items)`, so "one row per ticked
879
996
  weekday" is one authored statement, not a code loop:
880
997
 
@@ -935,14 +1052,26 @@ for reads / `effect` = `{ op, table, columns }` for writes), and the
935
1052
  collectMutationContext(def, schema); // { id: { type: 'string', column: 'tasks.id', note: 'upsert key — …' }, … }
936
1053
  collectQueryContext(dsl, schema); // same idea over a query's filter/compute/subqueries
937
1054
  mutationEffect(def); // [{ op: 'update', table: 'tasks', columns: ['done'] }]
1055
+ mutationMisfits(def, schema); // [] when it fits — else why not: ['"tasks.id" in the WHERE is not a column of "tasks".']
938
1056
  ```
939
1057
 
1058
+ `mutationMisfits` asks the column gates every write passes before it runs —
1059
+ table, written columns, WHERE, `$lookup`, conflict target, `$returned` —
1060
+ without running anything, an upsert as both statements it can become. Scope is
1061
+ not applied.
1062
+ moss's `migrateTables` asks it of every write entry before a migration run may
1063
+ commit.
1064
+
940
1065
  ### Authoring lint
941
1066
 
942
1067
  `lintMutation(def)` returns a list of issues (empty when clean). It flags an
943
1068
  `update`/`delete` whose WHERE binds no `$context` — such a write is not
944
- caller-bounded (its only row limit is the scope policy) — and any statement
945
- containing an `optional` condition. `seedCache` runs it on every mutation entry
1069
+ caller-bounded (its only row limit is the scope policy) — any statement
1070
+ containing an `optional` condition, and a `$returned` whose table is not
1071
+ written by exactly one earlier statement, or is written by an `insertEach` or
1072
+ a do-nothing `insert` (the four cases under
1073
+ [`$returned`](#returned--a-value-from-the-row-an-earlier-statement-wrote)).
1074
+ How many rows that statement returns is known only when it runs. `seedCache` runs it on every mutation entry
946
1075
  and throws on an issue, so an unkeyed write never ships: loud at boot, never at
947
1076
  runtime.
948
1077
 
@@ -1119,6 +1248,9 @@ await seedCache(engine.cache, [ordersRecent, taskSetDone]);
1119
1248
  (`{ $ref: '$.result' }`) is stored, so a replay never falls through to
1120
1249
  `mapToShape`.
1121
1250
  - A mutation is linted (`lintMutation`); a failing statement throws.
1251
+ - A read's `limit` is the author's: a seeded entry runs with the number it
1252
+ states even past `maxLimit` (see [Limit](#limit)). One that states none gets
1253
+ `defaultLimit` like any other query — state a `limit` on every list.
1122
1254
  - It is idempotent against a durable cache: a stored row that already matches
1123
1255
  its authored definition is left alone; anything else at that fingerprint
1124
1256
  converges to the authored definition, protected.
@@ -1337,7 +1469,11 @@ other than `GET`/`POST`/`PATCH`/`DELETE` answers 405.
1337
1469
  `fingerprint_protected`, a write with missing context → 400
1338
1470
  `missing_context` with the full derived signature in `details.expected`,
1339
1471
  other `VexError`s (including `locked` and `scope_denied`) → 400 with
1340
- `{ error, message, details }`. A reach the host cannot serve for this
1472
+ `{ error, message, details }`. A body that is not JSON → 400
1473
+ `invalid_request` (`Body must be JSON`): the Hono adapter parses the body
1474
+ itself and never reads the content type, so JSON sent as `text/plain` is
1475
+ read; the Express adapter is handed `req.body` as the host's parser left
1476
+ it. A reach the host cannot serve for this
1341
1477
  principal → 403 `scope_denied`. 500s are the host's faults: `missing_scope`
1342
1478
  (key names stay in the log), an entry demanding a reach with no
1343
1479
  `policyForReach`, a mutation fingerprint on a handler with no `mutations`
@@ -1346,7 +1482,9 @@ other than `GET`/`POST`/`PATCH`/`DELETE` answers 405.
1346
1482
  **`DELETE`** with `{ fingerprint }` → evict (a protected entry, read or mutation,
1347
1483
  is refused with 409 `fingerprint_protected`). Both are refused with 403 `locked` on a
1348
1484
  locked endpoint and answer 404 `cache_miss` for an unknown fingerprint.
1349
- Fingerprints ride the body — names contain `/`.
1485
+ Fingerprints ride the body — names contain `/` — and a body that is not
1486
+ JSON, or carries no fingerprint, is a 400 `invalid_request` before anything
1487
+ else is checked.
1350
1488
 
1351
1489
  Under the hood all of these call the framework-agnostic `handleDiscovery` /
1352
1490
  `handleQuery` / `handleFingerprintPatch` / `handleFingerprintDelete` from
@@ -1428,6 +1566,12 @@ try {
1428
1566
  }
1429
1567
  ```
1430
1568
 
1569
+ Under `require`, each entry point carries its own copy of the class. An error the
1570
+ engine itself throws is an `instanceof` the `VexError` exported beside it; one made
1571
+ by a hook from `@niscorp/vex/agent` (`agent_failed`, `unsatisfiable`) is not — it
1572
+ has the same `name`, `code` and `message`, so read `err.code` there. The framework
1573
+ handlers recognise both.
1574
+
1431
1575
  `VexError.code` is one of:
1432
1576
 
1433
1577
  | Code | Meaning |
@@ -1437,7 +1581,7 @@ try {
1437
1581
  | `missing_scope` | A `$scope` slot the host did not fill. The host's fault, not the request's: the HTTP handler answers 500 and keeps the key names in its log |
1438
1582
  | `scope_denied` | An entity is denied by the scope policy (`VexScopeError`) |
1439
1583
  | `missing_context` | A required context value was absent. Reads surface it softly via `meta.missingContext`; a WRITE hard-400s with the full derived signature in `details.expected` |
1440
- | `execution_error` | Schema not loaded, or a database/runtime failure |
1584
+ | `execution_error` | Schema not loaded, or a database/runtime failure. Also a batch whose `$returned` found no row, or several, where it needs exactly one — `details: { returned, rows }`, nothing written |
1441
1585
  | `agent_failed` | Generation was needed and there is no `generateDsl` hook, or the reference agent's run failed |
1442
1586
  | `cache_miss` | A fingerprint was sent alone and nothing is stored under it |
1443
1587
  | `fingerprint_protected` | A protected entry was asked to change: a named-slot request that no longer matches its stored one |