@mason-data/cli 0.6.2 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. package/README.md +71 -12
  2. package/dist/src/api/postgres/catalog.js +36 -1
  3. package/dist/src/api/postgres/server.js +133 -32
  4. package/dist/src/cdc/pg-wire.js +144 -38
  5. package/dist/src/cli/background.js +29 -5
  6. package/dist/src/cli/command.js +4 -3
  7. package/dist/src/cli/commands.js +259 -127
  8. package/dist/src/cli/demo.js +149 -0
  9. package/dist/src/cli/dev.js +113 -20
  10. package/dist/src/cli/doctor.js +6 -3
  11. package/dist/src/cli/guides.js +32 -8
  12. package/dist/src/cli/init.js +58 -18
  13. package/dist/src/cli/machine.js +8 -3
  14. package/dist/src/cli/main.js +2 -2
  15. package/dist/src/cli/mcp.js +726 -0
  16. package/dist/src/cli/preflight.js +16 -7
  17. package/dist/src/cli/project.js +5 -3
  18. package/dist/src/cli/registry.js +8 -5
  19. package/dist/src/cli/run-server.js +29 -106
  20. package/dist/src/cli/serve.js +6 -1
  21. package/dist/src/cli/source-edit.js +116 -0
  22. package/dist/src/cli/sources.js +6 -4
  23. package/dist/src/demo/seed.js +237 -0
  24. package/dist/src/demo/source.js +655 -0
  25. package/dist/src/duck/distinct-key.js +228 -0
  26. package/dist/src/duck/executor-worker.js +7 -2
  27. package/dist/src/duck/executor.js +67 -26
  28. package/dist/src/duck/explain.js +128 -9
  29. package/dist/src/duck/functions.js +60 -5
  30. package/dist/src/duck/gaps.js +9 -4
  31. package/dist/src/duck/jsonb-order.js +175 -0
  32. package/dist/src/duck/jsonb.js +10 -8
  33. package/dist/src/duck/kept-route.js +82 -2
  34. package/dist/src/duck/kept.js +206 -39
  35. package/dist/src/duck/pgtree.js +40 -0
  36. package/dist/src/duck/plan.js +21 -8
  37. package/dist/src/duck/rewrite.js +1168 -68
  38. package/dist/src/duck/store.js +132 -56
  39. package/dist/src/duck/sync.js +90 -25
  40. package/dist/src/duck/verified.js +6 -0
  41. package/dist/src/lib/timezone.js +3 -2
  42. package/dist/src/web/app.css +11 -0
  43. package/dist/src/web/app.js +167 -12
  44. package/dist/src/web/index.html +6 -1
  45. package/package.json +1 -1
package/README.md CHANGED
@@ -5,9 +5,10 @@ add, keeps the copy current, and serves it to BI tools and your app over the Pos
5
5
  same for you and for a coding agent (Claude Code, Codex, …): it prints for a person, or with `--json` as data, and its
6
6
  exit code says how it went ([for coding agents](#for-coding-agents)).
7
7
 
8
- It needs Node 22.13 or later and nothing else — no Docker, no database of its own; `mason doctor` checks this machine
9
- and says what is missing. On a terminal its output is coloured and wrapped to the window, and a command that takes a
10
- moment shows a spinner; piped, redirected, with `--json` or with `NO_COLOR` set, it is plain text.
8
+ There is nothing to clone or build: install this package and run `mason`. It needs Node 22.13 or later and nothing
9
+ else — no Docker, no database of its own; `mason doctor` checks this machine and says what is missing. On a terminal
10
+ its output is coloured and wrapped to the window, and a command that takes a moment shows a spinner; piped,
11
+ redirected, with `--json` or with `NO_COLOR` set, it is plain text.
11
12
 
12
13
  **A Mason project is your app's folder.** `mason init` makes it one: its sources file in `mason/sources/` — the
13
14
  database and the tables to copy — yours to commit; this machine's copy of your data and Mason's state in `.mason/`,
@@ -19,9 +20,9 @@ npm i -g @mason-data/cli
19
20
  cd my-app
20
21
  mason doctor # Node, memory, ports: what is missing, and what to do
21
22
  mason init # this folder becomes a project, on the database in .env's DATABASE_URL (checked first)
22
- mason start # Mason in the background, then returns
23
23
  mason tables # the tables: rows, keys, added; a table's columns with `mason tables orders`
24
- mason add orders customers # copy them, added to the sources file (--columns id,status: those only)
24
+ mason add orders customers # add them to the sources file (--columns id,status: those only)
25
+ mason start # Mason in the background, copying them, then returns
25
26
  mason wait # until every added table is copied and served
26
27
  mason user add metabase # a user for a BI tool: its password, printed once, and the URL to paste
27
28
  mason ui # the status page and the users, in your browser
@@ -33,6 +34,12 @@ without one it reads `DATABASE_URL` from the folder's `.env.local`, `.env` or th
33
34
  argument or it was only in the shell, and gitignores that file): change it there when the password changes. The
34
35
  database is required.
35
36
 
37
+ **No database at hand?** `mason init demo && mason start`, in a new folder, makes a project on a demo database Mason
38
+ hosts itself — a small shop (`customers`, `products`, `orders`, `order_items`; the first three added, `order_items`
39
+ left for your first `mason add order_items` after `mason start`), kept in `.mason/demo/`, served on a port of this machine, with a new order every few seconds (`MASON_DEMO_CHANGE_MS`, default 5000) so
40
+ `mason status` shows the copy keeping up. Nothing to install. It is read by polling only (it has no change log), and
41
+ `mason destroy` removes it with the copy.
42
+
36
43
  ## The project: its sources file, in your app
37
44
 
38
45
  ```
@@ -86,10 +93,11 @@ again and remove `mason/models/`.
86
93
  | `mason init [<postgres url>]` | Makes this folder a project: checks it can read the database (nothing is written when it cannot; `--skip-check` writes anyway), writes `.mason/config.json` (Mason's key, never the URL) and `mason/sources/<database>.yml` (keeping a sources file already there), adds the URL to `.env` when it was not there, and adds `.mason/` and that `.env` to `.gitignore`. The URL from the argument, else `DATABASE_URL` in `.env.local`, `.env` or the shell. `--name` sets the project's name (default: the folder's) — the database name a tool connects to — `--folder` where its files go, `--remote` allows a database on another machine. Refuses a folder already in a project, but for one with no sources file (an older Mason's), whose sources file alone it writes. |
87
94
  | `mason start` | Starts Mason for this folder's project in the background and returns once it is up. Run it again any time: it says so when Mason is already running. `--replication` reads the database's change log as production does (below); `--poll` goes back. `--self-signed-tls` serves the Postgres API over TLS; `--attach` stays in the foreground. |
88
95
  | `mason tables [<table>…] [--all]` | The tables this user can read against the sources file, one line each: rows, primary key, whether added (and with which columns), and why a table cannot be added yet (no primary key: the `--key` to add it by). Name a table (or pass `--all` for every one) for its columns: each one's type, what it references, and whether `mason add` would leave it out as a secret. It reads the project's database itself, so Mason need not be running. |
89
- | `mason add <table>… [--columns …] [--include …] [--key …]` | Copies tables into Mason, each one an entry of the sources file: every column (`orders: {}`), or `--columns id,status,created_at` for those (one table at a time; the key is written with them). Columns that look like secrets (`password`, `token`, `api_key`, `private_key`, …) are left out, named, and written as `exclude:`; `--include password_hash` adds one. `--key tenant_id,id` adds a table with no primary key, copied by those columns. Adding a table again with other columns changes its entry. |
90
- | `mason remove <table>…` | Takes tables out of the sources file — each one's entry, every other byte as it was — and Mason drops their copies. |
96
+ | `mason add <table>… [--columns …] [--include …] [--key …]` | Copies tables into Mason, each one an entry of the sources file: every column (`orders: {}`), or `--columns id,status,created_at` for those (one table at a time; the key is written with them). Columns that look like secrets (`password`, `token`, `api_key`, `private_key`, …) are left out, named, and written as `exclude:`; `--include password_hash` adds one. `--key tenant_id,id` adds a table with no primary key, copied by those columns. Adding a table again with other columns changes its entry. Mason need not be running: the entry is checked against the database as the run checks it, and the copy begins at `mason start` (with Mason running, at once). |
97
+ | `mason remove <table>…` | Takes tables out of the sources file — each one's entry, every other byte as it was — and Mason drops their copies: at once when it is running, else at the next `mason start`. Mason need not be running. |
91
98
  | `mason wait [--timeout <s>]` | Waits until every added table is copied and served, showing every added table on a line of its own: 100% for the finished ones, in the order they finished, and a bar for each still copying (rows, %, rate, time left) at the bottom. Exit 0 ready, 1 a copy failed (printed, with what to do), 124 timed out (default 600 s). |
92
99
  | `mason user add\|list\|remove <name>` | The users of the Postgres API: a name and a password that does not expire, as a BI tool's datasource form asks for them or an app's `DATABASE_URL` carries them. `add` prints the password once with the URL; `remove` refuses the next connection at once and ends the sessions it holds at their next statement, without a restart. |
100
+ | `mason mcp [--url <url>]` | An MCP server on stdio for Claude Code, Claude Desktop and other agents (below): the tables Mason serves with each copy's rows and how current it is, a table's columns, a statement's rows with their Postgres types or Mason's refusal with its way out, and EXPLAIN — connected as a `mason user` (`MASON_URL`, else `--url`), beside `mason start` or a `mason serve` elsewhere. |
93
101
  | `mason ui` | Opens the running Mason's web page in your browser: what `mason status` says, refreshed every 2 s, and the users of `mason user` (add one, its password shown once; remove one). It prints the URL with the run's token, which the page trades for a cookie on the first visit; `--no-open` prints it only, `--json` is `{ url, port }`. `mason start`'s alone: `mason serve` serves no page. |
94
102
  | `mason status` | What Mason is doing: the database and how its changes are read, the project folder, the copy's file, the Postgres API and how it listens, each table's copy, the statements answered slowly in the last day and the ones refused or failed, each with its reason. In replication mode, where the stream is. |
95
103
  | `mason sync <table>…` | Makes the copy of these tables current now: a table over the sync's row limit (1,000,000 rows) is copied and served but kept current only so, or by replication. |
@@ -99,7 +107,7 @@ again and remove `mason/models/`.
99
107
  | `mason serve` | Mason for a server or a container: `mason start --attach` configured by the environment alone, its state in one folder (a volume), its log on stdout, `GET /healthz`, a clean stop on SIGTERM. `mason serve --help` lists every setting with its default. |
100
108
 
101
109
  Every command documents itself: `mason --help` lists them, and `mason <command> --help` (or `mason help <command>`) says
102
- how to use one, with examples. Every command but `help` takes `--json`. A command means the project of the folder it
110
+ how to use one, with examples. Every command but `help` and `mcp` (whose stdout is the MCP protocol's) takes `--json`. A command means the project of the folder it
103
111
  runs in, or the nearest one above it.
104
112
 
105
113
  ## What `mason start` does
@@ -118,7 +126,7 @@ runs in, or the nearest one above it.
118
126
  your shell names the port, or moves the project to it (said). It listens on 127.0.0.1 unless `MASON_PG_HOST` names
119
127
  another address; off this machine it needs TLS (`--self-signed-tls`, or `MASON_PG_TLS_CERT` / `MASON_PG_TLS_KEY`).
120
128
  4. **It runs in the background**, copying what the sources file names: `mason add` and `mason remove` change it at once,
121
- a hand edit at the next `mason start`.
129
+ a hand edit (or an add or remove made while Mason was stopped) at the next `mason start`.
122
130
 
123
131
  Mason's own settings (`MASON_POLL_MS`, `MASON_DUCKDB_MEMORY`, `MASON_COPY_PARALLEL`, …) set in your shell reach Mason.
124
132
  A copy reads `MASON_COPY_PARALLEL` chunks at once (default 3), each sized to take about `MASON_COPY_CHUNK_SECONDS` to
@@ -151,7 +159,7 @@ what changed is written. A different database is refused: `mason destroy`, then
151
159
  `mason help agent` is the page an agent reads first, offline: the commands in the order a project takes them, every
152
160
  answer's shape, the exit codes and the failure codes. In short:
153
161
 
154
- - Every command but `help` takes `--json` and answers with **one JSON object on stdout**, always with
162
+ - Every command but `help` and `mcp` takes `--json` and answers with **one JSON object on stdout**, always with
155
163
  `next: [{ command, what }]` — the commands to run next, the same a person reads under `Next:`. The other keys are the
156
164
  command's own; `mason help <command>` names them on its `--json` line. Lists are wrapped (`{ tables: […], next }`).
157
165
  - A failure is `{ "error": { "code", "message", "why", "fix" }, "next": […] }` on stdout, and the **exit code** says it:
@@ -169,6 +177,53 @@ mason add orders customers --json && mason wait --json
169
177
  mason user add app --json # the password and URL for the app, as data
170
178
  ```
171
179
 
180
+ ### From an MCP client: `mason mcp`
181
+
182
+ A chat agent asks Mason with tools, over MCP, rather than through a shell. `mason mcp` is an MCP server on stdin and
183
+ stdout (newline-delimited JSON-RPC; stdout carries the protocol alone, what it says goes to stderr) that serves until
184
+ the client closes stdin; it takes no `--json`, and says a failure to start on stderr. It connects to the Postgres API as
185
+ a `mason user`, by the URL `mason user add` printed — `MASON_URL`, or `--url` — so it works beside `mason start` as against a `mason serve` elsewhere; TLS is as the URL's `sslmode`
186
+ says (a host off the machine: `require` when it names none). It never makes a user by itself: with no URL, it refuses
187
+ and says `mason user add claude`.
188
+
189
+ | Tool | Arguments | What it answers |
190
+ |---|---|---|
191
+ | `tables` | none | each table Mason serves, named as a statement writes it (`public.orders`, `public."Clients"`): `rows_about` (the copy's estimate), `kept` (how it is kept current, as `mason status` says it) and `kept_current_by` (`polling`, `logical replication`, `copying again`, or null when nothing keeps it current now) |
192
+ | `describe` | `table` (as `tables` names it, or a bare name in `public`) | its columns with their Postgres types and NOT NULL, its primary key, its foreign keys, comments, and the same rows and `kept` |
193
+ | `query` | `sql`, `max_rows` (200, at most 10,000), `timeout_ms` (30,000, at most 300,000) | `columns` (name, Postgres type), `rows` as Postgres's text, `truncated` when `max_rows` cut it; a refusal is a tool error with the SQLSTATE, message, DETAIL and HINT (the way out) |
194
+ | `explain` | `sql`, `analyze`, `timeout_ms` | Mason's EXPLAIN: each copy read with its rows and how current it is, the SQL it runs, the plan |
195
+
196
+ Its resources are `mason://help/agent` and `mason://help/mcp` (the two pages `mason help` prints). Every statement it
197
+ runs is in the request log under `application_name` `mason-mcp/<the client's name>` (`mason logs --requests`), a query
198
+ `max_rows` cut with the rows it sent; its catalog reads (the tables, a table's columns) are not, as no catalog read is.
199
+ A request the client cancels (`notifications/cancelled`) cancels its statement on the Postgres API.
200
+
201
+ Claude Code, in the project's folder with Mason running, the URL in the server's environment (the server's name
202
+ comes before `--env`, which takes every value after it):
203
+
204
+ ```sh
205
+ mason user add claude # the URL, printed once
206
+ claude mcp add mason --env MASON_URL="postgres://claude:<password>@127.0.0.1:5433/shop" -- mason mcp
207
+ # or on its command line, where ps shows it to this machine's users and claude mcp add prints it
208
+ claude mcp add mason -- mason mcp --url "postgres://claude:<password>@127.0.0.1:5433/shop"
209
+ ```
210
+
211
+ Claude Desktop, in `claude_desktop_config.json`. The app often starts a server without your shell's `PATH`, where
212
+ neither `mason` nor the `node` its first line asks for is found: name both by their absolute paths (`which node`,
213
+ `which mason`):
214
+
215
+ ```json
216
+ {
217
+ "mcpServers": {
218
+ "mason": {
219
+ "command": "/absolute/path/to/node",
220
+ "args": ["/absolute/path/to/mason", "mcp"],
221
+ "env": { "MASON_URL": "postgres://claude:<password>@127.0.0.1:5433/shop" }
222
+ }
223
+ }
224
+ }
225
+ ```
226
+
172
227
  ## Cleaning up
173
228
 
174
229
  Three things exist once a project has run: your **files** (the sources file, in your app), the **copy of your data** (a
@@ -180,7 +235,9 @@ keeps a different set:
180
235
  | Pause | `mason stop` | the data, the state, your files: the next `mason start` resumes in seconds |
181
236
  | Copy the data again, or free its disk | `mason destroy` | your files, `.mason/config.json` and the users: the next `mason start` copies everything again |
182
237
 
183
- Restarting the machine is a `mason stop` that nobody ran: the next `mason start` resumes.
238
+ Restarting the machine is a `mason stop` that nobody ran: the next `mason start` resumes. In a demo project
239
+ (`mason init demo`) the demo database is state too: `mason destroy` removes it with the copy, and the next `mason start`
240
+ seeds it again.
184
241
 
185
242
  ## Requirements
186
243
 
@@ -188,7 +245,9 @@ Restarting the machine is a `mason stop` that nobody ran: the next `mason start`
188
245
 
189
246
  - Node 22.13 or later. The SQL parser (`libpg-query`), the catalog (`PGlite`) and the engine (DuckDB) ship as
190
247
  WebAssembly or prebuilt binaries: no native build, no Python or compiler needed.
191
- - Memory: the copy uses up to half the machine's by default (`MASON_DUCKDB_MEMORY=4GiB` sets it).
248
+ - Memory: the copy may use half the container's memory limit by default, else half the machine's RAM
249
+ (`MASON_DUCKDB_MEMORY=4GB` sets it; `mason doctor` says it). Sizing a server:
250
+ [deploy/mason-data/README.md#memory](https://github.com/embeddable-hq/mason-data/blob/main/deploy/mason-data/README.md#memory).
192
251
  - Postgres 13 or later. Each table needs a primary key, or a `key:` in the sources file (polling only).
193
252
 
194
253
  ## Security
@@ -163,7 +163,42 @@ export function catalogDdl(schema) {
163
163
  statements.push(`COMMENT ON COLUMN ${qualified(table)}.${ident(c.name)} IS ${literal(parts.join("; "))};`);
164
164
  }
165
165
  }
166
- return [...statements, ...constraints];
166
+ return [...statements, ...constraints, ...sourceTypeNames(schema)];
167
+ }
168
+ /**
169
+ * THE NAME OF A TYPE BY THE OID THE WIRE SENDS FOR IT. A column of a type the source made (an enum, a citext) is
170
+ * sent with the SOURCE's OID (src/duck/plan.ts `userTypes`: what the source's own clients cached), which in this
171
+ * catalog means another type or none (both count from 16384): `format_type(<that oid>, …)` answered `???`, or another
172
+ * table's row type. Two readers ask by two OID spaces, and one OID can be in both:
173
+ * - `public.mason_wire_type(oid, integer)` names an OID the WIRE sent: a source OID as its type here, every other as
174
+ * Postgres does. `mason mcp` names a RowDescription's columns by it; this map is the one place a wire OID is named;
175
+ * - the catalog's `format_type(oid, integer)` — the `version()` override's way, search_path listing pg_catalog after
176
+ * public — keeps the catalog's meaning for an OID the catalog has (a reader of pg_attribute.atttypid asks by those)
177
+ * and names any other by the wire's map. `pg_catalog.format_type`, written out, is Postgres's own.
178
+ */
179
+ export function sourceTypeNames(schema) {
180
+ const names = new Map();
181
+ for (const t of schema.tables) {
182
+ if (t.rowTypeOid !== undefined)
183
+ names.set(t.rowTypeOid, qualified(t));
184
+ for (const c of t.columns) {
185
+ if (c.typeOid === undefined || c.spelled === undefined)
186
+ continue;
187
+ // an array column's own OID is its array type's; an element's array type is the element's spelling with []
188
+ names.set(c.typeOid, c.spelled);
189
+ if (c.typeArrayOid !== undefined && !c.spelled.endsWith("[]"))
190
+ names.set(c.typeArrayOid, `${c.spelled}[]`);
191
+ }
192
+ }
193
+ // with no type the source made, every OID the wire sends is Postgres's: the wire's name is format_type's
194
+ if (names.size === 0)
195
+ return [`CREATE OR REPLACE FUNCTION public.mason_wire_type(oid, integer) RETURNS text LANGUAGE sql STABLE AS $$ SELECT pg_catalog.format_type($1, $2) $$;`];
196
+ const values = [...names].map(([oid, spelled]) => `(${oid}::oid, ${literal(spelled)})`).join(", ");
197
+ return [`CREATE OR REPLACE FUNCTION public.mason_wire_type(oid, integer) RETURNS text LANGUAGE sql STABLE AS $$
198
+ SELECT coalesce((SELECT pg_catalog.format_type(pg_catalog.to_regtype(v.spelled), $2) FROM (VALUES ${values}) AS v(source, spelled) WHERE v.source = $1),
199
+ pg_catalog.format_type($1, $2)) $$;`,
200
+ `CREATE OR REPLACE FUNCTION public.format_type(oid, integer) RETURNS text LANGUAGE sql STABLE AS $$
201
+ SELECT CASE WHEN EXISTS (SELECT FROM pg_catalog.pg_type t WHERE t.oid = $1) THEN pg_catalog.format_type($1, $2) ELSE public.mason_wire_type($1, $2) END $$;`];
167
202
  }
168
203
  /**
169
204
  * The source's column defaults, one statement each, run after the schema's DDL by
@@ -1,3 +1,4 @@
1
+ import { copyFacts } from "../../duck/explain.js";
1
2
  import { explainedText, explainOptions } from "./explain.js";
2
3
  import * as crypto from "node:crypto";
3
4
  import * as net from "node:net";
@@ -781,8 +782,9 @@ class Session {
781
782
  * statement's ErrorResponse goes out), and for a refused password (CONNECT), through `options.record`. A catalog
782
783
  * read, a SET or a SHOW answered is the protocol's chatter, not recorded. Never throws: `record` buffers.
783
784
  */
784
- recordRequest(answer, sql, refusedUser) {
785
- const asked = this.asked;
785
+ recordRequest(answer, sql, refusedUser, of) {
786
+ // `of`: a statement recorded after its own messages (a portal dropped suspended), not the one being asked now
787
+ const asked = of === undefined ? this.asked : { sql: sql ?? "", method: of.method, startedAt: Date.now(), ...(of.jsonReads !== undefined ? { jsonReads: of.jsonReads } : {}) };
786
788
  const startedAt = (!(answer instanceof PgError) ? answer.startedAt : undefined) ?? asked?.startedAt ?? Date.now();
787
789
  // a refusal's statement is the one refuse() names (a simple Query's own statement, an Execute's portal's)
788
790
  const stored = storedPayload(sql ?? asked?.sql ?? "");
@@ -869,12 +871,57 @@ class Session {
869
871
  }
870
872
  /** A DuckDB statement's plan (Postgres's analysis and the DuckDB SQL), kept per text, search_path, TimeZone and
871
873
  * declared parameter types until the schema changes. Every statement DuckDB runs has one: never its text as written. */
872
- async duckPlan(raw, whole, paramOids) {
874
+ async duckPlan(raw, whole, paramOids, under = this.vars, wireBound = false) {
873
875
  const stmt = raw.stmt;
874
876
  const bytes = Buffer.from(whole, "utf8");
875
877
  const base = raw.stmt_location ?? 0;
876
878
  const text = bytes.subarray(base, raw.stmt_len ? base + raw.stmt_len : bytes.length).toString("utf8");
877
- return this.server.duckPlan(text, paramOids, this.vars.searchPath, this.vars.timeZone, () => this.server.options.duck.plan((b) => this.server.catalog.run(b), text, stmt, base, paramOids, this.vars.searchPath, this.vars.timeZone, this.server.userTypes()));
879
+ const { searchPath, timeZone } = under;
880
+ return this.server.duckPlan(text, paramOids, searchPath, timeZone, wireBound, () => this.server.options.duck.plan((b) => this.server.catalog.run(b), text, stmt, base, paramOids, searchPath, timeZone, this.server.userTypes(), wireBound));
881
+ }
882
+ /**
883
+ * A ROW COUNT ON THE WIRE IS A BOUND THE PLAN AT PARSE COULD NOT SEE (docs/DESIGN.md "A LEFT lateral of a list"): an
884
+ * Execute's row count (JDBC's setMaxRows or fetch size, DBeaver's, a native question's) arrives after the Parse, so a
885
+ * LEFT lateral of a list with no LIMIT in its SQL was left DuckDB's delimiter join, which reads the whole base before
886
+ * the first row and hands its rows back in its own order — the first rows sent were not Postgres's. Where the plan
887
+ * says such a lateral was left so (PlannedSql.unboundedLateral), the statement is planned again with the bound
888
+ * asserted, under the search_path and TimeZone of its Parse, kept on the statement and in the plans by text. The
889
+ * portal's result is begun ONCE, from whichever plan its first Execute picked; an Execute after a PortalSuspended
890
+ * reads on from that result, never runs it again. A plan made again over a schema moved since the Parse, whose
891
+ * result or parameters are not the ones the Parse described, is not run: the Parse's plan is (what a client was told
892
+ * of the result stands). Planning again is the statement's own time: a CancelRequest landing meanwhile aborts its
893
+ * controller (`running`, answered 57014 as the statement begins), and a session closed meanwhile runs nothing. A
894
+ * plan made again that Postgres or the rewrite refuses (a StatementError, never a cancel's or a timeout's 57014) is
895
+ * logged and the Parse's plan runs, kept: the Parse answered, and a row count must not make it fail.
896
+ */
897
+ async boundOnWire(statement, plan, controller) {
898
+ if (plan.sql.unboundedLateral !== true || statement.raw === null)
899
+ return plan;
900
+ if (statement.duckOnWire === undefined) {
901
+ this.running = controller;
902
+ let bounded;
903
+ try {
904
+ bounded = await this.duckPlan(statement.raw, statement.sql, statement.paramOids, statement.plannedUnder ?? this.vars, true);
905
+ }
906
+ catch (e) {
907
+ if (!(e instanceof StatementError) || e.fields.code === "57014")
908
+ throw e;
909
+ this.server.log(`session ${this.id}: a statement planned again for an Execute's row count was refused (${e.fields.code}: ${e.fields.message}): its Parse's plan runs`);
910
+ bounded = plan;
911
+ }
912
+ finally {
913
+ this.settled();
914
+ }
915
+ const shape = (p) => JSON.stringify([p.params, p.fields?.map((f) => [f.name, f.oid, f.typmod]) ?? null]);
916
+ statement.duckOnWire = shape(bounded) === shape(plan) ? bounded : plan;
917
+ if (statement.duckOnWire === plan && bounded !== plan)
918
+ this.server.log(`session ${this.id}: a statement planned again for an Execute's row count answers other columns than its Parse said (the schema moved): its Parse's plan runs`);
919
+ }
920
+ if (this.phase === "closed") {
921
+ this.server.log(`session ${this.id}: closed while its statement was planned again for an Execute's row count: it runs nothing`);
922
+ throw new PgError({ code: "08006", message: "the session closed while its statement was planned" });
923
+ }
924
+ return statement.duckOnWire;
878
925
  }
879
926
  /**
880
927
  * AN EXPLAIN (src/api/postgres/explain.ts, src/duck/explain.ts): its options read as Postgres reads them, then the
@@ -894,7 +941,17 @@ class Session {
894
941
  const found = await explainedText(text, t.query);
895
942
  if (found === null)
896
943
  throw new PgError({ code: "0A000", message: "Mason cannot find the statement this EXPLAIN explains in its text (a Mason gap: report it, with the statement)" });
897
- const plan = await this.server.options.duck.plan((b) => this.server.catalog.run(b), found.text, t.query, base + Buffer.byteLength(text.slice(0, found.offset)), paramOids, this.vars.searchPath, this.vars.timeZone, this.server.userTypes());
944
+ let plan;
945
+ try {
946
+ plan = await this.server.options.duck.plan((b) => this.server.catalog.run(b), found.text, t.query, base + Buffer.byteLength(text.slice(0, found.offset)), paramOids, this.vars.searchPath, this.vars.timeZone, this.server.userTypes());
947
+ }
948
+ catch (e) {
949
+ // a refusal of the statement explained is placed as Postgres places it: in the whole text, past `EXPLAIN …`
950
+ if (!(e instanceof StatementError) || e.fields.position === undefined)
951
+ throw e;
952
+ const before = bytes.subarray(0, base).toString("utf8").length + found.offset;
953
+ throw new StatementError({ ...e.fields, position: e.fields.position + before }, whole);
954
+ }
898
955
  return { options, plan };
899
956
  }
900
957
  /** an EXPLAIN of a DuckDB statement, run: its QUERY PLAN rows (ANALYZE runs the statement and sends none of its rows) */
@@ -904,7 +961,9 @@ class Session {
904
961
  this.running = controller;
905
962
  try {
906
963
  const values = await this.duckValues(e.plan, params, clock);
907
- const lines = await runner.explain(e.plan.sql, values, this.duckSession(), controller.signal, { options: e.options, fields: e.plan.fields, exact: e.plan.exact ?? [], planning: e.plan.timing ?? { analysisMs: 0, rewriteMs: 0 } });
964
+ const lines = await runner.explain(e.plan.sql, values, this.duckSession(), controller.signal, { options: e.options, fields: e.plan.fields, exact: e.plan.exact ?? [], planning: e.plan.timing ?? { analysisMs: 0, rewriteMs: 0 },
965
+ // how each copy is kept current, as the sync says it to mason status, read now (src/duck/sync.ts KeptCurrent)
966
+ copies: copyFacts(this.server.options.duck.copies?.() ?? []) });
908
967
  return { field: explainField(e.options, formatAt(resultFormats, 0)), rows: lines.map((l) => dataRow([Buffer.from(l)])) };
909
968
  }
910
969
  finally {
@@ -1054,9 +1113,20 @@ class Session {
1054
1113
  }
1055
1114
  }
1056
1115
  }
1057
- /** a portal gone (Close, Sync, a Bind of its name, DISCARD, the session's end): its DuckDB result with it */
1116
+ /**
1117
+ * A portal gone (Close, Sync, a Bind of its name, DISCARD, the session's end): its DuckDB result with it. One the
1118
+ * client stopped reading at a row count (an Execute's: JDBC's setMaxRows, `mason mcp`'s max_rows) was never read to
1119
+ * its end, so its row of the request log is written here, with the rows it sent: a statement answered is in the log
1120
+ * however far it was read.
1121
+ */
1058
1122
  dropPortal(name) {
1059
- this.portals.get(name)?.cursor?.answer.close();
1123
+ const portal = this.portals.get(name);
1124
+ const c = portal?.cursor;
1125
+ if (portal !== undefined && c !== undefined && c.suspended === true && c.recorded !== true && c.failed === undefined) {
1126
+ c.recorded = true;
1127
+ this.recordRequest({ rows: c.sent, startedAt: c.startedAt }, portal.statement.sql, undefined, { method: "EXECUTE", jsonReads: portal.statement.duck?.jsonReads });
1128
+ }
1129
+ c?.answer.close();
1060
1130
  this.portals.delete(name);
1061
1131
  }
1062
1132
  dropPortals() { for (const name of [...this.portals.keys()])
@@ -1132,6 +1202,9 @@ class Session {
1132
1202
  }
1133
1203
  if (this.skipToSync || m.type === "Flush")
1134
1204
  return;
1205
+ // its statement, read as the message arrives: a portal dropped while it runs (the session ended under a cancelled
1206
+ // statement) leaves the refusal its statement, never ""
1207
+ const sql = m.type === "Parse" ? m.sql : this.sqlOf(m);
1135
1208
  try {
1136
1209
  switch (m.type) {
1137
1210
  case "Parse": return this.sendAll(await this.parseMessage(m));
@@ -1142,7 +1215,6 @@ class Session {
1142
1215
  }
1143
1216
  }
1144
1217
  catch (err) {
1145
- const sql = m.type === "Parse" ? m.sql : this.sqlOf(m);
1146
1218
  const e = asPgError(err, sql);
1147
1219
  this.refuse(e, sql, m.type);
1148
1220
  this.skipToSync = true;
@@ -1204,6 +1276,7 @@ class Session {
1204
1276
  // Postgres's analysis says the result's columns and the parameters' types, and DuckDB's SQL is made from it
1205
1277
  if (statement.shape.kind === "duckdb") {
1206
1278
  statement.duck = await this.duckPlan(raw, m.sql, m.paramTypes);
1279
+ statement.plannedUnder = { searchPath: this.vars.searchPath, timeZone: this.vars.timeZone };
1207
1280
  // the implicit transaction begins with the Parse, as Postgres's does (start_xact_command): its literals read it
1208
1281
  statement.parsedNow = this.txNow ??= this.server.now();
1209
1282
  }
@@ -1382,35 +1455,50 @@ class Session {
1382
1455
  if (portal.cursor === undefined) {
1383
1456
  if (statement.duck === undefined)
1384
1457
  throw new PgError({ code: "XX000", message: "a DuckDB statement without its plan: report it (Mason's Postgres listener)" });
1458
+ // a row count on the wire bounds the rows: the plan made for it where it differs (boundOnWire)
1459
+ const plan = m.maxRows > 0 ? await this.boundOnWire(statement, statement.duck, controller) : statement.duck;
1385
1460
  const exec = this.txNow ??= this.server.now();
1386
- const answer = await this.duckAnswer(statement.duck, portal.params, portal.resultFormats, { parse: statement.parsedNow ?? exec, exec }, controller);
1461
+ const answer = await this.duckAnswer(plan, portal.params, portal.resultFormats, { parse: statement.parsedNow ?? exec, exec }, controller);
1387
1462
  portal.cursor = answer.cursor;
1388
1463
  portal.fields = answer.fields;
1389
1464
  }
1390
1465
  const c = portal.cursor;
1391
1466
  if (c.failed !== undefined)
1392
1467
  throw c.failed;
1393
- // every row: streamed
1394
- if (m.maxRows <= 0) {
1395
- try {
1396
- await this.streamRest(c);
1468
+ c.suspended = false;
1469
+ try {
1470
+ // every row: streamed
1471
+ if (m.maxRows <= 0) {
1472
+ try {
1473
+ await this.streamRest(c);
1474
+ }
1475
+ finally {
1476
+ c.answer.close();
1477
+ }
1478
+ c.recorded = true;
1479
+ this.recordRequest({ rows: c.sent, startedAt: c.startedAt });
1480
+ return [commandComplete(`SELECT ${c.sent}`)];
1397
1481
  }
1398
- finally {
1399
- c.answer.close();
1482
+ // at most maxRows (JDBC's fetch size, a cursor): suspended while a row is left, read one chunk ahead to know it
1483
+ while (!c.done && c.pending.length <= m.maxRows)
1484
+ await this.fill(c);
1485
+ const out = c.pending.splice(0, m.maxRows);
1486
+ c.sent += out.length;
1487
+ // suspended: its row is written at its last Execute, or when it is dropped (dropPortal)
1488
+ if (c.pending.length > 0) {
1489
+ c.suspended = true;
1490
+ return [...out, portalSuspended()];
1400
1491
  }
1492
+ c.answer.close();
1493
+ c.recorded = true;
1401
1494
  this.recordRequest({ rows: c.sent, startedAt: c.startedAt });
1402
- return [commandComplete(`SELECT ${c.sent}`)];
1403
- }
1404
- // at most maxRows (JDBC's fetch size, a cursor): suspended while a row is left, read one chunk ahead to know it
1405
- while (!c.done && c.pending.length <= m.maxRows)
1406
- await this.fill(c);
1407
- const out = c.pending.splice(0, m.maxRows);
1408
- c.sent += out.length;
1409
- if (c.pending.length > 0)
1410
- return [...out, portalSuspended()];
1411
- c.answer.close();
1412
- this.recordRequest({ rows: c.sent, startedAt: c.startedAt });
1413
- return [...out, commandComplete(`SELECT ${c.sent}`)];
1495
+ return [...out, commandComplete(`SELECT ${c.sent}`)];
1496
+ }
1497
+ catch (e) {
1498
+ // its refusal is its row (refuse)
1499
+ c.recorded = true;
1500
+ throw e;
1501
+ }
1414
1502
  }
1415
1503
  async closeMessage(m) {
1416
1504
  if (m.kind === "S") {
@@ -1548,7 +1636,8 @@ function relayedError(frame) {
1548
1636
  // the listener
1549
1637
  class Server {
1550
1638
  sessions = new Map();
1551
- /** the DuckDB plans by text, search_path and declared parameter types, for the schema they were made under */
1639
+ /** the DuckDB plans by text, search_path, TimeZone, declared parameter types and an Execute's bound (Session.boundOnWire), for
1640
+ * the schema they were made under */
1552
1641
  duckPlans = new Map();
1553
1642
  duckPlansOf = null;
1554
1643
  duckPlansAt = 0;
@@ -1578,8 +1667,17 @@ class Server {
1578
1667
  }
1579
1668
  return out;
1580
1669
  }
1581
- async duckPlan(text, paramOids, searchPath, timeZone, make) {
1582
- const schema = this.options.schema();
1670
+ /**
1671
+ * EVERY PLAN IS MADE OVER THE CATALOG OF THE SCHEMA IT IS KEPT UNDER: the catalog is replaced here (`schema`), not
1672
+ * only at a statement's classify, because a plan is also made at an Execute (Session.boundOnWire) — a re-plan that
1673
+ * read the schema raw analysed against the old catalog and was kept under the new one (a review finding of #23).
1674
+ * The ordering relied on: `schema()` calls `catalog.replace` synchronously, which queues the old instance's close on
1675
+ * the catalog's one chain (src/api/postgres/catalog.ts "SERIALISED") before `make` queues its analysis, so the
1676
+ * analysis starts the instance with the new DDL. A plan whose schema or generation moved while it was made is
1677
+ * returned to its caller and not kept.
1678
+ */
1679
+ async duckPlan(text, paramOids, searchPath, timeZone, wireBound, make) {
1680
+ const schema = this.schema();
1583
1681
  const generation = this.options.duck?.generation?.() ?? 0;
1584
1682
  if (schema !== this.duckPlansOf || generation !== this.duckPlansAt) {
1585
1683
  this.duckPlans.clear();
@@ -1587,11 +1685,14 @@ class Server {
1587
1685
  this.duckPlansAt = generation;
1588
1686
  }
1589
1687
  // the zone too: a plan holds the values Postgres parsed its literals to, under the session's zone
1590
- const key = `${searchPath}\u0000${timeZone}\u0000${paramOids.join(",")}\u0000${text}`;
1688
+ // and the bound: an Execute's row count plans a LEFT lateral of a list otherwise (Session.boundOnWire), kept beside
1689
+ const key = `${searchPath}\u0000${timeZone}\u0000${paramOids.join(",")}\u0000${wireBound ? "wire" : ""}\u0000${text}`;
1591
1690
  const hit = this.duckPlans.get(key);
1592
1691
  if (hit !== undefined)
1593
1692
  return hit;
1594
1693
  const plan = await make();
1694
+ if (this.duckPlansOf !== schema || this.duckPlansAt !== generation)
1695
+ return plan;
1595
1696
  // bounded: a tool that inlines its values sends a new text per value
1596
1697
  if (this.duckPlans.size >= 2000)
1597
1698
  this.duckPlans.delete(this.duckPlans.keys().next().value);