@mason-data/cli 0.6.2 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +71 -12
- package/dist/src/api/postgres/catalog.js +36 -1
- package/dist/src/api/postgres/server.js +133 -32
- package/dist/src/cdc/pg-wire.js +144 -38
- package/dist/src/cli/background.js +29 -5
- package/dist/src/cli/command.js +4 -3
- package/dist/src/cli/commands.js +259 -127
- package/dist/src/cli/demo.js +149 -0
- package/dist/src/cli/dev.js +113 -20
- package/dist/src/cli/doctor.js +6 -3
- package/dist/src/cli/guides.js +32 -8
- package/dist/src/cli/init.js +58 -18
- package/dist/src/cli/machine.js +8 -3
- package/dist/src/cli/main.js +2 -2
- package/dist/src/cli/mcp.js +726 -0
- package/dist/src/cli/preflight.js +16 -7
- package/dist/src/cli/project.js +5 -3
- package/dist/src/cli/registry.js +8 -5
- package/dist/src/cli/run-server.js +29 -106
- package/dist/src/cli/serve.js +6 -1
- package/dist/src/cli/source-edit.js +116 -0
- package/dist/src/cli/sources.js +6 -4
- package/dist/src/demo/seed.js +237 -0
- package/dist/src/demo/source.js +655 -0
- package/dist/src/duck/distinct-key.js +228 -0
- package/dist/src/duck/executor-worker.js +7 -2
- package/dist/src/duck/executor.js +67 -26
- package/dist/src/duck/explain.js +128 -9
- package/dist/src/duck/functions.js +60 -5
- package/dist/src/duck/gaps.js +9 -4
- package/dist/src/duck/jsonb-order.js +175 -0
- package/dist/src/duck/jsonb.js +10 -8
- package/dist/src/duck/kept-route.js +82 -2
- package/dist/src/duck/kept.js +206 -39
- package/dist/src/duck/pgtree.js +40 -0
- package/dist/src/duck/plan.js +21 -8
- package/dist/src/duck/rewrite.js +1168 -68
- package/dist/src/duck/store.js +132 -56
- package/dist/src/duck/sync.js +90 -25
- package/dist/src/duck/verified.js +6 -0
- package/dist/src/lib/timezone.js +3 -2
- package/dist/src/web/app.css +11 -0
- package/dist/src/web/app.js +167 -12
- package/dist/src/web/index.html +6 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -5,9 +5,10 @@ add, keeps the copy current, and serves it to BI tools and your app over the Pos
|
|
|
5
5
|
same for you and for a coding agent (Claude Code, Codex, …): it prints for a person, or with `--json` as data, and its
|
|
6
6
|
exit code says how it went ([for coding agents](#for-coding-agents)).
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
8
|
+
There is nothing to clone or build: install this package and run `mason`. It needs Node 22.13 or later and nothing
|
|
9
|
+
else — no Docker, no database of its own; `mason doctor` checks this machine and says what is missing. On a terminal
|
|
10
|
+
its output is coloured and wrapped to the window, and a command that takes a moment shows a spinner; piped,
|
|
11
|
+
redirected, with `--json` or with `NO_COLOR` set, it is plain text.
|
|
11
12
|
|
|
12
13
|
**A Mason project is your app's folder.** `mason init` makes it one: its sources file in `mason/sources/` — the
|
|
13
14
|
database and the tables to copy — yours to commit; this machine's copy of your data and Mason's state in `.mason/`,
|
|
@@ -19,9 +20,9 @@ npm i -g @mason-data/cli
|
|
|
19
20
|
cd my-app
|
|
20
21
|
mason doctor # Node, memory, ports: what is missing, and what to do
|
|
21
22
|
mason init # this folder becomes a project, on the database in .env's DATABASE_URL (checked first)
|
|
22
|
-
mason start # Mason in the background, then returns
|
|
23
23
|
mason tables # the tables: rows, keys, added; a table's columns with `mason tables orders`
|
|
24
|
-
mason add orders customers #
|
|
24
|
+
mason add orders customers # add them to the sources file (--columns id,status: those only)
|
|
25
|
+
mason start # Mason in the background, copying them, then returns
|
|
25
26
|
mason wait # until every added table is copied and served
|
|
26
27
|
mason user add metabase # a user for a BI tool: its password, printed once, and the URL to paste
|
|
27
28
|
mason ui # the status page and the users, in your browser
|
|
@@ -33,6 +34,12 @@ without one it reads `DATABASE_URL` from the folder's `.env.local`, `.env` or th
|
|
|
33
34
|
argument or it was only in the shell, and gitignores that file): change it there when the password changes. The
|
|
34
35
|
database is required.
|
|
35
36
|
|
|
37
|
+
**No database at hand?** `mason init demo && mason start`, in a new folder, makes a project on a demo database Mason
|
|
38
|
+
hosts itself — a small shop (`customers`, `products`, `orders`, `order_items`; the first three added, `order_items`
|
|
39
|
+
left for your first `mason add order_items` after `mason start`), kept in `.mason/demo/`, served on a port of this machine, with a new order every few seconds (`MASON_DEMO_CHANGE_MS`, default 5000) so
|
|
40
|
+
`mason status` shows the copy keeping up. Nothing to install. It is read by polling only (it has no change log), and
|
|
41
|
+
`mason destroy` removes it with the copy.
|
|
42
|
+
|
|
36
43
|
## The project: its sources file, in your app
|
|
37
44
|
|
|
38
45
|
```
|
|
@@ -86,10 +93,11 @@ again and remove `mason/models/`.
|
|
|
86
93
|
| `mason init [<postgres url>]` | Makes this folder a project: checks it can read the database (nothing is written when it cannot; `--skip-check` writes anyway), writes `.mason/config.json` (Mason's key, never the URL) and `mason/sources/<database>.yml` (keeping a sources file already there), adds the URL to `.env` when it was not there, and adds `.mason/` and that `.env` to `.gitignore`. The URL from the argument, else `DATABASE_URL` in `.env.local`, `.env` or the shell. `--name` sets the project's name (default: the folder's) — the database name a tool connects to — `--folder` where its files go, `--remote` allows a database on another machine. Refuses a folder already in a project, but for one with no sources file (an older Mason's), whose sources file alone it writes. |
|
|
87
94
|
| `mason start` | Starts Mason for this folder's project in the background and returns once it is up. Run it again any time: it says so when Mason is already running. `--replication` reads the database's change log as production does (below); `--poll` goes back. `--self-signed-tls` serves the Postgres API over TLS; `--attach` stays in the foreground. |
|
|
88
95
|
| `mason tables [<table>…] [--all]` | The tables this user can read against the sources file, one line each: rows, primary key, whether added (and with which columns), and why a table cannot be added yet (no primary key: the `--key` to add it by). Name a table (or pass `--all` for every one) for its columns: each one's type, what it references, and whether `mason add` would leave it out as a secret. It reads the project's database itself, so Mason need not be running. |
|
|
89
|
-
| `mason add <table>… [--columns …] [--include …] [--key …]` | Copies tables into Mason, each one an entry of the sources file: every column (`orders: {}`), or `--columns id,status,created_at` for those (one table at a time; the key is written with them). Columns that look like secrets (`password`, `token`, `api_key`, `private_key`, …) are left out, named, and written as `exclude:`; `--include password_hash` adds one. `--key tenant_id,id` adds a table with no primary key, copied by those columns. Adding a table again with other columns changes its entry. |
|
|
90
|
-
| `mason remove <table>…` | Takes tables out of the sources file — each one's entry, every other byte as it was — and Mason drops their copies. |
|
|
96
|
+
| `mason add <table>… [--columns …] [--include …] [--key …]` | Copies tables into Mason, each one an entry of the sources file: every column (`orders: {}`), or `--columns id,status,created_at` for those (one table at a time; the key is written with them). Columns that look like secrets (`password`, `token`, `api_key`, `private_key`, …) are left out, named, and written as `exclude:`; `--include password_hash` adds one. `--key tenant_id,id` adds a table with no primary key, copied by those columns. Adding a table again with other columns changes its entry. Mason need not be running: the entry is checked against the database as the run checks it, and the copy begins at `mason start` (with Mason running, at once). |
|
|
97
|
+
| `mason remove <table>…` | Takes tables out of the sources file — each one's entry, every other byte as it was — and Mason drops their copies: at once when it is running, else at the next `mason start`. Mason need not be running. |
|
|
91
98
|
| `mason wait [--timeout <s>]` | Waits until every added table is copied and served, showing every added table on a line of its own: 100% for the finished ones, in the order they finished, and a bar for each still copying (rows, %, rate, time left) at the bottom. Exit 0 ready, 1 a copy failed (printed, with what to do), 124 timed out (default 600 s). |
|
|
92
99
|
| `mason user add\|list\|remove <name>` | The users of the Postgres API: a name and a password that does not expire, as a BI tool's datasource form asks for them or an app's `DATABASE_URL` carries them. `add` prints the password once with the URL; `remove` refuses the next connection at once and ends the sessions it holds at their next statement, without a restart. |
|
|
100
|
+
| `mason mcp [--url <url>]` | An MCP server on stdio for Claude Code, Claude Desktop and other agents (below): the tables Mason serves with each copy's rows and how current it is, a table's columns, a statement's rows with their Postgres types or Mason's refusal with its way out, and EXPLAIN — connected as a `mason user` (`MASON_URL`, else `--url`), beside `mason start` or a `mason serve` elsewhere. |
|
|
93
101
|
| `mason ui` | Opens the running Mason's web page in your browser: what `mason status` says, refreshed every 2 s, and the users of `mason user` (add one, its password shown once; remove one). It prints the URL with the run's token, which the page trades for a cookie on the first visit; `--no-open` prints it only, `--json` is `{ url, port }`. `mason start`'s alone: `mason serve` serves no page. |
|
|
94
102
|
| `mason status` | What Mason is doing: the database and how its changes are read, the project folder, the copy's file, the Postgres API and how it listens, each table's copy, the statements answered slowly in the last day and the ones refused or failed, each with its reason. In replication mode, where the stream is. |
|
|
95
103
|
| `mason sync <table>…` | Makes the copy of these tables current now: a table over the sync's row limit (1,000,000 rows) is copied and served but kept current only so, or by replication. |
|
|
@@ -99,7 +107,7 @@ again and remove `mason/models/`.
|
|
|
99
107
|
| `mason serve` | Mason for a server or a container: `mason start --attach` configured by the environment alone, its state in one folder (a volume), its log on stdout, `GET /healthz`, a clean stop on SIGTERM. `mason serve --help` lists every setting with its default. |
|
|
100
108
|
|
|
101
109
|
Every command documents itself: `mason --help` lists them, and `mason <command> --help` (or `mason help <command>`) says
|
|
102
|
-
how to use one, with examples. Every command but `help` takes `--json`. A command means the project of the folder it
|
|
110
|
+
how to use one, with examples. Every command but `help` and `mcp` (whose stdout is the MCP protocol's) takes `--json`. A command means the project of the folder it
|
|
103
111
|
runs in, or the nearest one above it.
|
|
104
112
|
|
|
105
113
|
## What `mason start` does
|
|
@@ -118,7 +126,7 @@ runs in, or the nearest one above it.
|
|
|
118
126
|
your shell names the port, or moves the project to it (said). It listens on 127.0.0.1 unless `MASON_PG_HOST` names
|
|
119
127
|
another address; off this machine it needs TLS (`--self-signed-tls`, or `MASON_PG_TLS_CERT` / `MASON_PG_TLS_KEY`).
|
|
120
128
|
4. **It runs in the background**, copying what the sources file names: `mason add` and `mason remove` change it at once,
|
|
121
|
-
a hand edit at the next `mason start`.
|
|
129
|
+
a hand edit (or an add or remove made while Mason was stopped) at the next `mason start`.
|
|
122
130
|
|
|
123
131
|
Mason's own settings (`MASON_POLL_MS`, `MASON_DUCKDB_MEMORY`, `MASON_COPY_PARALLEL`, …) set in your shell reach Mason.
|
|
124
132
|
A copy reads `MASON_COPY_PARALLEL` chunks at once (default 3), each sized to take about `MASON_COPY_CHUNK_SECONDS` to
|
|
@@ -151,7 +159,7 @@ what changed is written. A different database is refused: `mason destroy`, then
|
|
|
151
159
|
`mason help agent` is the page an agent reads first, offline: the commands in the order a project takes them, every
|
|
152
160
|
answer's shape, the exit codes and the failure codes. In short:
|
|
153
161
|
|
|
154
|
-
- Every command but `help` takes `--json` and answers with **one JSON object on stdout**, always with
|
|
162
|
+
- Every command but `help` and `mcp` takes `--json` and answers with **one JSON object on stdout**, always with
|
|
155
163
|
`next: [{ command, what }]` — the commands to run next, the same a person reads under `Next:`. The other keys are the
|
|
156
164
|
command's own; `mason help <command>` names them on its `--json` line. Lists are wrapped (`{ tables: […], next }`).
|
|
157
165
|
- A failure is `{ "error": { "code", "message", "why", "fix" }, "next": […] }` on stdout, and the **exit code** says it:
|
|
@@ -169,6 +177,53 @@ mason add orders customers --json && mason wait --json
|
|
|
169
177
|
mason user add app --json # the password and URL for the app, as data
|
|
170
178
|
```
|
|
171
179
|
|
|
180
|
+
### From an MCP client: `mason mcp`
|
|
181
|
+
|
|
182
|
+
A chat agent asks Mason with tools, over MCP, rather than through a shell. `mason mcp` is an MCP server on stdin and
|
|
183
|
+
stdout (newline-delimited JSON-RPC; stdout carries the protocol alone, what it says goes to stderr) that serves until
|
|
184
|
+
the client closes stdin; it takes no `--json`, and says a failure to start on stderr. It connects to the Postgres API as
|
|
185
|
+
a `mason user`, by the URL `mason user add` printed — `MASON_URL`, or `--url` — so it works beside `mason start` as against a `mason serve` elsewhere; TLS is as the URL's `sslmode`
|
|
186
|
+
says (a host off the machine: `require` when it names none). It never makes a user by itself: with no URL, it refuses
|
|
187
|
+
and says `mason user add claude`.
|
|
188
|
+
|
|
189
|
+
| Tool | Arguments | What it answers |
|
|
190
|
+
|---|---|---|
|
|
191
|
+
| `tables` | none | each table Mason serves, named as a statement writes it (`public.orders`, `public."Clients"`): `rows_about` (the copy's estimate), `kept` (how it is kept current, as `mason status` says it) and `kept_current_by` (`polling`, `logical replication`, `copying again`, or null when nothing keeps it current now) |
|
|
192
|
+
| `describe` | `table` (as `tables` names it, or a bare name in `public`) | its columns with their Postgres types and NOT NULL, its primary key, its foreign keys, comments, and the same rows and `kept` |
|
|
193
|
+
| `query` | `sql`, `max_rows` (200, at most 10,000), `timeout_ms` (30,000, at most 300,000) | `columns` (name, Postgres type), `rows` as Postgres's text, `truncated` when `max_rows` cut it; a refusal is a tool error with the SQLSTATE, message, DETAIL and HINT (the way out) |
|
|
194
|
+
| `explain` | `sql`, `analyze`, `timeout_ms` | Mason's EXPLAIN: each copy read with its rows and how current it is, the SQL it runs, the plan |
|
|
195
|
+
|
|
196
|
+
Its resources are `mason://help/agent` and `mason://help/mcp` (the two pages `mason help` prints). Every statement it
|
|
197
|
+
runs is in the request log under `application_name` `mason-mcp/<the client's name>` (`mason logs --requests`), a query
|
|
198
|
+
`max_rows` cut with the rows it sent; its catalog reads (the tables, a table's columns) are not, as no catalog read is.
|
|
199
|
+
A request the client cancels (`notifications/cancelled`) cancels its statement on the Postgres API.
|
|
200
|
+
|
|
201
|
+
Claude Code, in the project's folder with Mason running, the URL in the server's environment (the server's name
|
|
202
|
+
comes before `--env`, which takes every value after it):
|
|
203
|
+
|
|
204
|
+
```sh
|
|
205
|
+
mason user add claude # the URL, printed once
|
|
206
|
+
claude mcp add mason --env MASON_URL="postgres://claude:<password>@127.0.0.1:5433/shop" -- mason mcp
|
|
207
|
+
# or on its command line, where ps shows it to this machine's users and claude mcp add prints it
|
|
208
|
+
claude mcp add mason -- mason mcp --url "postgres://claude:<password>@127.0.0.1:5433/shop"
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
Claude Desktop, in `claude_desktop_config.json`. The app often starts a server without your shell's `PATH`, where
|
|
212
|
+
neither `mason` nor the `node` its first line asks for is found: name both by their absolute paths (`which node`,
|
|
213
|
+
`which mason`):
|
|
214
|
+
|
|
215
|
+
```json
|
|
216
|
+
{
|
|
217
|
+
"mcpServers": {
|
|
218
|
+
"mason": {
|
|
219
|
+
"command": "/absolute/path/to/node",
|
|
220
|
+
"args": ["/absolute/path/to/mason", "mcp"],
|
|
221
|
+
"env": { "MASON_URL": "postgres://claude:<password>@127.0.0.1:5433/shop" }
|
|
222
|
+
}
|
|
223
|
+
}
|
|
224
|
+
}
|
|
225
|
+
```
|
|
226
|
+
|
|
172
227
|
## Cleaning up
|
|
173
228
|
|
|
174
229
|
Three things exist once a project has run: your **files** (the sources file, in your app), the **copy of your data** (a
|
|
@@ -180,7 +235,9 @@ keeps a different set:
|
|
|
180
235
|
| Pause | `mason stop` | the data, the state, your files: the next `mason start` resumes in seconds |
|
|
181
236
|
| Copy the data again, or free its disk | `mason destroy` | your files, `.mason/config.json` and the users: the next `mason start` copies everything again |
|
|
182
237
|
|
|
183
|
-
Restarting the machine is a `mason stop` that nobody ran: the next `mason start` resumes.
|
|
238
|
+
Restarting the machine is a `mason stop` that nobody ran: the next `mason start` resumes. In a demo project
|
|
239
|
+
(`mason init demo`) the demo database is state too: `mason destroy` removes it with the copy, and the next `mason start`
|
|
240
|
+
seeds it again.
|
|
184
241
|
|
|
185
242
|
## Requirements
|
|
186
243
|
|
|
@@ -188,7 +245,9 @@ Restarting the machine is a `mason stop` that nobody ran: the next `mason start`
|
|
|
188
245
|
|
|
189
246
|
- Node 22.13 or later. The SQL parser (`libpg-query`), the catalog (`PGlite`) and the engine (DuckDB) ship as
|
|
190
247
|
WebAssembly or prebuilt binaries: no native build, no Python or compiler needed.
|
|
191
|
-
- Memory: the copy
|
|
248
|
+
- Memory: the copy may use half the container's memory limit by default, else half the machine's RAM
|
|
249
|
+
(`MASON_DUCKDB_MEMORY=4GB` sets it; `mason doctor` says it). Sizing a server:
|
|
250
|
+
[deploy/mason-data/README.md#memory](https://github.com/embeddable-hq/mason-data/blob/main/deploy/mason-data/README.md#memory).
|
|
192
251
|
- Postgres 13 or later. Each table needs a primary key, or a `key:` in the sources file (polling only).
|
|
193
252
|
|
|
194
253
|
## Security
|
|
@@ -163,7 +163,42 @@ export function catalogDdl(schema) {
|
|
|
163
163
|
statements.push(`COMMENT ON COLUMN ${qualified(table)}.${ident(c.name)} IS ${literal(parts.join("; "))};`);
|
|
164
164
|
}
|
|
165
165
|
}
|
|
166
|
-
return [...statements, ...constraints];
|
|
166
|
+
return [...statements, ...constraints, ...sourceTypeNames(schema)];
|
|
167
|
+
}
|
|
168
|
+
/**
|
|
169
|
+
* THE NAME OF A TYPE BY THE OID THE WIRE SENDS FOR IT. A column of a type the source made (an enum, a citext) is
|
|
170
|
+
* sent with the SOURCE's OID (src/duck/plan.ts `userTypes`: what the source's own clients cached), which in this
|
|
171
|
+
* catalog means another type or none (both count from 16384): `format_type(<that oid>, …)` answered `???`, or another
|
|
172
|
+
* table's row type. Two readers ask by two OID spaces, and one OID can be in both:
|
|
173
|
+
* - `public.mason_wire_type(oid, integer)` names an OID the WIRE sent: a source OID as its type here, every other as
|
|
174
|
+
* Postgres does. `mason mcp` names a RowDescription's columns by it; this map is the one place a wire OID is named;
|
|
175
|
+
* - the catalog's `format_type(oid, integer)` — the `version()` override's way, search_path listing pg_catalog after
|
|
176
|
+
* public — keeps the catalog's meaning for an OID the catalog has (a reader of pg_attribute.atttypid asks by those)
|
|
177
|
+
* and names any other by the wire's map. `pg_catalog.format_type`, written out, is Postgres's own.
|
|
178
|
+
*/
|
|
179
|
+
export function sourceTypeNames(schema) {
|
|
180
|
+
const names = new Map();
|
|
181
|
+
for (const t of schema.tables) {
|
|
182
|
+
if (t.rowTypeOid !== undefined)
|
|
183
|
+
names.set(t.rowTypeOid, qualified(t));
|
|
184
|
+
for (const c of t.columns) {
|
|
185
|
+
if (c.typeOid === undefined || c.spelled === undefined)
|
|
186
|
+
continue;
|
|
187
|
+
// an array column's own OID is its array type's; an element's array type is the element's spelling with []
|
|
188
|
+
names.set(c.typeOid, c.spelled);
|
|
189
|
+
if (c.typeArrayOid !== undefined && !c.spelled.endsWith("[]"))
|
|
190
|
+
names.set(c.typeArrayOid, `${c.spelled}[]`);
|
|
191
|
+
}
|
|
192
|
+
}
|
|
193
|
+
// with no type the source made, every OID the wire sends is Postgres's: the wire's name is format_type's
|
|
194
|
+
if (names.size === 0)
|
|
195
|
+
return [`CREATE OR REPLACE FUNCTION public.mason_wire_type(oid, integer) RETURNS text LANGUAGE sql STABLE AS $$ SELECT pg_catalog.format_type($1, $2) $$;`];
|
|
196
|
+
const values = [...names].map(([oid, spelled]) => `(${oid}::oid, ${literal(spelled)})`).join(", ");
|
|
197
|
+
return [`CREATE OR REPLACE FUNCTION public.mason_wire_type(oid, integer) RETURNS text LANGUAGE sql STABLE AS $$
|
|
198
|
+
SELECT coalesce((SELECT pg_catalog.format_type(pg_catalog.to_regtype(v.spelled), $2) FROM (VALUES ${values}) AS v(source, spelled) WHERE v.source = $1),
|
|
199
|
+
pg_catalog.format_type($1, $2)) $$;`,
|
|
200
|
+
`CREATE OR REPLACE FUNCTION public.format_type(oid, integer) RETURNS text LANGUAGE sql STABLE AS $$
|
|
201
|
+
SELECT CASE WHEN EXISTS (SELECT FROM pg_catalog.pg_type t WHERE t.oid = $1) THEN pg_catalog.format_type($1, $2) ELSE public.mason_wire_type($1, $2) END $$;`];
|
|
167
202
|
}
|
|
168
203
|
/**
|
|
169
204
|
* The source's column defaults, one statement each, run after the schema's DDL by
|
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
import { copyFacts } from "../../duck/explain.js";
|
|
1
2
|
import { explainedText, explainOptions } from "./explain.js";
|
|
2
3
|
import * as crypto from "node:crypto";
|
|
3
4
|
import * as net from "node:net";
|
|
@@ -781,8 +782,9 @@ class Session {
|
|
|
781
782
|
* statement's ErrorResponse goes out), and for a refused password (CONNECT), through `options.record`. A catalog
|
|
782
783
|
* read, a SET or a SHOW answered is the protocol's chatter, not recorded. Never throws: `record` buffers.
|
|
783
784
|
*/
|
|
784
|
-
recordRequest(answer, sql, refusedUser) {
|
|
785
|
-
|
|
785
|
+
recordRequest(answer, sql, refusedUser, of) {
|
|
786
|
+
// `of`: a statement recorded after its own messages (a portal dropped suspended), not the one being asked now
|
|
787
|
+
const asked = of === undefined ? this.asked : { sql: sql ?? "", method: of.method, startedAt: Date.now(), ...(of.jsonReads !== undefined ? { jsonReads: of.jsonReads } : {}) };
|
|
786
788
|
const startedAt = (!(answer instanceof PgError) ? answer.startedAt : undefined) ?? asked?.startedAt ?? Date.now();
|
|
787
789
|
// a refusal's statement is the one refuse() names (a simple Query's own statement, an Execute's portal's)
|
|
788
790
|
const stored = storedPayload(sql ?? asked?.sql ?? "");
|
|
@@ -869,12 +871,57 @@ class Session {
|
|
|
869
871
|
}
|
|
870
872
|
/** A DuckDB statement's plan (Postgres's analysis and the DuckDB SQL), kept per text, search_path, TimeZone and
|
|
871
873
|
* declared parameter types until the schema changes. Every statement DuckDB runs has one: never its text as written. */
|
|
872
|
-
async duckPlan(raw, whole, paramOids) {
|
|
874
|
+
async duckPlan(raw, whole, paramOids, under = this.vars, wireBound = false) {
|
|
873
875
|
const stmt = raw.stmt;
|
|
874
876
|
const bytes = Buffer.from(whole, "utf8");
|
|
875
877
|
const base = raw.stmt_location ?? 0;
|
|
876
878
|
const text = bytes.subarray(base, raw.stmt_len ? base + raw.stmt_len : bytes.length).toString("utf8");
|
|
877
|
-
|
|
879
|
+
const { searchPath, timeZone } = under;
|
|
880
|
+
return this.server.duckPlan(text, paramOids, searchPath, timeZone, wireBound, () => this.server.options.duck.plan((b) => this.server.catalog.run(b), text, stmt, base, paramOids, searchPath, timeZone, this.server.userTypes(), wireBound));
|
|
881
|
+
}
|
|
882
|
+
/**
|
|
883
|
+
* A ROW COUNT ON THE WIRE IS A BOUND THE PLAN AT PARSE COULD NOT SEE (docs/DESIGN.md "A LEFT lateral of a list"): an
|
|
884
|
+
* Execute's row count (JDBC's setMaxRows or fetch size, DBeaver's, a native question's) arrives after the Parse, so a
|
|
885
|
+
* LEFT lateral of a list with no LIMIT in its SQL was left DuckDB's delimiter join, which reads the whole base before
|
|
886
|
+
* the first row and hands its rows back in its own order — the first rows sent were not Postgres's. Where the plan
|
|
887
|
+
* says such a lateral was left so (PlannedSql.unboundedLateral), the statement is planned again with the bound
|
|
888
|
+
* asserted, under the search_path and TimeZone of its Parse, kept on the statement and in the plans by text. The
|
|
889
|
+
* portal's result is begun ONCE, from whichever plan its first Execute picked; an Execute after a PortalSuspended
|
|
890
|
+
* reads on from that result, never runs it again. A plan made again over a schema moved since the Parse, whose
|
|
891
|
+
* result or parameters are not the ones the Parse described, is not run: the Parse's plan is (what a client was told
|
|
892
|
+
* of the result stands). Planning again is the statement's own time: a CancelRequest landing meanwhile aborts its
|
|
893
|
+
* controller (`running`, answered 57014 as the statement begins), and a session closed meanwhile runs nothing. A
|
|
894
|
+
* plan made again that Postgres or the rewrite refuses (a StatementError, never a cancel's or a timeout's 57014) is
|
|
895
|
+
* logged and the Parse's plan runs, kept: the Parse answered, and a row count must not make it fail.
|
|
896
|
+
*/
|
|
897
|
+
async boundOnWire(statement, plan, controller) {
|
|
898
|
+
if (plan.sql.unboundedLateral !== true || statement.raw === null)
|
|
899
|
+
return plan;
|
|
900
|
+
if (statement.duckOnWire === undefined) {
|
|
901
|
+
this.running = controller;
|
|
902
|
+
let bounded;
|
|
903
|
+
try {
|
|
904
|
+
bounded = await this.duckPlan(statement.raw, statement.sql, statement.paramOids, statement.plannedUnder ?? this.vars, true);
|
|
905
|
+
}
|
|
906
|
+
catch (e) {
|
|
907
|
+
if (!(e instanceof StatementError) || e.fields.code === "57014")
|
|
908
|
+
throw e;
|
|
909
|
+
this.server.log(`session ${this.id}: a statement planned again for an Execute's row count was refused (${e.fields.code}: ${e.fields.message}): its Parse's plan runs`);
|
|
910
|
+
bounded = plan;
|
|
911
|
+
}
|
|
912
|
+
finally {
|
|
913
|
+
this.settled();
|
|
914
|
+
}
|
|
915
|
+
const shape = (p) => JSON.stringify([p.params, p.fields?.map((f) => [f.name, f.oid, f.typmod]) ?? null]);
|
|
916
|
+
statement.duckOnWire = shape(bounded) === shape(plan) ? bounded : plan;
|
|
917
|
+
if (statement.duckOnWire === plan && bounded !== plan)
|
|
918
|
+
this.server.log(`session ${this.id}: a statement planned again for an Execute's row count answers other columns than its Parse said (the schema moved): its Parse's plan runs`);
|
|
919
|
+
}
|
|
920
|
+
if (this.phase === "closed") {
|
|
921
|
+
this.server.log(`session ${this.id}: closed while its statement was planned again for an Execute's row count: it runs nothing`);
|
|
922
|
+
throw new PgError({ code: "08006", message: "the session closed while its statement was planned" });
|
|
923
|
+
}
|
|
924
|
+
return statement.duckOnWire;
|
|
878
925
|
}
|
|
879
926
|
/**
|
|
880
927
|
* AN EXPLAIN (src/api/postgres/explain.ts, src/duck/explain.ts): its options read as Postgres reads them, then the
|
|
@@ -894,7 +941,17 @@ class Session {
|
|
|
894
941
|
const found = await explainedText(text, t.query);
|
|
895
942
|
if (found === null)
|
|
896
943
|
throw new PgError({ code: "0A000", message: "Mason cannot find the statement this EXPLAIN explains in its text (a Mason gap: report it, with the statement)" });
|
|
897
|
-
|
|
944
|
+
let plan;
|
|
945
|
+
try {
|
|
946
|
+
plan = await this.server.options.duck.plan((b) => this.server.catalog.run(b), found.text, t.query, base + Buffer.byteLength(text.slice(0, found.offset)), paramOids, this.vars.searchPath, this.vars.timeZone, this.server.userTypes());
|
|
947
|
+
}
|
|
948
|
+
catch (e) {
|
|
949
|
+
// a refusal of the statement explained is placed as Postgres places it: in the whole text, past `EXPLAIN …`
|
|
950
|
+
if (!(e instanceof StatementError) || e.fields.position === undefined)
|
|
951
|
+
throw e;
|
|
952
|
+
const before = bytes.subarray(0, base).toString("utf8").length + found.offset;
|
|
953
|
+
throw new StatementError({ ...e.fields, position: e.fields.position + before }, whole);
|
|
954
|
+
}
|
|
898
955
|
return { options, plan };
|
|
899
956
|
}
|
|
900
957
|
/** an EXPLAIN of a DuckDB statement, run: its QUERY PLAN rows (ANALYZE runs the statement and sends none of its rows) */
|
|
@@ -904,7 +961,9 @@ class Session {
|
|
|
904
961
|
this.running = controller;
|
|
905
962
|
try {
|
|
906
963
|
const values = await this.duckValues(e.plan, params, clock);
|
|
907
|
-
const lines = await runner.explain(e.plan.sql, values, this.duckSession(), controller.signal, { options: e.options, fields: e.plan.fields, exact: e.plan.exact ?? [], planning: e.plan.timing ?? { analysisMs: 0, rewriteMs: 0 }
|
|
964
|
+
const lines = await runner.explain(e.plan.sql, values, this.duckSession(), controller.signal, { options: e.options, fields: e.plan.fields, exact: e.plan.exact ?? [], planning: e.plan.timing ?? { analysisMs: 0, rewriteMs: 0 },
|
|
965
|
+
// how each copy is kept current, as the sync says it to mason status, read now (src/duck/sync.ts KeptCurrent)
|
|
966
|
+
copies: copyFacts(this.server.options.duck.copies?.() ?? []) });
|
|
908
967
|
return { field: explainField(e.options, formatAt(resultFormats, 0)), rows: lines.map((l) => dataRow([Buffer.from(l)])) };
|
|
909
968
|
}
|
|
910
969
|
finally {
|
|
@@ -1054,9 +1113,20 @@ class Session {
|
|
|
1054
1113
|
}
|
|
1055
1114
|
}
|
|
1056
1115
|
}
|
|
1057
|
-
/**
|
|
1116
|
+
/**
|
|
1117
|
+
* A portal gone (Close, Sync, a Bind of its name, DISCARD, the session's end): its DuckDB result with it. One the
|
|
1118
|
+
* client stopped reading at a row count (an Execute's: JDBC's setMaxRows, `mason mcp`'s max_rows) was never read to
|
|
1119
|
+
* its end, so its row of the request log is written here, with the rows it sent: a statement answered is in the log
|
|
1120
|
+
* however far it was read.
|
|
1121
|
+
*/
|
|
1058
1122
|
dropPortal(name) {
|
|
1059
|
-
this.portals.get(name)
|
|
1123
|
+
const portal = this.portals.get(name);
|
|
1124
|
+
const c = portal?.cursor;
|
|
1125
|
+
if (portal !== undefined && c !== undefined && c.suspended === true && c.recorded !== true && c.failed === undefined) {
|
|
1126
|
+
c.recorded = true;
|
|
1127
|
+
this.recordRequest({ rows: c.sent, startedAt: c.startedAt }, portal.statement.sql, undefined, { method: "EXECUTE", jsonReads: portal.statement.duck?.jsonReads });
|
|
1128
|
+
}
|
|
1129
|
+
c?.answer.close();
|
|
1060
1130
|
this.portals.delete(name);
|
|
1061
1131
|
}
|
|
1062
1132
|
dropPortals() { for (const name of [...this.portals.keys()])
|
|
@@ -1132,6 +1202,9 @@ class Session {
|
|
|
1132
1202
|
}
|
|
1133
1203
|
if (this.skipToSync || m.type === "Flush")
|
|
1134
1204
|
return;
|
|
1205
|
+
// its statement, read as the message arrives: a portal dropped while it runs (the session ended under a cancelled
|
|
1206
|
+
// statement) leaves the refusal its statement, never ""
|
|
1207
|
+
const sql = m.type === "Parse" ? m.sql : this.sqlOf(m);
|
|
1135
1208
|
try {
|
|
1136
1209
|
switch (m.type) {
|
|
1137
1210
|
case "Parse": return this.sendAll(await this.parseMessage(m));
|
|
@@ -1142,7 +1215,6 @@ class Session {
|
|
|
1142
1215
|
}
|
|
1143
1216
|
}
|
|
1144
1217
|
catch (err) {
|
|
1145
|
-
const sql = m.type === "Parse" ? m.sql : this.sqlOf(m);
|
|
1146
1218
|
const e = asPgError(err, sql);
|
|
1147
1219
|
this.refuse(e, sql, m.type);
|
|
1148
1220
|
this.skipToSync = true;
|
|
@@ -1204,6 +1276,7 @@ class Session {
|
|
|
1204
1276
|
// Postgres's analysis says the result's columns and the parameters' types, and DuckDB's SQL is made from it
|
|
1205
1277
|
if (statement.shape.kind === "duckdb") {
|
|
1206
1278
|
statement.duck = await this.duckPlan(raw, m.sql, m.paramTypes);
|
|
1279
|
+
statement.plannedUnder = { searchPath: this.vars.searchPath, timeZone: this.vars.timeZone };
|
|
1207
1280
|
// the implicit transaction begins with the Parse, as Postgres's does (start_xact_command): its literals read it
|
|
1208
1281
|
statement.parsedNow = this.txNow ??= this.server.now();
|
|
1209
1282
|
}
|
|
@@ -1382,35 +1455,50 @@ class Session {
|
|
|
1382
1455
|
if (portal.cursor === undefined) {
|
|
1383
1456
|
if (statement.duck === undefined)
|
|
1384
1457
|
throw new PgError({ code: "XX000", message: "a DuckDB statement without its plan: report it (Mason's Postgres listener)" });
|
|
1458
|
+
// a row count on the wire bounds the rows: the plan made for it where it differs (boundOnWire)
|
|
1459
|
+
const plan = m.maxRows > 0 ? await this.boundOnWire(statement, statement.duck, controller) : statement.duck;
|
|
1385
1460
|
const exec = this.txNow ??= this.server.now();
|
|
1386
|
-
const answer = await this.duckAnswer(
|
|
1461
|
+
const answer = await this.duckAnswer(plan, portal.params, portal.resultFormats, { parse: statement.parsedNow ?? exec, exec }, controller);
|
|
1387
1462
|
portal.cursor = answer.cursor;
|
|
1388
1463
|
portal.fields = answer.fields;
|
|
1389
1464
|
}
|
|
1390
1465
|
const c = portal.cursor;
|
|
1391
1466
|
if (c.failed !== undefined)
|
|
1392
1467
|
throw c.failed;
|
|
1393
|
-
|
|
1394
|
-
|
|
1395
|
-
|
|
1396
|
-
|
|
1468
|
+
c.suspended = false;
|
|
1469
|
+
try {
|
|
1470
|
+
// every row: streamed
|
|
1471
|
+
if (m.maxRows <= 0) {
|
|
1472
|
+
try {
|
|
1473
|
+
await this.streamRest(c);
|
|
1474
|
+
}
|
|
1475
|
+
finally {
|
|
1476
|
+
c.answer.close();
|
|
1477
|
+
}
|
|
1478
|
+
c.recorded = true;
|
|
1479
|
+
this.recordRequest({ rows: c.sent, startedAt: c.startedAt });
|
|
1480
|
+
return [commandComplete(`SELECT ${c.sent}`)];
|
|
1397
1481
|
}
|
|
1398
|
-
|
|
1399
|
-
|
|
1482
|
+
// at most maxRows (JDBC's fetch size, a cursor): suspended while a row is left, read one chunk ahead to know it
|
|
1483
|
+
while (!c.done && c.pending.length <= m.maxRows)
|
|
1484
|
+
await this.fill(c);
|
|
1485
|
+
const out = c.pending.splice(0, m.maxRows);
|
|
1486
|
+
c.sent += out.length;
|
|
1487
|
+
// suspended: its row is written at its last Execute, or when it is dropped (dropPortal)
|
|
1488
|
+
if (c.pending.length > 0) {
|
|
1489
|
+
c.suspended = true;
|
|
1490
|
+
return [...out, portalSuspended()];
|
|
1400
1491
|
}
|
|
1492
|
+
c.answer.close();
|
|
1493
|
+
c.recorded = true;
|
|
1401
1494
|
this.recordRequest({ rows: c.sent, startedAt: c.startedAt });
|
|
1402
|
-
return [commandComplete(`SELECT ${c.sent}`)];
|
|
1403
|
-
}
|
|
1404
|
-
|
|
1405
|
-
|
|
1406
|
-
|
|
1407
|
-
|
|
1408
|
-
|
|
1409
|
-
if (c.pending.length > 0)
|
|
1410
|
-
return [...out, portalSuspended()];
|
|
1411
|
-
c.answer.close();
|
|
1412
|
-
this.recordRequest({ rows: c.sent, startedAt: c.startedAt });
|
|
1413
|
-
return [...out, commandComplete(`SELECT ${c.sent}`)];
|
|
1495
|
+
return [...out, commandComplete(`SELECT ${c.sent}`)];
|
|
1496
|
+
}
|
|
1497
|
+
catch (e) {
|
|
1498
|
+
// its refusal is its row (refuse)
|
|
1499
|
+
c.recorded = true;
|
|
1500
|
+
throw e;
|
|
1501
|
+
}
|
|
1414
1502
|
}
|
|
1415
1503
|
async closeMessage(m) {
|
|
1416
1504
|
if (m.kind === "S") {
|
|
@@ -1548,7 +1636,8 @@ function relayedError(frame) {
|
|
|
1548
1636
|
// the listener
|
|
1549
1637
|
class Server {
|
|
1550
1638
|
sessions = new Map();
|
|
1551
|
-
/** the DuckDB plans by text, search_path
|
|
1639
|
+
/** the DuckDB plans by text, search_path, TimeZone, declared parameter types and an Execute's bound (Session.boundOnWire), for
|
|
1640
|
+
* the schema they were made under */
|
|
1552
1641
|
duckPlans = new Map();
|
|
1553
1642
|
duckPlansOf = null;
|
|
1554
1643
|
duckPlansAt = 0;
|
|
@@ -1578,8 +1667,17 @@ class Server {
|
|
|
1578
1667
|
}
|
|
1579
1668
|
return out;
|
|
1580
1669
|
}
|
|
1581
|
-
|
|
1582
|
-
|
|
1670
|
+
/**
|
|
1671
|
+
* EVERY PLAN IS MADE OVER THE CATALOG OF THE SCHEMA IT IS KEPT UNDER: the catalog is replaced here (`schema`), not
|
|
1672
|
+
* only at a statement's classify, because a plan is also made at an Execute (Session.boundOnWire) — a re-plan that
|
|
1673
|
+
* read the schema raw analysed against the old catalog and was kept under the new one (a review finding of #23).
|
|
1674
|
+
* The ordering relied on: `schema()` calls `catalog.replace` synchronously, which queues the old instance's close on
|
|
1675
|
+
* the catalog's one chain (src/api/postgres/catalog.ts "SERIALISED") before `make` queues its analysis, so the
|
|
1676
|
+
* analysis starts the instance with the new DDL. A plan whose schema or generation moved while it was made is
|
|
1677
|
+
* returned to its caller and not kept.
|
|
1678
|
+
*/
|
|
1679
|
+
async duckPlan(text, paramOids, searchPath, timeZone, wireBound, make) {
|
|
1680
|
+
const schema = this.schema();
|
|
1583
1681
|
const generation = this.options.duck?.generation?.() ?? 0;
|
|
1584
1682
|
if (schema !== this.duckPlansOf || generation !== this.duckPlansAt) {
|
|
1585
1683
|
this.duckPlans.clear();
|
|
@@ -1587,11 +1685,14 @@ class Server {
|
|
|
1587
1685
|
this.duckPlansAt = generation;
|
|
1588
1686
|
}
|
|
1589
1687
|
// the zone too: a plan holds the values Postgres parsed its literals to, under the session's zone
|
|
1590
|
-
|
|
1688
|
+
// and the bound: an Execute's row count plans a LEFT lateral of a list otherwise (Session.boundOnWire), kept beside
|
|
1689
|
+
const key = `${searchPath}\u0000${timeZone}\u0000${paramOids.join(",")}\u0000${wireBound ? "wire" : ""}\u0000${text}`;
|
|
1591
1690
|
const hit = this.duckPlans.get(key);
|
|
1592
1691
|
if (hit !== undefined)
|
|
1593
1692
|
return hit;
|
|
1594
1693
|
const plan = await make();
|
|
1694
|
+
if (this.duckPlansOf !== schema || this.duckPlansAt !== generation)
|
|
1695
|
+
return plan;
|
|
1595
1696
|
// bounded: a tool that inlines its values sends a new text per value
|
|
1596
1697
|
if (this.duckPlans.size >= 2000)
|
|
1597
1698
|
this.duckPlans.delete(this.duckPlans.keys().next().value);
|