postgresai 0.17.0 → 0.18.0-dev.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -76,6 +76,8 @@ Password input options (in priority order):
76
76
  - `PGAI_MON_PASSWORD` environment variable
77
77
  - if not provided: a strong password is generated automatically
78
78
 
79
+ Monitoring passwords must use printable ASCII characters; non-ASCII passwords are rejected until SASLprep normalization is supported. Role creation and password resets send a SCRAM-SHA-256 verifier instead of the cleartext password, including when the server uses `password_encryption=md5`. Printed SQL redacts the verifier as well. Server statement logging can still record the verifier; treat it as sensitive.
80
+
79
81
  By default, the generated password is printed **only in interactive (TTY) mode**. In non-interactive mode, you must either provide the password explicitly, or opt-in to printing it:
80
82
  - `--print-password` (dangerous in CI logs)
81
83
 
@@ -125,6 +127,52 @@ Notes:
125
127
  - All standard options work with Supabase mode (`--verify`, `--print-sql`, `--skip-optional-permissions`, etc.)
126
128
  - When using `--verify`, the tool checks if all required setup is in place
127
129
 
130
+ ### ClickHouse Managed Postgres
131
+
132
+ Hosts ending in `.pg.clickhouse.cloud` are auto-detected; `--provider clickhouse` forces it.
133
+ The plan is the same superuser plan as self-managed Postgres (`--print-sql` shows it).
134
+
135
+ ```bash
136
+ npx postgresai prepare-db 'postgres://postgres:...@xxx.pg.clickhouse.cloud:5432/postgres?channel_binding=require'
137
+ ```
138
+
139
+ Before any grant runs, one `-- scope:` line lists exactly what the run grants the
140
+ monitoring role, derived from the steps about to run (so it shrinks under `--skip-optional-permissions`
141
+ and is not printed on `--reset-password`, which grants nothing). With `--json` it goes to stderr.
142
+
143
+ For `prepare-db`, `channel_binding=require` in a URI or conninfo string enables SCRAM-SHA-256-PLUS when the server
144
+ offers it, disables the plaintext retry that `sslmode=prefer` would otherwise do, and is rejected
145
+ together with `sslmode=disable`. The mechanism actually negotiated is not enforced by the driver.
146
+
147
+ The collector cannot do channel binding. `mon targets add`, `mon local-install --db-url`, and
148
+ `pgai mon deploy` remove `channel_binding` from the saved/box URL. With `require`, they refuse
149
+ `sslmode=disable` or `allow`; keep `verify-full` with a warning; and upgrade `require`, `prefer`,
150
+ `verify-ca` or unset to `verify-full` after testing certificate and hostname verification with
151
+ Node's default trust store (or the CA file in `sslrootcert`). Authentication or other errors after
152
+ a verified TLS handshake do not block the upgrade. If verification fails, provide
153
+ `sslrootcert=<CA file>` with `sslmode=verify-full`, or explicitly accept the downgrade by removing
154
+ `channel_binding=require`. Successful upgrades print the collector warning and
155
+ `upgraded to sslmode=verify-full`. `channel_binding=prefer` or `disable` is removed with a note.
156
+
157
+ ### ClickHouse Cloud host metrics
158
+
159
+ For a ClickHouse Managed Postgres target, `mon local-install --db-url` and `mon targets add` also set up
160
+ host metrics (CPU, memory, disk, I/O) from the ClickHouse Cloud Prometheus endpoint when these are set:
161
+
162
+ ```bash
163
+ export CLICKHOUSE_ORG_ID='<org-id>' CLICKHOUSE_KEY_ID='<key-id>' CLICKHOUSE_KEY_SECRET='<key-secret>'
164
+ npx postgresai mon local-install --db-url 'postgres://postgres_ai_mon:...@xxx.pg.clickhouse.cloud:5432/postgres?sslmode=require'
165
+ ```
166
+
167
+ Use an organization API key with the **Basic Service API Reader** role: it is the least privilege that
168
+ works (it can list Postgres services and read `/prometheus`, and cannot change anything or create keys).
169
+ The CLI does not need an Admin key.
170
+
171
+ The key secret is stored in plaintext in `host-metrics/clickhouse-<name>.secret` in the monitoring
172
+ project directory (file 0600, directory 0700), because VictoriaMetrics reads it from a file for basic
173
+ auth. It is never printed or logged. `mon targets remove <name>` deletes it. To rotate the key, re-run
174
+ `mon targets add` with the new key.
175
+
128
176
  ### Verify and password reset
129
177
 
130
178
  Verify that everything is configured as expected (no changes):
@@ -141,6 +189,160 @@ npx postgresai prepare-db postgresql://admin@host:5432/dbname --reset-password -
141
189
 
142
190
  ## Quick start
143
191
 
192
+ ### One command: `pgai mon deploy`
193
+
194
+ ```bash
195
+ pgai mon deploy 'postgresql://postgres:<password>@<host>:5432/postgres'
196
+ ```
197
+
198
+ It signs you in if needed, creates the `postgres_ai_mon` role (with an admin URL; otherwise it
199
+ prints the SQL), provisions monitoring in PostgresAI Cloud, runs the express checkup (the checks
200
+ of `pgai checkup`, as `postgres_ai_mon`) while the box starts and prints its findings (in JSON:
201
+ `checkup`; skipped where PostgresAI keeps the role's password: no `checkup` key, no first report),
202
+ waits, and prints two links: the project's Health Matrix in the console
203
+ (`<console>/<org>/projects/<project>/health`, the next step) and Grafana, which signs you in
204
+ with your PostgresAI account (`https://<box>/login/generic_oauth?redirectTo=%2F`). Safe to re-run; a `--clickhouse-key` given on a
205
+ re-run is checked, and a rejected one is `action_required`. For ClickHouse Managed Postgres add `--clickhouse-key <key-id>:<key-secret>`
206
+ (a Basic Service API Reader key) for CPU, memory and disk. `--self-hosted` runs the stack on this
207
+ machine instead. When stdout is not a terminal the result is JSON with `status`, `health_url`,
208
+ `dashboard_url` and `next`. Then: `pgai mon instances list`, `pgai mon instances watch <id-or-name>`, `pgai mon instances delete <id-or-name>`,
209
+ and `pgai mon instances set <id-or-name> --vcpus <n>` to add or change the database server's vCPUs later (an org admin; AAS starts within a minute).
210
+
211
+ `pgai mon deploy` and `pgai dblab deploy` share their options and results: the database URL as the
212
+ argument (or `--db-url`), `--name` (default: from the database), `--location` (a Hetzner location;
213
+ default: any with capacity), `--wait <minutes>` / `--no-wait`, `--yes` (never prompt), `--json`, the
214
+ `status` words and exit codes below, and `instances list | watch | delete`, which take an id or a name
215
+ (a name two instances have is refused, with their ids). `watch` is also `status`.
216
+
217
+ With JSON output (`--json`, or stdout not a terminal) stderr is the progress stream: one JSON
218
+ object a line, each with `event`. Besides the steps, any other text is
219
+ `{"event":"log","level":"error"|"warn"|"info"|"debug","message":...}`: an error (also a missing
220
+ argument or a bad option; an unknown option is named without its value), a warning (`warn`: a line
221
+ with `Warning:`, also after a `[F001] ` prefix), the `--debug` request log (`debug`), and with
222
+ `--self-hosted` each line of `mon local-install`
223
+ (`"source":"mon local-install"`; `info` from its stdout, `error` from its stderr; the Grafana and
224
+ VictoriaMetrics logins it prints at its end are masked: `pgai mon show-grafana-credentials` shows them).
225
+
226
+ `pgai init` is the same for a person at a terminal: it signs in, asks for the database URL (and,
227
+ for ClickHouse, the API key; neither is shown as typed), then runs `pgai mon deploy`. Without a
228
+ terminal, or with `--json`, it only points to `pgai mon deploy`.
229
+
230
+ | Option | |
231
+ |---|---|
232
+ | `--provider <provider>` | `clickhouse`, `rds`, `supabase` or `self-managed`; default: detected from the host |
233
+ | `--clickhouse-key <key-id>:<key-secret>` | ClickHouse Cloud API key, for CPU, memory and disk |
234
+ | `--self-hosted` | run the stack on this machine (`mon local-install`) instead of PostgresAI Cloud |
235
+ | `--reset-password` | `postgres_ai_mon` exists and its password is lost: set a new one (see below) |
236
+ | `--db-url <url>` | the database URL, instead of the argument |
237
+ | `--name <name>` | the instance's name (default: `host[:port]/database`); ignored, with a note, for a database already monitored |
238
+ | `--location <location>` | Hetzner location `fsn1`, `nbg1` or `hel1` (default: any with capacity) |
239
+ | `--vcpus <n>` | `mon deploy` only: the database server's vCPUs. Without it there is no AAS (database load) data; at a terminal you are asked. On a database already monitored, it sets missing vCPUs (a set value is changed with `mon instances set`) |
240
+ | `--wait <minutes>` | how long to wait for the monitoring box; `0` does not wait (default 20) |
241
+ | `--no-wait` | the same as `--wait 0` |
242
+ | `-y, --yes` | never prompt; accept a billed box's price (`mon deploy` does not start the browser sign-in; `mon instances delete` does not ask) |
243
+ | `--coupon <code>` | promotion code for the organization's monitoring subscription |
244
+ | `--json` | JSON output, also on a terminal |
245
+
246
+ Without `--yes`, a non-interactive run for a billed box returns `action_required` with the price
247
+ and asks you to re-run with `--yes` to accept it.
248
+
249
+ | Environment | |
250
+ |---|---|
251
+ | `PGAI_API_KEY` | the API key, instead of signing in (agents, CI) |
252
+ | `PGAI_MON_PASSWORD` | the password of `postgres_ai_mon`. For a new role it is set; for an existing role it is checked by logging in, and never changed. Without it a new role gets a generated password. Not needed for another database on a server PostgresAI already monitors for the organization, with `sslmode=require` or `verify-*` in the URL (see below) |
253
+ | `PGPASSWORD` | the password for a URL without one. With a `postgres_ai_mon` URL it is the password the monitoring box gets, once the server has checked it |
254
+ | `PGSSLMODE` | the `sslmode` for a URL without one (`sslmode` in the URL wins, as in libpq) |
255
+ | `CLICKHOUSE_KEY_ID` + `CLICKHOUSE_KEY_SECRET` | instead of `--clickhouse-key`; read for ClickHouse hosts only |
256
+
257
+ `postgres_ai_mon` is one role for the whole server, so `mon deploy` never changes the password of an
258
+ existing one. If the role exists and you do not have its password (a deployment after
259
+ `pgai mon instances delete`, say), `pgai mon deploy <admin-url> --reset-password` sets a new one
260
+ (`PGAI_MON_PASSWORD`, else generated). It is refused while another database on the same server (the same host name, in this
261
+ organization) is monitored with the role, while a box's server is not known (it was created before
262
+ the platform kept it, under a name that does not say it), with `--self-hosted`, and for a database
263
+ already monitored. After the price is accepted, the run takes the platform's lock on the server (its host
264
+ name and port, in this organization) until the box is requested. A second `--reset-password` for
265
+ the same server exits 3 meanwhile. A run that stopped, or whose box request got no answer, frees the
266
+ server 15 minutes after it took the lock. A platform without the lock answers "set
267
+ `PGAI_MON_PASSWORD`" instead. Anything else that logs in as `postgres_ai_mon` (another organization,
268
+ another host name for the same server, a deploy without `--reset-password` running at the same
269
+ time) needs the new password.
270
+
271
+ Another database on a server PostgresAI already monitors for the organization needs neither: with
272
+ `sslmode=require` or `verify-*` in the URL, the role keeps its password, the box's URL carries
273
+ none, and PostgresAI fills in the one it kept from the first `mon deploy` (only over TLS, each query
274
+ parameter once, and only while a box on that server is not deleted, being deleted, or failed to
275
+ delete). The express checkup is skipped then: it runs only as `postgres_ai_mon`, never as the
276
+ admin. The full checkup follows on the box. On a server without TLS the kept password cannot be
277
+ used: set `PGAI_MON_PASSWORD`, or turn TLS on; if nobody has the password, delete the server's
278
+ other instances, then `--reset-password` with `PGAI_MON_PASSWORD` set to a new one, and deploy
279
+ them again with it (`mon deploy` names them).
280
+
281
+ When preparing an existing role with a supplied password (an admin URL with `PGAI_MON_PASSWORD`,
282
+ or the role's own URL), `mon deploy` also tries a random password as `postgres_ai_mon` to learn
283
+ whether the server checks passwords for this host. Successful PostgresAI Cloud preparation makes
284
+ two attempts: one in the check before the price, and one in preparation. Stopping after the check
285
+ (for example, without accepting the price) makes one. Successful `--self-hosted` preparation
286
+ makes one. Creating or resetting the role with an admin URL, using the platform's stored password,
287
+ or returning an existing box's status makes no random-password attempts. A rejected supplied
288
+ password stops before the random-password probe; that rejection is itself a failed login.
289
+ On a server that checks passwords, each random-password attempt adds one
290
+ `password authentication failed for user "postgres_ai_mon"` line to the server log and counts
291
+ toward failed-login policies (credcheck, fail2ban). On a server that does not (`trust`), `next`
292
+ says that the password was not checked; a password from `PGPASSWORD` is then not used: put it
293
+ in the URL.
294
+
295
+ A role that is not a superuser creates `postgres_ai_mon` only if it can run the whole
296
+ preparation: `CREATEROLE`, `CREATE` on the database, `pg_stat_statements` already installed, and
297
+ on PostgreSQL 16+ `ADMIN OPTION` on `pg_monitor` and `pg_read_all_stats`. Otherwise the SQL is
298
+ printed and nothing is created.
299
+
300
+ The host and the port are the ones in the URL: `host` or `port` in the query string is refused.
301
+ Of the query string, the monitoring box gets `sslmode` and `application_name`; `channel_binding`
302
+ is removed using the collector rule above, before a price is quoted or anything is changed.
303
+ Certificate files (`sslrootcert`, `sslcert`, `sslkey`) are used for the connections from this
304
+ machine only; with `sslmode=verify-ca` or `verify-full` and a private CA in `sslrootcert`, `next`
305
+ says that the box has no copy of that CA.
306
+
307
+ A run that fails does not leave such a role behind. The ClickHouse key and the service state are
308
+ checked before the database is touched. If the platform refuses the launch, or a later step of the
309
+ preparation fails, a role that this run created with a generated password is dropped again, so the
310
+ same command can be run again. (After a platform error that is not a refusal, such as a 5xx or a
311
+ timeout, the role stays: a monitoring box may be starting with it.)
312
+
313
+ An admin URL without a database connects to `PGDATABASE`, or else to the database named like the
314
+ user; the name of the connected database (`<host>[:<port>]/<database>`) uses that database.
315
+
316
+ `pgai mon deploy`, `pgai mon instances watch` and `pgai mon instances list` use the same words, and so
317
+ do `pgai dblab deploy` and `pgai dblab instances`:
318
+
319
+ | `status` | Meaning | Exit code |
320
+ |---|---|---|
321
+ | `ready` | monitoring works; `health_url` (when the project is found) and `dashboard_url` are set | 0 |
322
+ | `in_progress` | the monitoring box is being set up (also when `--wait` ran out); follow it with `pgai mon instances watch <id>` | 0 |
323
+ | `deleting` | a delete is in progress, also the result of `pgai mon instances delete` | 0 |
324
+ | `deleted` | it is gone | 0 |
325
+ | `inactive` | deactivated (billing) | 1 |
326
+ | `action_required` | do what `next` says, then re-run | 3 |
327
+ | `failed` | `next` has the reason | 1 |
328
+
329
+ `pgai init` exits with 130 when cancelled at a prompt (Ctrl-C, Ctrl-D).
330
+
331
+ `--self-hosted` starts `mon local-install` with the monitoring URL, the API key and the ClickHouse
332
+ key in its environment (`PGAI_DB_URL`, `PGAI_API_KEY`, `CLICKHOUSE_*`), not in its arguments.
333
+ `mon local-install` reads `PGAI_DB_URL` like `--db-url`, and `PGAI_API_KEY` only together with
334
+ `PGAI_DB_URL`: an exported `PGAI_API_KEY` alone does not change a plain or `--demo` install.
335
+ `mon targets add [name]` reads `PGAI_DB_URL` when argv has no `postgres://` / `postgresql://` URL
336
+ (a URL in argv wins); with it set, a lone argument other than a plain name (ASCII letters,
337
+ digits, `.`, `_`, `=`, `-`, with no `password=` / `pwd=`) is refused. Its default name is
338
+ `<host>-<database>`. Percent-encode the user name and the password one at a time (all but ASCII
339
+ letters, digits, `-`, `.`, `_` and `~`), and an `@` in the database name or the query. A user name
340
+ or password that pgx would misread or refuse is refused; an `@` after the host is refused, since it
341
+ cannot be told apart from one in a password; so is a control character anywhere in the URL.
342
+ Under sudo, pass `PGAI_DB_URL` on stdin, not with `--preserve-env`: sudo logs the variables it keeps.
343
+ Where sudoers enables I/O logging (`log_input`), sudo records stdin too: read the URL from a file
344
+ of mode 0600 inside the root shell instead.
345
+
144
346
  ### Authentication
145
347
 
146
348
  Authenticate via browser to obtain API key:
@@ -195,6 +397,18 @@ unaffected and need no flag.
195
397
  The MCP server follows the same rule — under a global token, `org_id` becomes a
196
398
  required tool argument instead of falling back to a stored default.
197
399
 
400
+ ### DBLab in PostgresAI cloud: `pgai dblab deploy`
401
+
402
+ ```bash
403
+ PGPASSWORD=<password> pgai dblab deploy 'postgresql://<user>@<host>:5432/<db>' --ssh-key <key name or id>
404
+ ```
405
+
406
+ The same options as `pgai mon deploy` (`--db-url`, `--name`, `--location`, `--wait`, `--no-wait`, `--yes`, `--json`),
407
+ plus `--size S|M|L` (default `S`), `--disk <GiB>` (default 50) and `--ssh-key` (required: one of the organization's
408
+ SSH keys, for reaching the server and its clones). `--name` defaults to the database's name; `--wait` to 120 minutes.
409
+ It shows the estimated monthly cost before asking; with no payment method it stops with the billing page (exit 3).
410
+ `pgai dblab instances list | watch (status) | delete` take an id or a name.
411
+
198
412
  ### Start monitoring
199
413
 
200
414
  Start monitoring with demo database:
@@ -256,6 +470,11 @@ When `--instance-id <uuid>` (or `PGAI_INSTANCE_ID`) is set, `local-install` forw
256
470
  ```bash
257
471
  postgresai mon targets list # List databases to monitor
258
472
  postgresai mon targets add <conn-string> <name> # Add database to monitor
473
+ # Same, URL kept out of argv and out of the sudo log (read from stdin, not exported),
474
+ # unless sudoers enables I/O logging (log_input), which records stdin:
475
+ printf '%s\n' "$URL" | sudo sh -c 'IFS= read -r PGAI_DB_URL; export PGAI_DB_URL; exec postgresai mon targets add <name>'
476
+ # With log_input, read it from a file of mode 0600 inside the root shell:
477
+ sudo sh -c 'IFS= read -r PGAI_DB_URL < /path/to/db-url; export PGAI_DB_URL; exec postgresai mon targets add <name>'
259
478
  postgresai mon targets remove <name> # Remove monitoring target
260
479
  postgresai mon targets test <name> # Test target connectivity
261
480
  ```
@@ -336,6 +555,7 @@ Tools exposed:
336
555
  - `update_issue_comment`: update an existing comment (args: `{ comment_id, org_id, content?, attachments?, debug? }`).
337
556
  - `upload_file`: upload a local file and return the storage URL plus a ready-to-paste markdown link (args: `{ path, org_id, debug? }`).
338
557
  - `download_file`: download a file from storage (args: `{ url, org_id, output_path?, debug? }`).
558
+ - `deploy_monitoring`: `pgai mon deploy` as a tool (args: `{ database_url, org_id, provider?, clickhouse_key?, yes?, coupon?, debug? }`). Returns the same JSON (`status`: `ready`, `in_progress`, `deleting`, `deleted`, `inactive`, `action_required` or `failed`; `health_url`; `dashboard_url`; `next`). For a billed box, the tool first returns the price with `action_required`; an explicit call with `yes: true` accepts it. `coupon` supplies a promotion code. It does not wait for the monitoring box: call it again to see the status. `database_url` must carry its password, and only the query parameters `sslmode`, `channel_binding`, `application_name` (and `password`); `PGAI_MON_PASSWORD`, `PGPASSWORD` and `CLICKHOUSE_KEY_ID` + `CLICKHOUSE_KEY_SECRET` of the server process are not used, a TLS failure is not retried in plaintext, and `--self-hosted` is CLI-only. For another database on a server PostgresAI already monitors for the organization, an admin `database_url` with `sslmode=require` (or `verify-*`) is enough: PostgresAI fills in the password of `postgres_ai_mon` it keeps.
339
559
 
340
560
  #### `attachments` parameter (issue/comment tools)
341
561
 
@@ -592,7 +812,7 @@ PGPASSWORD=... postgresai checkup \
592
812
  `postgresai checkup <ID> <conn>`); stdout is then a one-key object.
593
813
  - **stderr** carries only human-readable diagnostics (progress, warnings,
594
814
  errors). It never contains report JSON — machine consumers should read stdout
595
- only. Do not parse stderr as JSON.
815
+ only. Do not parse stderr of `checkup` as JSON (that of `pgai mon deploy` is JSON lines, see above).
596
816
  - **Exit codes**: `0` on success; non-zero when the run fails (connection
597
817
  failure, insufficient permissions, an unknown/unavailable check ID, or a
598
818
  failing check). On a non-zero exit, no JSON report object is written to