postgresai 0.17.0-dev.1 → 0.17.0-dev.3

package/README.md

@@ -76,6 +76,8 @@ Password input options (in priority order):
 - `PGAI_MON_PASSWORD` environment variable
 - if not provided: a strong password is generated automatically
 
+Monitoring passwords must use printable ASCII characters; non-ASCII passwords are rejected until SASLprep normalization is supported. Role creation and password resets send a SCRAM-SHA-256 verifier instead of the cleartext password, including when the server uses `password_encryption=md5`. Printed SQL redacts the verifier as well. Server statement logging can still record the verifier; treat it as sensitive.
+
 By default, the generated password is printed **only in interactive (TTY) mode**. In non-interactive mode, you must either provide the password explicitly, or opt-in to printing it:
 - `--print-password` (dangerous in CI logs)
 
@@ -125,6 +127,42 @@ Notes:
 - All standard options work with Supabase mode (`--verify`, `--print-sql`, `--skip-optional-permissions`, etc.)
 - When using `--verify`, the tool checks if all required setup is in place
 
+### ClickHouse Managed Postgres
+
+Hosts ending in `.pg.clickhouse.cloud` are auto-detected; `--provider clickhouse` forces it.
+The plan is the same superuser plan as self-managed Postgres (`--print-sql` shows it).
+
+```bash
+npx postgresai prepare-db 'postgres://postgres:...@xxx.pg.clickhouse.cloud:5432/postgres?channel_binding=require'
+```
+
+Before any grant runs, one `-- scope:` line lists exactly what the run grants the
+monitoring role, derived from the steps about to run (so it shrinks under `--skip-optional-permissions`
+and is not printed on `--reset-password`, which grants nothing). With `--json` it goes to stderr.
+
+`channel_binding=require` in a URI or conninfo string enables SCRAM-SHA-256-PLUS when the server
+offers it, disables the plaintext retry that `sslmode=prefer` would otherwise do, and is rejected
+together with `sslmode=disable`. The mechanism actually negotiated is not enforced by the driver.
+
+### ClickHouse Cloud host metrics
+
+For a ClickHouse Managed Postgres target, `mon local-install --db-url` and `mon targets add` also set up
+host metrics (CPU, memory, disk, I/O) from the ClickHouse Cloud Prometheus endpoint when these are set:
+
+```bash
+export CLICKHOUSE_ORG_ID='<org-id>' CLICKHOUSE_KEY_ID='<key-id>' CLICKHOUSE_KEY_SECRET='<key-secret>'
+npx postgresai mon local-install --db-url 'postgres://postgres_ai_mon:...@xxx.pg.clickhouse.cloud:5432/postgres?sslmode=require'
+```
+
+Use an organization API key with the **Basic Service API Reader** role: it is the least privilege that
+works (it can list Postgres services and read `/prometheus`, and cannot change anything or create keys).
+The CLI does not need an Admin key.
+
+The key secret is stored in plaintext in `host-metrics/clickhouse-<name>.secret` in the monitoring
+project directory (file 0600, directory 0700), because VictoriaMetrics reads it from a file for basic
+auth. It is never printed or logged. `mon targets remove <name>` deletes it. To rotate the key, re-run
+`mon targets add` with the new key.
+
 ### Verify and password reset
 
 Verify that everything is configured as expected (no changes):
@@ -141,6 +179,90 @@ npx postgresai prepare-db postgresql://admin@host:5432/dbname --reset-password -
 
 ## Quick start
 
+### One command: `pgai connect`
+
+```bash
+pgai connect 'postgresql://postgres:<password>@<host>:5432/postgres'
+```
+
+It signs you in if needed, creates the `postgres_ai_mon` role (with an admin URL; otherwise it
+prints the SQL), provisions monitoring in PostgresAI Cloud, waits, and prints the dashboard URL.
+Safe to re-run. For ClickHouse Managed Postgres add `--clickhouse-key <key-id>:<key-secret>`
+(a Basic Service API Reader key) for CPU, memory and disk. `--self-hosted` runs the stack on this
+machine instead. When stdout is not a terminal the result is JSON with `status`, `dashboard_url`
+and `next`. Then: `pgai databases`, `pgai status <name>`, `pgai disconnect <name>`.
+
+`pgai init` is the same for a person at a terminal: it signs in, asks for the database URL (and,
+for ClickHouse, the API key, which is not shown as typed), then runs `pgai connect`. Without a
+terminal, or with `--json`, it only points to `pgai connect`.
+
+| Option | |
+|---|---|
+| `--provider <provider>` | `clickhouse`, `rds`, `supabase` or `self-managed`; default: detected from the host |
+| `--clickhouse-key <key-id>:<key-secret>` | ClickHouse Cloud API key, for CPU, memory and disk |
+| `--self-hosted` | run the stack on this machine (`mon local-install`) instead of PostgresAI Cloud |
+| `--wait <minutes>` | how long to wait for the monitoring box; `0` does not wait (default 20) |
+| `-y, --yes` | never prompt (`connect` does not start the browser sign-in; `disconnect` does not ask) |
+| `--json` | JSON output, also on a terminal |
+
+| Environment | |
+|---|---|
+| `PGAI_API_KEY` | the API key, instead of signing in (agents, CI) |
+| `PGAI_MON_PASSWORD` | the password of `postgres_ai_mon`. For a new role it is set; for an existing role it is checked by logging in, and never changed. Without it a new role gets a generated password |
+| `PGPASSWORD` | the password for a URL without one. With a `postgres_ai_mon` URL it is the password the monitoring box gets, once the server has checked it |
+| `PGSSLMODE` | the `sslmode` for a URL without one (`sslmode` in the URL wins, as in libpq) |
+| `CLICKHOUSE_KEY_ID` + `CLICKHOUSE_KEY_SECRET` | instead of `--clickhouse-key`; read for ClickHouse hosts only |
+
+`postgres_ai_mon` is one role for the whole server, so `connect` never changes the password of an
+existing one. If the role exists and you do not have its password, change it explicitly and
+update every monitoring box that uses it:
+`pgai prepare-db <admin-url> --reset-password --password <new-password>`, then
+`PGAI_MON_PASSWORD=<new-password> pgai connect <admin-url>`.
+
+When the role exists (an admin URL with `PGAI_MON_PASSWORD`, or the role's own URL), `connect`
+also logs in once as `postgres_ai_mon` with a random password, to learn whether the server checks
+passwords for this host at all. On a server that does, this is one
+`password authentication failed for user "postgres_ai_mon"` line in the server log per run, and it
+counts toward failed-login policies (credcheck, fail2ban). On a server that does not (`trust`),
+`next` says that the password was not checked; a password from `PGPASSWORD` is then not used:
+put it in the URL.
+
+A role that is not a superuser creates `postgres_ai_mon` only if it can run the whole
+preparation: `CREATEROLE`, `CREATE` on the database, `pg_stat_statements` already installed, and
+on PostgreSQL 16+ `ADMIN OPTION` on `pg_monitor` and `pg_read_all_stats`. Otherwise the SQL is
+printed and nothing is created.
+
+The host and the port are the ones in the URL: `host` or `port` in the query string is refused.
+Of the query string, the monitoring box gets `sslmode`, `channel_binding` and `application_name`.
+Certificate files (`sslrootcert`, `sslcert`, `sslkey`) are used for the connections from this
+machine only; with `sslmode=verify-ca` or `verify-full` and a private CA in `sslrootcert`, `next`
+says that the box has no copy of that CA.
+
+A run that fails does not leave such a role behind. The ClickHouse key and the service state are
+checked before the database is touched. If the platform refuses the launch, or a later step of the
+preparation fails, a role that this run created with a generated password is dropped again, so the
+same command can be run again. (After a platform error that is not a refusal, such as a 5xx or a
+timeout, the role stays: a monitoring box may be starting with it.)
+
+An admin URL without a database connects to `PGDATABASE`, or else to the database named like the
+user; the name of the connected database (`<host>[:<port>]/<database>`) uses that database.
+
+| `status` | Meaning | Exit code |
+|---|---|---|
+| `connected` | monitoring works; `dashboard_url` is set | 0 |
+| `provisioning` | the monitoring box is being set up; check with `pgai status <name>` | 0 |
+| `disconnecting` | a disconnect is in progress (`pgai status`) | 0 |
+| `disconnected` | the result of `pgai disconnect` | 0 |
+| `action_required` | do what `next` says, then re-run | 3 |
+| `failed` | `next` has the reason | 1 |
+
+`pgai init` exits with 130 when cancelled at a prompt (Ctrl-C, Ctrl-D).
+
+`--self-hosted` starts `mon local-install` with the monitoring URL, the API key and the ClickHouse
+key in its environment (`PGAI_DB_URL`, `PGAI_API_KEY`, `CLICKHOUSE_*`), not in its arguments.
+`mon local-install` reads `PGAI_DB_URL` like `--db-url`, and `PGAI_API_KEY` only together with
+`PGAI_DB_URL`: an exported `PGAI_API_KEY` alone does not change a plain or `--demo` install.
+
 ### Authentication
 
 Authenticate via browser to obtain API key:
@@ -250,7 +372,7 @@ postgresai mon health [--wait <sec>]  # Check monitoring services health
 
 When `--instance-id <uuid>` (or `PGAI_INSTANCE_ID`) is set, `local-install` forwards the id to the platform, which **adopts** the already-provisioned monitoring instance instead of self-registering a duplicate under an auto-created `postgres-ai-monitoring` project. The CLI then persists the adopted instance's real project to `.pgwatch-config`, so checkup reports upload alongside the rest of that instance's health data. Adoption is awaited (with one automatic retry); if it fails, the CLI warns and reports fall back to the default project until you re-run `local-install`. Without the flag, the legacy self-registration path is byte-for-byte unchanged.
 
-`local-install` writes `.env` in the monitoring directory. It preserves existing `REPLICATOR_PASSWORD` and `VM_AUTH_*` values or generates new random ones when missing; `VM_AUTH_USERNAME` defaults to `vmauth` when absent. The replication password is used by the demo PostgreSQL standby replication user, and the VM auth credentials are required before Docker Compose can provision Grafana datasources. If you run `docker compose` directly or maintain `.env` yourself, set both VM auth values before upgrading. For rotation, run `VM_AUTH_PASSWORD="$(openssl rand -base64 18)" ./scripts/rotate-vm-auth.sh` from the monitoring directory so `.env`, `sink-prometheus`, and `grafana` update together.
+`local-install` writes `.env` in the monitoring directory. It preserves existing `REPLICATOR_PASSWORD`, `VM_AUTH_*` and the VictoriaMetrics admin-endpoint keys (`VM_DELETE_AUTH_KEY`, `VM_SNAPSHOT_AUTH_KEY`, `VM_FORCE_MERGE_AUTH_KEY`, `VM_PPROF_AUTH_KEY`, added in 0.17) or generates new random ones when missing. The admin keys live on the `sink-prometheus` command line, so `mon update` and `mon update-config` also recreate that container to apply them; `mon restart` (`docker compose restart`) does not; `VM_AUTH_USERNAME` defaults to `vmauth` when absent. The replication password is used by the demo PostgreSQL standby replication user, and the VM auth credentials are required before Docker Compose can provision Grafana datasources. If you run `docker compose` directly or maintain `.env` yourself, set both VM auth values before upgrading. For rotation, run `VM_AUTH_PASSWORD="$(openssl rand -base64 18)" ./scripts/rotate-vm-auth.sh` from the monitoring directory so `.env`, `sink-prometheus`, and `grafana` update together.
 
 #### Monitoring target databases (`mon targets` subgroup)
 ```bash
@@ -271,6 +393,35 @@ postgresai mon check                          # System readiness check
 postgresai mon shell <service>                # Open shell to monitoring service
 ```
 
+### PromQL queries (`promql`)
+
+Ask a monitoring instance a PromQL question through the platform. The platform
+never connects to the instance: it queues the query as a job, the instance picks
+it up on its next poll, runs it against its own metric store, and posts the
+answer back.
+
+```bash
+# instant query
+pgai promql 'up' --instance <instance-uuid>
+
+# range query
+pgai promql 'sum(rate(pgwatch_db_stats_xact_commit[5m]))' \
+  --instance <instance-uuid> \
+  --range --start 2026-09-21T00:00:00Z --end 2026-09-21T01:00:00Z --step 60
+
+# on the box itself, --instance is read from .pgwatch-config
+pgai promql 'up'
+```
+
+`--at <time>` evaluates an instant query at a given moment (ignored with
+`--range`). `--json` prints every point instead of the first and last per series.
+The wait is sized from the instance's poll pacing; `--timeout` overrides it. A
+result too large for the byte budget is trimmed and flagged `truncated`.
+
+Requires platform-all !809
+(https://gitlab.com/postgres-ai/platform-all/-/merge_requests/809); until it is
+deployed the command returns a PGRST202 404.
+
 ### MCP server (`mcp` group)
 
 ```bash
@@ -299,14 +450,15 @@ required** — the server will not assume an organization. Call `orgs_list` (or
 run `pgai orgs`) to discover the available ids.
 
 Tools exposed:
-- `list_issues`: returns the same JSON as `postgresai issues list` (args: `{ org_id, status?, hidden_only?, limit?, offset?, debug? }`).
+- `list_issues`: returns the same JSON as `postgresai issues list` (args: `{ org_id, status?, hidden_only?, limit?, offset?, debug? }`). Unlike the CLI, omitting `status` returns open and closed issues; pass `status: "open"` to match the CLI default.
 - `view_issue`: view a single issue with its comments (args: `{ issue_id, org_id, debug? }`).
-- `create_issue`: create a new issue (args: `{ title, description?, org_id, attachments?, debug? }`).
-- `update_issue`: update title/description/status/labels (args: `{ issue_id, org_id, title?, description?, status?, labels?, attachments?, debug? }`).
+- `create_issue`: create a new issue (args: `{ title, description?, org_id, project_id?, labels?, attachments?, is_hidden?, debug? }`).
+- `update_issue`: update title/description/status/labels/is_hidden (args: `{ issue_id, org_id, title?, description?, status?, labels?, attachments?, is_hidden?, debug? }`).
 - `post_issue_comment`: post a comment (args: `{ issue_id, org_id, content?, parent_comment_id?, attachments?, debug? }`).
 - `update_issue_comment`: update an existing comment (args: `{ comment_id, org_id, content?, attachments?, debug? }`).
 - `upload_file`: upload a local file and return the storage URL plus a ready-to-paste markdown link (args: `{ path, org_id, debug? }`).
 - `download_file`: download a file from storage (args: `{ url, org_id, output_path?, debug? }`).
+- `connect_database`: `pgai connect` as a tool (args: `{ database_url, org_id, provider?, clickhouse_key?, debug? }`). Returns the same JSON (`status`: `connected`, `provisioning`, `disconnecting`, `action_required` or `failed`; `dashboard_url`; `next`). It does not wait for the monitoring box: call it again to see the status. `database_url` must carry its password, and only the query parameters `sslmode`, `channel_binding`, `application_name` (and `password`); `PGAI_MON_PASSWORD`, `PGPASSWORD` and `CLICKHOUSE_KEY_ID` + `CLICKHOUSE_KEY_SECRET` of the server process are not used, a TLS failure is not retried in plaintext, and `--self-hosted` is CLI-only.
 
 #### `attachments` parameter (issue/comment tools)
 
@@ -350,11 +502,14 @@ sensitive.
 ### Issues management (`issues` group)
 
 ```bash
-postgresai issues list                                       # List issues (shows: id, title, status, created_at; is_hidden only when set)
+postgresai issues list                                       # List OPEN issues (shows: id, title, status, created_at; is_hidden only when set)
+postgresai issues list --status closed                       # Only closed issues (--status all for both)
 postgresai issues list --hidden-only                         # Only hidden issues (PostgresAI staff)
 postgresai issues view <issueId>                             # View issue details and comments
-postgresai issues create --org-id <id> --title <t>           # Create a new issue
+postgresai issues create --org-id <id> <title>               # Create a new issue
+postgresai issues create --org-id <id> <title> --hidden      # Create a hidden issue (PostgresAI staff)
 postgresai issues update <issueId> [--title ... --status ...]# Update an existing issue
+postgresai issues update <issueId> --hidden|--no-hidden      # Hide / unhide an issue (PostgresAI staff)
 postgresai issues post-comment <issueId> <content>           # Post a comment to an issue
 postgresai issues update-comment <commentId> <content>       # Update an existing comment
 postgresai issues files upload <path>                        # Upload a file, print URL + markdown
@@ -370,7 +525,11 @@ postgresai issues files download <url> [-o <path>]           # Download a file
 Hidden issues are staff-internal. `issues list` and `issues view` mark them
 with `is_hidden: true`; the key is omitted entirely otherwise, so ordinary
 issues look exactly as they always have. `--hidden-only` lists just the hidden
-ones, filtered server-side.
+ones, filtered server-side. `issues create --hidden` creates one, and
+`issues update --hidden` / `--no-hidden` hides or unhides an existing issue
+(MCP: `is_hidden` on `create_issue` / `update_issue`). The CLI does no staff
+check of its own: the platform refuses a visibility change for a non-staff
+credential with a plain error, and the CLI prints it and exits non-zero.
 
 Staff access is granted per credential, not per person, and a credential that
 does not qualify simply sees nothing — `--hidden-only` returns an empty list