datagrok-tools 6.5.7 → 6.5.8

package/CHANGELOG.md

@@ -1,5 +1,36 @@
 # Datagrok-tools changelog
 
+## 6.5.8 (2026-09-10)
+
+* `grok s` — correctness pass from the CLI audit: every server failure now exits 1 with one line on stderr (a 200 carrying an `ApiError` — missing entity, duplicate login or group name, unknown function — used to print as data with exit 0); `--limit`/`--offset` are honoured for users, groups, connections and functions (their public routes ignore paging, so `list` and the new `count` verb go through the internal routers); `files list` works (the public files route only downloads; listing goes through the connection's file route) and prints `path, kind, size`; `tables list|get|delete` no longer throw, and `tables download|get|delete` accept the bare table name, the full `Project:Table` name or the id; `describe <entity|type>` works (registry record plus the fields of a live sample; it called a nonexistent endpoint before); positional `functions run 'F(1, 2)'` maps the values onto the function's inputs instead of silently sending `{"0": 1}` (which ran the function with nulls); `raw` takes API-relative paths (a leading `/api` is accepted), sends a body from `--json <file>` or `--data '<json>'`, and fails on non-2xx — so it, and `healthcheck`, now work against a bare-Datlas `--host` too; `shares list` accepts a name and shows the grants inherited through project links; `groups` name resolution prefers an exact match (`admin` no longer collides with `Administrators`) and `--user` works on `list-members`/`list-memberships`; `connections delete` of an unknown id fails instead of reporting success; a bad `--host` URL or alias reports the reason without the help dump. `users delete` refuses with a pointer at `users block`: the platform has no user deletion, and removing the entity record leaves the login unusable. `functions delete` covers scripts and queries.
+
+* GROK-20298: `grok s domains` — entity-mapped domain schemas and their rows from the CLI, addressed as `<schema>` or `<schema>.<table>`: `list`, `get` (manifest, a table's columns, or one row), `query` (`--filter`, `--columns`, `--sort`, `--expand`, `--limit`, `--offset`), `count`, `aggregate` (`--measures 'count,sum(x) as t' --group-by a`), `insert` / `update` (from `--json` or inline `col=value`, `--version` for optimistic concurrency), `delete` (one row, `--filter` bulk, or a whole user schema with `--force`), `upload` (csv, d42, json; `--upsert`, `--no-all-or-nothing`), `download` (csv to stdout or `-O out.csv|out.d42`), `transaction`, `audit`, `capabilities`, `grants` / `grant` / `revoke` (groups by name), `create`, and `apply` (`--dry-run`, `--confirm-destructive`, `--if-version`). Validation errors print per row; a destructive plan awaiting confirmation is printed with the error.
+* GROK-20298: `grok api` — the `autoNumber` column key (`true`, or `{"scope": "<ref column>", "start": N}`; int columns only, implies `immutable`): the engine assigns the number on insert when the payload leaves it blank and keeps a supplied one, so `<Table>Row` emits the column without `?` while `<Table>Insert` keeps it optional. `domain-schema.schema.json` validates the key.
+* GROK-20298: `grok api` — a domain column may reference any registered table with `"ref": "<Schema>.<table>"` (`Core.queries`, `Core.users`, another plugin's `grit.issue`); the target is resolved at build time — Core tables from the sealed `Core.json` shipped with the CLI, other schemas from a dependency's `databases/<schema>/schema.json` in `node_modules` — and the referrer's `<Table>Expand` entry lists the target's columns with the same types a same-schema ref gets. An unresolvable target fails the run naming the missing package. `domain-schema.schema.json` accepts the qualified form.
+* `grok s pull/push/migrate/diff` + `grok s bundle ls`: move UI-built entities between Datagrok instances through a git-friendly bundle directory — connections (never their secrets), queries, scripts, dashboards and spaces with their tables, views, layouts, files, jobs, notebooks and models, plus the groups, memberships, grants and tags they need. Consecutive pulls merge into one bundle; a push is idempotent (an unchanged bundle writes nothing), merges project relations instead of replacing them, and resolves name clashes per `--on-conflict fail|skip|duplicate|adopt`. `grok s migrate <selection> --from dev --to prod` does both ends in one command, and `--creds creds.yaml` (with `${ENV}` substitution) supplies the target-side connection secrets that never travel in a bundle.
+* `grok s --admin` — asks the server for an admin session for the duration of one command, so a stand-wide pull sees entities the key's own account cannot: other people's team spaces, and personal spaces addressed by the user's project id. The server authorises it (`START_ADMIN_SESSION`) and signs the flag into the token; a dev key mints a fresh session per invocation, so it cannot outlive the command.
+* `grok s` — a session the server rejects mid-run is replaced rather than losing the run: the developer key takes a fresh one and the request is replayed once, restoring admin mode if the run asked for it. A stand serving several isolates can reject a session one of them does not know, which killed a full pull of release after 11 minutes with a bare "Invalid session" and an empty bundle.
+* `grok s pull` — a table whose data file is missing on the source answers 200 with an ApiError body as `text/plain`, which used to be written into the bundle as the table's `.d42` and only fail on the far stand at push time. It is now `warn(no_data)` at pull time, and absent bytes no longer count as an incomplete pull: the entity travels, and a datasync table that was never materialised refreshes on the target.
+* `grok s pull/push` — a project the platform keeps for itself never travels: the namespace an installed package occupies (which collides by name on any target with the same packages) and the wrapper it maintains around a saved entity (which the target accepts and then does not have). On a real 1.27.9 → 1.27.9 push these were 84 name conflicts and 113 failed saves.
+* `grok s pull/migrate` — a request now has a 60s deadline and retries a dropped socket (`GROK_HTTP_TIMEOUT` / `GROK_HTTP_RETRIES`), and an entity the server will not answer for is a `warn(fetch_failed)` row naming the route instead of a `fetch failed` that discards the whole run.
+* `grok s migrate` — refuses to push a bundle whose pull dropped entities, instead of promoting less than was asked for, and byte transfers get their own 10-minute deadline (`GROK_HTTP_BYTES_TIMEOUT`) so a large table is not cut off by the request one.
+* `grok s push/migrate` — an id the bundle points at but does not carry is now resolved against the target: `info(orphan_ref)` when the target holds that id itself, and `warn(dependency_missing)` naming the entity that needs it when neither side has it. A reference left dangling on the source used to plan as a clean `create` and then fail the save with the server's own opaque error.
+* `grok s push/migrate` — `warn(namespace_not_preserved)` when an entity lands outside the namespace it was pulled from, which happens when its owning space is not in the bundle.
+* `grok s migrate --by-namespace` — moves a whole instance one space at a time instead of in a single bundle. Placement is exclusive on the server, so one bundle holding every space has its projects taking entities from each other; scoped to a space that contention stays inside it. Checks the instance-level prerequisites once up front (users and packages the target lacks) and refuses a whole-instance run until they are fixed, keeps going when a space fails, and records what finished so a re-run resumes. Three spaces, 1,946 entities, 4m44s; the same content inside a 15,778-entity bundle did not finish.
+* `grok s migrate --by-namespace` — a full run ends with a sweep for the leaf entities no space owns (a layout under no namespace would otherwise never travel), refuses a name in `--only`/`--skip` that matches no space rather than migrating nothing, needs `--admin` so it cannot enumerate a subset of the instance and call it whole, and leaves the platform's own `System` space alone. A part whose pull dropped entities is not pushed and stays re-runnable; `--dry-run` records nothing.
+* `grok s push` — a share file now answers to `--on-conflict` instead of overwriting whatever the target has, and reports `update` rather than `create` when it replaces one. `skip(dead_on_source)` is no longer applied to a timeout or a dropped connection, which said nothing about the entity. Placement falls back to the project's own relations when `/projects/relations` fails on a project linking domain-table rows, instead of reading it as holding nothing.
+* `grok s push/migrate` — relation placement is idempotent. Containment is exclusive on the server (a non-link relation deletes the entity's other non-link rows), so a project the plan called `identical` was skipped for relation writing while the projects that did write took its entities away, and a stale `isLink` flag carried over from the target was deleted by the real owner and never re-examined. Pushing the same bundle twice used to cut a stand from 273 placed relation rows to 90; three consecutive pushes now leave the same state, and a run repairs what an earlier one stripped.
+* `grok s pull/push` — a reference the source itself can no longer resolve is recorded in the bundle (`warn(source_dangling_refs)`) and whatever depends on it is reported as `skip(dead_on_source)` rather than `failed`, so a stand full of saved views whose tables were deleted no longer fails the run. A reference the source keeps and the target lacks stays a real failure.
+* `grok s pull` — outside references are resolved against the entity as the bundle stores it, not as the server returns it: embedded copies used to look like rows the bundle carried, which both hid dead references and left package entities pointing at ids the target has never seen.
+* `grok s pull` — selection, dependency expansion and reference resolution overlap their requests instead of one round trip at a time; a 90-day slice of a large stand went from tens of minutes to minutes, and a whole-instance pull is now bounded by the server rather than by latency.
+* `grok s pull/push/migrate` — progress on stderr for the long walks, so a stand-sized run is distinguishable from a hang; `--output json` keeps one parseable document.
+* `grok s pull` — a datasync table's share file travels with it, and `--namespace X` selects `X` itself as well as its contents, so what is pulled has somewhere to be placed.
+* `grok s pull` — the default selection now lists root spaces, which only the `--type space` alias asked for before.
+* `grok s pull` — a datasync table backed by a query or script (`Ns:Name(...)`) brings it along, as file-backed ones already did.
+* `grok s pull` — `AllUsers`, `Administrators` and `Developers` stay on the instance that owns them (their ids differ across stands, so pushing one was a name conflict); grants on them still travel by name.
+* `grok s <entity> <verb>` — a usage error under `--output json` no longer prints the help block on stdout (it goes to stderr for the other formats), so the JSON error on stderr is all a caller has to parse.
+* `grok s` — errors are one plain-text line on stderr (the Dart stack only with `--verbose`), and a JSON object only under `--output json`.
+
 ## 6.5.7 (2026-08-28)
 
 * GROK-20789: `grok s packages` — full package lifecycle from the CLI: `install <name>... [--version <v>]` (server pulls released versions from the configured package repository / npm, `latest` by default), `uninstall`, `update <name>... | --all` (preserves `latest` auto-update tracking; pins pinned packages to the concrete registry-latest), `outdated`, `versions`, `set-version`, and `share`. Multi-package install/update runs sequentially with a per-package status table; a nonexistent package or version is a per-package `error` and exit code 1. Also fixed the generic `packages delete`, which was hitting a nonexistent `/public/v1` endpoint.

package/CLAUDE.md

@@ -234,15 +234,17 @@ grok s groups save --json group.json --save-relations
 
 # Share entities
 grok s shares add "JohnDoe:MyConnection" Chemists,Admins --access Edit
-grok s shares list <entity-uuid>
+grok s shares list "JohnDoe:MyConnection"      # by name or UUID; includes inherited grants
 
-# List / inspect entities
+# List / count / inspect entities (--limit / --offset page, --filter is a smart filter)
 grok s users list
+grok s users count --filter 'status = "active"'
 grok s packages list --filter "MyPlugin"       # check if a plugin is published
 grok s connections list --output json
 grok s functions list --filter "Chem"          # find registered functions
-grok s connections get <id>
-grok s connections delete <id>
+grok s connections get <id-or-name>
+grok s connections delete <id-or-name>
+# users cannot be deleted (no server API) — `grok s users block <login>` instead
 grok s connections save --json conn.json --save-credentials   # create or update
 grok s connections test "JohnDoe:MyConnection"                # test by id or name
 grok s connections test --json conn.json                      # test a connection defined in JSON
@@ -275,11 +277,21 @@ grok s groups list-members Admins --admin               # admin members only
 grok s groups list-members Admins --no-admin            # non-admin members only
 grok s groups list-memberships alice                    # groups alice belongs to
 
-# Hit any API endpoint directly
-grok s raw GET /api/users/current
-grok s raw GET /api/packages/dev/MyPlugin
-
-# Describe entity JSON schema
+# Domain schemas and their rows (entity-mapped domain tables)
+grok s domains list                            # registered schemas; `list grit` lists its tables
+grok s domains get grit.issue                  # a table's columns (`get grit` = manifest, `get grit.issue <id>` = one row)
+grok s domains query grit.issue --filter 'status = "open"' --sort '!created_on' --limit 20
+grok s domains insert grit.issue title="Crash" status=open
+grok s domains upload grit.issue ./issues.csv --upsert
+grok s domains grant grit.issue Chemists --access Edit
+grok s domains apply myschema --json schema.json --dry-run   # user-managed schema DDL
+
+# Hit any API endpoint directly (API-relative path; a leading /api is accepted; exit 1 on failure)
+grok s raw GET /users/current
+grok s raw GET /packages/dev/MyPlugin
+grok s raw POST /public/v1/functions/Sin/call --data '{"x": 1}'
+
+# Fields of an entity type (registry record + a live sample)
 grok s describe connections
 
 # Target a specific server
@@ -292,7 +304,10 @@ grok s users list --output quiet | xargs ...   # pipe IDs
 ```
 
 **Windows Git Bash:** prefix raw paths with `MSYS_NO_PATHCONV=1` to prevent POSIX→Windows
-path conversion: `MSYS_NO_PATHCONV=1 grok s raw GET /api/users/current`
+path conversion: `MSYS_NO_PATHCONV=1 grok s raw GET /users/current`
+
+**Tests:** `npm run test:server` (unit, mocked client) and `HOST=<alias> npm run test:integration`
+(a live server; read-only apart from a `files.put` round trip).
 
 **Implementation:** `bin/commands/server.ts`, `bin/utils/node-dapi.ts` (Node.js REST client),
 `bin/utils/server-output.ts` (formatters). The Node.js dapi bypasses the Dart interop layer