datagrok-tools 6.5.6 → 6.5.8

package/.devcontainer/docker-compose.yaml

@@ -8,6 +8,29 @@
 # On Windows, use `grok claude` instead — it handles path conversion
 # and Docker socket mapping automatically.
 
+# Shared by every jkg_* gateway below. They differ only in image and network alias.
+x-jkg: &jkg
+  platform: linux/amd64
+  environment:
+    GROK_PARAMETERS: |
+      {
+        "queueSettings": {
+          "useQueue": true,
+          "amqpHost": "rabbitmq",
+          "amqpUser": "guest",
+          "amqpPassword": "guest",
+          "amqpPort": 5672,
+          "pipeHost": "grok_pipe",
+          "pipePort": 3000,
+          "pipeKey": "test-key",
+          "maxConcurrentCalls": 16
+        }
+      }
+  restart: unless-stopped
+  depends_on:
+    rabbitmq:
+      condition: service_started
+
 services:
   postgres:
     image: pgvector/pgvector:pg17
@@ -118,33 +141,54 @@ services:
     restart: unless-stopped
     profiles: ["full"]
 
-  jupyter_kernel_gateway:
-    image: datagrok/jupyter_kernel_gateway:${DG_VERSION:-latest}
-    platform: linux/amd64
-    depends_on:
-      rabbitmq:
-        condition: service_started
-    environment:
-      GROK_PARAMETERS: |
-        {
-          "queueSettings": {
-            "useQueue": true,
-            "amqpHost": "rabbitmq",
-            "amqpUser": "guest",
-            "amqpPassword": "guest",
-            "amqpPort": 5672,
-            "pipeHost": "grok_pipe",
-            "pipePort": 3000,
-            "pipeKey": "test-key",
-            "maxConcurrentCalls": 16
-          }
-        }
+  # Per-language kernel gateways, which replaced the single all-languages
+  # datagrok/jupyter_kernel_gateway image. They version with the Scripting package, not with
+  # the platform, so they take their own JKG_VERSION. `scripting`/`full` starts Python and R;
+  # add `jkg_octave`, `jkg_julia` or `jkg_nodejs` when a package needs those languages.
+  jkg_python:
+    <<: *jkg
+    image: datagrok/jkg_python:${JKG_VERSION:-latest}
     networks:
       dg:
         aliases:
-          - jupyter_kernel_gateway
-    restart: unless-stopped
-    profiles: ["scripting", "full"]
+          - jkg_python
+    profiles: ["scripting", "full", "jkg_python"]
+
+  jkg_r:
+    <<: *jkg
+    image: datagrok/jkg_r:${JKG_VERSION:-latest}
+    networks:
+      dg:
+        aliases:
+          - jkg_r
+    profiles: ["scripting", "full", "jkg_r"]
+
+  jkg_octave:
+    <<: *jkg
+    image: datagrok/jkg_octave:${JKG_VERSION:-latest}
+    networks:
+      dg:
+        aliases:
+          - jkg_octave
+    profiles: ["jkg_octave"]
+
+  jkg_julia:
+    <<: *jkg
+    image: datagrok/jkg_julia:${JKG_VERSION:-latest}
+    networks:
+      dg:
+        aliases:
+          - jkg_julia
+    profiles: ["jkg_julia"]
+
+  jkg_nodejs:
+    <<: *jkg
+    image: datagrok/jkg_nodejs:${JKG_VERSION:-latest}
+    networks:
+      dg:
+        aliases:
+          - jkg_nodejs
+    profiles: ["jkg_nodejs"]
 
   world:
     image: datagrok/demo_db_postgres:12-1.1.0

package/CHANGELOG.md

@@ -1,11 +1,50 @@
 # Datagrok-tools changelog
 
-## 6.5.6 (WIP)
+## 6.5.8 (2026-09-10)
+
+* `grok s` — correctness pass from the CLI audit: every server failure now exits 1 with one line on stderr (a 200 carrying an `ApiError` — missing entity, duplicate login or group name, unknown function — used to print as data with exit 0); `--limit`/`--offset` are honoured for users, groups, connections and functions (their public routes ignore paging, so `list` and the new `count` verb go through the internal routers); `files list` works (the public files route only downloads; listing goes through the connection's file route) and prints `path, kind, size`; `tables list|get|delete` no longer throw, and `tables download|get|delete` accept the bare table name, the full `Project:Table` name or the id; `describe <entity|type>` works (registry record plus the fields of a live sample; it called a nonexistent endpoint before); positional `functions run 'F(1, 2)'` maps the values onto the function's inputs instead of silently sending `{"0": 1}` (which ran the function with nulls); `raw` takes API-relative paths (a leading `/api` is accepted), sends a body from `--json <file>` or `--data '<json>'`, and fails on non-2xx — so it, and `healthcheck`, now work against a bare-Datlas `--host` too; `shares list` accepts a name and shows the grants inherited through project links; `groups` name resolution prefers an exact match (`admin` no longer collides with `Administrators`) and `--user` works on `list-members`/`list-memberships`; `connections delete` of an unknown id fails instead of reporting success; a bad `--host` URL or alias reports the reason without the help dump. `users delete` refuses with a pointer at `users block`: the platform has no user deletion, and removing the entity record leaves the login unusable. `functions delete` covers scripts and queries.
+
+* GROK-20298: `grok s domains` — entity-mapped domain schemas and their rows from the CLI, addressed as `<schema>` or `<schema>.<table>`: `list`, `get` (manifest, a table's columns, or one row), `query` (`--filter`, `--columns`, `--sort`, `--expand`, `--limit`, `--offset`), `count`, `aggregate` (`--measures 'count,sum(x) as t' --group-by a`), `insert` / `update` (from `--json` or inline `col=value`, `--version` for optimistic concurrency), `delete` (one row, `--filter` bulk, or a whole user schema with `--force`), `upload` (csv, d42, json; `--upsert`, `--no-all-or-nothing`), `download` (csv to stdout or `-O out.csv|out.d42`), `transaction`, `audit`, `capabilities`, `grants` / `grant` / `revoke` (groups by name), `create`, and `apply` (`--dry-run`, `--confirm-destructive`, `--if-version`). Validation errors print per row; a destructive plan awaiting confirmation is printed with the error.
+* GROK-20298: `grok api` — the `autoNumber` column key (`true`, or `{"scope": "<ref column>", "start": N}`; int columns only, implies `immutable`): the engine assigns the number on insert when the payload leaves it blank and keeps a supplied one, so `<Table>Row` emits the column without `?` while `<Table>Insert` keeps it optional. `domain-schema.schema.json` validates the key.
+* GROK-20298: `grok api` — a domain column may reference any registered table with `"ref": "<Schema>.<table>"` (`Core.queries`, `Core.users`, another plugin's `grit.issue`); the target is resolved at build time — Core tables from the sealed `Core.json` shipped with the CLI, other schemas from a dependency's `databases/<schema>/schema.json` in `node_modules` — and the referrer's `<Table>Expand` entry lists the target's columns with the same types a same-schema ref gets. An unresolvable target fails the run naming the missing package. `domain-schema.schema.json` accepts the qualified form.
+* `grok s pull/push/migrate/diff` + `grok s bundle ls`: move UI-built entities between Datagrok instances through a git-friendly bundle directory — connections (never their secrets), queries, scripts, dashboards and spaces with their tables, views, layouts, files, jobs, notebooks and models, plus the groups, memberships, grants and tags they need. Consecutive pulls merge into one bundle; a push is idempotent (an unchanged bundle writes nothing), merges project relations instead of replacing them, and resolves name clashes per `--on-conflict fail|skip|duplicate|adopt`. `grok s migrate <selection> --from dev --to prod` does both ends in one command, and `--creds creds.yaml` (with `${ENV}` substitution) supplies the target-side connection secrets that never travel in a bundle.
+* `grok s --admin` — asks the server for an admin session for the duration of one command, so a stand-wide pull sees entities the key's own account cannot: other people's team spaces, and personal spaces addressed by the user's project id. The server authorises it (`START_ADMIN_SESSION`) and signs the flag into the token; a dev key mints a fresh session per invocation, so it cannot outlive the command.
+* `grok s` — a session the server rejects mid-run is replaced rather than losing the run: the developer key takes a fresh one and the request is replayed once, restoring admin mode if the run asked for it. A stand serving several isolates can reject a session one of them does not know, which killed a full pull of release after 11 minutes with a bare "Invalid session" and an empty bundle.
+* `grok s pull` — a table whose data file is missing on the source answers 200 with an ApiError body as `text/plain`, which used to be written into the bundle as the table's `.d42` and only fail on the far stand at push time. It is now `warn(no_data)` at pull time, and absent bytes no longer count as an incomplete pull: the entity travels, and a datasync table that was never materialised refreshes on the target.
+* `grok s pull/push` — a project the platform keeps for itself never travels: the namespace an installed package occupies (which collides by name on any target with the same packages) and the wrapper it maintains around a saved entity (which the target accepts and then does not have). On a real 1.27.9 → 1.27.9 push these were 84 name conflicts and 113 failed saves.
+* `grok s pull/migrate` — a request now has a 60s deadline and retries a dropped socket (`GROK_HTTP_TIMEOUT` / `GROK_HTTP_RETRIES`), and an entity the server will not answer for is a `warn(fetch_failed)` row naming the route instead of a `fetch failed` that discards the whole run.
+* `grok s migrate` — refuses to push a bundle whose pull dropped entities, instead of promoting less than was asked for, and byte transfers get their own 10-minute deadline (`GROK_HTTP_BYTES_TIMEOUT`) so a large table is not cut off by the request one.
+* `grok s push/migrate` — an id the bundle points at but does not carry is now resolved against the target: `info(orphan_ref)` when the target holds that id itself, and `warn(dependency_missing)` naming the entity that needs it when neither side has it. A reference left dangling on the source used to plan as a clean `create` and then fail the save with the server's own opaque error.
+* `grok s push/migrate` — `warn(namespace_not_preserved)` when an entity lands outside the namespace it was pulled from, which happens when its owning space is not in the bundle.
+* `grok s migrate --by-namespace` — moves a whole instance one space at a time instead of in a single bundle. Placement is exclusive on the server, so one bundle holding every space has its projects taking entities from each other; scoped to a space that contention stays inside it. Checks the instance-level prerequisites once up front (users and packages the target lacks) and refuses a whole-instance run until they are fixed, keeps going when a space fails, and records what finished so a re-run resumes. Three spaces, 1,946 entities, 4m44s; the same content inside a 15,778-entity bundle did not finish.
+* `grok s migrate --by-namespace` — a full run ends with a sweep for the leaf entities no space owns (a layout under no namespace would otherwise never travel), refuses a name in `--only`/`--skip` that matches no space rather than migrating nothing, needs `--admin` so it cannot enumerate a subset of the instance and call it whole, and leaves the platform's own `System` space alone. A part whose pull dropped entities is not pushed and stays re-runnable; `--dry-run` records nothing.
+* `grok s push` — a share file now answers to `--on-conflict` instead of overwriting whatever the target has, and reports `update` rather than `create` when it replaces one. `skip(dead_on_source)` is no longer applied to a timeout or a dropped connection, which said nothing about the entity. Placement falls back to the project's own relations when `/projects/relations` fails on a project linking domain-table rows, instead of reading it as holding nothing.
+* `grok s push/migrate` — relation placement is idempotent. Containment is exclusive on the server (a non-link relation deletes the entity's other non-link rows), so a project the plan called `identical` was skipped for relation writing while the projects that did write took its entities away, and a stale `isLink` flag carried over from the target was deleted by the real owner and never re-examined. Pushing the same bundle twice used to cut a stand from 273 placed relation rows to 90; three consecutive pushes now leave the same state, and a run repairs what an earlier one stripped.
+* `grok s pull/push` — a reference the source itself can no longer resolve is recorded in the bundle (`warn(source_dangling_refs)`) and whatever depends on it is reported as `skip(dead_on_source)` rather than `failed`, so a stand full of saved views whose tables were deleted no longer fails the run. A reference the source keeps and the target lacks stays a real failure.
+* `grok s pull` — outside references are resolved against the entity as the bundle stores it, not as the server returns it: embedded copies used to look like rows the bundle carried, which both hid dead references and left package entities pointing at ids the target has never seen.
+* `grok s pull` — selection, dependency expansion and reference resolution overlap their requests instead of one round trip at a time; a 90-day slice of a large stand went from tens of minutes to minutes, and a whole-instance pull is now bounded by the server rather than by latency.
+* `grok s pull/push/migrate` — progress on stderr for the long walks, so a stand-sized run is distinguishable from a hang; `--output json` keeps one parseable document.
+* `grok s pull` — a datasync table's share file travels with it, and `--namespace X` selects `X` itself as well as its contents, so what is pulled has somewhere to be placed.
+* `grok s pull` — the default selection now lists root spaces, which only the `--type space` alias asked for before.
+* `grok s pull` — a datasync table backed by a query or script (`Ns:Name(...)`) brings it along, as file-backed ones already did.
+* `grok s pull` — `AllUsers`, `Administrators` and `Developers` stay on the instance that owns them (their ids differ across stands, so pushing one was a name conflict); grants on them still travel by name.
+* `grok s <entity> <verb>` — a usage error under `--output json` no longer prints the help block on stdout (it goes to stderr for the other formats), so the JSON error on stderr is all a caller has to parse.
+* `grok s` — errors are one plain-text line on stderr (the Dart stack only with `--verbose`), and a JSON object only under `--output json`.
+
+## 6.5.7 (2026-08-28)
+
+* GROK-20789: `grok s packages` — full package lifecycle from the CLI: `install <name>... [--version <v>]` (server pulls released versions from the configured package repository / npm, `latest` by default), `uninstall`, `update <name>... | --all` (preserves `latest` auto-update tracking; pins pinned packages to the concrete registry-latest), `outdated`, `versions`, `set-version`, and `share`. Multi-package install/update runs sequentially with a per-package status table; a nonexistent package or version is a per-package `error` and exit code 1. Also fixed the generic `packages delete`, which was hitting a nonexistent `/public/v1` endpoint.
+* `grok s` — handler errors now print a clean `{"error": ...}` and exit 1 instead of a raw stack trace with exit 255 (handler dispatch was not awaited, so rejections escaped the error boundary).
+* `--version` values are no longer numerically coerced (`--version 1.10` used to arrive as `1.1`; affected `grok claude` too).
+* `grok test` — a Playwright pass that fails before running specs now reports a failing test instead of nothing. It returned an empty CSV, and the caller drops an empty Playwright result whenever the Puppeteer pass produced rows, so a suite whose config would not load (or that collected zero specs) was indistinguishable from a clean run — Build-Deploy #1277 printed "Playwright: no JSON report produced" and then "Tests passed", exit 0, for six packages. Every early exit now writes one failing row to `test-report-playwright.csv` and the merged report, and a report containing no specs is treated as a failure rather than a pass.
+
+## 6.5.6 (2026-08-20)
 
 * `grok report` (`ticket`, `comment`, `label`, `attach`) — support Atlassian **service-account** tokens alongside user API tokens, chosen by the token itself. A user token (`ATATT...`) keeps HTTP Basic with `JIRA_USER` + `JIRA_TOKEN` against the site. A service-account token (`ATSTT...`, minted at admin.atlassian.com) is a scoped token that the site host will not accept — Basic returns 401 and Bearer returns 403 `"Failed to parse Connect Session Auth Token"` — so it is sent as Bearer to the API gateway at `api.atlassian.com/ex/jira/<cloudId>`, with the cloud id read once from the site's public `/_edge/tenant_info` (override with `$JIRA_CLOUD_ID`). `$JIRA_URL` and `--jira-url` still mean the SITE; the gateway address is derived from it, never substituted for it. A service token needs no `JIRA_USER`, so the credential checks no longer demand one.
 
-## 6.5.5 (WIP)
+## 6.5.5 (2026-08-12)
 
+* `grok test` — added a Node (browserless) pass: tests annotated `{node: true}` run headless under the js-api Node runtime before the browser launches; the browser pass excludes them and is skipped entirely when nothing browser-only matches. New flags: `--skip-node`, `--node-only`. Packages opt in by exporting `testNode()` from `package-test.ts`; others keep the previous behavior.
 * GROK-20298: `grok api --ui` — a relation-bearing table's `<Table>Ui` handle carries `<Table>Update` as its fifth generic too (`<Table>Ui.client`, `.table()` and the `<Schema>UiDb` property), so `<table>Ui.client.update(id, {labels: [...]})` compiles instead of failing the excess-property check.
 * GROK-20298: `grok api` — typed many-to-many relations: a table's `relations` block is validated with the server's rules (via/target declared in the same manifest, junction distinct from both sides, self-referential relations explicit, FK sides unique-or-explicit, junction `businessKey` covering both) and emitted as an expand-map entry `'<name>': {<name>?: {id: string; name: string}[]}`, a `<name>?: string[]` link set on `<Table>Insert`, and a new `<Table>Update` type (`Partial<<Table>Row>` plus the link sets) used by the transaction op union and passed as the client's fifth generic, so `update()` takes the link sets directly. Manifests without relations generate byte-identically.
 * GROK-20298: `grok api --ui` — also emits the typed SCHEMA handle: `export function <schema>UiDb(): Promise<<Schema>UiDb>` (a typed `domains.db(...)`), and the `<Schema>UiDb` interface types each table property (`const db = await gritUiDb(); db.issues.form({values: {...}})` is compile-checked).
@@ -32,10 +71,6 @@
 
 * `grok test` — the Node-pass report now merges into the browser report by column name. The line-wise merge assumed identical column order, so node rows landed misaligned (string values in the integer `ms` column) and the CI test-report upload failed with `invalid input syntax for type integer`.
 
-## 6.5.5 (WIP)
-
-* `grok test` — added a Node (browserless) pass: tests annotated `{node: true}` run headless under the js-api Node runtime before the browser launches; the browser pass excludes them and is skipped entirely when nothing browser-only matches. New flags: `--skip-node`, `--node-only`. Packages opt in by exporting `testNode()` from `package-test.ts`; others keep the previous behavior.
-
 ## 6.4.8 (2026-07-22)
 
 * GROK-20452: `grok publish` — always `docker build` the tools-generated worker dirs (`dockerfiles/queue`, `dockerfiles/celery`, `dockerfiles/<app>-celery`) instead of reusing a cached local tag. The "found local image" shortcut froze the worker at whatever stock base the daemon had when the tag was first built (CI ran day-old worker code with a fresh base available); the layer cache keeps an unchanged rebuild near-instant.

package/CLAUDE.md

@@ -234,15 +234,17 @@ grok s groups save --json group.json --save-relations
 
 # Share entities
 grok s shares add "JohnDoe:MyConnection" Chemists,Admins --access Edit
-grok s shares list <entity-uuid>
+grok s shares list "JohnDoe:MyConnection"      # by name or UUID; includes inherited grants
 
-# List / inspect entities
+# List / count / inspect entities (--limit / --offset page, --filter is a smart filter)
 grok s users list
+grok s users count --filter 'status = "active"'
 grok s packages list --filter "MyPlugin"       # check if a plugin is published
 grok s connections list --output json
 grok s functions list --filter "Chem"          # find registered functions
-grok s connections get <id>
-grok s connections delete <id>
+grok s connections get <id-or-name>
+grok s connections delete <id-or-name>
+# users cannot be deleted (no server API) — `grok s users block <login>` instead
 grok s connections save --json conn.json --save-credentials   # create or update
 grok s connections test "JohnDoe:MyConnection"                # test by id or name
 grok s connections test --json conn.json                      # test a connection defined in JSON
@@ -251,6 +253,16 @@ grok s connections test --json conn.json                      # test a connectio
 grok s functions run 'Chem:smilesToMw("ccc")'
 grok s functions run 'Pkg:fn({a:5,b:22})'
 
+# Manage packages (server pulls released versions from the package repository / npm)
+grok s packages install Chem Bio PowerGrid     # install latest of each
+grok s packages install Chem --version 1.14.0  # pin a version
+grok s packages outdated                       # installed vs registry-latest
+grok s packages update --all                   # upgrade everything outdated
+grok s packages versions Chem                  # published versions + current/latest flags
+grok s packages set-version Chem 1.13.0        # activate a specific version
+grok s packages uninstall Chem                 # repo entry stays installable
+grok s packages share Chem Chemists --access View
+
 # Browse file storage
 grok s files list "System:AppData" -r          # list files recursively
 grok s files list "System:AppData/MyPlugin"
@@ -265,11 +277,21 @@ grok s groups list-members Admins --admin               # admin members only
 grok s groups list-members Admins --no-admin            # non-admin members only
 grok s groups list-memberships alice                    # groups alice belongs to
 
-# Hit any API endpoint directly
-grok s raw GET /api/users/current
-grok s raw GET /api/packages/dev/MyPlugin
-
-# Describe entity JSON schema
+# Domain schemas and their rows (entity-mapped domain tables)
+grok s domains list                            # registered schemas; `list grit` lists its tables
+grok s domains get grit.issue                  # a table's columns (`get grit` = manifest, `get grit.issue <id>` = one row)
+grok s domains query grit.issue --filter 'status = "open"' --sort '!created_on' --limit 20
+grok s domains insert grit.issue title="Crash" status=open
+grok s domains upload grit.issue ./issues.csv --upsert
+grok s domains grant grit.issue Chemists --access Edit
+grok s domains apply myschema --json schema.json --dry-run   # user-managed schema DDL
+
+# Hit any API endpoint directly (API-relative path; a leading /api is accepted; exit 1 on failure)
+grok s raw GET /users/current
+grok s raw GET /packages/dev/MyPlugin
+grok s raw POST /public/v1/functions/Sin/call --data '{"x": 1}'
+
+# Fields of an entity type (registry record + a live sample)
 grok s describe connections
 
 # Target a specific server
@@ -282,7 +304,10 @@ grok s users list --output quiet | xargs ...   # pipe IDs
 ```
 
 **Windows Git Bash:** prefix raw paths with `MSYS_NO_PATHCONV=1` to prevent POSIX→Windows
-path conversion: `MSYS_NO_PATHCONV=1 grok s raw GET /api/users/current`
+path conversion: `MSYS_NO_PATHCONV=1 grok s raw GET /users/current`
+
+**Tests:** `npm run test:server` (unit, mocked client) and `HOST=<alias> npm run test:integration`
+(a live server; read-only apart from a `files.put` round trip).
 
 **Implementation:** `bin/commands/server.ts`, `bin/utils/node-dapi.ts` (Node.js REST client),
 `bin/utils/server-output.ts` (formatters). The Node.js dapi bypasses the Dart interop layer