datagrok-tools 6.7.8 → 6.7.10

package/.devcontainer/Dockerfile.pkg_dev

@@ -1,4 +1,4 @@
-FROM node:22-bookworm-slim
+FROM node:22.23.4-bookworm-slim
 
 # System deps + Chrome stable
 RUN apt-get update && apt-get install -y --no-install-recommends \

package/.turbo/turbo-build.log

@@ -1,5 +1,5 @@
 
-> datagrok-tools@6.7.8 build /home/runner/work/public/public/tools
+> datagrok-tools@6.7.10 build /home/runner/work/public/public/tools
 > node build.js
 
-Successfully compiled 51 files with Babel.
+Successfully compiled 55 files with Babel.

package/CHANGELOG.md

@@ -1,5 +1,38 @@
 # Datagrok-tools changelog
 
+## 6.7.10 (2026-11-06)
+
+* Added `grok src`: git recipes across the repos of the datagrok umbrella (`clone`, `add`, `sparse`, `switch`, `status`, `pull`, `at`, `bump`, `wt add`, `doctor`)
+* Kept positionals, `--url` and `--dir` verbatim (`1.10` no longer becomes 1.1), and an explicit boolean such as `--force` now reaches every command instead of being dropped
+* Added `grok s o groups list|get|add|set|rm|put|reset`, `endpoints list|rm`, `logs match '<filter>'` (with `--space`) and `logs activations [stop]`: log groups and endpoints of Settings > Logger
+* `grok publish`: A local container build that fails for the host platform on Apple Silicon is retried for linux/amd64, so a package whose base image has no arm64 variant (the Chembl ChEMBL database) builds
+* `grok test`: A package built on u2 or ems runs with u2 strict — a u2 misuse throws a `U2UsageError` naming the fix, and one the page caught and logged fails the run when it is the package's own (a stack frame in its bundle, or none to tell; another package's prints as a warning); `U2_STRICT=report` lists them instead without failing
+* `grok check`: A package built on u2 or ems fails on a problem of u2's consumer check whatever its version (a 0.x package only printed it), so a CSS rule over a new `.u2-*` class or `[data-u2…]` attribute fails the check
+* Added `grok s o problems severity <id|kind:key> <info|warning|critical|none>`: pins the severity a problem's raises use (`none` unpins it)
+* Added `grok connect-folder`: connects a local folder as a connection, served from a Docker container; the agent image tag is pulled on every connect, so a moved tag replaces a stale local agent
+* `grok connect-folder`: Runs only the Docker Hub `datagrok/proxy_storage` agent image whatever image the server names; `--image` runs another, such as a copy in a private registry
+* Added `grok s ... --log [<spec>]` on every `grok s` command (`grok s o` included): its requests share a `datlas-log-id`, every server instance keeps what they log at the spec (`verbose`, `standard`, levels, debug flags or `all`) in memory only, and at the end the command gathers it and prints it to stderr
+* Added `grok s functions run ... --debug`: a debug call (`?debug=true`) whose own log, a query's stages as the query debugger shows them, is kept and printed the same way
+* Added `grok s o logs request <logId>`: gathers what the requests of one `--log` command logged
+* `grok s status`: server build, uptime, DB pool and HTTP connections from `/info/server`; for admins also the requests in flight (`?extended=true`)
+* `grok api`: Fixed double plurals in table accessors (`categorieses` → `categories`)
+* Build: Fixed `package-api.ts` of a decorator package lagging one build behind `package.g.ts`
+* Added a `restore` arm per soft-delete table to the `<Schema>TransactionOp` that `grok api` generates, so the typed `transaction()` accepts the restore ops the server runs
+* Made `grok api` type a required column with a `default` as optional on `<Table>Insert`: the server fills it when an insert omits it
+* Fixed `grok api` emitting a choices type that does not compile, or holds a different value, when a choice contains `'` or `\`
+
+## 6.7.9 (2026-10-16)
+
+* Upgraded Puppeteer to v25, which drops the vulnerable `extract-zip` and `basic-ftp` transitive dependencies; Node.js 22.12+ is now required
+* `grok add app --domain`, `grok api --ui`: scaffold against `@datagrok-libraries/ems`
+* GROK-20884: Added `grok s o errors get <signature|problem>`: each occurrence of an error with its time, user, function and session, so an error incident leads to its calls
+* GROK-20884: Added `grok s o logs container [<name|id>]`: a docker container's log lines cut by `--since`/`--from`/`--to` and `--grep`; no name lists the containers
+* GROK-20884: `grok s o problems get` prints one field per line with every detail in full, and `errors top --limit` returns that many rows instead of the top of every source
+* GROK-20884: Added `grok s observe` (alias `grok s o`): problems and their alerts, problem rules, logger settings with expiring user and group entries, errors, the timeline and cloud logs (CloudWatch and the log archive)
+* GROK-20884: Added `grok s o logs capture <login|group> [--for 30m]`: saves every API request of a user or a group until the entry expires (a `logger set` of `saveHttpRequests`; an entry that expires later keeps its time); `logs capture stop` removes an expiring entry that holds only the capture, `logs capture` lists the active ones
+* GROK-20298: `grok s domains apply` sends the manifest JSON's `version` as the now-required `ifVersion` of a user-managed schema, accepts `--if-version current` to take the registered one, refuses such an apply without either, and forwards `friendlyName`, `description` and `storage` from the JSON for a user-managed schema (a package schema keeps its key set)
+* GROK-20298: `domain-schema.schema.json` accepts the root `storage` key (`{kind: external, connection, schema?, catalog?}`) and the remote-name keys `table`/`column`; `grok api` types an external table's rows with `id` only, an empty expand map, and emits no `<Schema>TransactionOp`/`transaction` for an external schema
+
 ## 6.7.8 (2026-10-13)
 
 * Fixed `grok api` (and every command other than `grok s`) exiting with code 1 and its usage text in 6.7.7: the new `--send-email` flag reached every command as `send-email: false`, which `grok api` rejected as an unknown option

package/CLAUDE.md

@@ -33,6 +33,7 @@ The CLI uses a modular command pattern. Each command is a separate module that:
 - `claude.ts` - Launch a Dockerized dev environment with Datagrok + Claude Code
 - `migrate.ts` - Update legacy packages
 - `init.ts` - Apply configuration to existing packages
+- `src.ts` - Git recipes across the repos of the datagrok umbrella (`grok src`)
 
 The commands `api`, `check`, `link`, `publish`, and `test` support the `--all` flag to run recursively across all packages in the current directory.
 
@@ -121,6 +122,18 @@ When a package has a `dockerfiles/` directory, `grok publish` automatically mana
 
 **Practical implication:** editing any file inside `dockerfiles/<name>/` will cause `grok publish` to rebuild that container image on the next publish, without needing `--rebuild-docker`.
 
+**Publish traps:**
+- `grok publish` runs `npm run build` / `pnpm run build` itself (`buildPackage`); `--skip-check`
+  only skips checks, and `--rebuild` skips the local build rather than forcing one.
+- With no local image and a transient error on the fallback lookup, CLI <= 6.5.10 published
+  `image: null` and the container went to `error`; 6.5.11+ builds instead.
+- A republish renames a package's containers to `<name>_1`.
+- Outside the pnpm workspace, a fresh `npm install` in a package may pull ts-loader 9.6.x /
+  webpack 5.109, which cannot resolve `datagrok-api/*` (tsc can); reuse a known-good
+  `node_modules` (ts-loader 9.5.7 / webpack 5.106) and junction `node_modules/datagrok-api` to a
+  built js-api (`cmd /c mklink /J`).
+- Long publishes outlive a 10-minute agent tool limit; run them detached and poll a log.
+
 ### Testing Framework
 
 Tests use Puppeteer for headless browser automation:

package/GROK_S.md

@@ -36,6 +36,8 @@ browser or a logged-in session.
 | See what fields an entity type has                          | `grok s describe connections`                          |
 | Hit any undocumented endpoint                               | `grok s raw GET /users/current` / `raw POST <path> --data '{...}'` |
 | Check server + per-module health                            | `grok s healthcheck [--module <name>]`                 |
+| Server build, uptime, DB pool, requests in flight           | `grok s status`                                        |
+| Problems, alerts, problem rules, logger settings, errors, cloud logs | `grok s observe problems list --alert-status open` (alias `grok s o`) |
 | Bulk operations in one round-trip                           | `grok s batch <entity> <verb> --json items.json`       |
 | Move entities dev to prod (bundle, or instance to instance) | `grok s pull ... --out ./bundle` / `grok s migrate ... --from dev --to prod` |
 | Browse / query / edit domain-table rows                    | `grok s domains query grit.issue --filter 'status = "open"'` / `domains insert` / `domains upload` |
@@ -136,6 +138,18 @@ grok s connections test --json conn.json              # test before saving
 
 See `public/packages/Chembl/connections/` for connection JSON examples.
 
+Connection tags over REST: the entity JSON carries `entityTags`
+(`[{id, tag, owner: {id: <All users group id>}}]`, no `#type` on the owner), not `tags`.
+`grok s connections save --json` without `entityTags` **clears** the tags (saveTags deletes rows
+not in the list); `POST /entities/save_tags {id, entityTags}` sets them. Filter with
+`whereTag([...])` (`?tags=x`); `whereSmart('tag = "x"')` returns nothing for connections over REST.
+A `Web` connection's parameters must be wrapped (`{"url": {"value": "https://..."}}`); a bare
+string makes the test route answer 400 "Invalid argument(s): value".
+
+A connection saved over REST goes through `Project.add(uniqueName: true)` and can come back
+renamed `<name>_1` for a package's connection — see `core/docs/PROJECTS_AND_PACKAGES.MD`
+"Package connections on a stand".
+
 ## Group membership
 
 `grok s groups add-members` is idempotent — it resolves each member (by login, name,
@@ -161,6 +175,12 @@ ambiguous the command prints every matching group and exits non-zero — pass a
 disambiguate, or add `--user` (accepted by all four membership verbs) to restrict lookup to
 personal groups.
 
+> **Trap:** `grok s groups add-members <group> <users...>` without `--admin` sets `isAdmin=false`
+> on members that are already group admins (reported as `updated`, not `noop`). Before a bulk add
+> run `grok s groups list-members <group> --admin` and exclude those users, or re-add them with
+> `--admin`. Pass the group UUID — `list-members Administrators` by name has returned
+> "(no results)" while the id worked.
+
 ## User administration
 
 ```bash
@@ -198,6 +218,7 @@ grants inherited through the entity's project links (`inherited: true`).
 grok s functions run 'Chem:smilesToMw("CCO")'                  # positional args
 grok s functions run 'Pkg:fn({smiles:"CCO", radius:2})'        # named args
 grok s functions run Pkg:fn --json params.json                 # big input from a file
+grok s functions run 'MyPkg:MyQuery()' --debug                 # a debug call: the query's own log on stderr
 ```
 
 The server binds arguments by parameter name, so positional values are mapped onto the
@@ -411,12 +432,19 @@ grok s domains audit inventory --limit 50                        # DDL and row e
 grok s domains delete inventory --force                          # purge: data, audit, registry, grants
 ```
 
-`schema.json` may be a full manifest or a partial apply body — only `tables`, `extend`,
-`propertySchemas` and `dropTables` are sent; `name`, `version` and `description` are dropped.
-Named tables replace their definition wholesale; untouched tables stay as the registry has
-them. A plan that drops or narrows anything is refused until `--confirm-destructive` is
-passed, and the plan is printed with the refusal. `--if-version` fails the apply when the
-schema's apply counter (or `ext_version` on a package schema) has moved. Package-deployed
+`schema.json` may be a full manifest or a partial apply body — `tables`, `extend`,
+`propertySchemas`, `dropTables` and `access` are sent, plus `friendlyName`, `description` and
+`storage` when the schema is user-managed (`name` is dropped; `version` becomes `ifVersion` and
+`incarnation` becomes `ifIncarnation` unless `--if-version` says otherwise — `current` takes
+both from the schema registered right now, so a name deleted and re-created since the file was
+written is a conflict, not a fresh `1`). Named tables
+replace their definition wholesale; untouched tables stay as the registry has them. A plan
+that drops or narrows anything is refused until `--confirm-destructive` is passed, and the
+plan is printed with the refusal. `--if-version` fails the apply when the schema's apply
+counter (or `ext_version` on a package schema) has moved; a user-managed schema REQUIRES the
+token (the server answers `version-required` without it) — pass the version the edit was made
+against, or `--if-version current` to take the one registered right now. Dropping a table
+needs Delete on that table, dropping its columns Edit on it. Package-deployed
 schemas cannot be applied to or deleted here — `apply` on one is the user-extension path
 (needs `Extend`), and the manifest is owned by `grok publish`. `delete <schema>` requires
 `--force` because it takes every row with it.
@@ -471,6 +499,110 @@ module the server does not report exits 1. Requires a valid dev key (standard `g
 For an anonymous liveness probe — load balancer, k8s readiness — hit `/admin/health` directly;
 it's on the server's unauthenticated allowlist.
 
+### Server status
+
+```bash
+grok s status                                  # build, uptime, DB pool, HTTP connections
+grok s status --output json                    # {server, extended}
+grok s status --output quiet                   # just the version
+```
+
+Reads `GET /info/server` and `GET /info/server?extended=true`: `Version`, `Branch`, `Commit`,
+build `Date`, `Uptime`, `Requests` served, `Used Connections` / `Pool Connections` of the DB pool,
+`Isolate`, `Service network`, `webRoot`, and the `Http Server` connection counts. For an admin the
+extended call replaces the active-connection count with the requests in flight per endpoint
+(`{"GET /api/...": n}`), printed as a table; for anyone else the two responses are the same.
+
+## Problems, logging and errors (`grok s observe`, alias `grok s o`)
+
+Thin commands over the server's routes; `grok s observe` with no command prints every verb and option.
+
+```bash
+grok s o problems list --alert-status open                 # the open alerts; --status, --state, --kind, --since 7d
+grok s o problems get 'health:Jupyter@<server>'            # by id or kind:key; history <id> for its records
+grok s o problems status connection:<connection id> --data '{"status": "muted", "reason": "maintenance", "until": "2026-10-04T06:00:00Z"}'
+grok s o problems ack 'health:Jupyter@<server>' --reason x  # resolve <id> --reason ... closes the alert
+grok s o problems severity dashboard:<project id> critical --reason 'board meeting'   # pin; none unpins
+grok s o rules get > rules.json                             # Settings > Alerts problem rules; rules put --json rules.json
+grok s o rules test                                         # evaluates the saved rules once, now
+grok s o logger get exportSettings                          # the logger settings, or one path of them
+grok s o logger set --user alice --set debugFlags='["db"]' --for 30m --reason "ticket 123"   # an expiring entry, from All users
+grok s o logger set --user alice --remove                  # delete that entry
+grok s o logger history --limit 5                           # log-settings-changed audit records with their diff
+grok s o logs capture alice --for 2h --reason "ticket 123" # logger set of saveHttpRequests on an expiring entry; 30m by default
+grok s o logs capture                                       # the active captures; logs capture stop alice removes one
+grok s o errors top --since 7d --by signature --limit 20    # GET /admin/metrics errors
+grok s o errors top --since 24h --source server           # ranked by distinct users: system errors (scheduler, probes) need --source
+grok s o errors get error-incident:<key>                    # each occurrence of a problem's error: time, user, function, session
+grok s o errors get <signature> --since 1d                  # or of a signature from errors top --by signature
+grok s o timeline --session <id>                            # its events and requests of the last 15 min
+grok s o timeline --session <id> --from 2026-10-09T10:00     # 10 min from then; --to sets the end, at most 2 h later
+grok s o logs cloud --since 2h --filter ERROR                # CloudWatch, the instance's own group; --group, --from/--to
+grok s o logs archive list --prefix cw/ --since 1d           # keys in the log archive (WORM bucket); --since filters the returned page only
+grok s o logs archive read <key>                             # one archived object decoded to events
+grok s o logs container                                      # docker containers and their status
+grok s o logs container scripting-python_1 --since 1h --grep 'Kernel (started|shutdown)'   # one container's log lines
+grok s o logs match 'level in (error, warning) and not text ~ ResizeObserver' --since 2h   # what a log filter matches
+grok s o logs match --level error --package Chembl --grep 'time(d )?out'   # flags add conditions joined by and; --grep is a regex
+grok s o logs match --level audit --space Chem   # audit of entities in the Chem space and its subspaces (case-sensitive)
+grok s o logs match 'space startsWith Chem: or space endsWith :Assays'   # space: the entity's space path; = ~ =~ startsWith endsWith
+grok s o groups                                             # log groups (Settings > Logger): name, kind, levels, filter, where to
+grok s o groups add --name Chembl --levels error,warning --filter 'package = Chembl' --to db,grafana:otlp
+grok s o groups set Chembl --when 'level = error and package = Chembl' --scope session --for 15m --before 60s --flags db
+grok s o groups set Chembl --enabled false                  # only the flags given change; groups rm Chembl deletes it
+grok s o groups put --json groups.json                      # replace every log group (the GROK_PARAMETERS settings.logger.logGroups shape)
+grok s o groups reset                                       # back to the default groups; the audit group can only be switched off, not removed
+grok s o endpoints                                          # where groups can send records; endpoints rm grafana
+grok s o logs activations                                   # triggered groups on now; logs activations stop <group id> <key>
+```
+
+Problems need `ManageAlerts`, errors, another session's timeline and `logs cloud`/`logs archive` `ViewTelemetry`,
+`rules test` both, logger and rules changes, `groups`/`endpoints` changes, `logs activations stop` and `logs capture`
+`EditPluginsSettings`; `logs match` and `logs activations` need `ViewTelemetry`, and changing the audit group or an
+endpoint it sends to also needs `EditAuditLogging`. `logs match` takes the whole filter as one quoted argument; `--grep`
+is a regex and `--space` matches the space and its subspaces, both case-sensitive, as `=~` is. `groups add|set|rm`
+need a server that returns its default log groups; on an older one, replace them all with `groups put`. `capture` is an expiring
+`userGroupSettings` entry with `saveHttpRequests` on (seeded from the All Users entry, with `usage` saved); on an entry
+that already expires later it keeps that time. `capture stop` removes only an expiring entry that holds nothing but the
+capture; for any other entry use `logger set --user <login> --remove`. Read what it recorded with
+`timeline --session <id>`. `logs container` reads the container's latest `--limit` lines (20000) and only then cuts `--since`/`--from`/`--to` (UTC, matched across the container's own offset) and `--grep`, so a window older than those lines comes back empty: raise `--limit`. `logs archive list --since` filters only the page the server returned (`--limit`), so narrow
+the listing with `--prefix` first. `logs` read with the server's AWS role, or with `--connection <id>`. In durations `m` means minutes (in `grok s pull --since` it means months).
+
+### Logging one command (`--log`)
+
+`logger set --user` changes what a user logs for a while; `--log` makes the server keep what the
+requests of one command log, and prints it at the end. It works on every `grok s` command, `grok s o`
+included:
+
+```bash
+grok s functions run UsageAnalysis:Groups --log                # verbose: every level
+grok s functions run UsageAnalysis:Groups --log verbose,db     # and every SQL statement (the db debug flag)
+grok s users list --log debug,storage,credentials              # levels and flags; flags alone add debug
+grok s functions run 'MyPkg:MyQuery()' --debug                 # the query's own debug log, plus errors and warnings
+grok s users list --log --output json 2>log.jsonl              # one JSON object per record
+```
+
+Every request of the command carries one `datlas-log-id` (a new UUID per command) and the spec in
+`datlas-log`. A command is often several requests, and each can land on a different isolate and
+server instance: each keeps what its request logged at the spec in memory, in a buffer for that id.
+Those records never reach the database, the console or a sync destination, and what the server logs
+the usual way does not change. When the command ends, `GET /log/requests/<logId>` gathers the
+buffers from every isolate of every instance over the message queue, hands them over and forgets
+them; the CLI prints them to stderr, so stdout keeps the result. The gather also adds what the
+server saved the usual way for the command's session at the levels of the spec (errors, warnings, audit),
+skipping copies of records it already has, so an instance that died before it could answer still
+shows its saved errors. A buffer nobody gathers is dropped after an hour. No permission is needed; another user's id takes `ViewTelemetry`.
+`grok s o logs request <logId>` gathers one id by hand.
+
+The spec is comma-separated: `verbose`, `standard`, the levels (`error`, `warning`, `info`, `debug`,
+`audit`, `usage`) and the debug flags of the logger settings (`db`, `socket`, `storage`, `credentials`,
+`docker`, `auth`, ... or `all`).
+
+`functions run --debug` sends `?debug=true`: the call runs as a debug call (not cached), the way the
+query debugger in the UI runs it, and its own log (connection, the query's stages) is kept with the
+rest; without `--log` the spec is `error,warning`. Only what runs in Datlas is kept: a script that
+runs on a queue worker adds nothing here.
+
 ## Describing an entity type
 
 ```bash

package/README.md

@@ -151,3 +151,17 @@ For more information on configuring connections, refer to the [Connections](http
   | debug          | Sets a debug breakpoint before each test execution, allowing you to debug the test. Only works in `gui` mode                          |
 
 - `link` command is used for public plugins development to link `datagrok-api` and libraries.
+- `connect-folder` makes a local folder a Datagrok connection. A folder agent in Docker dials out to the server (no
+  inbound port) with a key of its own that can only serve this folder; your keypair (`grok login <server>` first)
+  never leaves the machine. Connecting the same folder again resumes the same connection. Requires Docker and the
+  "Connect local folders" permission.
+
+  ```shell
+  grok connect-folder <folder> [--host <alias>] [--read-only] [--foreground] [--image <ref>] [--force]
+  grok connect-folder list [--host <alias>] [--output json]
+  grok connect-folder start|stop|logs <connection> [-f]
+  grok connect-folder rm <connection>
+  ```
+
+  `<connection>` is the connection name, a connection id prefix (8+ characters), or the folder. `stop` keeps the
+  folder disconnected after a reboot; `rm` removes the container, the key and the connection.

package/bin/commands/add.js

@@ -16,8 +16,7 @@ function _interopRequireWildcard(e, t) { if ("function" == typeof WeakMap) var r
  * answer key): u2 is not on npm, so the dependency is the workspace link, and `u2core` is the one
  * external `@datagrok/build-config` does not know about. The css imports and the tsconfig need
  * nothing — the shared bundler config carries the loaders and the tsconfig extends the base. */
-const u2Dependency = 'workspace:^';
-const u2DgModule = '@datagrok-libraries/u2/src/dg/index.js';
+const emsDgModule = '@datagrok-libraries/ems/src/dg';
 const u2RspackExternal = `'datagrok-api/u2core': 'DG.U2'`;
 const u2RspackConfig = [`const {bundler} = require('@datagrok/build-config');`, '', `module.exports = bundler({externals: {${u2RspackExternal}}});`, ''].join('\n');
 function add(args) {
@@ -110,7 +109,7 @@ function add(args) {
    * existing imports (which webpack's externals depend on). */
   function addU2Imports() {
     const contents = _fs.default.readFileSync(packageEntry, 'utf8');
-    if (contents.includes(u2DgModule)) return;
+    if (contents.includes(emsDgModule)) return;
     const eol = contents.includes('\r\n') ? '\r\n' : '\n';
     const imports = _fs.default.readFileSync(_path.default.join(templateDir, 'entity-template', 'domain-app-imports.js'), 'utf8').trimEnd().split(/\r?\n/);
     const lines = contents.split(/\r?\n/);
@@ -125,7 +124,8 @@ function add(args) {
   function addU2Wiring() {
     const packageObj = JSON.parse(_fs.default.readFileSync(packagePath, 'utf8'));
     packageObj.dependencies = packageObj.dependencies ?? {};
-    packageObj.dependencies['@datagrok-libraries/u2'] ??= u2Dependency;
+    packageObj.dependencies['@datagrok-libraries/ems'] ??= 'workspace:^';
+    packageObj.dependencies['@datagrok-libraries/u2'] ??= 'workspace:^';
     _fs.default.writeFileSync(packagePath, JSON.stringify(packageObj, null, 2), 'utf8');
     if (!_fs.default.existsSync(rspackConfigPath)) _fs.default.writeFileSync(rspackConfigPath, u2RspackConfig, 'utf8');else if (!_fs.default.readFileSync(rspackConfigPath, 'utf8').includes('datagrok-api/u2core')) color.warn(`rspack.config.js is already there — add \`externals: {${u2RspackExternal}}\` to it by hand`);
   }

package/bin/commands/api.js

@@ -24,20 +24,30 @@ const dbFile = 'db.ts';
 const dbUiFile = 'db-ui.ts';
 /** Where `db-ui.ts` takes the u2 domain stack from — the same specifier Grit and Stockroom use
  * (u2 is a relative-path dependency without a `dg` subpath export). */
-const u2DgModule = '@datagrok-libraries/u2/src/dg/index.js';
+const emsDgModule = '@datagrok-libraries/ems/src/dg';
 
-/** Naive English plural of a (snake_case) table name — `issue` → `issues`, `box` →
- * `boxes`, `category` → `categories`. */
+/** The platform's `StringUtils.toSingular` (ddt utils.dart): regular forms only; a noun already
+ * singular in `-is`/`-us`/`-ss` passes through. */
+function singularizeTableName(name) {
+  if (name.length <= 2 || !name.endsWith('s') || /(ss|is|us)$/.test(name)) return name;
+  if (name.endsWith('ies') && name.length > 3) return name.slice(0, -3) + 'y';
+  return /(xes|zzes|ches|shes|sses|tuses)$/.test(name) ? name.slice(0, -2) : name.slice(0, -1);
+}
+
+/** The platform's `StringUtils.toPlural` (ddt utils.dart) — `issue` → `issues`, `box` →
+ * `boxes`, `category` → `categories`, `analysis` → `analyses`. */
 function pluralizeTableName(name) {
+  if (name.endsWith('is')) return name.slice(0, -2) + 'es';
   if (/(s|x|z|ch|sh)$/.test(name)) return name + 'es';
-  if (/[^aeiou_]y$/.test(name)) return name.slice(0, -1) + 'ies';
+  if (/[^aeiou]y$/.test(name)) return name.slice(0, -1) + 'ies';
   return name + 's';
 }
 
 /** The camelCase plural property name a table gets on the schema client and under the
- * schema handle's `tables` (`issue_label` → `issueLabels`). */
+ * schema handle's `tables` (`issue_label` → `issueLabels`, `categories` → `categories`): the
+ * plural of the singular, as the platform derives a table's plural name. */
 function tableProp(tableName) {
-  return utils.snakeToCamelCase(pluralizeTableName(tableName), false);
+  return utils.snakeToCamelCase(pluralizeTableName(singularizeTableName(tableName)), false);
 }
 const domainSchemaPath = _path.default.join(_path.default.dirname(_path.default.dirname(__dirname)), 'domain-schema.schema.json');
 /** The platform's own schema, whose tables a qualified ref may target (`Core.users`, ...). */
@@ -229,7 +239,7 @@ function generateDomainClients(packageDir = curDir, options) {
   _fs.default.writeFileSync(_path.default.join(genDir, dbFile), normEol(content).replace(/\n/g, '\r\n'), 'utf8');
   color.log(`Successfully generated file src/generated/${dbFile}${sep}`, 'success');
   if (ui) {
-    const uiContent = annotationForDbUiFile + `import {domains, DomainTable} from '${u2DgModule}';${sep}` + wrapTokens(`import {`, dbImports, ', ', `} from './${dbFile.replace(/\.ts$/, '')}';`) + sep + sep + uiParts.join(sep);
+    const uiContent = annotationForDbUiFile + `import {domains, DomainTable} from '${emsDgModule}';${sep}` + wrapTokens(`import {`, dbImports, ', ', `} from './${dbFile.replace(/\.ts$/, '')}';`) + sep + sep + uiParts.join(sep);
     _fs.default.writeFileSync(_path.default.join(genDir, dbUiFile), normEol(uiContent).replace(/\n/g, '\r\n'), 'utf8');
     color.log(`Successfully generated file src/generated/${dbUiFile}${sep}`, 'success');
   }
@@ -265,6 +275,10 @@ function findDependencyManifest(packageDir, schemaName) {
 function generateDomainSchemaCode(manifest, manifestPath, emittedTypes, packageDir) {
   const decls = [];
   const systemColumnNames = new Set(domainSystemColumns.map(([name]) => name));
+  // An external table's rows carry `id` (the encoded business key) and nothing else
+  // system-side; nothing is expanded and nothing is written (external-bindings §3).
+  const external = manifest.storage?.kind === 'external';
+  const systemColumns = external ? domainSystemColumns.filter(([name]) => name === 'id') : domainSystemColumns;
   const tableNames = Object.keys(manifest.tables);
   // Keyed by table name; a qualified ref target ('Core.queries', 'grit.issue') lands here
   // under its qualified key so the expand map can list its columns — it gets no accessor.
@@ -273,7 +287,9 @@ function generateDomainSchemaCode(manifest, manifestPath, emittedTypes, packageD
   // different sets fall back to a `<alias><PascalTable>` name (deterministic).
   const choicesAliases = {};
   const choicesAlias = (tableName, columnName, values) => {
-    const valueSet = values.map(v => `'${v}'`).join(' | ');
+    // JSON.stringify escapes every special character; inside single quotes only ' needs one more
+    const literals = values.map(v => `'${JSON.stringify(v).slice(1, -1).replace(/'/g, `\\'`)}'`);
+    const valueSet = literals.join(' | ');
     let alias = utils.snakeToCamelCase(tableName) + utils.snakeToCamelCase(columnName);
     if (choicesAliases[alias] !== undefined && choicesAliases[alias] !== valueSet) alias += utils.snakeToCamelCase(tableName);
     if (choicesAliases[alias] !== undefined && choicesAliases[alias] !== valueSet) {
@@ -282,7 +298,7 @@ function generateDomainSchemaCode(manifest, manifestPath, emittedTypes, packageD
     }
     if (choicesAliases[alias] === undefined) {
       choicesAliases[alias] = valueSet;
-      decls.push(wrapTokens(`export type ${alias} = `, values.map(v => `'${v}'`), ' | ', ';'));
+      decls.push(wrapTokens(`export type ${alias} = `, literals, ' | ', ';'));
     }
     return alias;
   };
@@ -360,6 +376,7 @@ function generateDomainSchemaCode(manifest, manifestPath, emittedTypes, packageD
         insertType: column.type === 'datetime' ? 'Dayjs | string' : tsType,
         required: column.required === true,
         autoNumber: column.autoNumber != null,
+        hasDefault: column.default != null,
         ref: ref
       });
       return true;
@@ -376,6 +393,7 @@ function generateDomainSchemaCode(manifest, manifestPath, emittedTypes, packageD
     }
     return columns;
   }
+  const accessorTables = new Map();
   for (const tableName of tableNames) {
     const typeName = utils.snakeToCamelCase(tableName);
     if (emittedTypes.has(typeName)) {
@@ -383,6 +401,12 @@ function generateDomainSchemaCode(manifest, manifestPath, emittedTypes, packageD
       return null;
     }
     emittedTypes.add(typeName);
+    const accessor = tableProp(tableName);
+    if (accessorTables.has(accessor)) {
+      color.error(`${manifestPath}: tables '${accessorTables.get(accessor)}' and '${tableName}' ` + `both get the accessor '${accessor}' (rename one of them)`);
+      return null;
+    }
+    accessorTables.set(accessor, tableName);
     const columns = collectColumns(tableName, manifest, false);
     if (columns == null) return null;
     tableColumns[tableName] = columns;
@@ -427,12 +451,12 @@ function generateDomainSchemaCode(manifest, manifestPath, emittedTypes, packageD
     const columns = tableColumns[tableName];
     const relations = tableRelations[tableName];
     const rowLines = [`/** Row of \`${manifest.name}.${tableName}\`. */`, `export interface ${typeName}Row {`];
-    for (const [name, tsType] of domainSystemColumns) rowLines.push(`  ${name}: ${tsType};`);
+    for (const [name, tsType] of systemColumns) rowLines.push(`  ${name}: ${tsType};`);
     for (const c of columns) rowLines.push(`  ${c.name}${c.required || c.autoNumber ? '' : '?'}: ${c.tsType};`);
     rowLines.push('}');
     decls.push(rowLines.join(sep));
     const insertLines = [`/** Insert payload for \`${manifest.name}.${tableName}\`. */`, `export interface ${typeName}Insert {`];
-    for (const c of columns) insertLines.push(`  ${c.name}${c.required ? '' : '?'}: ${c.insertType};`);
+    for (const c of columns) insertLines.push(`  ${c.name}${c.required && !c.hasDefault ? '' : '?'}: ${c.insertType};`);
     for (const r of relations) insertLines.push(`  /** Link set: \`${r.target}\` row ids. */`, `  ${r.name}?: string[];`);
     if (table.idempotency === true) insertLines.push('  idempotencyKey?: string;');
     insertLines.push('}');
@@ -447,13 +471,13 @@ function generateDomainSchemaCode(manifest, manifestPath, emittedTypes, packageD
       updateLines.push('};');
       decls.push(updateLines.join(sep));
     }
-    decls.push(formatColumnUnion(`${typeName}Column`, [...domainSystemColumns.map(([name]) => name), ...columns.map(c => c.name)]));
+    decls.push(formatColumnUnion(`${typeName}Column`, [...systemColumns.map(([name]) => name), ...columns.map(c => c.name)]));
 
     // Expand map: MUST stay a `type` alias — object-type literals carry the implicit
     // index signature the `TExpand extends {[key: string]: {}}` constraint needs
     // (an interface would fail it).
     const expandEntries = [];
-    for (const c of columns) {
+    for (const c of external ? [] : columns) {
       if (c.ref == null) continue;
       const fields = tableColumns[c.ref].map(tc => `'${c.name}.${tc.name}'?: ${tc.tsType}`);
       expandEntries.push(wrapTokens(`  '${c.name}': {`, fields, '; ', '};', '    '));
@@ -463,15 +487,16 @@ function generateDomainSchemaCode(manifest, manifestPath, emittedTypes, packageD
     // The link shape stays INLINE (structurally DG.DomainRelationLink): generated
     // files must compile against the datagrok-api the package depends on, which may
     // predate that interface.
-    for (const r of relations) expandEntries.push(`  '${r.name}': {${r.name}?: {id: string; name: string}[]};`);
-    for (const childName of tableNames) {
+    for (const r of external ? [] : relations) expandEntries.push(`  '${r.name}': {${r.name}?: {id: string; name: string}[]};`);
+    for (const childName of external ? [] : tableNames) {
       const fks = tableColumns[childName].filter(c => c.ref === tableName);
       if (fks.length === 0) continue;
       const childType = utils.snakeToCamelCase(childName);
       if (fks.length === 1) expandEntries.push(`  'details:${childName}': {${childName}?: ${childType}Row[]};`);else for (const fk of fks) expandEntries.push(`  'details:${childName}.${fk.name}': {${childName}?: ${childType}Row[]};`);
     }
     decls.push(expandEntries.length === 0 ? `export type ${typeName}Expand = {};` : [`/** Expand keys of \`${manifest.name}.${tableName}\` → fields each adds to the row ` + `(consumed by query()/builder). */`, `export type ${typeName}Expand = {`, ...expandEntries, '};'].join(sep));
-    txArms.push(`  {op: 'insert'; table: '${tableName}'; ref?: string; values: DG.DomainTxValues<${typeName}Insert>; ` + `onDuplicate?: 'error'} |`, `  {op: 'update'; table: '${tableName}'; id: string; ` + `values: DG.DomainTxValues<${relations.length > 0 ? `${typeName}Update` : `Partial<${typeName}Row>`}>; ` + `expectedVersion?: number} |`, `  {op: 'delete'; table: '${tableName}'; id: string} |`);
+    if (!external) txArms.push(`  {op: 'insert'; table: '${tableName}'; ref?: string; values: DG.DomainTxValues<${typeName}Insert>; ` + `onDuplicate?: 'error'} |`, `  {op: 'update'; table: '${tableName}'; id: string; ` + `values: DG.DomainTxValues<${relations.length > 0 ? `${typeName}Update` : `Partial<${typeName}Row>`}>; ` + `expectedVersion?: number} |`, `  {op: 'delete'; table: '${tableName}'; id: string} |`);
+    if (!external && table.softDelete !== false) txArms.push(`  {op: 'restore'; table: '${tableName}'; id: string} |`);
 
     // No datetime config: the client resolves datetime columns from the domain
     // registry itself (the Dayjs typing in <Table>Row stays true for every client).
@@ -480,10 +505,14 @@ function generateDomainSchemaCode(manifest, manifestPath, emittedTypes, packageD
     clients.push(`  get ${tableProp(tableName)}() {`, `    return grok.dapi.domains.table<${typeName}Row, ${typeName}Insert, ` + `${typeName}Column, ${typeName}Expand` + `${relations.length > 0 ? `, ${typeName}Update` : ''}>('${manifest.name}.${tableName}');`, '  },');
   }
   const schemaType = utils.snakeToCamelCase(manifest.name);
-  const lastArm = txArms.pop();
-  txArms.push(lastArm.replace(/ \|$/, ';'));
-  decls.push([`export type ${schemaType}TransactionOp =`, ...txArms].join(sep));
-  decls.push([`/** Typed clients for the \`${manifest.name}\` domain schema tables ` + `(lazy — no import-time side effects). */`, `export const ${utils.snakeToCamelCase(manifest.name, false)}Db = {`, ...clients, `  transaction<T extends ${schemaType}TransactionOp[]>(ops: [...T]):`, `      Promise<{[K in keyof T]: DG.DomainOpResultFor<T[K]>}> {`, `    return grok.dapi.domains.transaction('${manifest.name}', ops) as`, `      Promise<{[K in keyof T]: DG.DomainOpResultFor<T[K]>}>;`, '  },', '};'].join(sep));
+  const transaction = [];
+  if (!external) {
+    const lastArm = txArms.pop();
+    txArms.push(lastArm.replace(/ \|$/, ';'));
+    decls.push([`export type ${schemaType}TransactionOp =`, ...txArms].join(sep));
+    transaction.push(`  transaction<T extends ${schemaType}TransactionOp[]>(ops: [...T]):`, `      Promise<{[K in keyof T]: DG.DomainOpResultFor<T[K]>}> {`, `    return grok.dapi.domains.transaction('${manifest.name}', ops) as`, `      Promise<{[K in keyof T]: DG.DomainOpResultFor<T[K]>}>;`, '  },');
+  }
+  decls.push([`/** Typed clients for the \`${manifest.name}\` domain schema tables ` + `(lazy — no import-time side effects). */`, `export const ${utils.snakeToCamelCase(manifest.name, false)}Db = {`, ...clients, ...transaction, '};'].join(sep));
   return decls.join(sep.repeat(2)) + sep;
 }
 

package/bin/commands/check.js

@@ -13,6 +13,7 @@ exports.checkImportStatements = checkImportStatements;
 exports.checkNpmIgnore = checkNpmIgnore;
 exports.checkPackageFile = checkPackageFile;
 exports.checkSourceMap = checkSourceMap;
+exports.checkU2Usage = checkU2Usage;
 exports.extractExternals = extractExternals;
 var _fs = _interopRequireDefault(require("fs"));
 var _path = _interopRequireDefault(require("path"));
@@ -20,6 +21,7 @@ var _ignoreWalk = _interopRequireDefault(require("ignore-walk"));
 var utils = _interopRequireWildcard(require("../utils/utils"));
 var color = _interopRequireWildcard(require("../utils/color-utils"));
 var _const = require("../utils/const");
+var _child_process = require("child_process");
 function _interopRequireWildcard(e, t) { if ("function" == typeof WeakMap) var r = new WeakMap(), n = new WeakMap(); return (_interopRequireWildcard = function (e, t) { if (!t && e && e.__esModule) return e; var o, i, f = { __proto__: null, default: e }; if (null === e || "object" != typeof e && "function" != typeof e) return f; if (o = t ? n : r) { if (o.has(e)) return o.get(e); o.set(e, f); } for (const t in e) "default" !== t && {}.hasOwnProperty.call(e, t) && ((i = (o = Object.defineProperty) && Object.getOwnPropertyDescriptor(e, t)) && (i.get || i.set) ? o(f, t, i) : f[t] = e[t]); return f; })(e, t); }
 const warns = ['Latest package version', 'Datagrok API version should contain', 'is served by the platform'];
 const forbiddenNames = ['function', 'class', 'export'];
@@ -91,6 +93,8 @@ function runChecks(packagePath, soft = false, noExit = false) {
     if (externals) errors.push(...checkImportStatements(packagePath, jsTsFiles, externals));
   }
   errors.push(...checkDatagrokApiImports(packagePath, jsTsFiles));
+  const u2Errors = checkU2Usage(packagePath, json);
+  errors.push(...u2Errors);
   warnings.push(...checkHeavyImports(packagePath, jsTsFiles, _path.default.basename(packagePath)));
   if (!soft) errors.push(...checkSourceMap(packagePath));
   errors.push(...checkNpmIgnore(packagePath));
@@ -111,7 +115,8 @@ function runChecks(packagePath, soft = false, noExit = false) {
   if (errors.length) {
     console.log(`Checking package ${_path.default.basename(packagePath)}...`);
     showError(errors);
-    if (soft || json.version.startsWith('0') || errors.every(w => warns.some(ww => w.includes(ww)))) return true;
+    // the u2 consumer ratchet holds pre-1.0 packages too
+    if (soft || u2Errors.length === 0 && (json.version.startsWith('0') || errors.every(w => warns.some(ww => w.includes(ww))))) return true;
     if (noExit) return false;else process.exit(1);
   }
   color.log(`Checking package ${_path.default.basename(packagePath)}...\t\t\t\u2713 OK`);
@@ -213,6 +218,43 @@ function checkDatagrokApiImports(packagePath, files) {
   }
   return errors;
 }
+
+/** u2's consumer check for a package built on u2 or ems: the u2 lint rules and the shrink-only counts of
+ * `consumers-baseline.json`. Prints the ratchet line; growth and enforced rules are errors. */
+function checkU2Usage(packagePath, json) {
+  const deps = {
+    ...json.dependencies,
+    ...json.devDependencies
+  };
+  if (!('@datagrok-libraries/u2' in deps) && !('@datagrok-libraries/ems' in deps)) return [];
+  let script;
+  try {
+    script = _path.default.join(_path.default.dirname(require.resolve('@datagrok-libraries/u2/package.json', {
+      paths: [packagePath]
+    })), 'scripts', 'check-consumer.mjs');
+  } catch {
+    return [];
+  }
+  if (!_fs.default.existsSync(script)) return [];
+  const run = (0, _child_process.spawnSync)(process.execPath, [script, packagePath, '--json'], {
+    encoding: 'utf-8',
+    maxBuffer: 64 * 1024 * 1024,
+    timeout: 300000
+  });
+  let report;
+  try {
+    report = JSON.parse(run.stdout);
+  } catch {
+    return [`u2 consumer check could not run: ${(run.stderr || run.stdout || String(run.error)).trim()}`];
+  }
+  const errors = [];
+  for (const tree of report.trees) {
+    for (const line of tree.summary) console.log(line);
+    for (const note of tree.notes) color.log(`  ${note}`, 'warn');
+    errors.push(...tree.problems);
+  }
+  return errors;
+}
 function checkHeavyImports(packagePath, files, packageName = '') {
   const warnings = [];
   const isTestPackage = /tests?$/i.test(packageName);