@altimateai/altimate-code 0.12.6 → 0.12.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (5) hide show
  1. package/CHANGELOG.md +80 -0
  2. package/bin/altimate-code +7 -1
  3. package/package.json +10 -10
  4. package/skills/altimate-setup/SKILL.md +2 -2
  5. package/skills/data-question/SKILL.md +192 -0
package/CHANGELOG.md CHANGED
@@ -5,6 +5,86 @@ All notable changes to this project will be documented in this file.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [0.12.8] - 2026-10-23
9
+
10
+ **Claude Code can work through your Altimate workspace.** With the Altimate Code plugin installed and the project linked, tasks Claude hands to altimate-code run with the workspace's skills, knowledge and shared memory, and a headless `run` no longer hangs waiting on a stdin nobody writes to. Also in this release: the curl and PowerShell installers add an `altimate-code` command, the analyst can read workspace memory, `learn status` shows whether a lesson keeps being corrected, `schema_inspect` shows what column values look like, and SQL policy checks fail closed on statements they cannot analyze.
11
+
12
+ **Heads-up for support:**
13
+ - **`run` waits at most 3 s on a socket stdin.** When `run` already has a message or `--command` and stdin is a socket (an agent's shell tool, a Node `spawn`), it waits 3 s for the first byte, then continues without stdin and prints one line on stderr; input sent later is ignored. Files and shell pipelines (`dbt build | altimate-code run …`) are still read to the end. `ALTIMATE_RUN_STDIN_WAIT_MS` changes the wait (`0` waits for EOF), and `< /dev/null` (`< NUL` in cmd.exe) skips it. On Windows a pipe is not a socket, so a caller that never closes stdin should redirect it (#1577).
14
+ - **`run` may end with a line about unsent memory**, giving the count and `altimate-code workspace sync`. It goes to stderr; scripts that parse stderr will see it.
15
+ - **Plain `run` prompts are no longer quoted.** Arguments containing spaces used to reach the model wrapped in quotes; they now arrive as typed. `--command` keeps its quoting.
16
+ - **`check` policy and semantic checks fail closed.** altimate-core 0.9.0 accepts table-valued functions (`TABLE(fn())`, Snowflake `LATERAL FLATTEN`, DuckDB `read_csv(...)`) that 0.8.0 rejected as invalid, but cannot plan them. Policy now reports `policy_not_evaluated` and treats the statement as blocked; semantics reports it as skipped instead of clean.
17
+ - **`schema_inspect` sends sampled column values to the model provider** by default (see Added). `values: false` turns it off per call, and `ALTIMATE_DISABLE_SCHEMA_VALUES=1` turns it off for every call.
18
+ - **`error_fingerprint` telemetry carries a masked `error_message`**: the first line of the error with every word outside a fixed list of common error terms replaced by `<x>`, at most 200 characters. No identifiers, paths, values or SQL.
19
+ - **Read and bash deny rules match equivalent path spellings** (relative, absolute, after `cd`, through a symlink). A deny rule can now stop a command that used to slip past it; allow rules are unchanged.
20
+ - **Warehouse connections are kept per project directory.** A `serve` process serving two projects no longer shares one project's connections with the other. Global and project connection files load as before; a running server still does not re-read an edited `connections.json`.
21
+ - **`datamate_manager` is not offered in projects linked to a workspace**; integrations there come from the workspace's engine. Existing `datamate-*` entries still load, with a one-time notice naming their file.
22
+
23
+ ### Added
24
+
25
+ - **Claude Code plugin path to workspaces.** The plugin hands work to `altimate-code run` in the linked workspace; the first run in a project pulls the workspace's skills; `workspace list [--json]` lists the workspaces an account can link to. See the Claude Code guide. (#1552)
26
+ - **`altimate-code` from the standalone installers.** curl installs link `altimate-code` to `altimate`; PowerShell installs add `altimate-code.cmd` and a Git Bash shim. Re-running the installer adds it to an existing install. Scripts on Windows should call `altimate`, since the `.cmd` shim passes arguments through cmd.exe. (#1562)
27
+ - **Column value profiles in `schema_inspect`.** On SQLite, DuckDB, PostgreSQL and MongoDB it adds, per column, distinct count, null share, the most frequent values with counts, uniqueness in the sample and common value shapes, from a bounded sample, so filters match the stored format. MongoDB nested fields are listed by path. (#1542)
28
+ - **`ALTIMATE_DISABLE_SCHEMA_VALUES`** turns those value profiles off for every call, whatever the model asks for. (#1578)
29
+ - **Lesson recurrence in `learn status`.** It shows whether a lesson that was shown still drew a correction on the same topic, and flags recurring lessons to reflection. (#1556)
30
+ - **`/integrations` in the TUI** opens the workspace's integration status directly, or offers the link or the engine install when those are missing. (#1464)
31
+ - **Workspace sync state for every client.** When skills and memory were last checked and what changed is recorded and published, so the IDE extension can show it as the TUI sidebar does. (#1429)
32
+ - **Analyst data analysis** gets query deadlines, result envelopes, a scratch bridge (`sql_execute` `query_from`) and verification checks. (#1441)
33
+ - **`answer_commit` asks once for a missed item.** A commit with fewer fields than the request asked for ("the X and its Y") is rejected once, naming the clause; an unchanged recommit is accepted. (#1553)
34
+ - **`client_connected` telemetry** marks the first client to attach to a running server. (#1528)
35
+
36
+ ### Changed
37
+
38
+ - **The analyst can read workspace memory** (read, refresh and audit) and is steered to workspace skills. Saving memory blocks stays with builder; `training_save` still works. (#1570)
39
+ - **altimate-core 0.9.0**, with policy and semantic checks failing closed as described above. The messages now say to review such statements by hand. (#1555, #1585)
40
+ - **`datamate_manager` is off in linked projects.** (#1431)
41
+ - **Plain `run` prompt arguments are no longer quoted.** (#1561)
42
+
43
+ ### Fixed
44
+
45
+ - **Headless `run` no longer hangs under an agent's shell.** (#1552)
46
+ - **"database is locked" when several CLI processes start on a fresh data directory.** Setup is serialized with a lock that the OS releases if its holder dies; after 5 s of waiting, CLI commands such as `run` and `serve` print the lock they are waiting on to stderr; the TUI writes it to the log. (#1541, #1580)
47
+ - **Warehouse connections no longer leak between projects in one server**, including SSH tunnels, which are now owned by the connect attempt that opened them. (#1567)
48
+ - **`warehouse_list` with no connections points at the real config locations** and the `ALTIMATE_CODE_CONN_*` variables. (#1559)
49
+ - **Workspace memory and link checks match a repo's remote across its URL forms**, using the same comparison as the workspace service. (#1462)
50
+ - **A message sent while the assistant is busy attaches the workspace engine.** (#1461)
51
+ - **Prompt caching works across working directories.** The environment section moved after the cached instructions, and tool schemas no longer embed the directory. (#1517)
52
+ - **Gemini:** tool schemas no longer contain empty-string enum values, which Gemini rejected with a 400 on every request; small calls on Gemini 4 use thinking levels instead of a budget it rejects. (#1532, #1534)
53
+ - **Large MongoDB reads are fast again.** Byte-bounded cursors read in 1,000-document batches instead of one document per round trip. (#1480)
54
+ - **DuckDB schema indexing** stays in the current catalog, no longer duplicates schemas from attached catalogs, and stops at a 30 s default deadline with partial results. (#1514)
55
+ - **The analyst reports tool failures as failures**, accepts valid read-only syntax and retrieves long text in full. (#1448)
56
+ - **`run --model` reports a failed provider list as itself**, with the HTTP status, not "Model not found". (#1472)
57
+ - **A failed background file scan is handled** instead of surfacing as an unhandled rejection. (#1471)
58
+ - **The read-only SQL check blocks PostgreSQL replication-slot copy and sync functions.** `pg_copy_logical_replication_slot`, `pg_copy_physical_replication_slot` and `pg_sync_replication_slots` were not on the side-effect list, so a top-level call passed as read-only; the create and drop slot functions were already blocked. Found in this release's review.
59
+ - **`uninstall` on Windows gives PowerShell commands, and says so**, and now also offers to remove the empty `~/.altimate/bin` directory (previously only a legacy `~/.opencode/bin`, with a POSIX `rmdir … 2>/dev/null` line even on Windows). Found in this release's review.
60
+
61
+ ## [0.12.7] - 2026-10-13
62
+
63
+ **Workspace skills and knowledge come first.** In a linked project the agent prefers the workspace's own skills over lookalikes in the repo or your home folder, and checks the workspace's knowledge base for best practices without being asked. Linking now ends with a starter that says what the workspace provides and suggests prompts. Also in this release: opt-in automatic promotion for `learn`, session traces for the IDE extension, and Snowflake reconnects written to `opencode.log`.
64
+
65
+ **Heads-up for support:**
66
+ - **The start-screen workspace box is back.** 0.12.6 removed it because it stopped every keybinding once it loaded. The cause is fixed, and Ctrl+C, Ctrl+P and Esc were checked on the start screen of the release build, linked, unlinked and with the workspace service unreachable. If a signed-in user still reports dead keys, `ALTIMATE_DISABLE_WORKSPACE=1` remains the stopgap.
67
+ - **The skill list changes for everyone, not only workspace users.** Each skill the agent sees now carries its source (`workspace`, `project`, `built-in`, `personal` or `other`), and the agent is told to prefer them in that order. So a skill in the repository now outranks a personal one with a similar purpose. It is guidance to the agent, not a filter: every skill is still listed and can be asked for by name.
68
+ - **The agent may search the workspace knowledge base on its own.** When the linked workspace has knowledge, the agent consults it for best practices and team conventions without the user asking. Unlinking the project, or turning knowledge off for the workspace, stops it.
69
+ - **`serve` has new read-only routes for the IDE extension** (see Added). Traces hold prompts and tool output, including SQL and query results. Like every `serve` route, they need no password unless `OPENCODE_SERVER_PASSWORD` is set, so set one on a shared machine.
70
+
71
+ ### Changed
72
+
73
+ - **Workspace skills and knowledge are preferred.** Every skill in the agent's list is labelled with where altimate-code found it, and one rule tells the agent to load the highest source that fits instead of asking which team's skill to follow. A project skill with exactly the same name as a workspace skill still replaces it. Listings stay in name order, so built-in skills are not pushed out of the skill tool's list. With knowledge on, the workspace section also tells the agent to check the knowledge base before answering best-practice questions. (#1435)
74
+
75
+ ### Added
76
+
77
+ - **A starter after linking.** `altimate-code link` ends with what the workspace provides (skills, knowledge, integrations, memory setting) and up to three prompts to try; the TUI shows the same after a link and from `/workspace` → **Get started**, where picking a prompt fills the input without sending it. It makes no model call, and anything not yet known is left out rather than shown as "none". `GET /altimate/workspace/starter` serves it to the IDE extension. (#1435)
78
+ - **`learn` can promote lessons without a person, if you opt in.** `learn enable --auto-promote` (or `learn.auto_promote: true`, or `ALTIMATE_LEARN_AUTO_PROMOTE=1`) promotes a lesson an automatic reflection staged when every check passes: it came from a correction, review or CI signal; it is exactly what was staged; it validates like a manual promote; it does not weaken a verification step; it does not edit or remove a lesson a person approved or pinned; and it stays within 3 changes per promotion and 5 per day (`learn.auto_promote_max_changes`, `learn.auto_promote_daily`). Otherwise the candidate stays staged and `learn status` says why. Off by default, needs capture and automatic reflection on, and `learn rollback` undoes a promotion. (#1418)
79
+ - **Session traces for the IDE extension.** `serve` adds `GET /altimate/trace` (a page of traces, newest first) and `GET /altimate/trace/<session>/view` (the trace viewer, without Copy Link), so the extension's `/traces` can list and open them. Browser-originated requests are refused, as for the workspace routes. (#1415)
80
+
81
+ ### Fixed
82
+
83
+ - **The TUI's keybindings keep working with the start-screen workspace box.** The box rendered nothing until the project's link was resolved; each time its lines arrived, the start screen remounted it empty, and it resolved again without end, which held up every keybinding. It now keeps a root element from its first render. (#1435)
84
+ - **Snowflake reconnects are in `opencode.log`.** When Snowflake closes a session and the driver reopens it, the log gains `reconnecting`, then `reconnected` (with the reason, how long it took, how many session settings were restored, and whether temporary tables or a transaction may have been lost) or `reconnect failed` (a warning with the masked error). The lines name the account, and the connection when only one uses it; they never contain SQL. Before, a session that kept dropping left no trace. (#1427)
85
+ - **A trace containing `<!--<script` no longer opens as an empty page.** The viewer embeds the trace in an inline script, and that text in tool output (a fetched web page, say) stopped the script from running. Affected the TUI's `/traces` too. Found in this release's testing.
86
+ - **The trace viewer page cannot call the server's API.** Served from `serve` for the IDE, the page now carries a content security policy that allows its own script but no requests or form posts, so script injected through trace content cannot reach other routes. Found in this release's review.
87
+
8
88
  ## [0.12.6] - 2026-10-06
9
89
 
10
90
  Hotfix for 0.12.5. **Heads-up for support:** anyone on 0.12.5 who is signed in to Altimate gets a TUI where keybindings do nothing and Ctrl+C does not quit; they have to close the terminal. Upgrading fixes it, and so does `ALTIMATE_DISABLE_WORKSPACE=1` as a stopgap.
package/bin/altimate-code CHANGED
@@ -65,7 +65,13 @@ function run(target) {
65
65
  console.error(result.error.message)
66
66
  process.exit(1)
67
67
  }
68
- const code = typeof result.status === "number" ? result.status : 0
68
+ // A child killed by a signal (e.g. the kernel OOM killer) has no exit status; report it the way
69
+ // a shell would (128 + signal number) instead of exiting 0 as if the run had succeeded.
70
+ if (result.signal) {
71
+ console.error(`altimate-code terminated by ${result.signal}`)
72
+ process.exit(128 + (os.constants.signals[result.signal] ?? 1))
73
+ }
74
+ const code = typeof result.status === "number" ? result.status : 1
69
75
  process.exit(code)
70
76
  }
71
77
 
package/package.json CHANGED
@@ -7,20 +7,20 @@
7
7
  "scripts": {
8
8
  "postinstall": "bun ./postinstall.mjs || node ./postinstall.mjs"
9
9
  },
10
- "version": "0.12.6",
10
+ "version": "0.12.8",
11
11
  "license": "MIT",
12
12
  "dependencies": {
13
- "@altimateai/altimate-core": "0.7.0"
13
+ "@altimateai/altimate-core": "0.9.0"
14
14
  },
15
15
  "optionalDependencies": {
16
- "@altimateai/altimate-code-windows-x64-baseline": "0.12.6",
17
- "@altimateai/altimate-code-linux-x64": "0.12.6",
18
- "@altimateai/altimate-code-linux-x64-baseline": "0.12.6",
19
- "@altimateai/altimate-code-darwin-x64": "0.12.6",
20
- "@altimateai/altimate-code-linux-arm64": "0.12.6",
21
- "@altimateai/altimate-code-windows-x64": "0.12.6",
22
- "@altimateai/altimate-code-darwin-x64-baseline": "0.12.6",
23
- "@altimateai/altimate-code-darwin-arm64": "0.12.6"
16
+ "@altimateai/altimate-code-windows-x64-baseline": "0.12.8",
17
+ "@altimateai/altimate-code-linux-x64": "0.12.8",
18
+ "@altimateai/altimate-code-linux-x64-baseline": "0.12.8",
19
+ "@altimateai/altimate-code-darwin-x64": "0.12.8",
20
+ "@altimateai/altimate-code-linux-arm64": "0.12.8",
21
+ "@altimateai/altimate-code-windows-x64": "0.12.8",
22
+ "@altimateai/altimate-code-darwin-x64-baseline": "0.12.8",
23
+ "@altimateai/altimate-code-darwin-arm64": "0.12.8"
24
24
  },
25
25
  "peerDependencies": {
26
26
  "pg": ">=8",
package/skills/altimate-setup/SKILL.md CHANGED
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: altimate-setup
3
- description: Configure Altimate platform credentials for datamate and API access
3
+ description: Configure Altimate platform credentials for API and workspace access
4
4
  ---
5
5
 
6
6
  # Altimate Setup
@@ -28,4 +28,4 @@ Guide the user through configuring their Altimate platform credentials.
28
28
  ```
29
29
  Then set permissions to owner-only: `chmod 600 ~/.altimate/altimate.json`
30
30
 
31
- 4. **Validate**: Call the `datamate_manager` tool with `operation: "list"` to verify the credentials work. Report success or failure to the user.
31
+ 4. **Validate**: If the `datamate_manager` tool is available, call it with `operation: "list"` to verify the credentials work, and report success or failure to the user. If it is not (it is turned off in a project linked to an Altimate workspace), ask the user to run `/workspace` → Refresh instead, which calls the Altimate API with these credentials and reports an error if they don't work.
package/skills/data-question/SKILL.md ADDED
@@ -0,0 +1,192 @@
1
+ ---
2
+ name: data-question
3
+ description: Answer an analytical question from connected databases, explore data to answer a question, or perform cross-database analysis.
4
+ ---
5
+
6
+ # Data Question
7
+
8
+ Use the available read-only tools to answer the user's question. Source databases and project files stay unchanged; when exposed, `sql_materialize` may create session-local tables in the `scratch` warehouse. All examples below use synthetic tables. Adapt names and dialects after inspection. This skill is self-contained.
9
+
10
+ ## Workflow
11
+
12
+ 1. **Inspect.** Use `warehouse_list`, `schema_inspect` and small `sql_execute` queries as needed. Inspect actual keys and text before choosing joins, casts or parsing rules: schemas alone hide whitespace, case, prefixes and mixed formats. Preserve meaningful leading zeros; check that normalization does not merge distinct identities.
13
+ 2. **Plan.** State the population, row grain, denominator, units and material assumptions. Decide which joins and aggregation levels preserve them. Keep this brief; a trivial lookup does not need a checklist or several validation scans.
14
+ 3. **Compute in-engine.** Filter and aggregate close to the source. Derive large intermediate sets with joins/CTEs or the scratch bridge below, rather than copying rows into `IN (...)` or `VALUES`. Small user-supplied constants are fine. Keep LIMIT, sampling and approximate functions exploratory unless the user requests an estimate, or an exact answer would need per-row model labeling of a large population (see "Batch semantic labels, then count"); then say the answer is an estimate. For exact final answers, aggregate the full intended population in SQL.
15
+ 4. **Verify relevant invariants.** Check key uniqueness, before/after join counts, missing matches and null share when they affect the answer. Use `sql_check` for these invariants and extraction coverage; for cross-warehouse comparisons, first materialize both sides into `scratch` with `sql_materialize`. Reconcile components with totals and preserve weighting. `SUM(group_mean * contributing_count) / SUM(contributing_count)` recovers the overall mean for disjoint, complete groups using unrounded means and counts of the same non-null measure; `AVG(group_mean)` gives each group equal weight. `sql_analyze` is optional. If available, a read-only `data-explorer` via `task` can independently re-derive a difficult result.
16
+ 5. **Answer.** Explain the basis in one or two sentences (population, grain, caveats), retaining useful tables or explanation before it. Finish with `**Answer:** <requested value(s), with units>`. If evidence or available tools are insufficient, finish with `**Answer:** unable to determine — <why>`. Never guess a missing result.
17
+
18
+ Select needed columns and useful partition filters to control cost; LIMIT need not reduce scanned bytes. Report measured costs, not invented estimates. After a timeout, revise the query or split work along exhaustive, disjoint partitions; a failed query is not a zero count.
19
+
20
+ ## Cross-engine bridge
21
+
22
+ **Prerequisites:** both source warehouses are connected; `scratch` is available and uses DuckDB; `sql_materialize` is exposed. Source queries use their own engine's dialect. The synthetic source tables here are `orders(order_id, customer_code, amount, status)` on `sales_db` and `customers(customer_code, region)` on `crm_db`, with text customer codes.
23
+
24
+ After inspecting examples and checking normalization collisions, invoke `sql_materialize` twice:
25
+
26
+ - `source_warehouse: "sales_db"`, `target_table: "orders_stage"`, `sql`:
27
+
28
+ ```sql
29
+ SELECT order_id, LOWER(TRIM(customer_code)) AS customer_key, amount
30
+ FROM orders
31
+ WHERE status = 'paid'
32
+ ```
33
+
34
+ - `source_warehouse: "crm_db"`, `target_table: "customers_stage"`, `sql`:
35
+
36
+ ```sql
37
+ SELECT LOWER(TRIM(customer_code)) AS customer_key, region
38
+ FROM customers
39
+ ```
40
+
41
+ These are complete derived relations, without exploratory LIMITs. Check reported materialization completeness/counts against the intended scope. A partial transfer cannot support an exact cross-engine answer. Query with `sql_execute` on warehouse `scratch`, using the returned table names:
42
+
43
+ ```sql
44
+ SELECT c.region, COUNT(*) AS paid_orders,
45
+ SUM(o.amount) AS total_amount, AVG(o.amount) AS mean_order_amount
46
+ FROM orders_stage AS o
47
+ LEFT JOIN customers_stage AS c ON c.customer_key = o.customer_key
48
+ GROUP BY c.region;
49
+ ```
50
+
51
+ First verify the customer key is unique and quantify unmatched orders with the next pattern. Do not silently remove unmatched orders; decide how they affect the requested population.
52
+
53
+ **If the bridge is unavailable:** use existing views, federation or an already connected engine that can access both relations. Independent aggregates can be combined only if sufficient for the requested statistic (for example, sums and contributing counts over disjoint partitions). Otherwise state that the cross-engine join cannot be completed with available tools. Do not invent a local scripting/write capability or hand-transcribe a large key list.
54
+
55
+ ## Fan-out and nulls
56
+
57
+ **Prerequisites:** ordinary SELECT/CTE support; this pattern works in DuckDB and SQLite. `order_id` is the intended unique left-side grain. Run on the warehouse holding both staged relations, or adapt to source tables on one engine.
58
+
59
+ ```sql
60
+ SELECT customer_key, COUNT(*) AS occurrences
61
+ FROM customers_stage
62
+ GROUP BY customer_key
63
+ HAVING COUNT(*) > 1;
64
+
65
+ WITH joined AS (
66
+ SELECT o.order_id, o.customer_key AS source_key,
67
+ c.customer_key AS matched_key
68
+ FROM orders_stage AS o
69
+ LEFT JOIN customers_stage AS c ON c.customer_key = o.customer_key
70
+ )
71
+ SELECT
72
+ (SELECT COUNT(*) FROM orders_stage) AS rows_before,
73
+ COUNT(*) AS rows_after,
74
+ COUNT(DISTINCT order_id) AS distinct_orders_after,
75
+ SUM(CASE WHEN matched_key IS NULL THEN 1 ELSE 0 END) AS unmatched_rows,
76
+ 1.0 * SUM(CASE WHEN source_key IS NULL THEN 1 ELSE 0 END)
77
+ / NULLIF(COUNT(*), 0) AS source_key_null_share
78
+ FROM joined;
79
+ ```
80
+
81
+ Submit the two SELECTs separately if the connector accepts only one statement. For a many-to-one left join, row counts should match; also verify left-side IDs are unique and non-null. Resolve unintended fan-out at the correct business grain, not by adding an arbitrary DISTINCT. The reported null share is at joined grain; interpret it as source share only after ruling out fan-out. Empty-population ratios are undefined, not zero.
82
+
83
+ ## Many similarly shaped tables
84
+
85
+ **Prerequisites:** inspect the engine/version and table schemas. `information_schema`, DuckDB `query_table` and `UNION ALL BY NAME` are not portable to SQLite. Confirm compatible types and that all intended tables are included exactly once; name-based unions fill absent columns with NULL. Reconcile union counts with the sum of source counts and account for overlapping row identities.
86
+
87
+ For DuckDB, first generate a single SELECT from the catalog of synthetic `orders_*` partitions:
88
+
89
+ ```sql
90
+ WITH catalog AS (
91
+ SELECT '"' || replace(table_catalog, '"', '""') || '"."' ||
92
+ replace(table_schema, '"', '""') || '"."' ||
93
+ replace(table_name, '"', '""') || '"' AS qualified_name
94
+ FROM information_schema.tables
95
+ WHERE table_catalog = current_database()
96
+ AND table_schema = 'main' AND table_type = 'BASE TABLE'
97
+ AND starts_with(table_name, 'orders_')
98
+ )
99
+ SELECT COUNT(*) AS table_count,
100
+ 'SELECT COUNT(*) AS order_count, SUM(amount) AS total_amount ' ||
101
+ 'FROM query_table([' ||
102
+ string_agg('''' || replace(qualified_name, '''', '''''') || '''',
103
+ ', ' ORDER BY qualified_name) || '], true)' AS sql
104
+ FROM catalog;
105
+ ```
106
+
107
+ Check the catalog scope and `table_count`, then execute the complete generated `sql`. `query_table([...], true)` requires a constant list and unions by name; it does not accept a catalog subquery as its argument. It adds no source-table column. If provenance matters, or table names contain embedded double quotes, generate ordinary quoted SELECTs joined with `UNION ALL BY NAME` instead. For two known synthetic partitions:
108
+
109
+ ```sql
110
+ SELECT 'orders_jan' AS source_table, * FROM "main"."orders_jan"
111
+ UNION ALL BY NAME
112
+ SELECT 'orders_feb' AS source_table, * FROM "main"."orders_feb";
113
+ ```
114
+
115
+ For a large family, generate those SELECT branches from the catalog, escaping SQL string literals and identifiers separately; do not enumerate tables from memory. Reserve a provenance column name absent from the input tables. The star here intentionally aligns the inspected partition schemas; project needed columns for the final computation.
116
+
117
+ SQLite has no `query_table` or `UNION ALL BY NAME`. Query `sqlite_master` to build a complete statement with quoted identifiers and a consistent explicit column order:
118
+
119
+ ```sql
120
+ SELECT COUNT(*) AS table_count,
121
+ 'WITH combined AS (' ||
122
+ group_concat(
123
+ 'SELECT ' || quote(name) || ' AS source_table, order_id, amount ' ||
124
+ 'FROM main."' || replace(name, '"', '""') || '"',
125
+ ' UNION ALL '
126
+ ) || ') SELECT COUNT(*) AS order_count, SUM(amount) AS total_amount ' ||
127
+ 'FROM combined' AS sql
128
+ FROM main.sqlite_master
129
+ WHERE type = 'table' AND substr(name, 1, 7) = 'orders_';
130
+ ```
131
+
132
+ Inspect the catalog, then execute the returned statement. Every SQLite partition must have `order_id` and `amount` with compatible meaning/types, or each generated branch needs explicit casts/NULL placeholders. `UNION ALL` retains rows; `UNION` would silently deduplicate them. Respect the engine's compound-query limit; if batching is necessary, combine disjoint batches using sufficient statistics (sum/count, not averages of averages).
133
+
134
+ In both generators, zero matching tables yields `table_count = 0` and `sql = NULL`: investigate coverage instead of claiming a zero total. Never execute clipped generated SQL. Read its spill if available, or generate smaller catalog batches if `read` also clips a long line. These queries generate SQL from metadata, not lists of copied business keys.
135
+
136
+ Dialect references: [DuckDB query_table](https://www.duckdb.org/docs/current/guides/sql_features/query_and_query_table_functions), [DuckDB union-by-name option](https://www.duckdb.org/docs/lts/sql/functions/utility), [SQLite compound SELECT](https://www.sqlite.org/lang_select.html). All essential examples are above; no external file is required.
137
+
138
+ ## Batch semantic labels, then count
139
+
140
+ Treat inferred labels as new data, not ground truth. Define a fixed rubric, allowed labels and uncertainty policy before classifying. For synthetic `messages(message_id, body)`, a rubric might classify a request to reverse a charge as `refund`, a question about shipping progress as `delivery`, and ambiguous text as `uncertain`. Specify tie-breaking for multi-topic messages.
141
+
142
+ Inspect text, then define the complete population of stable IDs and count it before classifying. Labeling with a model costs one call per batch, so it grows with the table and can cost far more than the analysis itself. Without the user's confirmation, label at most 500 rows in total by any path: your own reading, subagents, or warehouse AI functions (for example classification functions in Snowflake, Databricks or BigQuery), which are billed per row too. To label more, first tell the user the row count, how the labels would be produced and the size of the job (about one model call per 50 rows, or a per-row charge on their warehouse bill for warehouse AI functions), and wait for their confirmation. When the population is larger than 500 rows, by default label a documented random sample of up to 500 rows (roughly ±4.5 percentage points at 95% confidence for a proportion) and report the result as an estimate with its sample size, not as an exact count. Review uncertain cases consistently.
143
+
144
+ To reuse the checks below, form `labels(message_id, label, rubric_version)` for the labeled IDs; `population` must cover the same intended IDs (for a sample, the sampled IDs). Small audited annotation batches can use an available permitted ingestion path. When `scratch` and `sql_materialize` are available, these small batches can be expressed as a SELECT relation and materialized there. For example (DuckDB/SQLite SELECT syntax):
145
+
146
+ ```sql
147
+ WITH annotations(message_id, label, rubric_version) AS (
148
+ VALUES (101, 'refund', 'v1'),
149
+ (102, 'delivery', 'v1'),
150
+ (103, 'uncertain', 'v1')
151
+ )
152
+ SELECT * FROM annotations;
153
+ ```
154
+
155
+ These literals are newly inferred annotations, not a copied intermediate database result. Do not scale this into hand-copying thousands of IDs. When the full population cannot be labeled within these limits, label a sample or report the coverage limitation. `sql_materialize` alone does not imply an arbitrary file writer. Never claim labels or counts you did not compute and persist; with no persistence tool, a small complete annotation CTE can still be checked and counted in a read-only query.
156
+
157
+ **Prerequisites for the following SQL:** DuckDB/SQLite; `population(message_id)` contains exactly the intended unique IDs (the sampled IDs for a sample) and `labels` contains the annotation records in the same engine. They may be scratch tables or CTEs. Run each SELECT separately as needed:
158
+
159
+ ```sql
160
+ -- Must return no rows: includes duplicates across batches.
161
+ SELECT message_id, COUNT(*) AS copies
162
+ FROM labels GROUP BY message_id HAVING COUNT(*) <> 1;
163
+
164
+ -- Missing IDs, extra IDs and invalid records must all be zero.
165
+ SELECT
166
+ (SELECT COUNT(*) FROM population AS p
167
+ WHERE NOT EXISTS (SELECT 1 FROM labels AS l
168
+ WHERE l.message_id = p.message_id)) AS missing_ids,
169
+ (SELECT COUNT(*) FROM labels AS l
170
+ WHERE NOT EXISTS (SELECT 1 FROM population AS p
171
+ WHERE p.message_id = l.message_id)) AS extra_ids,
172
+ (SELECT COUNT(*) FROM labels
173
+ WHERE message_id IS NULL OR label IS NULL
174
+ OR label NOT IN ('refund', 'delivery', 'uncertain')
175
+ OR rubric_version IS NULL OR rubric_version <> 'v1') AS invalid_records;
176
+
177
+ -- Only count after the invariants pass; retain uncertain in the denominator.
178
+ SELECT label, COUNT(*) AS messages
179
+ FROM labels GROUP BY label ORDER BY label;
180
+ ```
181
+
182
+ Verify the label counts sum to the `population` count (the sample size when you labeled a sample). Use these computed counts in the answer; do not re-count labels by eye or combine unverified child summaries. For a sample, report proportions with the sample size and the size of the full population, and do not present scaled-up counts as exact.
183
+
184
+ ## Reading spilled results
185
+
186
+ When `sql_execute` reports `Preview: N of M returned rows ... saved to <path>`, N is the displayed preview and M is the connector-returned count, not necessarily the number of matches. A row-limit warning still applies to the spill. If a JSONL path is supplied, inspect it with `read`, for example:
187
+
188
+ ```json
189
+ {"filePath":"<absolute path reported by sql_execute>","offset":1,"limit":50}
190
+ ```
191
+
192
+ Inspect any metadata/schema record before interpreting row records; preserve encoded types and distinguish NULL from the string "NULL". Continue using line offsets from the read output, accounting for metadata lines. Read may itself clip long lines or total output: recover only the needed fields with a SQL projection/filter instead of guessing missing cell contents. For exact totals, compute `COUNT(*)`, SUM or other requested aggregates in the engine; a spill cannot recover rows discarded by a connector limit. If no spill is available, narrow or aggregate the query and respect its returned-row limit.