@kaelio/ktx 0.14.0 → 0.16.0

package/dist/skills/sigma_ingest/SKILL.md

@@ -0,0 +1,189 @@
+---
+name: sigma_ingest
+description: Extract durable ktx wiki knowledge from staged Sigma data model specs and workbook summaries. Load for WorkUnits with unitKey sigma-data-models or sigma-workbooks.
+callers: [memory_agent]
+---
+
+# Sigma Ingest
+
+Sigma ingest turns staged data model specs and workbook summaries into durable ktx wiki knowledge. The deterministic `project()` step has already written semantic-layer YAML for all warehouse-table data model elements before this skill runs — do not re-write those SL sources.
+
+## Work unit structure
+
+Sigma produces at minimum two work units per ingest run:
+
+- `sigma-data-models` or `sigma-data-models-N`
+  - `rawFiles`: `data-models/<id>.json` files (one per data model in this batch)
+  - `peerFileIndex`: `workbooks/<id>.json` files + `sigma-manifest.json` + `sigma-projection-config.json`
+  - When the workspace has more than 50 data models, split into batches: `sigma-data-models-0`, `sigma-data-models-1`, … with `displayLabel` like `"Sigma: data models (1/8)"`. When ≤50 data models, the unitKey is simply `sigma-data-models` with no suffix.
+- `sigma-workbooks` or `sigma-workbooks-N`
+  - `rawFiles`: `workbooks/<id>.json` files (one per workbook in this batch)
+  - `peerFileIndex`: `data-models/<id>.json` files + `sigma-manifest.json` + `sigma-projection-config.json`
+  - When the workspace has more than 2000 workbooks, split into batches: `sigma-workbooks-0`, `sigma-workbooks-1`, … with `displayLabel` like `"Sigma: workbooks (1/4)"`. When ≤2000 workbooks, the unitKey is simply `sigma-workbooks` with no suffix.
+
+`sigma-manifest.json` and `sigma-projection-config.json` are never in `rawFiles`. They live at the staged dir root and always appear in `peerFileIndex`.
+
+## Staged file shapes
+
+**`data-models/<id>.json`** — one per data model (in `rawFiles` for data-model units):
+```json
+{
+  "sigmaId": "abc-123",
+  "name": "Revenue Model",
+  "path": "Finance/Revenue Model",
+  "latestVersion": 3,
+  "updatedAt": "2026-01-15T00:00:00Z",
+  "isArchived": false,
+  "spec": {
+    "name": "Revenue Model",
+    "pages": [{
+      "id": "p1",
+      "name": "Main",
+      "elements": [{
+        "id": "elem1",
+        "kind": "table",
+        "name": "Opportunities",
+        "hidden": false,
+        "source": {
+          "kind": "warehouse-table",
+          "connectionId": "<sigma-internal-uuid>",
+          "path": ["DATABASE", "SCHEMA", "OPPORTUNITIES"]
+        },
+        "columns": [
+          { "id": "c1", "name": "Deal Amount", "formula": "[OPPORTUNITIES/Amount]", "description": "Net contract value in USD" },
+          { "id": "c2", "name": "Total ARR", "formula": "Sum([OPPORTUNITIES/ARR])", "description": "Annualised recurring revenue" }
+        ]
+      }]
+    }]
+  }
+}
+```
+
+`source.kind` discriminates:
+- `warehouse-table` — element maps directly to a warehouse table. Has `connectionId` and `path` (array of path segments forming the fully-qualified table name). `project()` writes an SL source when `connectionMappings` covers this `connectionId`.
+- `table` — element is a derived view layered on top of another element; identified by `source.elementId`. No warehouse path. Wiki-only.
+
+**`workbooks/<id>.json`** — one per workbook, in `rawFiles` for workbook units (summary only; no spec endpoint exists):
+```json
+{
+  "sigmaId": "wb-abc",
+  "name": "ARR Tracker",
+  "path": "Finance/Dashboards",
+  "latestVersion": 2,
+  "updatedAt": "2026-01-16T00:00:00Z",
+  "isArchived": false,
+  "workbookUrlId": "57a96EMo3G...",
+  "description": "Tracks ARR by segment and cohort for the finance team"
+}
+```
+
+**Peer files (available via `peerFileIndex`, not `rawFiles`):**
+
+**`sigma-manifest.json`** — fetch summary; use for provenance only.
+
+**`sigma-projection-config.json`** — written by `fetch()`, contains two fields the skill must read:
+
+- `connectionMappings`: `{sigmaInternalUuid: ktxWarehouseConnectionId}`. Use the mapped warehouse connection ID for `entity_details` when verifying warehouse identifiers found in data model specs.
+- `workbookFilter`: the filter settings that were active when workbooks were last fetched:
+  - `includeArchived` (default `false`) — when `false`, archived workbooks are not in `workbooks/`; `isArchived: true` files will only appear when this was `true`.
+  - `includeExplorations` (default `false`) — when `false`, exploration-type workbooks (unsaved analyses) are excluded; treat present workbooks as intentional, curated reports.
+  - `updatedSince` (optional ISO 8601 string) — when set, only workbooks updated on or after this date are staged; the set is a recent-changes slice, not the full workspace. Do not infer that absent workbooks were deleted.
+
+`sigma-manifest.json` also reflects any active `dataModelFilter`. When `dataModelFilter.updatedSince` was set during fetch, `dataModelCount` reflects only matching models, not the full workspace. Do not infer that absent data models were deleted.
+
+Read `sigma-projection-config.json` first and keep `workbookFilter` in scope while processing the WorkUnit.
+
+## Required workflow
+
+1. Read every `rawFiles` entry for the WorkUnit.
+2. Read `sigma-projection-config.json` from the staged dir to get `connectionMappings`.
+3. For each data model file: extract business semantics from element names, column descriptions, and the domain context of the model. Skip hidden elements and hidden columns.
+4. For each workbook file: extract business domain knowledge from the name and description. When `workbookFilter.updatedSince` is set, treat the staged set as a recent-changes slice — absent workbooks were not deleted, they were simply outside the filter window.
+5. Use `discover_data` before writing to find existing wiki pages on the same topic.
+6. Write wiki candidates with `context_candidate_write`. Do not call `wiki_write` directly from a Sigma WorkUnit; Stage 4 reconciliation promotes candidates.
+7. Do not write or edit SL sources. The `project()` step owns all SL output for Sigma.
+
+## Identifier Verification Protocol
+
+Before writing a wiki page or SL source on any topic:
+
+1. `discover_data({query: "<topic>"})` - see what wikis, SL sources, and raw
+   tables already exist. Prefer updating existing pages over creating new ones.
+
+Before emitting any `schema.table` or `schema.table.column` into a wiki body,
+SL source, `tables:` frontmatter, `sl_refs`, or `emit_unmapped_fallback`:
+
+2. `entity_details({connectionId, targets: [{display: "<identifier>"}]})` -
+   confirm the identifier resolves; inspect native types, FK/PK, and
+   sampleValues. Use the warehouse `connectionId` from `connectionMappings` in
+   `sigma-projection-config.json`, not the Sigma connection ID. If
+   `connectionMappings` has no entry for the element's `source.connectionId`,
+   skip `entity_details` — there is no mapped warehouse to verify against —
+   and wrap any identifier references with `[unverified - from <rawPath>]`.
+3. For literal values from the source, such as status codes or plan tiers,
+   check whether they appear in `entity_details` sampleValues for the relevant
+   column. If sampleValues is short or the sample may have missed real values,
+   run a `sql_execution` probe with the same warehouse connection id:
+   `sql_execution({connectionId, sql: "SELECT DISTINCT <col> FROM <ref> LIMIT 50"})`.
+4. If the candidate identifier still does not resolve, do one of:
+   - Use `sql_execution({connectionId, sql: "SELECT 1 FROM <ref> LIMIT 0"})`.
+     If it errors, the identifier is fictional.
+   - Wrap the identifier in `[unverified - from <rawPath>]` in the wiki body,
+     citing the exact raw path that mentioned it.
+   - When recording `emit_unmapped_fallback` with `no_physical_table`, include
+     the failing probe error in `clarification`.
+5. Never copy `<schema>.<table>` placeholder strings from these instructions
+   into output.
+
+## Data model elements
+
+### Warehouse-table elements (`source.kind === "warehouse-table"`)
+
+`project()` writes an SL source for a warehouse-table element **only when** the element's `source.connectionId` has an entry in `connectionMappings`. When no mapping exists, no SL source is written and the element is wiki-only.
+
+To determine whether an SL source exists: check whether `connectionMappings[element.source.connectionId]` resolves. If it does, use `sl_discover` to find the source by its slugified name (`<dataModelName>_<elementName>`), then:
+
+- Read the existing SL source with `sl_read_source` to understand what columns and measures are captured.
+- Write a wiki candidate about the business domain if the element name, column descriptions, or data model description reveals durable knowledge not already in the wiki.
+- `sl_refs` in the wiki candidate should point to the already-written SL source name.
+
+If `connectionMappings` has no entry for the element's `source.connectionId`, treat the element as wiki-only — do not attempt `sl_discover` or `sl_read_source` for it, as no source was written.
+
+### Joins within a data model
+
+Joins are not projected in v1; `joins: []` is always written by `project()`. `Lookup()` formulas may be described in wiki prose instead.
+
+### Non-warehouse elements (`source.kind === "table"`)
+
+These reference another element by `elementId` — they are derived views layered on top of a warehouse-table element. They have no warehouse path of their own. Do not attempt SL writes for these elements. They may produce wiki candidates if their column names or descriptions reveal business semantics not captured by the underlying warehouse-table element.
+
+## Workbooks
+
+Workbooks have summary metadata only. There is no spec endpoint.
+
+Extract business domain knowledge from:
+- `name`: the workbook's primary topic (e.g. "ARR Tracker" → ARR tracking concepts)
+- `description`: business context and intended audience
+- `path`: team or functional area (e.g. `Finance/Dashboards`)
+
+Write wiki candidates when the name or description reveals a reusable business concept, metric definition, or domain convention. Write one candidate per distinct concept, not one per workbook.
+
+Skip workbooks whose name or description contains no durable business semantics (e.g. "Untitled Workbook", "Test Dashboard").
+
+## Capture rules
+
+Write wiki candidates for:
+- Metric definitions mentioned in element names or column descriptions (e.g. "Net ARR", "Churned MRR")
+- Domain conventions such as cohort definitions, segment taxonomies, or fiscal calendar rules
+- Relationships between business entities revealed by data model joins
+
+Skip:
+- Visualization settings, layout, colors, chart types
+- Owner names, folder paths, and version numbers as wiki narrative
+- Hidden elements and hidden columns
+- Data model names that are purely technical with no business meaning
+- When `workbookFilter.includeExplorations` is `false` (the default), all staged workbooks are intentional reports — no extra exploration filter needed. When it is `true`, workbooks without a description or with a generic auto-generated name are likely ephemeral explorations; skip those.
+
+## Usage signals
+
+Sigma workbooks carry `latestVersion` but no usage counts. Treat a higher `latestVersion` as weak evidence of continued maintenance; do not include version numbers in wiki prose.

package/dist/sql.js

@@ -1,5 +1,5 @@
 import { FEDERATED_CONNECTION_ID } from './context/connections/federation.js';
-import { executeProjectReadOnlySql } from './context/connections/project-sql-executor.js';
+import { executeProjectRawSql } from './context/connections/project-sql-executor.js';
 import { assertSqlQueryableConnection } from './context/connections/dialects.js';
 import { resolveConfiguredConnection } from './context/connections/resolve-connection.js';
 import { loadKtxProject } from './context/project/project.js';
@@ -91,6 +91,8 @@ export async function runKtxSql(args, io = process, deps = {}) {
         const connection = isFederated ? undefined : resolveConfiguredConnection(project.config, args.connectionId);
         driver = isFederated ? 'duckdb' : String(connection?.driver ?? 'unknown').toLowerCase();
         demoConnection = isFederated ? false : isDemoConnection(args.connectionId, connection);
+        // Fail fast before creating the SQL-analysis daemon port; executeProjectRawSql
+        // re-asserts this for every caller.
         if (!isFederated) {
             assertSqlQueryableConnection(args.connectionId, connection?.driver);
         }
@@ -103,25 +105,18 @@ export async function runKtxSql(args, io = process, deps = {}) {
             }));
         const analysisPort = createSqlAnalysis();
         const dialect = isFederated ? 'duckdb' : sqlAnalysisDialectForDriver(connection?.driver);
-        const validation = await analysisPort.validateReadOnly(args.sql, dialect);
-        if (!validation.ok) {
-            throw new Error(validation.error ?? 'SQL is not read-only.');
-        }
-        const referencedTableCount = await safeReferencedTableCount(analysisPort, args.sql, dialect);
         const createScanConnector = deps.createScanConnector ?? createKtxCliScanConnector;
-        const result = await executeProjectReadOnlySql({
+        const result = await executeProjectRawSql({
             project,
-            input: {
-                connectionId: args.connectionId,
-                projectDir: args.projectDir,
-                connection,
-                sql: args.sql,
-                maxRows: args.maxRows,
-            },
+            connectionId: args.connectionId,
+            sql: args.sql,
+            maxRows: args.maxRows,
+            sqlAnalysis: analysisPort,
             createConnector: (connectionId) => createScanConnector(project, connectionId),
             executeFederated: deps.executeFederated,
             runId: 'cli-sql',
         });
+        const referencedTableCount = await safeReferencedTableCount(analysisPort, args.sql, dialect);
         const mode = resolveOutputMode({ explicit: args.output, json: args.json, io });
         printSqlResult(resultOutput(args.connectionId, result), mode, io);
         await emitTelemetryEvent({

package/dist/status-project.js

@@ -236,6 +236,7 @@ function buildConnectionStatus(name, conn, env) {
             const hint = envHint(conn.credentials_json);
             return warn(hint ? `credentials missing (env: ${hint})` : 'credentials not set', hint ? `Set ${hint}` : 'Rerun `ktx setup`');
         }
+        case 'duckdb':
         case 'sqlite': {
             const path = conn.path;
             if (typeof path === 'string' && path.length > 0)
@@ -361,7 +362,7 @@ async function buildQueryHistoryStatus(project, options) {
     return statuses;
 }
 const ADAPTER_DRIVER_REQUIREMENT = {
-    'live-database': ['postgres', 'mysql', 'snowflake', 'bigquery', 'clickhouse', 'sqlite', 'sqlserver'],
+    'live-database': ['postgres', 'mysql', 'snowflake', 'bigquery', 'clickhouse', 'sqlite', 'duckdb', 'sqlserver'],
     dbt: ['dbt', 'dbt-core', 'dbt-cloud'],
     notion: ['notion'],
     metabase: ['metabase'],

package/dist/telemetry/events.d.ts

@@ -66,8 +66,8 @@ export declare const telemetryEventSchemas: {
             project: "project";
             models: "models";
             sources: "sources";
-            context: "context";
             runtime: "runtime";
+            context: "context";
             secrets: "secrets";
             agents: "agents";
             "demo-tour": "demo-tour";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kaelio/ktx",
-  "version": "0.14.0",
+  "version": "0.16.0",
   "description": "Standalone ktx context layer for data agents",
   "author": {
     "name": "Kaelio",
@@ -36,6 +36,8 @@
     "@ai-sdk/devtools": "0.0.18",
     "@ai-sdk/google-vertex": "^4.0.134",
     "@anthropic-ai/claude-agent-sdk": "0.3.146",
+    "@aws-sdk/client-athena": "^3.1068.0",
+    "@aws-sdk/client-glue": "^3.1068.0",
     "@clack/core": "1.3.1",
     "@clack/prompts": "1.4.0",
     "@clickhouse/client": "^1.18.5",