@kinetica/admin-agent 0.2.2 → 0.2.4

package/README.md

@@ -332,14 +332,14 @@ The `--bundle` flag points the agent at an **extracted** support-bundle director
 
 Available against an extracted `gpudb_sysinfo` support bundle (see [Offline Bundle Mode](#offline-bundle-mode)). All read-only; the search/timeline tools stream and bound their output so a large rank log (tens of MB, hundreds of thousands of lines) never blows up the context.
 
-| Tool                           | Description                                                                                                                                                                 |
-| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `kinetica_load_bundle`         | Attach an extracted bundle directory; without a path it opens a directory picker (a model-supplied path needs operator confirmation)                                        |
-| `kinetica_bundle_list_files`   | Inventory: detected version, ranks + services present, file counts/sizes by kind, plus a layout-match verdict + per-file confidence for off-shape bundles — call this first |
-| `kinetica_bundle_log_timeline` | Per-time-bucket severity counts across ranks (the incident shape) — call before searching                                                                                   |
-| `kinetica_bundle_search_logs`  | Bounded log search by regex, min-severity, time window, and rank / host-manager / component (reads both rolling and Loki-export logs)                                       |
-| `kinetica_bundle_read_config`  | Read the bundle's real on-disk `gpudb.conf`, with optional section/key filter                                                                                               |
-| `kinetica_bundle_read_sysinfo` | OS/process/version diagnostic files (memory, CPU, disk, GPU, network, process args)                                                                                         |
+| Tool                           | Description                                                                                                                                                                                                                                                                                 |
+| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `kinetica_load_bundle`         | Attach an extracted bundle directory; without a path it opens a directory picker (a model-supplied path needs operator confirmation)                                                                                                                                                        |
+| `kinetica_bundle_list_files`   | Inventory: detected version, ranks + services present, file counts/sizes by kind, plus a layout-match verdict + per-file confidence for off-shape bundles — call this first                                                                                                                 |
+| `kinetica_bundle_log_timeline` | Per-time-bucket severity counts across ranks (the incident shape) — call before searching                                                                                                                                                                                                   |
+| `kinetica_bundle_search_logs`  | Bounded log search by regex, min-severity, time window, and rank / host-manager / component (reads both rolling and Loki-export logs); `include_multiline` stitches a multi-line record — e.g. a full `Executing SQL:` query whose embedded newlines span many lines — back onto each match |
+| `kinetica_bundle_read_config`  | Read the bundle's real on-disk `gpudb.conf`, with optional section/key filter                                                                                                                                                                                                               |
+| `kinetica_bundle_read_sysinfo` | OS/process/version diagnostic files (memory, CPU, disk, GPU, network, process args)                                                                                                                                                                                                         |
 
 ### Reporting
 
@@ -417,7 +417,7 @@ References provide domain knowledge (not diagnostic runbooks). Create a `.md` fi
 
 **Playbooks** (6): memory-pressure, gpu-out-of-memory, query-contention, resource-group-exhaustion, stale-rank, config-drift
 
-**References** (9):
+**References** (10):
 
 - `gpudb-conf` — master config file structure, section index, tiered storage semantics
 - `tiered-objects` — `ki_tiered_objects` schema, ID format, diagnostic queries
@@ -427,11 +427,12 @@ References provide domain knowledge (not diagnostic runbooks). Create a `.md` fi
 - `mutation-safety` — pre-execution checklist for rebalance, alter-config, and DDL paths
 - `sql-alter-table` — Kinetica 7.2 ALTER TABLE grammar, column property flags, shard-key immutability
 - `sql-create-index` — column index syntax, chunk skip index, when to use which
+- `sql-dialect` — PostgreSQL-baseline mental model + a "false friends" table of cross-dialect SQL that looks valid but fails in Kinetica (e.g. `TRY_CAST`/`SAFE_CAST`, backtick quoting, `NUMERIC` vs `DECIMAL`); steers remediation SQL away from SQL Server/Snowflake/Oracle idioms
 - `version-quirks-7.2` — endpoint/property differences between 7.2.x and earlier releases
 
 Plus a **bundle-scoped reference** (`support-bundle` — bundle layout, the two per-rank log families, raw + Loki-JSONL log-line formats, severity ordering, file parsing, crash-SQL forensics, and how to work an off-shape bundle via the `layout_match`/confidence signals) that lives in `knowledge/references/bundle/`. It loads in **every** session — even a pure live one — so that a bundle attached mid-session via `kinetica_load_bundle` has its parsing knowledge ready in the (build-once) prompt; the corpus is cached, so the cost to a session that never attaches a bundle is negligible.
 
-> **Heads up — prompt budget:** all playbooks and references are front-loaded into a single system prompt at startup, so its token cost grows with the knowledge corpus. A startup tripwire (`agent/prompt-budget.ts`) prints the assembled prompt size under `DEBUG` and warns on stderr once it exceeds ~20,000 estimated tokens. Current baseline is ~13.4k tokens (6 playbooks + 9 references). If you add substantial knowledge and trip that warning, treat it as the cue to switch from "load everything" to keyword-based playbook selection.
+> **Heads up — prompt budget:** all playbooks and references are front-loaded into a single system prompt at startup, so its token cost grows with the knowledge corpus. A startup tripwire (`agent/prompt-budget.ts`) prints the assembled prompt size under `DEBUG` and warns on stderr once it exceeds ~20,000 estimated tokens. Current baseline is ~15.5k tokens (6 playbooks + 10 references). If you add substantial knowledge and trip that warning, treat it as the cue to switch from "load everything" to keyword-based playbook selection.
 
 ## Development
 

package/dist/admin-agent.js

@@ -4034,6 +4034,8 @@ function parseLogLine(line) {
 
 // src/bundle/log-search.ts
 var DEFAULT_MAX_MATCHES = 200;
+var MULTILINE_MAX_LINES = 300;
+var MULTILINE_MAX_CHARS = 2e4;
 var REGEX_SCAN_MAX = 8192;
 var GRANULARITY_LEN = {
   day: 10,
@@ -4075,6 +4077,23 @@ function matchesFilters(parsed, query3, regex, minRank) {
     return false;
   return true;
 }
+function buildMatch(lineNumber, parsed) {
+  return {
+    lineNumber,
+    ...parsed.timestamp !== void 0 ? { timestamp: parsed.timestamp } : {},
+    ...parsed.severity !== void 0 ? { severity: parsed.severity } : {},
+    ...parsed.rank !== void 0 ? { rank: parsed.rank } : {},
+    message: parsed.message,
+    raw: parsed.raw
+  };
+}
+function finalizeMultiline(pending) {
+  if (pending.extra.length === 0) return pending.base;
+  const joined = pending.extra.join("\n");
+  const suffix = pending.truncated ? "\n\u2026 [continuation truncated]" : "";
+  return { ...pending.base, message: `${pending.base.message}
+${joined}${suffix}` };
+}
 async function searchLogFile(filePath, query3) {
   const maxMatches = query3.maxMatches ?? DEFAULT_MAX_MATCHES;
   const minRank = query3.minSeverity !== void 0 ? severityRank(query3.minSeverity) : -Infinity;
@@ -4096,9 +4115,17 @@ async function searchLogFile(filePath, query3) {
     ...query3.fromTs !== void 0 ? { fromTs: floorTimestamp(query3.fromTs) } : {},
     ...query3.toTs !== void 0 ? { toTs: ceilTimestamp(query3.toTs) } : {}
   };
+  const coalesce = query3.coalesceMultiline === true;
   const matches = [];
   let totalMatched = 0;
   let linesScanned = 0;
+  let pending;
+  const flushPending = () => {
+    if (pending) {
+      matches.push(finalizeMultiline(pending));
+      pending = void 0;
+    }
+  };
   try {
     const rl = (0, import_node_readline.createInterface)({
       input: (0, import_node_fs4.createReadStream)(filePath, { encoding: "utf-8" }),
@@ -4107,20 +4134,29 @@ async function searchLogFile(filePath, query3) {
     for await (const line of rl) {
       linesScanned++;
       const parsed = parseLogLine(line);
+      if (pending) {
+        if (parsed.timestamp === void 0) {
+          if (!pending.truncated && pending.extra.length < MULTILINE_MAX_LINES && pending.chars + line.length + 1 <= MULTILINE_MAX_CHARS) {
+            pending.extra.push(line);
+            pending.chars += line.length + 1;
+          } else {
+            pending.truncated = true;
+          }
+          continue;
+        }
+        flushPending();
+      }
       if (!matchesFilters(parsed, boundedQuery, regex, minRank)) continue;
       totalMatched++;
       if (matches.length < maxMatches) {
-        matches.push({
-          lineNumber: linesScanned,
-          ...parsed.timestamp !== void 0 ? { timestamp: parsed.timestamp } : {},
-          ...parsed.severity !== void 0 ? { severity: parsed.severity } : {},
-          ...parsed.rank !== void 0 ? { rank: parsed.rank } : {},
-          message: parsed.message,
-          raw: parsed.raw
-        });
+        const base = buildMatch(linesScanned, parsed);
+        if (coalesce) pending = { base, extra: [], chars: 0, truncated: false };
+        else matches.push(base);
       }
     }
+    flushPending();
   } catch (err) {
+    flushPending();
     const message = err instanceof Error ? err.message : String(err);
     return {
       matches,
@@ -4508,7 +4544,8 @@ function toLineQuery(q) {
     ...q.minSeverity !== void 0 ? { minSeverity: q.minSeverity } : {},
     ...q.fromTs !== void 0 ? { fromTs: q.fromTs } : {},
     ...q.toTs !== void 0 ? { toTs: q.toTs } : {},
-    ...q.maxMatches !== void 0 ? { maxMatches: q.maxMatches } : {}
+    ...q.maxMatches !== void 0 ? { maxMatches: q.maxMatches } : {},
+    ...q.coalesceMultiline !== void 0 ? { coalesceMultiline: q.coalesceMultiline } : {}
   };
 }
 function toTimelineLineQuery(q) {
@@ -4847,6 +4884,9 @@ var BundleSearchLogsSchema = import_zod20.z.object({
   host_manager: import_zod20.z.boolean().describe("Search the host-manager (hm) log \u2014 a singleton service, not a rank.").optional(),
   component: import_zod20.z.string().optional(),
   include_components: import_zod20.z.boolean().optional(),
+  include_multiline: import_zod20.z.boolean().describe(
+    "Reconstruct multi-line log records: append continuation lines (those with no timestamp) to each match. Use this to capture a full SQL statement on an 'Executing SQL:' line \u2014 the query often spans many lines because the SQL has embedded newlines, and a plain match shows only its first line. Works on the rolling core logs (logs-local/); Loki per-rank tails (logs/rankN.log) keep only the statement's first line, so there are no continuation lines to stitch there."
+  ).optional(),
   max_matches: import_zod20.z.number().int().min(1).max(1e3).optional()
 });
 async function bundleSearchLogs(source, args = {}) {
@@ -4859,6 +4899,7 @@ async function bundleSearchLogs(source, args = {}) {
     ...args.host_manager !== void 0 ? { hostManager: args.host_manager } : {},
     ...args.component !== void 0 ? { component: args.component } : {},
     ...args.include_components !== void 0 ? { includeComponents: args.include_components } : {},
+    ...args.include_multiline !== void 0 ? { coalesceMultiline: args.include_multiline } : {},
     ...args.max_matches !== void 0 ? { maxMatches: args.max_matches } : {}
   };
   const result = await source.searchLogs(query3);

package/knowledge/references/bundle/support-bundle.md

@@ -20,6 +20,7 @@ Severity order for filtering is `WARN < UERR < ERROR < FATAL`, so `min_severity=
 
 - The logs are large (a rank log can exceed 100k lines). NEVER ask for a whole file. Use `kinetica_bundle_log_timeline` to localize, then `kinetica_bundle_search_logs` with a tight time window + severity to extract only relevant lines. The match cap is shared across files — if you see "capped", narrow the query rather than asking for more.
 - You can pass a timeline bucket label straight into `from_ts`/`to_ts` (e.g. `2026-06-11 15` searches that whole hour) — partial timestamps are widened to cover the full period.
+- A single log record can span multiple physical lines when a logged value (notably a SQL statement) contains embedded newlines — the continuation lines have no timestamp. A plain search returns only the first line. Pass `include_multiline: true` to stitch the continuation lines back onto each match and recover the whole record. See "Finding a crash's triggering SQL".
 - Timestamps are plain local strings without a timezone; compare them lexically and treat cross-rank timing cautiously.
 - **Ranks vs. the host manager:** `rank` selects a numeric rank (`r0`, `r1`, …) only. The host manager (`core-gpudb-rolling-hm.log`) is a singleton service, NOT a rank — search or timeline it with `host_manager: true`, never `rank: "hm"`. By default both `log_timeline` and `search_logs` already cover the host manager along with the numeric ranks; `kinetica_bundle_list_files` lists it under `services_present`.
 
@@ -29,11 +30,13 @@ When a worker rank segfaults mid-query, that rank's log holds the **backtrace**
 
 Workflow, given a `JobId` from a worker's crash stack:
 
-1. `kinetica_bundle_search_logs` with `rank: "r0"` and `regex` = the JobId. r0 logs the `/execute/sql` receipt (submitting user), the `Sql/SqlDriver.cpp … Executing SQL:` line, and per-operation endpoint lines.
-2. The per-operation lines (`Endpoint_aggregate_group_by.cpp`, filter/join endpoints) carry `table:`, `column_names:`/`aliases:` (the SELECT list), and `expr:` (the full WHERE predicate) — reconstruct the query from these.
-3. **Quirk:** if `Found plan for the SQL in cache` precedes it, the `Executing SQL:` line is truncated to just `SELECT`. Use the per-operation endpoint lines (step 2) — their predicate survives a cache hit. A `datetime()`/timestamp filter showing up here often _is_ the input that triggered a parser segfault.
+1. `kinetica_bundle_search_logs` with `rank: "r0"`, `regex` = the JobId, **and `include_multiline: true`**. r0 logs the `/execute/sql` receipt (submitting user), the `Sql/SqlDriver.cpp … Executing SQL:` line, and per-operation endpoint lines.
+2. **Read the full statement straight from the `Executing SQL:` line — do not reconstruct it.** Kinetica logs the SQL verbatim, so a real query spans MANY physical lines: `FROM …`, `JOIN …`, `WHERE …` each land on their own line with no timestamp prefix. Those continuation lines belong to the same log record; `include_multiline: true` stitches them back onto the match so you see the WHOLE query. WITHOUT it, a match is only the first physical line (e.g. `… Executing SQL: SELECT c."circuitId", c."circuitRef"`) and you would wrongly report the query as "truncated." Quote the statement verbatim from this single (now multi-line) match.
+3. **Cache-hit fallback only:** if `Found plan for the SQL in cache` precedes the job, Kinetica logs `Executing SQL:` as just the statement keyword (e.g. a bare `SELECT` or `EXECUTE PROCEDURE …`) with no continuation lines to stitch. ONLY then fall back to the per-operation endpoint lines (`Endpoint_aggregate_group_by.cpp`, filter/join endpoints): they carry `table:`, `column_names:`/`aliases:` (the SELECT list), and `expr:` (the full WHERE predicate), whose values survive a cache hit. A `datetime()`/timestamp filter showing up here often _is_ the input that triggered a parser segfault.
 
-See `rank-architecture.md` (Where queries are logged) for why this locality holds.
+**Where the full multi-line query actually lives:** `include_multiline` recovers the whole statement only from the **rolling core logs** (`logs-local/core-gpudb-rolling-r0.log`), where the SQL's embedded newlines are preserved as continuation lines. The **Loki per-rank tail** (`logs/rank0.log`) keeps only the statement's first physical line — promtail captures each line as its own record, so the `FROM`/`JOIN`/`WHERE` lines are simply not in that export, and nothing can stitch them. So this workflow depends on r0 being present under `logs-local/`. If r0 exists only as a Loki tail (rare for the coordinator, but possible for a Loki-only bundle), the complete query may not be in the bundle at all — say so rather than reporting the first line as the whole query, and fall back to step 3's endpoint lines.
+
+See `rank-architecture.md` (Where queries are logged) for why this locality holds, and "Two log families" below for rolling-vs-Loki precedence.
 
 ### Files of interest
 

package/knowledge/references/sql-dialect.md

@@ -0,0 +1,100 @@
+---
+title: Kinetica SQL Dialect — PostgreSQL Baseline & False Friends
+category: sql-syntax
+keywords:
+  [
+    sql-dialect,
+    postgresql,
+    false-friends,
+    try-cast,
+    safe-cast,
+    cast,
+    convert,
+    remediation-sql,
+    datediff,
+    timestamp,
+    nested-aggregate,
+    identifiers,
+    backticks,
+    decimal,
+    numeric,
+  ]
+---
+
+## Mental Model — Start from PostgreSQL
+
+Kinetica SQL is **PostgreSQL-compatible**: treat standard PostgreSQL syntax,
+functions, and behavior as the baseline. The deviations documented here (and in
+`version-quirks-7.2.md`) **override** that baseline. When no Kinetica-specific
+rule applies, the PostgreSQL form is the safe default.
+
+**The common failure when recommending remediation SQL is importing idioms from
+OTHER dialects** — SQL Server, Snowflake, Oracle, MySQL. Those are not the
+baseline; PostgreSQL is. The table below lists imports that look valid but fail
+in Kinetica.
+
+> Dialect facts adapted from the official `kineticadb/agent-skills` knowledge
+> base (Apache-2.0).
+
+## False Friends — Looks Valid, FAILS in Kinetica
+
+Do NOT put any of these in a remediation suggestion. Use the Kinetica form.
+
+| Looks valid (other dialect)            | Why it fails                                                | Use instead                                                                     |
+| -------------------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------- |
+| `TRY_CAST(x AS t)` / `SAFE_CAST(x, t)` | No error-tolerant cast exists in Kinetica                   | `CAST(x AS t)` or `CONVERT(x, t)`; shorthand `INT(x)`, `DOUBLE(x)`, `STRING(x)` |
+| `` `ident` `` (backtick quoting)       | Backticks are not a valid identifier quote                  | ANSI double quotes: `"ident"`                                                   |
+| `ts1 - ts2` (timestamp subtraction)    | Timestamp arithmetic with `-` is not supported              | `DATEDIFF('unit', ts1, ts2)`                                                    |
+| `NUMERIC(p, s)`                        | The type is named `DECIMAL`, not `NUMERIC`                  | `DECIMAL(p, s)` (max precision 27, max scale 18)                                |
+| `SUM(COUNT(*))` (nested aggregates)    | Fails with "Aggregate expressions cannot be nested"         | Separate into CTEs — window/aggregate in different stages                       |
+| `ANALYZE TABLE t`                      | No cost-based optimizer stats (see `version-quirks-7.2.md`) | No equivalent — do NOT suggest a "refresh table stats" step                     |
+| `SELECT ... ;` (trailing semicolon)    | A trailing `;` is rejected                                  | Omit the trailing semicolon                                                     |
+| `ORDER BY <array_col>`                 | Cannot sort by an `array<...>` column                       | Index an element (`ORDER BY "col"[1]`) or sort by a scalar column               |
+
+`TRY_CAST` / `SAFE_CAST` warrant special note: they come from SQL Server,
+Snowflake, and BigQuery, and Kinetica has no cast variant that returns NULL on
+conversion failure. If a value might not convert cleanly, filter the source rows
+(`WHERE` / `CASE`) before casting rather than reaching for a non-existent
+`TRY_*` function.
+
+## Type Conversion — the Valid Forms
+
+- Standard `CAST(expr AS type)` and `CONVERT(expr, type)` both work.
+- Shorthand cast functions: `INT(expr)`, `LONG(expr)`, `DOUBLE(expr)`,
+  `FLOAT(expr)`, `DECIMAL(expr)`, `STRING(expr)`, `ULONG(expr)`.
+- `JSON_EXTRACT_VALUE` always returns TEXT — you MUST cast for numeric use:
+  `CAST(JSON_EXTRACT_VALUE("payload", '$.count') AS INTEGER) > 100`.
+
+## Date / Time — Use Functions, Not Arithmetic
+
+| Kinetica form                        | Replaces (PostgreSQL / other)     |
+| ------------------------------------ | --------------------------------- |
+| `DATEDIFF('unit', start, end)`       | `EXTRACT(EPOCH FROM end - start)` |
+| `DATEADD('unit', amount, ts)`        | `ts + INTERVAL '...'`             |
+| `TIME_BUCKET(INTERVAL 'n' UNIT, ts)` | `date_bin()`                      |
+
+Units: `SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER, YEAR` (also
+`MICROSECOND` / `MILLISECOND`). INTERVAL syntax: `INTERVAL '30' MINUTE`.
+
+## Identifier & Statement Hygiene
+
+- **Double quotes only** for identifiers (`"my_col"`) — never backticks.
+- **Identifiers are case-sensitive** — `"UserID"` ≠ `"userid"`. Verify column
+  names against the discovered schema before recommending SQL.
+- **Fully-qualify table names** — `"schema"."table"`.
+- **No trailing semicolons.**
+
+## Kinetica Conveniences (valid, non-obvious)
+
+- `SELECT * EXCLUDE (col1, col2)` — wildcard minus specific columns.
+- `IF(cond, a, b)` — ternary (PostgreSQL has only `CASE`).
+- `NVL(x, default)` / `NVL2(x, not_null, null_val)` — null handling.
+- `DECODE(expr, m1, v1, ..., default)` — pattern matching.
+
+## When Unsure — Verify Empirically
+
+The live database is the source of truth. Before recommending any remediation
+SQL whose syntax you are not certain Kinetica supports, validate it with
+`kinetica_explain_query` against the live instance. If it cannot be validated
+(or there is no live connection), label the suggestion as unverified rather than
+asserting it is correct.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kinetica/admin-agent",
-  "version": "0.2.2",
+  "version": "0.2.4",
   "description": "Autonomous diagnostic agent for Kinetica databases",
   "license": "Apache-2.0",
   "author": "Kinetica",