@really-knows-ai/foundry 2.3.2 → 3.0.1

package/dist/docs/work-spec.md

@@ -0,0 +1,283 @@
+# WORK.md Spec
+
+WORK.md is created at the start of a foundry flow on a work branch. It is the shared state between all stages in all foundry cycles. It is transient — it exists only for the duration of the foundry flow.
+
+## Branch namespaces
+
+WORK.md and its sibling YAML files live on one of two branch kinds:
+
+- **`work/<flowId>-<description>`** — the standard flow run. Created
+  from `main`. On `foundry_git_finish`, the work branch is preserved
+  as `archive/work/<flowId>-<description>-<hash>` and squash-merged to
+  the base branch with a signed commit embedding the canonical Foundry
+  attestation block.
+- **`dry-run/<parentConfig>/<flowId>-<description>`** — a trial run
+  used to test in-progress config edits against a real flow. Created
+  from a `config/*` branch. On `foundry_git_finish`, the dry-run
+  branch is force-deleted and a forensic snapshot
+  (`README.md`, `work/WORK*`, `diff.patch`, `trace.jsonl`) is written
+  to `.snapshots/<run-id>/` on the parent `config/*` working tree.
+  No merge, no commit.
+
+WORK.md is identical on both kinds; the dispatching distinction is
+made entirely by `foundry_git_finish`. The third namespace,
+`config/<description>`, owns schema/config mutation and never carries
+WORK files.
+
+## Frontmatter
+
+```yaml
+---
+flow: <flow-id>
+cycle: <current-cycle-id>
+stages: [forge:write-haiku, quench:check-syllables, appraise:evaluate-quality]
+max-iterations: 3
+human-appraise: false
+deadlock-appraise: true
+deadlock-iterations: 5
+models:
+  forge: anthropic/claude-opus-4.7
+  appraise: openai/gpt-5
+assay:
+  extractors: [list-routes, list-models]
+---
+```
+
+Fields:
+- `flow` — the foundry flow being executed.
+- `cycle` — the current cycle id.
+- `stages` — the ordered route for this cycle. Each entry uses `base:alias` format where `base` is the stage type (`forge`, `quench`, `appraise`, `human-appraise`, or `assay`) and `alias` is a human-readable name for what that stage does in this cycle. The list is derived from the cycle and artefact type: `forge` and `appraise` are always included; `quench` is included iff any applicable law declares validators; `human-appraise` is included iff the cycle sets `human-appraise: true`; and `assay` is included iff the cycle declares an `assay.extractors` block.
+- `max-iterations` — how many forge passes before the cycle is blocked (default: 3).
+- `human-appraise` — run human-appraise every iteration (default: `false`).
+- `deadlock-appraise` — route to human-appraise when LLM appraisers deadlock (default: `true`).
+- `deadlock-iterations` — deadlock threshold (default: 5).
+- `models` — optional per-stage model overrides; individual appraisers may further override via their own `model` field.
+- `assay.extractors` — optional list of extractor names (defined under `foundry/memory/extractors/`) to run at iteration 0 before the first forge. Requires `foundry/memory/` to be initialized; cycle fails to load otherwise.
+
+The `stages` list is the happy path. Sort follows it, loops back to `forge` when unresolved feedback demands it, and may insert `human-appraise` on deadlock. If `assay` is configured, it runs once at iteration 0 before the route begins.
+
+### Who sets what
+
+- `flow`, `cycle` — set by the `flow` skill via `foundry_workfile_create` at flow start and updated as the flow advances between cycles.
+- `goal` — written once by the `flow` skill when `WORK.md` is created.
+- `stages`, `max-iterations`, `human-appraise`, `deadlock-appraise`, `deadlock-iterations`, `models`, `assay` — set by `foundry_orchestrate` on the first call of each cycle (via internal `workfile_configure_from_cycle`, reading the cycle definition).
+
+## Sections
+
+### Goal
+
+Free text describing what the foundry flow is producing and any context the human provided. Written once at foundry flow start, not modified after.
+
+### Artefacts
+
+A table tracking every artefact produced by the foundry flow. The generator (`createWorkfile` in `src/scripts/lib/workfile.js`) writes the table immediately after the `# Goal` body — there is no `# Artefacts` heading. The orchestrator's internal finalise step appends rows for matching output files; authoring tools should not edit the artefacts table directly.
+
+```markdown
+| File | Type | Cycle | Status |
+|------|------|-------|--------|
+| petitions/login-change.md | petition | write-petition | draft |
+| features/login-change.feature | gherkin | petition-to-gherkin | draft |
+```
+
+Statuses:
+- `draft` — artefact exists but has not cleared all stages
+- `done` — artefact has cleared all stages
+- `blocked` — artefact hit iteration limit or a violation
+
+## WORK.feedback.yaml
+
+The flow run owns one `WORK.feedback.yaml` file alongside `WORK.md` and
+`WORK.history.yaml`. It records every feedback item created during the current run,
+and the full state-transition history of each. Tracked in git, committed
+per-stage on the work branch, deleted by `foundry_git_finish` before the
+squash-merge (same lifecycle as `WORK.history.yaml`).
+
+### Schema
+
+Top-level: `{ items: [Item...] }`.
+
+Each `Item`:
+
+| Field | Type | Required | Mutable? |
+|-------|------|----------|----------|
+| `id` | string (ULID, 26 chars) | yes | no |
+| `file` | string | yes | no |
+| `tag` | string (no leading `#`) | yes | no |
+| `text` | string | yes | no |
+| `source` | string (`base:alias`) | yes | no |
+| `history` | array, length >= 1 | yes | prepend-only |
+
+`source` bases include `quench`, `appraise`, and `human-appraise`. Extractor failure marks the workfile failed, so `assay` does not appear as a feedback source.
+
+Each history snapshot:
+
+| Field | Type | Required | Notes |
+|-------|------|----------|-------|
+| `state` | enum | yes | `open \| actioned \| wont-fix \| rejected \| deadlocked \| resolved` |
+| `stage` | string (`base:alias`) or literal `sort` | yes | Who performed the transition |
+| `cycle` | string | yes | Cycle id at the time of the transition |
+| `timestamp` | ISO-8601 UTC with ms | yes | |
+| `reason` | string | conditional | Required on `rejected`, `wont-fix`, `deadlocked`, `resolved`; forbidden on `open`; optional on `actioned` |
+
+`history[0]` is always the current state; new snapshots are prepended.
+`resolved` is terminal.
+
+### State machine
+
+Feedback items flow through a six-state lifecycle: `open` (newly raised), `actioned` (forge has addressed it), `wont-fix` (forge declined subjective feedback with justification), `rejected` (appraiser or human overruled the wont-fix), `deadlocked` (sort detected repeated forge/appraise iterations on the same item), and `resolved` (approved by the item's originating stage or human override). Transitions are source-based: the legal moves depend on what stage created the item and who is trying to transition it. The feedback state machine is the engine that routes work between cycles.
+
+The six states and the legal transitions are:
+
+| From \ Caller | forge (any source) | source-stage (quench / appraise / human-appraise where stageId === item.source) | sort | human-appraise (override authority, any source) |
+|---|---|---|---|---|
+| `open` | -> `actioned` always; -> `wont-fix` only if `item.source` base is `appraise` | — | -> `deadlocked` (if depth >= threshold) | — |
+| `rejected` | -> `actioned` always; -> `wont-fix` only if `item.source` base is `appraise` | — | -> `deadlocked` (if depth >= threshold) | — |
+| `actioned` | — | -> `{resolved, rejected}` | -> `deadlocked` (if depth >= threshold) | -> `{resolved, rejected}` |
+| `wont-fix` | — | -> `{resolved, rejected}` | -> `deadlocked` (if depth >= threshold) | -> `{resolved, rejected}` |
+| `deadlocked` | — | — | — | -> `{resolved, rejected}` |
+| `resolved` | — | — | — | — (terminal) |
+
+Notes:
+
+- `source-stage` column applies when the caller's stage id exactly matches `item.source` (e.g. `appraise:write-check` resolving an item it created). `human-appraise` override authority (last column) applies regardless of `item.source` and is the only path that can transition out of `deadlocked`.
+- **Forge `wont-fix` scope.** When `item.source` base is `quench` (objective validation failure) or `human-appraise` (direct user instruction), forge may not `wont-fix` — it must `actioned`. Only `appraise`-sourced items are wont-fix-able by forge. This replaces the earlier tag-based restriction on `validation` / `human` tags.
+- `tag` is categorical and display-only. The state machine consults `source`, not tags; `validation` / `human` tag-based restrictions are legacy and do not apply.
+- **Reason required on** `rejected`, `wont-fix`, `deadlocked`, `resolved`. **Forbidden on** `open`. **Optional on** `actioned` (the code change is the reason).
+- Sort is the only writer of `state: deadlocked`; it writes these via its internal pass, not through the plugin API.
+
+This section is the authoritative specification of the feedback state machine.
+
+### Transitions are made via the plugin API
+
+No direct yaml editing. Every state change goes through one of:
+
+- `foundry_feedback_add` (creates items from `quench`, `appraise`, and `human-appraise`)
+- `foundry_feedback_action` (forge: open/rejected -> actioned)
+- `foundry_feedback_wontfix` (forge: open/rejected -> wont-fix)
+- `foundry_feedback_resolve` (source stage: actioned/wont-fix -> resolved/rejected; or human-appraise deadlock override)
+
+### Persistence
+
+Writes are atomic: `io.writeFile(path + '.tmp', body); io.rename(tmp, path)`.
+A crash between the two steps leaves the live file untouched.
+
+## Who writes what
+
+| Section | Written by | Updated by |
+|---------|-----------|------------|
+| Frontmatter (`flow`, `cycle`) | `foundry_workfile_create` (flow skill) | updated in place as the flow advances between cycles |
+| Frontmatter (`stages`, `max-iterations`, `human-appraise`, `deadlock-appraise`, `deadlock-iterations`, `models`) | `foundry_orchestrate` (first call of each cycle, internally) | reset on each new cycle |
+| Goal | `foundry_workfile_create` (flow skill) | nobody |
+| Artefacts | the orchestrator's internal finalize step (after forge closes) | `foundry_artefacts_set_status` (orchestrator → `done`/`blocked`) |
+| `WORK.feedback.yaml` | `foundry_feedback_add` (`quench` / `appraise` / `human-appraise`) | `foundry_feedback_action` / `foundry_feedback_wontfix` (forge), `foundry_feedback_resolve` (source stage / human-appraise override); sort writes only deadlocked snapshots |
+| `WORK.history.yaml` | `foundry_orchestrate` | `foundry_orchestrate` |
+
+Note: `foundry_artefacts_add` no longer exists as a public tool — artefact registration is automatic via the orchestrator's internal finalize step, which scans the git diff and registers files matching the output type's `file-patterns` as `draft`.
+
+## WORK.history.yaml
+
+A separate file (`WORK.history.yaml`) alongside WORK.md. Append-only log of every stage execution.
+
+```yaml
+- timestamp: "2026-04-17T14:32:01.000Z"
+  cycle: write-petition
+  stage: forge:draft-petition
+  iteration: 1
+  comment: Initial petition draft created
+  seq: 0
+  open_feedback: 0
+
+- timestamp: "2026-04-17T14:32:45.000Z"
+  cycle: write-petition
+  stage: quench:validate-petition
+  iteration: 1
+  comment: 2 validation issues found
+  seq: 1
+  open_feedback: 2
+
+- timestamp: "2026-04-17T14:33:12.000Z"
+  cycle: write-petition
+  stage: forge:draft-petition
+  iteration: 2
+  comment: Addressed 2 validation issues
+  seq: 2
+  open_feedback: 2
+
+- timestamp: "2026-04-17T14:33:30.000Z"
+  cycle: write-petition
+  stage: quench:validate-petition
+  iteration: 2
+  comment: Validation passed
+  seq: 3
+  open_feedback: 0
+
+- timestamp: "2026-04-17T14:34:00.000Z"
+  cycle: write-petition
+  stage: appraise:review-petition
+  iteration: 2
+  comment: No issues found, cycle complete
+  seq: 4
+  open_feedback: 0
+```
+
+### Fields
+
+| Field | Type | Required | Notes |
+|-------|------|----------|-------|
+| `cycle` | string | yes | |
+| `stage` | string or literal `sort` | yes | |
+| `iteration` | integer | yes | Count of completed forge stages for the cycle at the time of write |
+| `comment` | string | yes | |
+| `timestamp` | ISO-8601 UTC with ms | yes | |
+| `seq` | integer | yes on write | Monotonic per file; sort tiebreaker for same-ms entries |
+| `route` | string | conditional | Only on `stage: sort` entries; records the route decision. Throws if set on a non-sort entry |
+| `open_feedback` | integer | yes on write | Count of non-resolved items in `WORK.feedback.yaml` at the time of write; deadlocked items are counted |
+
+### Rules
+
+- Append-only — never edit or delete entries.
+- Every stage produces an entry when it completes.
+- Sort reads this to determine what has happened in the current cycle.
+- Iteration is derived from counting forge entries for the current cycle.
+
+### Who writes
+
+History entries are written by `foundry_orchestrate` after each stage closes (via its internal `foundry_history_append` — the tool is not registered publicly). Sub-agents never append history directly.
+
+### Lifecycle
+
+`WORK.history.yaml` is tracked in git and committed per-stage on the work
+branch. `foundry_git_finish` deletes it before the squash-merge so the
+history does not leak into the base branch.
+
+If the yaml is malformed on read (parse failure or non-array root), the
+flow is marked failed via `markWorkfileFailed` and the error is re-thrown
+to the caller. Mirrors the P0 #3 failed-flow pattern used by the memory
+sync writer.
+
+## Example
+
+A complete WORK.md mid-foundry flow:
+
+```markdown
+---
+flow: make-haiku
+cycle: haiku-creation
+stages: [forge:write-haiku, quench:check-syllables, appraise:evaluate-quality]
+max-iterations: 3
+human-appraise: false
+deadlock-appraise: true
+deadlock-iterations: 5
+---
+
+# Goal
+
+Write a haiku about autumn rain. Should evoke loneliness
+and the sound of rain on leaves.
+
+| File | Type | Cycle | Status |
+|------|------|-------|--------|
+| petitions/autumn-rain-haiku.md | petition | haiku-ideation | done |
+| haiku/autumn-rain.md | haiku | haiku-creation | draft |
+
+```

package/dist/scripts/lib/artefacts.js

@@ -0,0 +1,151 @@
+/**
+ * Artefacts table utilities for WORK.md.
+ *
+ * Parses, adds rows to, and updates status in the markdown artefacts table.
+ */
+
+// --- Table line classifiers ---
+
+function isTableHeader(line) {
+  return line.startsWith('| File');
+}
+
+function isTableSeparator(line) {
+  return line.startsWith('|---');
+}
+
+function isTableRow(line) {
+  return line.startsWith('|');
+}
+
+function parseTableRow(line) {
+  const cols = line.split('|').slice(1, -1).map(c => c.trim());
+  return cols.length >= 4 ? cols : null;
+}
+
+// --- Status validation ---
+
+function validateStatus(newStatus) {
+  if (newStatus === 'draft') {
+    throw new Error('status draft not permitted; artefacts are registered automatically during orchestration');
+  }
+  if (!['done', 'blocked'].includes(newStatus)) {
+    throw new Error(`invalid status: ${newStatus}`);
+  }
+}
+
+// --- Table boundary detection ---
+
+function findTableHeader(lines) {
+  for (let i = 0; i < lines.length; i++) {
+    if (isTableHeader(lines[i].trim())) return i;
+  }
+  return -1;
+}
+
+function findTableSeparator(lines, afterIdx) {
+  for (let i = afterIdx + 1; i < lines.length; i++) {
+    if (isTableSeparator(lines[i].trim())) return i;
+  }
+  return -1;
+}
+
+function getTableBounds(lines) {
+  const headerIdx = findTableHeader(lines);
+  if (headerIdx < 0) return null;
+  const sepIdx = findTableSeparator(lines, headerIdx);
+  if (sepIdx < 0) return null;
+  return { headerIdx, sepIdx };
+}
+
+function findTableEnd(lines, startIdx) {
+  for (let i = startIdx; i < lines.length; i++) {
+    const stripped = lines[i].trim();
+    if (!isTableRow(stripped)) return i;
+  }
+  return lines.length;
+}
+
+function formatTableRow(cols) {
+  return '| ' + cols.join(' | ') + ' |';
+}
+
+/**
+ * Parse the artefacts markdown table from text.
+ * @param {string} text
+ * @returns {Array<{file: string, type: string, cycle: string, status: string}>}
+ */
+export function parseArtefactsTable(text) {
+  const lines = text.split('\n');
+  const bounds = getTableBounds(lines);
+  if (!bounds) return [];
+
+  const artefacts = [];
+  const endIdx = findTableEnd(lines, bounds.sepIdx + 1);
+
+  for (let i = bounds.sepIdx + 1; i < endIdx; i++) {
+    const cols = parseTableRow(lines[i].trim());
+    if (cols) {
+      artefacts.push({
+        file: cols[0],
+        type: cols[1],
+        cycle: cols[2],
+        status: cols[3],
+      });
+    }
+  }
+
+  return artefacts;
+}
+
+/**
+ * Add a row to the artefacts table.
+ * @param {string} text - Full WORK.md text
+ * @param {{file: string, type: string, cycle: string, status: string}} row
+ * @returns {string} Updated text
+ */
+export function addArtefactRow(text, { file, type, cycle, status }) {
+  const lines = text.split('\n');
+  const bounds = getTableBounds(lines);
+
+  if (!bounds) {
+    throw new Error('Artefacts table not found');
+  }
+
+  const endIdx = findTableEnd(lines, bounds.sepIdx + 1);
+  const insertAt = endIdx > bounds.sepIdx + 1 ? endIdx - 1 : bounds.sepIdx;
+  const newRow = `| ${file} | ${type} | ${cycle} | ${status} |`;
+  lines.splice(insertAt + 1, 0, newRow);
+  return lines.join('\n');
+}
+
+/**
+ * Update the status column for a specific file in the artefacts table.
+ * @param {string} text - Full WORK.md text
+ * @param {string} file - File name to match
+ * @param {string} newStatus - New status value
+ * @returns {string} Updated text
+ */
+export function setArtefactStatus(text, file, newStatus) {
+  validateStatus(newStatus);
+
+  const lines = text.split('\n');
+  const bounds = getTableBounds(lines);
+
+  if (!bounds) {
+    throw new Error(`File not found in artefacts table: ${file}`);
+  }
+
+  const endIdx = findTableEnd(lines, bounds.sepIdx + 1);
+
+  for (let i = bounds.sepIdx + 1; i < endIdx; i++) {
+    const cols = parseTableRow(lines[i].trim());
+    if (cols && cols[0] === file) {
+      cols[3] = newStatus;
+      lines[i] = formatTableRow(cols);
+      return lines.join('\n');
+    }
+  }
+
+  throw new Error(`File not found in artefacts table: ${file}`);
+}

package/dist/scripts/lib/assay/loader.js

@@ -0,0 +1,151 @@
+import yaml from 'js-yaml';
+import { memoryPaths } from '../memory/paths.js';
+
+/**
+ * Extractor Contract
+ * ==================
+ *
+ * An extractor is a project-authored executable (script, binary, etc.) that
+ * emits JSONL (newline-delimited JSON) describing entities and edges to upsert
+ * into flow memory.
+ *
+ * Output Format:
+ * - One JSON object per line (JSONL/NDJSON format)
+ * - Pretty-printed multi-line JSON is NOT supported
+ * - Blank lines and lines starting with '#' are ignored
+ * - Each object must have a "kind" field: "entity" or "edge"
+ *
+ * Entity format:
+ *   {"kind":"entity","type":"<entity-type>","name":"<id>","value":"<string ≤ 4KB>"}
+ *
+ * Edge format:
+ *   {"kind":"edge","from":{"type":"...","name":"..."},"edge":"<edge-type>","to":{"type":"...","name":"..."}}
+ *
+ * Exit codes:
+ * - 0 on success
+ * - Non-zero on failure (aborts the assay stage)
+ *
+ * Environment:
+ * - Extractors inherit the agent's full environment, including any API tokens
+ *   or credentials present in the agent process
+ * - Extractors are project-authored, committed code; they are trusted paths
+ * - Keep environment variable handling internal to extraction logic
+ */
+
+const IDENT = /^[a-z][a-z0-9_-]*$/;
+const MAX_TIMEOUT_MS = 600_000;
+
+function validateTimeoutMs(ms) {
+  if (!Number.isFinite(ms) || ms <= 0) throw new Error(`timeout must be a positive number (ms) or duration string`);
+  if (ms > MAX_TIMEOUT_MS) throw new Error(`timeout must not exceed 600000ms (10 minutes)`);
+  return ms;
+}
+
+function parseNumberTimeout(v) {
+  return validateTimeoutMs(v);
+}
+
+function unitToMs(unit) {
+  if (unit === 'ms') return 1;
+  if (unit === 's') return 1_000;
+  if (unit === 'm') return 60_000;
+  throw new Error(`timeout: impossible unit ${unit}`);
+}
+
+function parseStringTimeout(v) {
+  const trimmed = v.trim();
+  const m = trimmed.match(/^(\d+)(ms|s|m)?$/);
+  if (!m) throw new Error(`timeout: unrecognised duration '${trimmed}' (expected e.g. "500ms", "30s", "2m")`);
+  const n = Number(m[1]);
+  const unit = m[2] ?? 'ms';
+  const ms = n * unitToMs(unit);
+  return validateTimeoutMs(ms);
+}
+
+function parseTimeout(v) {
+  if (v === undefined || v === null) return 60_000;
+  if (typeof v === 'number') return parseNumberTimeout(v);
+  if (typeof v !== 'string') throw new Error(`timeout must be a duration string (e.g. "30s") or a number of ms`);
+  return parseStringTimeout(v);
+}
+
+function stripBom(text) {
+  return text.charCodeAt(0) === 0xFEFF ? text.slice(1) : text;
+}
+
+function findFrontmatterEnd(lines) {
+  for (let i = 1; i < lines.length; i++) {
+    if (lines[i] === '---') return i;
+  }
+  return -1;
+}
+
+function parseFrontmatterYaml(fmText) {
+  const fm = yaml.load(fmText) ?? {};
+  if (typeof fm !== 'object' || Array.isArray(fm)) throw new Error(`frontmatter must be a mapping`);
+  return fm;
+}
+
+function splitFrontmatter(text) {
+  const stripped = stripBom(text);
+  const lines = stripped.split(/\r?\n/);
+  if (lines[0] !== '---') throw new Error(`missing frontmatter: file must start with '---'`);
+  const end = findFrontmatterEnd(lines);
+  if (end === -1) throw new Error(`missing frontmatter: no closing '---'`);
+  const fmText = lines.slice(1, end).join('\n');
+  const body = lines.slice(end + 1).join('\n').replace(/^\s+/, '');
+  const fm = parseFrontmatterYaml(fmText);
+  return { fm, body };
+}
+
+function validateCommand(fm, name) {
+  if (typeof fm.command !== 'string' || !fm.command.trim()) {
+    throw new Error(`extractor '${name}': 'command' is required and must be a non-empty string`);
+  }
+}
+
+function validateMemoryWrite(fm, name) {
+  const writeList = fm?.memory?.write;
+  if (!Array.isArray(writeList) || writeList.length === 0) {
+    throw new Error(`extractor '${name}': 'memory.write' is required and must be a non-empty array of entity type names`);
+  }
+  return writeList;
+}
+
+function validateEntityTypes(writeList, name) {
+  for (const t of writeList) {
+    if (typeof t !== 'string' || !IDENT.test(t)) {
+      throw new Error(`extractor '${name}': memory.write entry '${t}' is not a valid entity type identifier`);
+    }
+  }
+}
+
+export async function loadExtractor(foundryDir, name, io) {
+  if (!IDENT.test(name)) throw new Error(`invalid extractor name '${name}' (expected lowercase identifier)`);
+  const p = memoryPaths(foundryDir);
+  const path = p.extractorFile(name);
+  if (!(await io.exists(path))) throw new Error(`extractor not found: ${name} (expected at ${path})`);
+  const text = await io.readFile(path);
+  const { fm, body } = splitFrontmatter(text);
+  validateCommand(fm, name);
+  const writeList = validateMemoryWrite(fm, name);
+  validateEntityTypes(writeList, name);
+  const timeoutMs = parseTimeout(fm.timeout);
+  return {
+    name,
+    command: fm.command,
+    memoryWrite: writeList,
+    timeoutMs,
+    body: body,
+  };
+}
+
+export async function listExtractors(foundryDir, io) {
+  const p = memoryPaths(foundryDir);
+  if (!(await io.exists(p.extractorsDir))) return [];
+  const entries = await io.readDir(p.extractorsDir);
+  return entries
+    .filter((f) => f.endsWith('.md'))
+    .map((f) => f.slice(0, -3))
+    .sort();
+}

package/dist/scripts/lib/assay/parse-jsonl.js

@@ -0,0 +1,102 @@
+import { MAX_VALUE_BYTES } from '../memory/validate.js';
+
+const ENTITY_FIELDS = new Set(['kind', 'type', 'name', 'value']);
+const EDGE_FIELDS = new Set(['kind', 'from', 'edge', 'to']);
+
+function checkFields(obj, allowed, lineNo, kind) {
+  for (const k of Object.keys(obj)) {
+    if (!allowed.has(k)) {
+      throw new Error(`extractor output line ${lineNo}: unknown field '${k}' on ${kind} row`);
+    }
+  }
+}
+
+function req(obj, key, lineNo, kind) {
+  if (obj[key] === undefined || obj[key] === null || obj[key] === '') {
+    throw new Error(`extractor output line ${lineNo}: ${kind}.${key} is required`);
+  }
+}
+
+function parseEntityRow(obj, lineNo) {
+  checkFields(obj, ENTITY_FIELDS, lineNo, 'entity');
+  req(obj, 'type', lineNo, 'entity');
+  req(obj, 'name', lineNo, 'entity');
+  if (typeof obj.value !== 'string') {
+    throw new Error(`extractor output line ${lineNo}: entity.value is required and must be a string`);
+  }
+  const bytes = Buffer.byteLength(obj.value, 'utf-8');
+  if (bytes > MAX_VALUE_BYTES) {
+    throw new Error(`extractor output line ${lineNo}: entity.value is ${bytes} bytes (max ${MAX_VALUE_BYTES}, too large)`);
+  }
+  return { kind: 'entity', type: obj.type, name: obj.name, value: obj.value };
+}
+
+function validateRef(obj, fieldName, lineNo) {
+  if (!obj[fieldName] || typeof obj[fieldName] !== 'object') {
+    throw new Error(`extractor output line ${lineNo}: edge.${fieldName} is required and must be an object {type,name}`);
+  }
+  req(obj[fieldName], 'type', lineNo, `edge.${fieldName}`);
+  req(obj[fieldName], 'name', lineNo, `edge.${fieldName}`);
+}
+
+function checkRefSize(ref, fieldName, lineNo) {
+  const bytes = Buffer.byteLength(ref.name, 'utf-8');
+  if (bytes > MAX_VALUE_BYTES) {
+    throw new Error(`extractor output line ${lineNo}: edge.${fieldName}.name is ${bytes} bytes (max ${MAX_VALUE_BYTES}, too large)`);
+  }
+}
+
+function parseEdgeRow(obj, lineNo) {
+  checkFields(obj, EDGE_FIELDS, lineNo, 'edge');
+  validateRef(obj, 'from', lineNo);
+  validateRef(obj, 'to', lineNo);
+  req(obj, 'edge', lineNo, 'edge');
+  checkRefSize(obj.from, 'from', lineNo);
+  checkRefSize(obj.to, 'to', lineNo);
+  return {
+    kind: 'edge',
+    edge_type: obj.edge,
+    from_type: obj.from.type,
+    from_name: obj.from.name,
+    to_type: obj.to.type,
+    to_name: obj.to.name,
+  };
+}
+
+function isSkippableLine(trimmed) {
+  return trimmed === '' || trimmed.startsWith('#');
+}
+
+function parseJsonLine(trimmed, lineNo) {
+  try {
+    return JSON.parse(trimmed);
+  } catch (err) {
+    throw new Error(`extractor output line ${lineNo}: invalid JSON (${err.message}). Extractors must output one JSON object per line (JSONL/NDJSON format), not pretty-printed multi-line JSON.`, { cause: err });
+  }
+}
+
+function validateParsedObject(obj, lineNo) {
+  if (!obj || typeof obj !== 'object' || Array.isArray(obj)) {
+    throw new Error(`extractor output line ${lineNo}: expected a JSON object`);
+  }
+}
+
+function dispatchKind(obj, lineNo) {
+  if (obj.kind === 'entity') return parseEntityRow(obj, lineNo);
+  if (obj.kind === 'edge') return parseEdgeRow(obj, lineNo);
+  throw new Error(`extractor output: unknown kind '${obj.kind}' at line ${lineNo}`);
+}
+
+export function parseExtractorOutput(text) {
+  if (!text) return [];
+  const lines = text.split(/\r?\n/);
+  const out = [];
+  for (let i = 0; i < lines.length; i++) {
+    const trimmed = lines[i].trim();
+    if (isSkippableLine(trimmed)) continue;
+    const obj = parseJsonLine(trimmed, i + 1);
+    validateParsedObject(obj, i + 1);
+    out.push(dispatchKind(obj, i + 1));
+  }
+  return out;
+}