@exellix/ai-tasks 8.4.2 → 8.4.3

package/.docs/narrix-context-consumption-gap.md

@@ -0,0 +1,184 @@
+# Narrix Context Consumption Gap (produced, but not grounding the final answer)
+
+## Summary
+Narrix is being executed and its structured output is present in runtime memory, but the task execution / prompt-building layer does not reliably consume it as *primary grounding context* for the final answer.
+
+The result is a common symptom:
+- Narrix-generated structured facts exist at runtime
+- but the final LLM output behaves like a generic web-based explanation (or otherwise does not strongly reflect the locally grounded facts)
+
+This doc explains:
+1. What is going wrong
+2. Why it happens (where the handoff mismatch occurs)
+3. What needs to be done (implementation checklist)
+4. How to validate the fix (acceptance checks)
+
+---
+
+## What we observed (symptom)
+When Narrix produces structured scoping/discovery data:
+- the data appears in runtime memory (so Narrix "works")
+- but the downstream component that builds the final prompt does not consistently embed that data in the main context block the LLM relies on
+
+The practical "acceptance" symptom:
+- Structured local facts are available
+- but the generated answer does not visibly/consistently use those facts as grounding evidence
+
+---
+
+## Root causes (where the mismatch happens)
+
+### 1) Context injection is opt-in (`includeContextInPrompt`)
+In this repository, the executor only attaches a task-level `context` markdown string when:
+- `request.includeContextInPrompt === true`
+
+When context is not requested, the executor receives:
+- `context: ""`
+
+So even if Narrix output exists in memory, it will not be embedded into the LLM message unless the pipeline enables context generation.
+
+Relevant code path:
+- `src/core/task-sdk.ts` (`_executeDirect`): context becomes empty when `includeContextInPrompt` is not true.
+
+### 2) `executionPipeline` + `synthesized-context` can replace the normal context path (`overrideContext`)
+When `executionPipeline` is used, PRE steps may set an `overrideContext` string.
+
+If `overrideContext` is set, `_executeDirect` uses that string directly and skips the normal path that would append Narrix context sections.
+
+Net effect:
+- Narrix may be present in memory
+- but the specific prompt context used for the main LLM call may not contain the Narrix-derived grounding you expect
+
+Relevant code path:
+- `src/core/task-sdk.ts` (execution pipeline: PRE step `synthesized-context`)
+- `src/core/task-sdk.ts` (`_executeDirect`: if `overrideContext !== undefined`, it uses it as-is)
+
+### 3) Narrix attachment vs `taskMemory.narrix` for `synthesized-context`
+There are two different Narrix "output locations" depending on which Narrix flow is used:
+
+1. **Task-level Narrix pre-processor (`request.narrix`)**
+   - Output is attached to `executionMemory._narrix` (or `executionMemory[attachToField]`)
+   - `_runSynthesizedContextPreStep()` reads `input.executionMemory[attachToField]` where `attachToField` defaults to `"_narrix"`.
+
+2. **`narrix-then-direct` (`executionType: "narrix-then-direct"`)**
+   - Output is written into `taskMemory.narrix` (array)
+   - It also sets a convenience field on the skill input as `input.narrixContext`
+
+**Current behavior (synthesis PRE step):** `_runSynthesizedContextPreStep` builds `narrixAttachment` from `executionMemory[attachToField]` when it has the enriched attachment shape, **otherwise** it **coerces** `taskMemory.narrix` via `coerceTaskMemoryNarrixToAttachment()` (see `src/core/task-sdk.ts`). So synthesis source material can include Narrix from either location. Use an explicit `contextSourcePolicy` (e.g. `memory-web`) when you must **exclude** Narrix from serialized memory while still using web + memory.
+
+Relevant code paths:
+- `src/core/task-sdk.ts` (`_runSynthesizedContextPreStep`): `narrixAttachment` resolution + `resolveSourceMaterial`
+- `src/core/task-sdk.ts` (`narrix-then-direct` branch): writes Narrix output to `taskMemory.narrix` and sets `input.narrixContext`
+
+---
+
+## Why this leads to "generic answers"
+LLMs tend to default to the most salient, consistently provided instruction/context structure.
+
+If the final `context` string passed to the executor:
+- is empty, or
+- is synthesized from the wrong subset of available information, or
+- contains web-style evidence but not Narrix grounding, or
+- omits the Narrix sections that the prompt / skill was designed to lean on,
+
+then the model will still produce a coherent explanation, but it will be less constrained by the locally grounded structured facts.
+
+This matches the observed behavior: Narrix data exists, yet output does not strongly reflect it.
+
+---
+
+## What needs to be done (implementation checklist)
+This section is written as "what the fix should accomplish", not as a specific patch.
+
+### 1) Ensure Narrix grounding is embedded whenever Narrix is requested
+Decision: when Narrix is in use, the system should not silently omit the `context` field.
+
+Recommended behavior:
+- If a request includes Narrix (either `request.narrix` or `executionType: "narrix-then-direct"`), default to:
+  - `includeContextInPrompt: true`
+- Or throw a clear error when `synthesized-context` PRE is configured but context generation is not enabled.
+
+Why:
+- Narrix being created without being injected into `enrichedInput.context` is effectively a no-op for grounding.
+
+### 2) Make `synthesized-context` consume Narrix from the correct location(s)
+`synthesized-context` PRE step must treat Narrix output as "available source material" regardless of whether it lives in:
+- `executionMemory._narrix` (task-level pre-processor flow)
+- `taskMemory.narrix` (narrix-then-direct flow)
+
+Options:
+- Convert `taskMemory.narrix` into the `_narrix`-shaped attachment used by `resolveSourceMaterial()`
+- Or extend `resolveSourceMaterial()` / serialization so synthesis can consume `taskMemory.narrix` directly
+
+Why:
+- Today, synthesized-context may be blind to narrix-then-direct's canonical output location.
+
+### 3) Prevent `overrideContext` from accidentally dropping Narrix sections
+When `overrideContext` is used, it should still include Narrix grounding sections (or Narrix-serialized source material) when Narrix is available.
+
+Two non-exclusive strategies:
+- Synthesis must embed Narrix source material (fix #2), so `overrideContext` naturally contains it
+- Or post-process `overrideContext` to append the Narrix grounding markdown section when Narrix exists
+
+Why:
+- Avoids a failure mode where Narrix exists in memory, but the PRE step's override ignores it.
+
+### 4) Clarify the template contract (do not rely on `{{narrix}}` tokens)
+In this repo, the executor passes Narrix grounding via:
+- `enrichedInput.context` (generated markdown string)
+- and for `narrix-then-direct`, also via `input.narrixContext` convenience field
+
+Therefore, adding `{{narrix}}` tokens to templates will not automatically help unless templates are explicitly wired to the actual fields this SDK populates.
+
+Recommended action:
+- Make the skill/prompt contract reference the real fields this SDK provides:
+  - `context` (main visible grounding block)
+  - `input.narrixContext` (for `narrix-then-direct`)
+  - `taskMemory.narrix` (if a skill reads from memory directly)
+
+---
+
+## Acceptance checks (how to validate the fix)
+
+### A. Context string must include Narrix-derived content
+For a request where Narrix output is expected to be available:
+- the `enrichedInput.context` passed to the executor must contain Narrix grounding markers, such as:
+  - `## Scoping and discovery` (for `_narrix`-attachment flow), or
+  - `## Narrix Scoping` (for `taskMemory.narrix` flow)
+
+At minimum:
+- the context must be non-empty when Narrix is requested
+
+### B. Pipeline with `synthesized-context` must not lose Narrix grounding
+When using `executionPipeline` with a PRE step of type `synthesized-context`:
+- the synthesized context returned by the PRE step must be generated from Narrix output (from the correct location)
+- and the resulting main LLM call must receive a `context` that reflects that grounding
+
+### C. "No Narrix requested" should behave exactly as today
+Regression guardrails:
+- if Narrix is not requested, context behavior should remain unchanged (default empty or context-generator behavior)
+- no accidental placeholder injection should appear
+
+---
+
+## Debug / triage checklist for callers
+When investigating a live case:
+1. Confirm Narrix flow type:
+   - task-level pre-processor: `request.narrix`
+   - narrix-then-direct: `executionType: "narrix-then-direct"` + `narrixInput`
+2. Confirm pipeline usage:
+   - is `executionPipeline` set?
+   - is there a PRE step of type `synthesized-context`?
+3. Confirm context opt-in:
+   - is `includeContextInPrompt: true` in the final request?
+4. Inspect the constructed executor input:
+   - verify `enrichedInput.context` is non-empty and contains Narrix grounding markers
+   - verify `input.narrixContext` exists for narrix-then-direct
+
+---
+
+## Related references
+- `.docs/narrix-context-downstream-report.md` (Task context contract and how ai-tasks controls `context`)
+- `.docs/synthesized-context-strategy-spec.md` (synthesized-context design and constraints)
+- `src/core/task-sdk.ts` (all relevant wiring: context generation, pipeline PRE override, and narrix-then-direct / narrix pre-processor injection)
+

package/.docs/narrix-context-downstream-report.md

@@ -0,0 +1,30 @@
+# Task context in LLM messages
+
+All fixes for task context are implemented **here (ai-tasks)**. This document describes what we send and how we control it.
+
+---
+
+## What ai-tasks sends
+
+- Ai-tasks passes an **enriched input** to the executor with a **`context`** field: a single string (markdown) that may contain task-level context for the LLM.
+- **Context is opt-in.** We add context only when `request.includeContextInPrompt === true`. By default we send no context (empty string), so the LLM gets system + user only unless the step explicitly requests context.
+- **NARRIX is the source of scoping/discovery context** in the LLM prompt for this flow. When NARRIX is in play and context is requested, ai-tasks builds context only from NARRIX (`buildNarrixPreProcessorContextMarkdown`): a **"## Scoping and discovery"** section when there is real content (signals, stories), or nothing when empty. We do not pass _narrix to the context generator (xontext), so the context map cannot emit the "Narrix Pre-Processor" placeholder; placeholder emission is avoided and NARRIX context is added only in ai-tasks.
+- When NARRIX is not in play and context is requested, context comes from the context generator plus any `taskMemory.narrix` section. The text uses neutral headings only; no internal product or system names appear in the context string.
+
+---
+
+## Summary (all fixes here)
+
+| Item | Action |
+|------|--------|
+| What is in context | When requested: NARRIX in play → only "## Scoping and discovery" from _narrix (or empty); NARRIX not in play → context generator + taskMemory.narrix. |
+| Opt-in only | Context is added only when `includeContextInPrompt === true`; default is no context. |
+| No placeholder | We do not use the context generator for context when NARRIX is in play, so the "Narrix Pre-Processor" placeholder is never emitted; NARRIX context is built only in ai-tasks. |
+
+---
+
+## References (for implementors)
+
+- **Contract:** `enrichedInput.context` is a markdown string. It is non-empty only when `includeContextInPrompt === true`. When NARRIX is in play, it contains at most a "## Scoping and discovery" subsection from _narrix (or is empty). Placeholder emission is controlled by ai-tasks (we do not pass _narrix to the context generator when NARRIX is in play).
+- **Opt-in:** `request.includeContextInPrompt === true` is required for any context to be sent; default is no context.
+- **No empty/placeholder:** When NARRIX is in play we build context only from NARRIX in ai-tasks; we never add the "Narrix Pre-Processor" placeholder or an empty block.

package/.docs/narrix-ingest-and-packs-library-spec.md

@@ -0,0 +1,240 @@
+# Narrix Ingest, Packs-Library, and Related Components — Low-Level Spec for Planning
+
+This document describes **`@narrices/narrix-ingest`**, **`@narrices/narrix-packs-library`**, and related ai-tasks wiring so you can plan a component that decides **what and whether to search the web** (or other sources) to obtain the **complete story / information** needed for a Narrix run.
+
+**Audience:** Low-level spec for new component planning (e.g. “pre-narrix info gatherer” or “gap-driven web search”).  
+**Scope:** Contract and data flow only; implementation details of ingest/packs-library live in those packages.
+
+---
+
+## 1. Overview: From Input to “Complete Story”
+
+- **Narrix** turns raw input (record / text / docs / chat) into **stories** and **signals** via: **ingest** (input → CNI) and **runner** (CNI + datasetId → entity, stories, signals, passes).
+- **“Complete story”** in this codebase means: a successful Narrix run (`ok: true`) with **stories** and **signals** produced and, optionally, no **gaps** (no missing mapping, schema, or unknown dataset).
+- To know **whether** you have enough info to run Narrix (or need to search the web first), you need:
+  1. **What ingest needs** — per-medium input shapes and where they come from.
+  2. **What packs-library provides** — which datasets exist, which schema/roleMap each needs, and whether routing/schema can be “patched” at runtime (smart mode).
+  3. **What “incomplete” looks like** — gap detection (errors, unknown dataset, missing schema, etc.) and how that can drive a decision to fetch more info.
+
+Sections below specify each piece so a new component can rely on them.
+
+---
+
+## 2. `@narrices/narrix-ingest` (narrix-ingest)
+
+### 2.1 Role
+
+- **Ingest** converts **raw input** into a single internal format (**CNI**) that the **runner** consumes.
+- ai-tasks does **not** implement ingest; it uses the default export from `@narrices/narrix-ingest` and calls `ingest.toCni({ kind, input })`.
+- Adapter selection (records vs text vs docs vs chat) is **ingest’s responsibility**; ai-tasks only passes `kind` and the corresponding `input` shape.
+
+### 2.2 Where It’s Used in ai-tasks
+
+| File | Usage |
+|------|--------|
+| `src/narrix/narrixClient.ts` | Imports `defaultIngest` and type `NarrixIngest`; exposes `getNarrixClient()` → `{ ingest, runner }`. |
+| `src/narrix/runNarrixForRecord.ts` | `ingest.toCni({ kind: "records", input: { record, recordType, roleMap } })`. |
+| `src/narrix/runNarrixForText.ts` | `ingest.toCni({ kind: "text", input: { text, sourceMeta, meta } })`. |
+| `src/narrix/runNarrixForDocs.ts` | `ingest.toCni({ kind: "docs", input: { pages, docId, title, sourceMeta } })`. |
+| `src/narrix/runNarrixForChat.ts` | `ingest.toCni({ kind: "chat", input: { messages, threadId, title, participants, sourceMeta } })`. |
+
+### 2.3 Required API (Consumer Contract)
+
+- **Export:** `defaultIngest` (or equivalent that yields a `NarrixIngest` instance).
+- **Type:** `NarrixIngest`.
+- **Method:** `ingest.toCni(options)` where `options` has:
+  - `kind`: `"records"` | `"text"` | `"docs"` | `"chat"`
+  - `input`: object shape below for that `kind`.
+
+**Return:**
+
+- **Success:** `{ cni: <opaque> }` (no `error: true`). ai-tasks passes `cni` to `runner.run({ cni, datasetId, sourceMeta })`.
+- **Error:** `{ error: true, code: string, message: string }`. ai-tasks surfaces `code` and `message` in the run output.
+
+### 2.4 Exact `toCni` Call Shapes (What Ingest Needs)
+
+These are the **minimum** inputs ingest must receive to produce CNI. If your component needs to “have the complete info” before calling Narrix, it must ensure these are available (or know that they will be filled by someone else).
+
+| `kind` | `input` shape | Notes |
+|--------|----------------|--------|
+| **records** | `{ record: Record<string, unknown>; recordType: string; roleMap: RoleMap }` | `recordType` = entityKind from pack. `roleMap` comes from packs-library (schema → roleMap). |
+| **text** | `{ text: string; sourceMeta: { kind: "text"; datasetId: string }; meta?: Record<string, unknown> }` | Free-form text + datasetId. |
+| **docs** | `{ pages: NarrixDocPage[]; docId?: string; title?: string; sourceMeta: { kind: "doc"; datasetId: string } }` | `pages[].text` (and optional pageId, pageNumber, index, mime, meta). |
+| **chat** | `{ messages: NarrixChatMessage[]; threadId?: string; title?: string; participants?: string[]; sourceMeta: { kind: "chat"; datasetId: string } }` | Messages with `role`, `text`, optional id/ts/authorId/meta. |
+
+For **records**, the only way to get `roleMap` and `recordType` in ai-tasks is via **packs-library** (see next section). So “do I have enough to run record ingest?” depends on: (1) having a `record` object, (2) having a `datasetId` that resolves to a pack/processor/schema in packs-library.
+
+### 2.5 Implications for “Complete Info” / Web Search
+
+- **Records:** You need the **record** payload and a **datasetId** that the packs-library knows (pack + processor + schema). If the dataset is unknown or schema is missing, ingest or the next step (resolveRoleMap / runner) will fail — that’s when gap handling or a “fetch more info” step (e.g. web search to discover or enrich data) can be relevant.
+- **Text/Docs/Chat:** You need the **content** (text, pages, messages) and a **datasetId**. Same idea: if the dataset isn’t in the registry, resolution fails; otherwise ingest only needs that content + datasetId in sourceMeta.
+
+So for planning a “what/if to search” component:
+
+- **Ingest itself** does not define “complete story”; it only requires the above shapes. “Completeness” is determined by: (a) packs-library having routing + schema for the dataset, and (b) runner succeeding and optionally (c) no gaps in the sense of System-2 (see Section 5).
+
+---
+
+## 3. `@narrices/narrix-packs-library` (narrix-packs-library)
+
+### 3.1 Role
+
+- **Packs-library** provides the **registry of Narrix packs** and helpers to resolve **datasetId → pack, processor, entityKind, record schema**, and **roleMap**.
+- It also provides **smart-mode APIs** (when available): Xronox-backed “remember” operations so routing and schema can be updated at runtime (System-2).
+
+### 3.2 Where It’s Used in ai-tasks
+
+| File | Usage |
+|------|--------|
+| `src/narrix/resolveRoleMap.ts` | `PACK_REGISTRY`, `findPackByDatasetId(datasetId, registry)`, `getRecordSchema(packId, entityKind, registry)`; `schemaToRoleMap(schema)` from `@narrices/narrix-adapter-records`. |
+| `src/narrix/system2/smartInit.ts` | Dynamic import: `createXronoxFromEnv`, `configurePacksLibrarySmartMode` (smart mode init). |
+| `src/narrix/system2/applyPatchPlan.ts` | Dynamic import: `rememberDatasetPackRouting`, `rememberDatasetProcessorRouting`, `rememberRecordSchemaOverride` (apply System-2 patch plan). |
+
+### 3.3 Static API (Compile-Time Exports)
+
+- **`PACK_REGISTRY`** — `Record<string, NarrixPack>`. All packs known to the library (default registry).
+- **`findPackByDatasetId(datasetId, registry?)`** — Returns the pack that has a processor whose `route.datasetIds` includes `datasetId`, or undefined.
+- **`getRecordSchema(packId, entityKind, registry?)`** — Returns the record schema for that pack/entityKind, or undefined.
+
+From these, ai-tasks derives:
+
+- **Pack id** and **processor** for the given `datasetId`.
+- **entityKind** (from processor’s `entityKind` or `id`).
+- **roleMap** = `schemaToRoleMap(schema)` (adapter-records); required for `toCni({ kind: "records", input: { record, recordType, roleMap } })`.
+
+If `findPackByDatasetId` returns nothing → “pack not found for datasetId” (unknown dataset). If `getRecordSchema` returns nothing → “record schema missing”. Both are **gaps** that can trigger System-2 or a “need more info” decision.
+
+### 3.4 Pack/Processor Shape (What ai-tasks Assumes)
+
+- **Pack:** has `id` and `processors` (array).
+- **Processor:** has `id`, `route?.datasetIds` (string[]), `entityKind` (or fallback to `id`). The processor that “handles” a datasetId is the one whose `route.datasetIds` includes that datasetId.
+- **Schema:** shape is opaque to ai-tasks; only that `getRecordSchema(packId, entityKind, registry)` returns something that `schemaToRoleMap` accepts.
+
+### 3.5 Smart-Mode APIs (Runtime / Optional)
+
+These may not be present in all versions; ai-tasks uses dynamic import and runtime checks.
+
+- **Init:** `createXronoxFromEnv()`, `configurePacksLibrarySmartMode({ xronox, enableWrites, role })`.
+- **Remember (used by System-2 applyPatchPlan):**
+  - `rememberDatasetPackRouting(scope, { datasetId, packId })`
+  - `rememberDatasetProcessorRouting(scope, { datasetId, packId, processorId, entityKind })`
+  - `rememberRecordSchemaOverride(scope, { packId, entityKind, schemaOverride, mergeStrategy })`
+
+**Scope** is `NarrixSmartScope`: `{ scopeType: "domain" | "agent", key: string }`. So “what/if to search” can be independent of smart mode; smart mode is about **persisting** fixes (new dataset routing, schema overrides) so the next run has “complete” routing/schema without searching again.
+
+### 3.6 Implications for “Complete Info” / Web Search
+
+- To know **whether** a given `datasetId` can be run:
+  - Call `findPackByDatasetId(datasetId, PACK_REGISTRY)` (or a custom registry). If null → **unknown dataset**; a planner might decide to search the web (or ask a user) to map this dataset to a pack or to create a new pack entry.
+- To know **what** structure a record must have for that dataset:
+  - After resolving the pack/processor, call `getRecordSchema(packId, entityKind, registry)`. If null → **missing schema**; a planner might search for schema docs or use LLM to infer a schema and then use System-2 remember (if available) or report that schema is missing.
+- **Stories/signals** semantics: produced by the **runner**, not packs-library. Packs-library only provides **routing and schema** so that ingest and runner can run. “Complete story” in the sense of “we have stories/signals” is an **output** of the pipeline; “complete info to run” is “we have datasetId + pack + schema + input payload”.
+
+---
+
+## 4. Related Components in ai-tasks (Relevant for “Complete Story” and Search)
+
+### 4.1 resolveRoleMap (`src/narrix/resolveRoleMap.ts`)
+
+- **Input:** `datasetId`, optional `registry` (default `PACK_REGISTRY`).
+- **Output:** `{ roleMap, packId, entityKind }` or throws if pack not found, processor not found, or schema missing.
+- **Use:** All four media runners call this before `ingest.toCni` (for records they pass `roleMap` and `entityKind` as `recordType`; for text/docs/chat they still resolve pack/entityKind for runner and meta).
+- **For planning:** A component that “checks if we can run” can call `resolveRoleMapForDataset(datasetId)` and catch errors to distinguish “unknown dataset” vs “missing schema” vs “processor not found”.
+
+### 4.2 narrixClient (`src/narrix/narrixClient.ts`)
+
+- **`getNarrixClient(): Promise<{ ingest, runner }>`** — Singleton. Ingest = `defaultIngest`; runner = `createRunner({ runtime: { onProcessorNotMatched: "attachError", onMissingMapping: "attachError", deterministicSort: true } })`.
+- Use this to get `ingest` and `runner` when you actually run the pipeline.
+
+### 4.3 Runner Contract (`@narrices/narrix-runner`)
+
+- **`runner.run({ cni, datasetId, sourceMeta })`** → Promise of `{ entity, signals, stories, passes, meta }`.
+- **entity:** `{ entityKind, entityId?, entityKey }`.
+- **stories:** array of objects (e.g. with `narrativeTypeId`); “story” in the product sense.
+- **signals:** array of objects (e.g. with `code`).
+- **meta:** includes `runId`, `producedAt`, `processorId`, `packVersion`, etc.
+- Success/failure: runner may throw or return; ai-tasks also interprets `output.ok: false` and meta.errors/diagnostics for gap detection.
+
+### 4.4 System-2: Gaps and Patch Plan (`src/narrix/system2/`)
+
+- **detectGaps(output)** — Pure. Considers `output.ok === false` and scans meta.errors, diagnostics, passes for keywords:
+  - **processorNotMatched**, **missingMapping**, **missingSchema**, **unknownDataset** (and generic “error”).
+- **GapHints:** `{ processorNotMatched?, missingMapping?, missingSchema?, unknownDataset?, details? }`.
+- **buildSystem2Input(input, result, gapHints)** — Builds a string for the LLM that produces a **PatchPlan** (actions: rememberDatasetPackRouting, rememberDatasetProcessorRouting, rememberRecordSchemaOverride; rerun: boolean).
+- **applyPatchPlan(plan, smartScope, enableWrites)** — Calls packs-library remember* APIs; if no scope or writes disabled, actions are skipped.
+- **runWithSystem2** — Runs System-1, then if gaps (or ALWAYS mode), calls LLM for patch plan, applies via packs-library, optionally reruns.
+
+So “incomplete” in code is explicitly: **hasGap** from `detectGaps` (error, processorNotMatched, missingMapping, missingSchema, unknownDataset). A “what/if to search” component could use the same hints: e.g. if **unknownDataset** → maybe search for which pack/dataset to use; if **missingSchema** → maybe search for schema or use LLM to propose one.
+
+### 4.5 Types (Unified Input / Output)
+
+- **NarrixRunInput** — Discriminated by `medium`: `record` | `text` | `docs` | `chat`, plus `datasetId` and the corresponding payload (record, text, document, thread).
+- **NarrixRunOutput** — Success: `ok: true`, `entity`, `signals`, `stories`, `passes`, `meta`. Failure: `ok: false`, `error`, `message`, `meta?`.
+- **Stories** are the main “narrative” output; they carry `narrativeTypeId` in scoping/filtering. So “complete story” can mean: we have at least one story, or we have stories for expected narrativeTypeIds (depending on product requirements).
+
+---
+
+## 5. When to Search the Web (or Gather More Info) — Spec for a New Component
+
+Using the above, here is a **low-level spec** for a component that decides **what and if to search** to get the complete story/info.
+
+### 5.1 Prerequisites You Can Check (Without Running Narrix)
+
+1. **Dataset known to packs-library**
+   - Call `findPackByDatasetId(datasetId, PACK_REGISTRY)` (or resolveRoleMap and catch).
+   - If **no pack** → **unknown dataset**. Options: (a) search the web (or docs) for “which pack/dataset handles X”, (b) ask user, (c) let System-2 try to remember a new routing (if smart mode and LLM suggest one).
+2. **Schema available for record runs**
+   - For `medium === "record"`, after resolving pack/processor, call `getRecordSchema(packId, entityKind, registry)`.
+   - If **no schema** → **missing schema**. Options: (a) search for schema/docs for that pack or entityKind, (b) use LLM to propose a schema and then rememberRecordSchemaOverride (if smart mode), (c) fail with a clear “schema missing” for this dataset.
+3. **Input payload present**
+   - Records: need a `record` object (and optionally recordId). Text: need `text`. Docs: need `document.pages`. Chat: need `thread.messages`.
+   - If payload is **empty or clearly placeholder** → might decide to “search” or “enrich” (e.g. fetch from API or web) before calling Narrix.
+
+### 5.2 After a Narrix Run (Gap-Driven)
+
+1. **Run Narrix** (ingest → runner) and get **NarrixRunOutput**.
+2. **Run `detectGaps(output)`** to get `hasGap`, `reasons`, `gapHints`.
+3. **If hasGap:**
+   - **unknownDataset** → Same as 5.1: decide whether to search for pack/dataset mapping or to call System-2 (if enabled) to remember new routing.
+   - **missingSchema** → Search for schema or use LLM + rememberRecordSchemaOverride.
+   - **processorNotMatched** / **missingMapping** → May need to search for mapping rules, or let System-2 propose rememberDatasetPackRouting / rememberDatasetProcessorRouting.
+4. **If !hasGap but “story” is empty:** Product decision: is “no stories” acceptable or do we need to “enrich” input (e.g. web search) and re-run? This is not currently a “gap” in code; a new component could define a policy (e.g. “if stories.length === 0 and policy says require-stories, then try to enrich and re-run”).
+
+### 5.3 What “Complete Story / Info” Means (Checklist for Spec)
+
+- **Enough to run ingest:** You have the correct `kind` and `input` shape for that medium (see Section 2.4), and for records you have `roleMap` and `recordType` (from packs-library).
+- **Enough to run runner:** You have `cni` from ingest and `datasetId` (and optional sourceMeta).
+- **Enough to consider “complete”:** Either (a) run succeeded and you accept the stories/signals you got, or (b) run failed with gaps and you have a policy: e.g. “on unknownDataset we search once for pack mapping”, “on missingSchema we search for schema or call System-2”, “on empty stories we optionally enrich and re-run”.
+
+A new component can implement:
+- **Pre-run:** “Resolve dataset + schema; if missing, search (web/docs/LLM) and optionally update registry or smart overlay.”
+- **Post-run:** “If detectGaps says hasGap, classify by gapHints and trigger the right search or System-2 path; optionally retry after applyPatchPlan.”
+
+---
+
+## 6. Quick Reference: Packages and Files
+
+| Concern | Package / File | Key exports / functions |
+|--------|----------------|---------------------------|
+| Ingest | `@narrices/narrix-ingest` | `defaultIngest`, `NarrixIngest`, `toCni({ kind, input })` |
+| Packs registry | `@narrices/narrix-packs-library` | `PACK_REGISTRY`, `findPackByDatasetId`, `getRecordSchema` |
+| RoleMap for records | `@narrices/narrix-adapter-records` (via resolveRoleMap) | `schemaToRoleMap(schema)` |
+| Resolve dataset → pack/schema | `src/narrix/resolveRoleMap.ts` | `resolveRoleMapForDataset(datasetId, registry?)` |
+| Client | `src/narrix/narrixClient.ts` | `getNarrixClient()` → `{ ingest, runner }` |
+| Runner | `@narrices/narrix-runner` | `createRunner(options)`, `runner.run({ cni, datasetId, sourceMeta })` |
+| Gaps | `src/narrix/system2/detectGaps.ts` | `detectGaps(output)` → `{ hasGap, reasons, gapHints }` |
+| System-2 input for LLM | `src/narrix/system2/buildSystem2Input.ts` | `buildSystem2Input(input, result, gapHints)` |
+| Patch plan | `src/narrix/system2/patchPlan.ts` | `validatePatchPlan(raw)`, types `PatchPlan`, `PatchPlanAction` |
+| Apply patch | `src/narrix/system2/applyPatchPlan.ts` | `applyPatchPlan(plan, smartScope, enableWrites)` |
+| Smart init | `src/narrix/system2/smartInit.ts` | `ensureSmartModeInitialized({ enableWrites, role })` |
+| Scope for smart writes | `src/narrix/system2/resolveSmartScope.ts` | `resolveSmartScope(system2Options, ctx)` → `NarrixSmartScope | null` |
+| Types | `src/narrix/types.ts` | `NarrixRunInput`, `NarrixRunOutput`, `NarrixSystem2Options`, etc. |
+
+---
+
+## 7. Summary for New Component Planning
+
+- **narrix-ingest** defines **what input shape** is needed per medium to produce CNI; it does not define “complete story” — that’s runner output and gap semantics.
+- **narrix-packs-library** defines **which datasets exist**, **how to get roleMap/schema** for records, and **how to persist new routing/schema** (smart mode) so future runs have “complete” config.
+- **Gap detection** (unknown dataset, missing schema, processor not matched, missing mapping) is the **signal** for “we don’t have the complete info yet”; a new component can use it to decide **what** to search (e.g. pack name, schema, mapping) and **whether** to search (e.g. only on unknownDataset, or also on empty stories by policy).
+- This doc is sufficient to specify a **low-level component** that: (1) before run: checks dataset/schema and optionally triggers web (or other) search to fill gaps; (2) after run: uses `detectGaps` and optionally System-2 or search to fix gaps and retry; (3) defines “complete story” as run success + optional story/signal presence policy.

package/.docs/narrix-record-input-current-design.md

@@ -0,0 +1,48 @@
+# NARRIX record input: how it works today
+
+This document describes **where ai-tasks gets the record** (or more generally the input) that it passes to NARRIX when running it. It is a snapshot of the current design so it can be compared with a desired design later.
+
+---
+
+## Two ways NARRIX is run
+
+1. **Task-level pre-processor** — When `request.narrix` is set (graph/node config), ai-tasks runs NARRIX before the task and injects the result into `executionMemory` (and `jobMemory`). The input to NARRIX is **always a single record** (medium `"record"`). The record is **resolved from the request** using a fixed priority list; there is no separate “pointer” or “record ref” from memory.
+
+2. **NARRIX_THEN_DIRECT** — When `executionType === "narrix-then-direct"`, the client must pass `request.narrixInput`. That can be a full `NarrixRunInput` (record, text, docs, or chat) **inline**, or `{ $path: "jobMemory.<path>" }` so the value is read from `jobMemory` at that path. So for this path, the “where” is explicit: either the request body or jobMemory at the given path.
+
+The design issue discussed here applies mainly to **path 1 (pre-processor)**: the record is not pointed to by memory; it is **scavenged** from several possible locations.
+
+---
+
+## Pre-processor: where the record comes from today
+
+The pre-processor **never** uses `request.narrixInput`. It builds a `NarrixRunInputRecord` itself:
+
+- `datasetId` comes from `request.narrix.datasetId`.
+- `record` and `sourceMeta` come from **resolvePreProcessorInput(request)**.
+
+That function ([`src/narrix/resolvePreProcessorInput.ts`](../src/narrix/resolvePreProcessorInput.ts)) looks for a **raw record** in this **strict priority order**:
+
+| Order | Source | Meaning |
+|-------|--------|--------|
+| 1 | `executionMemory.input.raw` | Current input’s `raw` field in execution memory. |
+| 2 | `executionMemory.inputs.<first key>.raw` | First key in a keyed map of inputs; that entry’s `raw`. |
+| 3 | `jobMemory.record` | Job-level “record” field. |
+| 4 | `jobMemory.currentRecord` | Job-level “current record” field. |
+| 5 | `request.input.record` | Task input’s `record` field. |
+| 6 | `request.input.raw` | Task input’s `raw` field. |
+
+**sourceMeta** is taken from (in order): `executionMemory.input.sourceMeta`, else `jobMemory.sourceMeta`, else `{}`.
+
+If none of these locations holds a valid record (object), the function returns `null` and the SDK throws: *"NARRIX pre-processor: raw record not found. Set executionMemory.input.raw, executionMemory.inputs.<key>.raw, jobMemory.record, jobMemory.currentRecord, or input.record / input.raw."*
+
+---
+
+## What’s wrong with this design (as identified)
+
+- **No single contract.** The runner/caller does not know which field “is” the record; ai-tasks just checks a fixed list. Different callers may set different places, and behavior depends on **order** (e.g. if both `jobMemory.currentRecord` and `request.input.record` are set, the former wins because it’s checked first).
+- **Memory does not “point” to the record.** There is no agreed convention like “the record for this task is always at jobMemory.currentRecord” or “executionMemory.input.raw is the canonical record.” The code only “finds” a record in one of several slots.
+- **Pre-processor vs NARRIX_THEN_DIRECT are inconsistent.** For NARRIX_THEN_DIRECT, the input is explicit (`narrixInput` or `$path`). For the pre-processor, the input is implicit (whatever is found by the resolution order above).
+- **Keying is arbitrary.** Using “first key” of `executionMemory.inputs` is undefined if key order matters; the contract is unclear.
+
+This document does not define how it *should* work; it only records how it *is* done today so that a better design can be specified and implemented later.

package/.docs/pacakge.md

@@ -0,0 +1,48 @@
+**don’t use CommonJS** 🙂
+
+### Use **ESM only**
+
+```json
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "module": "NodeNext",
+    "moduleResolution": "NodeNext"
+  }
+}
+```
+
+### Why this is the *right* choice **now**
+
+* **Native ESM everywhere** (Node 18/20+, Bun, Deno, modern bundlers)
+* Clean `import` / `export`, no interop hacks
+* Future-proof for:
+
+  * tree-shaking
+  * edge / workers
+  * package `exports`
+* Matches how *new* libraries are written in 2025–2026
+
+### What *not* to do
+
+* ❌ `"module": "commonjs"` → legacy, friction, dead-end
+* ❌ `"module": "ES2020"` without `NodeNext` → subtle resolution bugs with Node ESM
+
+### Minimal `package.json` you should use
+
+```json
+{
+  "type": "module",
+  "exports": {
+    ".": "./dist/index.js"
+  }
+}
+```
+
+### One important rule (don’t trip on this)
+
+* Use **explicit file extensions** in imports:
+
+  ```ts
+  import { x } from "./utils.js";
+  ```

package/.docs/possible-components/README.md

@@ -0,0 +1,11 @@
+# Possible components (specs)
+
+Architecture-only documentation for extracting **Pre-Task** and **Post-Task** components and **re-integrating** them into `ai-tasks`.
+
+| Folder | Use when you… |
+|--------|----------------|
+| [`pre-component/`](pre-component/README.md) | Build or maintain everything that runs **before MAIN**. **Implementation entry:** [`pre-component/builder-guide.md`](pre-component/builder-guide.md). |
+| [`post-component/`](post-component/README.md) | Build or maintain everything that runs **after MAIN**. **Implementation entry:** [`post-component/builder-guide.md`](post-component/builder-guide.md). |
+| [`integration/`](integration/README.md) | Shared platform (**single source**), phased checklist, merge-back into `ai-tasks`. |
+
+**Rule:** Shared rules (aifunctions-js, Activix, identity, Xynthesis layering) live **only** in [`integration/platform.md`](integration/platform.md). Pre/post **builder guides** summarize obligations and link there for full detail—no second full copy of platform prose.

package/.docs/possible-components/integration/README.md

@@ -0,0 +1,10 @@
+# Integration — shared platform & merge-back into `ai-tasks`
+
+Use this folder for **shared** concerns. For **building** each component, start with [`../pre-component/builder-guide.md`](../pre-component/builder-guide.md) and [`../post-component/builder-guide.md`](../post-component/builder-guide.md) (complete deps, env, files, acceptance).
+
+| Doc | Purpose |
+|-----|---------|
+| [`platform.md`](platform.md) | **Only** place for aifunctions-js, Activix, `identity`, Xynthesis layering (no duplicate in pre/post). |
+| [`roadmap-and-checklists.md`](roadmap-and-checklists.md) | Phased A–D checklist + pre layering diagram. |
+| [`reintegrate-into-ai-tasks.md`](reintegrate-into-ai-tasks.md) | Where to wire extracted packages back into this repo. |
+| [`gaps-when-merging.md`](gaps-when-merging.md) | Versioning, barrels, DI, docs to update when merging. |

package/.docs/possible-components/integration/gaps-when-merging.md

@@ -0,0 +1,16 @@
+# Gaps when merging packages back into `ai-tasks`
+
+Items that often bite during **integration**, distinct from per-component template/code lists.
+
+| Gap | Mitigation |
+|-----|------------|
+| **Version skew** | Pin `@athenices/xynthesis`, `aifunctions-js`, `@woroces/ai-skills`, `@narrices/*` to versions tested together. |
+| **Barrel exports** | Update `src/index.ts` / `package.json` `exports` if pre/post move; avoid duplicate symbols. |
+| **Singleton globals** | `setScopingGateway`, `setSynthesisInvoker`, web scoper singleton — document thread-safety or replace with instance DI. |
+| **Cwd-relative templates** | Post templates use `process.cwd()`; in monorepo tests, set `AUDIT_TEMPLATES_PATH` / `POLISH_TEMPLATES_PATH` to repo root. |
+| **dist vs src** | If consumers import `dist-test`, align build pipeline so specs match published paths. |
+| **Documentation drift** | Update this tree if `task-sdk` flow changes; keep **one** platform doc ([`platform.md`](platform.md)). |
+
+## Optional shared library
+
+Consider a tiny **`@woroces/ai-task-platform`** (or similar) owning: Activix+identity helper, `runLlmTextCall` wrapper, and SynthesisInvoker factory — imported by both pre and post packages and by `ai-tasks`.