@exellix/ai-tasks 8.4.1 → 8.4.3

package/.docs/memory-narrix-adapter-developer-guide.md

@@ -0,0 +1,402 @@
+# Memory Narrix Adapter — developer guide
+
+This document is for the developer who will **implement** the Memory Narrix Adapter.
+It assumes no prior knowledge of Narrix, ai-tasks memory, or the current implementation.
+You should be able to build the component using **only**:
+
+- `memory-narrix-adapter-requirements.md` (what the adapter must do), and
+- this guide (background, shapes, and concrete examples).
+
+You do **not** need to read the rest of the codebase to get started.
+
+---
+
+## High-level picture
+
+- Ai-tasks is a task runner that keeps **memory** for tasks and jobs (JSON-like objects).
+- Narrix is an **enrichment engine** that takes some input (a record, text, docs, or chat) and returns enriched data (signals, stories, etc.).
+- The **Memory Narrix Adapter** sits in between:
+  - It receives **memory** (an arbitrary object) plus a few required parameters.
+  - It **interprets** that memory and decides how to turn it into a Narrix input.
+  - It either returns a **valid Narrix input** or a **structured “bad input” result** explaining why it cannot.
+
+You are **not** responsible for calling Narrix, only for constructing the input object Narrix expects.
+
+---
+
+## What Narrix expects (input shapes)
+
+Narrix supports four input “media”. The adapter’s job is to output **exactly one** of these shapes or say “bad input”.
+
+```ts
+// Unified input (NarrixRunInput)
+
+type NarrixRunInputRecord = {
+  medium: "record";
+  datasetId: string;
+  record: Record<string, unknown>;
+  // Optional; passed through to Narrix runner
+  sourceMeta?: Record<string, unknown>;
+};
+
+type NarrixRunInputText = {
+  medium: "text";
+  datasetId: string;
+  text: string;
+  meta?: Record<string, unknown>;
+};
+
+type NarrixRunInputDocs = {
+  medium: "docs";
+  datasetId: string;
+  document: {
+    pages: { text: string; pageId?: string; pageNumber?: number; index?: number; mime?: string; meta?: Record<string, unknown> }[];
+    docId?: string;
+    title?: string;
+  };
+};
+
+type NarrixRunInputChat = {
+  medium: "chat";
+  datasetId: string;
+  thread: {
+    messages: { role: "user" | "assistant" | "system" | "tool" | "other"; text: string; id?: string; ts?: number; authorId?: string; meta?: Record<string, unknown> }[];
+    threadId?: string;
+    title?: string;
+    participants?: string[];
+  };
+};
+
+type NarrixRunInput =
+  | NarrixRunInputRecord
+  | NarrixRunInputText
+  | NarrixRunInputDocs
+  | NarrixRunInputChat;
+```
+
+You do not need to know how Narrix uses these; just construct one of these shapes.
+
+---
+
+## What “memory” looks like (conceptually)
+
+Ai-tasks has three main memory levels. In practice these are just JSON-like objects; you do **not** need to know all fields:
+
+- **jobMemory** — Long-lived data for the job (e.g. “current record”, job-wide metadata).
+- **taskMemory** — Data produced by this task or previous tasks (e.g. prior Narrix runs).
+- **executionMemory** — Execution-specific data (inputs, raw payloads, sourceMeta, internal state).
+
+When ai-tasks calls the adapter, it can pass:
+
+- Any one of these objects, or
+- A **merged** object that contains pieces from multiple levels, or
+- A completely different object that still has useful data.
+
+You should treat the memory input as:
+
+```ts
+type MemoryLike = unknown; // could be any JSON-like object
+```
+
+Your job is to **inspect** this object and decide:
+
+- What kind of thing it looks like (record, text, docs, chat, or “unknown”), and
+- Which **subset** (if any) should be used as Narrix input.
+
+You may also receive:
+
+- A `datasetId` (always required).
+- An optional **medium hint** (e.g. `"record"` or `"text"`) that you should respect when reasonable.
+
+---
+
+## Suggested adapter API (for implementation)
+
+The requirements doc deliberately does not fix the exact API.
+This is a suggested shape you can use when implementing the component.
+
+```ts
+type MemoryNarrixAdapterInput = {
+  datasetId: string;
+  // Optional hint from caller; when present, prefer this medium
+  mediumHint?: "record" | "text" | "docs" | "chat";
+  // Arbitrary memory object (job + task + execution, or any JSON-like object)
+  memory: unknown;
+};
+
+type InterpretationKind = "record" | "text" | "docs" | "chat" | "mixed" | "unknown";
+
+type InterpretationStrategy = "selection" | "transformation" | "serialization" | "none";
+
+type MemoryNarrixAdapterSuccess = {
+  ok: true;
+  narrixInput: NarrixRunInput;
+  interpretation: {
+    kind: InterpretationKind;
+    // e.g. "jobMemory.currentRecord", "executionMemory.input.raw", "entire memory serialized as text"
+    sourceDescription: string;
+    // "guessed" vs "hinted" vs "exact"
+    confidence: "guessed" | "hinted" | "exact";
+    strategy: InterpretationStrategy;
+  };
+};
+
+type MemoryNarrixAdapterFailure = {
+  ok: false;
+  error: "NO_USABLE_INPUT" | "UNSUPPORTED_SHAPE" | "INCONSISTENT_DATA";
+  message: string;
+  attempted: string[]; // short descriptions of strategies you tried
+  foundSummary: string; // short description of what was in memory
+  interpretation?: {
+    kind: InterpretationKind;
+    strategy: InterpretationStrategy;
+  };
+};
+
+type MemoryNarrixAdapterResult = MemoryNarrixAdapterSuccess | MemoryNarrixAdapterFailure;
+```
+
+You can adjust names, but the **ideas** must all be present:
+
+- `NarrixRunInput` on success.
+- Structured, non-throwing “bad input” on failure.
+- Interpretation metadata (what it looked at, what it thought it was, and how it decided).
+
+---
+
+## What the adapter must actually do (step-by-step)
+
+When called, the adapter should:
+
+1. **Inspect the memory object.**  
+   - If it is not an object at all (e.g. `null`, a number, a string), you will probably end up returning a “bad input” result unless you decide to treat it as text.
+   - If it is an object, look at its fields and nested structures.
+
+2. **Optionally respect the medium hint.**  
+   - If `mediumHint` is `"record"`, first try to find a record-shaped object.
+   - If `mediumHint` is `"text"`, first try to extract or serialize text.
+   - If no hint is provided, you decide the order of strategies (see examples below).
+
+3. **Try strategies, in a deterministic order.** For example:
+   - **Record strategy:** Look for a single JSON object that can be treated as a record.
+   - **Chat strategy:** Look for something that looks like a list of chat messages.
+   - **Docs strategy:** Look for something that looks like a list of pages with text.
+   - **Text strategy:** If nothing else matches, consider serializing the whole memory to text.
+
+4. **If a strategy succeeds, produce `NarrixRunInput`.**  
+   - Make sure the resulting object matches one of the Narrix run input shapes exactly.
+   - Include `datasetId` (from the adapter input).
+   - For `record` medium, also set `sourceMeta` if you can infer it from memory.
+
+5. **Always return interpretation metadata.**  
+   - Even on success, indicate where you got the data and how confident you were.
+   - On failure, list what you tried and what you found.
+
+6. **Never throw for “bad input”.**  
+   - If the memory is unusable, return a `MemoryNarrixAdapterFailure`.
+   - The caller (ai-tasks) is responsible for turning that into an error response.
+
+---
+
+## How this replaces current behavior
+
+Today, ai-tasks has multiple scattered pieces of logic:
+
+- `resolvePreProcessorInput` — scans specific fields to find a `record` and `sourceMeta`.
+- `normalizeInput` — validates and normalizes any Narrix input into the four media.
+- `resolveNarrixInput` — resolves `{ $path: "jobMemory.currentRecord" }` into a concrete object.
+
+The Memory Narrix Adapter **replaces all of that** with a single component:
+
+- Ai-tasks will give you **datasetId**, optional **medium hint**, and a **memory object**.
+- You return either a valid `NarrixRunInput` plus interpretation, or a structured “bad input”.
+- Ai-tasks no longer needs to know where the record lives or how to normalize inputs.
+
+You do **not** need to exactly copy today’s priority order; you only need to meet the requirements in `memory-narrix-adapter-requirements.md`.
+
+---
+
+## Example scenarios
+
+These examples are illustrative, not exhaustive. They show how to think about memory and what the adapter might do.
+
+### 1. Simple record in memory
+
+**Input:**
+
+```ts
+const input: MemoryNarrixAdapterInput = {
+  datasetId: "vuln-dataset",
+  mediumHint: "record",
+  memory: {
+    jobMemory: {
+      currentRecord: { id: "v-123", severity: "high", title: "Example vuln" },
+      sourceMeta: { source: "scanner-x" },
+    },
+  },
+};
+```
+
+**Expected behavior:**
+
+- Detect that `jobMemory.currentRecord` is a record-shaped object.
+- Treat this as `medium: "record"`.
+- Use `jobMemory.sourceMeta` as `sourceMeta`.
+
+**Output (success):**
+
+```ts
+{
+  ok: true,
+  narrixInput: {
+    medium: "record",
+    datasetId: "vuln-dataset",
+    record: { id: "v-123", severity: "high", title: "Example vuln" },
+    sourceMeta: { source: "scanner-x" },
+  },
+  interpretation: {
+    kind: "record",
+    sourceDescription: "jobMemory.currentRecord",
+    confidence: "hinted",
+    strategy: "selection",
+  },
+}
+```
+
+### 2. Chat-like memory without a hint
+
+**Input:**
+
+```ts
+const input: MemoryNarrixAdapterInput = {
+  datasetId: "support-chat",
+  memory: {
+    thread: {
+      messages: [
+        { role: "user", text: "hi" },
+        { role: "assistant", text: "hello" },
+      ],
+    },
+  },
+};
+```
+
+**Expected behavior:**
+
+- Detect that `thread.messages` is an array of `{ role, text }`.
+- Classify this as `kind: "chat"`.
+- Build a `NarrixRunInputChat`.
+
+**Output (success, shape only):**
+
+```ts
+{
+  ok: true,
+  narrixInput: {
+    medium: "chat",
+    datasetId: "support-chat",
+    thread: {
+      messages: [
+        { role: "user", text: "hi" },
+        { role: "assistant", text: "hello" },
+      ],
+    },
+  },
+  interpretation: {
+    kind: "chat",
+    sourceDescription: "thread.messages",
+    confidence: "guessed",
+    strategy: "selection",
+  },
+}
+```
+
+### 3. Whole memory serialized as text
+
+**Input:**
+
+```ts
+const input: MemoryNarrixAdapterInput = {
+  datasetId: "debug-dataset",
+  memory: {
+    jobMemory: { /* many fields */ },
+    taskMemory: { /* many fields */ },
+    executionMemory: { /* many fields */ },
+  },
+};
+```
+
+**Expected behavior:**
+
+- None of the specific strategies (record, chat, docs) find a good match.
+- As a last resort, decide to **serialize** the entire memory object into a text blob (e.g. JSON string).
+- Return a `medium: "text"` input.
+
+**Output (success, shape only):**
+
+```ts
+{
+  ok: true,
+  narrixInput: {
+    medium: "text",
+    datasetId: "debug-dataset",
+    text: "<serialized memory here>",
+  },
+  interpretation: {
+    kind: "text",
+    sourceDescription: "entire memory serialized",
+    confidence: "guessed",
+    strategy: "serialization",
+  },
+}
+```
+
+### 4. Bad input
+
+**Input:**
+
+```ts
+const input: MemoryNarrixAdapterInput = {
+  datasetId: "any",
+  memory: 42,
+};
+```
+
+**Expected behavior:**
+
+- No usable structure (it’s just a number).
+- Return a failure result.
+
+**Output (failure, shape only):**
+
+```ts
+{
+  ok: false,
+  error: "NO_USABLE_INPUT",
+  message: "Memory is not an object; cannot derive Narrix input.",
+  attempted: ["checked for record", "checked for chat", "checked for docs", "considered text serialization"],
+  foundSummary: "memory is a number (42)",
+  interpretation: {
+    kind: "unknown",
+    strategy: "none",
+  },
+}
+```
+
+---
+
+## How to work with these docs
+
+You should read:
+
+- `memory-narrix-adapter-requirements.md` — what the component is **responsible for** and how ai-tasks will use it.
+- This developer guide — background on Narrix input shapes, memory, suggested API, and example behaviors.
+
+You do **not** need to match today’s implementation details (priority order, exact property names) unless you choose to.
+Your implementation should focus on:
+
+- Producing valid Narrix inputs,
+- Providing clear interpretation metadata,
+- Returning structured failures instead of throwing, and
+- Being easy to extend as new memory shapes or strategies appear.
+

package/.docs/memory-narrix-adapter-requirements.md

@@ -0,0 +1,112 @@
+# Memory Narrix Adapter — requirements definition
+
+This document defines requirements for a **component** (not a one-off function) whose job is to take memory and other required inputs and produce what NARRIX needs: either a valid NARRIX input, a decision about which subset of the input to use, or an explicit "bad input" result. The component is intended to be extended over time; ai-tasks will call it and remove the current ad-hoc resolution logic once the adapter exists.
+
+---
+
+## Name and responsibility
+
+- **Name:** Memory Narrix Adapter (or equivalent; the name should reflect "memory to NARRIX input").
+- **Responsibility:** Bring NARRIX the information it needs from memory. Given required inputs and **any** object (e.g. job memory, task memory, execution memory, the whole memory, or a mix), the adapter either:
+  1. **Decides** which subset of the input to use as NARRIX input (intelligent selection), or  
+  2. **Transforms** what it gets into something that conforms to the NARRIX protocol and works as input, or  
+  3. **Reports** that the input is bad / unusable.
+
+The adapter is an intelligent adapter: it understands the NARRIX protocol, understands the shape of memory, and produces a clear outcome (usable input or "bad input").
+
+---
+
+## Knowing or guessing what we're looking at
+
+The adapter must be able to **know or at least guess what it is looking at** based on what it finds in the memory. That is:
+
+- **Need:** Callers (and NARRIX) benefit from understanding the *nature* of the data: e.g. "this is a single record," "this looks like a chat thread," "this is job-level context," "this is a document with pages," "this is a large memory object that could be serialized as text." The adapter inspects the provided object and, from structure and content, infers or classifies what kind of input it is. That interpretation drives which NARRIX medium and shape to use and whether the input is usable at all.
+- **Input:** The same memory/arbitrary object the adapter receives. No extra "type" hint is required from the caller (though the caller **may** provide a medium hint — see below); the adapter figures it out from what's there.
+- **Output:** The adapter's result exposes this interpretation: which part of memory was used, what it classified the content as, and whether that was inferred ("guess") or known. This supports debugging, observability, and correct handling without the caller having to know memory layout.
+
+The Memory Narrix Adapter doesn't only select or transform — it **interprets** what it found. The new Narrix component will implement how that interpretation works (heuristics, schema hints, conventions); this document states the need and that inputs/outputs must support it.
+
+---
+
+## Inputs to the component
+
+- **`datasetId`** — Required. Determines which Narrix pack/entity to use. Always provided by the caller (e.g. from `request.narrix.datasetId` or equivalent config).
+- **Medium hint (optional)** — The caller may suggest a medium (e.g. `"record"`, `"text"`). When provided, the adapter prefers that medium; when not provided, it auto-detects from what it finds. This supports both "I know what this is" and "figure it out" use cases.
+- **Memory / arbitrary object** — The adapter receives "memory" (or any object): e.g. full or partial `jobMemory`, `taskMemory`, `executionMemory`, the **entire memory** as one object, or a merged view. It can be anything.
+
+The key requirement: the adapter can take **any** such object and attempt to derive NARRIX input from it.
+
+### Whole memory as input
+
+A critical scenario: the adapter receives the **entire memory** (job + task + execution, or all of them merged). It does not just look for a specific field; it can decide to use the whole thing (e.g. serialize it for text medium), or pick the most relevant part. The adapter understands that memory has levels (job, task, execution) and can reason about which level or part is most useful for NARRIX.
+
+---
+
+## NARRIX protocol (what the adapter targets)
+
+The adapter's output must align with the NARRIX run input contract. Valid inputs are one of four media:
+
+- **record** — `{ medium: "record", datasetId, record, sourceMeta? }`
+- **text** — `{ medium: "text", datasetId, text, meta? }`
+- **docs** — `{ medium: "docs", datasetId, document: { pages, docId?, title? } }`
+- **chat** — `{ medium: "chat", datasetId, thread: { messages, threadId?, title?, participants? } }`
+
+The adapter's job is to produce something that fits one of these shapes (or to say it cannot).
+
+### sourceMeta
+
+When producing a record-medium input, the adapter must also resolve **`sourceMeta`** (metadata about the record's origin). Today this is found separately from the record itself (e.g. `executionMemory.input.sourceMeta` or `jobMemory.sourceMeta`). The adapter takes over this responsibility: it knows where to look for sourceMeta and includes it in the output when relevant.
+
+### Legacy formats
+
+The adapter must handle **legacy shapes** that lack a `medium` field, e.g. `{ datasetId, recordType, record }`. Today these are normalized in `task.ts` `normalizeInput()`. The adapter subsumes that normalization: if it receives something that looks like a legacy input, it interprets and converts it.
+
+---
+
+## Outputs of the component
+
+The component produces one of:
+
+1. **Usable NARRIX input**  
+   A concrete `NarrixRunInput` (one of the four media above), either by:
+   - **Selection:** Choosing a subset of the provided object (e.g. "use the record found at this path") and returning it in the right shape, or  
+   - **Transformation:** Converting the given object (or a part of it) into a valid `NarrixRunInput` (e.g. serializing memory into text, extracting fields to form a record, restructuring chat messages).
+
+2. **Bad input**  
+   A structured result (not a thrown error) indicating the provided memory/object cannot be used to form valid NARRIX input. Must include at minimum:
+   - **What was attempted** — what the adapter tried to do (e.g. "looked for a record, tried text serialization").
+   - **Why it failed** — reason (e.g. "no record-shaped data found, memory is empty, incompatible shape").
+   - **What was found** — what was actually in the memory (summary, not the full object), so the caller can produce a useful error message.
+
+3. **Interpretation metadata** (always present, on both success and failure)  
+   - Which part of memory was used (or inspected).
+   - What the content was classified as (record, chat, document, text, mixed, unknown).
+   - Whether the classification was inferred ("guess") or known (e.g. medium hint was provided).
+   - Which strategy was applied (selection, transformation, serialization).
+
+---
+
+## Behavior (requirements)
+
+- **Extensible** — New strategies, memory shapes, or media can be added without ai-tasks having to know the details. New behavior goes in one place.
+- **Deterministic where possible** — Same inputs produce the same decision/transformation unless the design explicitly allows non-determinism.
+- **No hidden global state** — Decisions are based only on the provided inputs and the provided memory object.
+- **Clear contract** — Inputs and outputs are well-defined so ai-tasks (or any other caller) can integrate without depending on implementation details.
+- **Subsumes existing normalization** — The adapter replaces both `resolvePreProcessorInput` (pre-processor record resolution) and `normalizeInput` (media validation/normalization in task.ts). One component, one place.
+
+---
+
+## Integration (after the component exists)
+
+- Ai-tasks will **use** the Memory Narrix Adapter when it needs to run NARRIX. Both execution paths (pre-processor and NARRIX_THEN_DIRECT) can use it. Ai-tasks passes `datasetId`, optional medium hint, and the relevant memory.
+- Ai-tasks will **remove** the old resolution logic:
+  - `resolvePreProcessorInput` (the 6-step priority-list record scavenger).
+  - `resolveNarrixInput` in task-sdk (the `$path` resolver for NARRIX_THEN_DIRECT).
+  - `normalizeInput` in task.ts (the media validation/normalization).
+- The exact API (method name, parameters, return type) will be defined when the component is implemented; this document specifies the responsibility and the input/output semantics.
+
+---
+
+## Relation to current design
+
+The current behavior (where the record comes from, and its problems) is described in [narrix-record-input-current-design.md](./narrix-record-input-current-design.md). The Memory Narrix Adapter is intended to replace that with a single, explicit component that can interpret memory, select or transform it into NARRIX input (or report bad input), and be extended over time.