@exellix/ai-tasks 8.1.16 → 8.2.0

package/documenations/upstream-feature-requests/logxer-failure-classification-and-causal-diagnostics.md

@@ -0,0 +1,403 @@
+# `@x12i/logxer` — failure classification, causal diagnostics, and provider-route observability
+
+Status: open  
+Owner: `@x12i/logxer`  
+Filed by: `@exellix/ai-tasks` (8.x+)  
+Context: graph-engine `TASK_RUN_FAILED` investigation — wrapper errors hid root causes (provider auth, prompt render, content registry). ai-tasks added client-side workarounds; logxer should own the contract.
+
+---
+
+## Summary
+
+When orchestration layers wrap failures (`TASK_RUN_FAILED: <skillKey>`, empty `parsed`, schema validation), operators need logs that answer:
+
+1. Is **this line** the root **cause** or a downstream **symptom**?
+2. If symptom — **what event / log / record** should I inspect next?a
+3. For LLM failures — **which adapter/route** was intended (`openai-direct` vs `openrouter`, model id, likely env keys)?
+
+Today logxer supports `errorCode` / `warnCode`, `diagnostics.summary`, and `evidence[]` via `fieldEvidence`. That works, but every consumer re-implements classification, causal linking, and provider-route shaping — and typed `LogDiagnostics` is too narrow for structured failure payloads.
+
+---
+
+## Lessons learned (from `@exellix/ai-tasks` + graph-engine)
+
+| Observation | Impact |
+|-------------|--------|
+| Graph-engine throws `TASK_RUN_FAILED: <skillKey>` while the real message lives in `res.error.message`, Activix, or ai-tasks logs | Console shows **symptom only** |
+| `runTaskDiagnostics.level: basic` hides gateway/provider detail | Operators cannot see OpenAI vs OpenRouter without trace mode |
+| Fast failures (~300ms) on MAIN often mean pre-LLM causes (auth, content miss, empty prompt) | Duration + phase are useful classification signals |
+| `LogDiagnostics` currently accepts `{ summary }` only | Packages push structured data into `evidence[]` strings — hard to query |
+| Same failure produces logs in ai-tasks, ai-skills, gateway, graph-engine, Activix | No standard **causal link** between symptom and cause entries |
+| Each package would duplicate `FailureRole` / `FailureEvent` types | Needs a shared logxer contract |
+
+ai-tasks interim workaround (until logxer ships): `classifyRunTaskFailure`, `RunTaskExecutionError`, `failureClassification` on errors, `fieldEvidence("failureRole", …)` on `RUN_TASK_EXECUTION_FAILED`.
+
+---
+
+## FR-1 — First-class `failureClassification` on coded diagnostics
+
+### Problem
+
+Consumers encode cause/symptom as ad-hoc evidence strings (`failureRole`, `failureEvent`, `failurePointsTo`). UIs and Mongo queries cannot rely on a stable schema.
+
+### Proposed API
+
+Extend coded diagnostic meta (`errorCode` / `warnCode` payload) with optional:
+
+```ts
+type FailureRole = "cause" | "symptom" | "unknown";
+
+type FailureClassification = {
+  role: FailureRole;
+  /** Stable machine id, e.g. "provider_auth", "graph_engine_task_run_wrapper" */
+  event: string;
+  /** When role === "symptom": event id(s) or operator hint for next lookup */
+  pointsTo?: string | string[];
+  note: string;
+  confidence: "high" | "medium" | "low";
+};
+
+type CodedDiagnosticMeta = {
+  source?: string;
+  debugKind?: DebugLogAbstract;
+  diagnostics?: LogDiagnostics; // see FR-2
+  evidence?: EvidenceEntry[];
+  failureClassification?: FailureClassification;
+};
+```
+
+Optional helper:
+
+```ts
+logxer.errorCode("RUN_TASK_EXECUTION_FAILED", {
+  diagnostics: { summary: "…" },
+  failureClassification: {
+    role: "cause",
+    event: "prompt_template_empty",
+    note: "Prompt template resolved to empty text",
+    confidence: "high",
+  },
+});
+```
+
+### Acceptance
+
+- Serialized log records include `failureClassification` at a stable JSON path.
+- `getJobLogs({ filter: { failureRole: "symptom" } })` works (see FR-4).
+- Packages may omit it; logxer does not require catalog entries to define classification.
+
+---
+
+## FR-2 — Expand `LogDiagnostics` for structured failure detail (not summary-only)
+
+### Problem
+
+TypeScript consumers hit `Object literal may only specify known properties` when adding `phase`, `llmRoute`, `rootError`, etc. under `diagnostics`. Workaround: flatten everything into `evidence[]` — loses type safety and queryability.
+
+### Proposed shape
+
+```ts
+type LogDiagnostics = {
+  /** Human one-liner (required for coded diagnostics) */
+  summary: string;
+  /** Optional structured detail bag — validated as JSON-serializable */
+  detail?: Record<string, unknown>;
+};
+```
+
+Recommended **documented** keys in `detail` (not enforced by logxer, but listed in docs):
+
+| Key | Purpose |
+|-----|---------|
+| `phase` | Orchestration phase (`main_skill`, `pipeline_pre`, …) |
+| `stage` | Finer step id (`main-skill`, `pre-synthesis-markdown`, …) |
+| `rootError` | `{ name, message, code?, stack? }` — underlying error |
+| `llmRoute` | See FR-3 |
+| `durationMs` | Call latency |
+| `skillKey`, `graphId`, `nodeId`, `jobId`, `taskId` | Correlation |
+
+### Acceptance
+
+- `diagnostics.detail` round-trips through runtime capture and `getJobLogs`.
+- Catalog linter allows optional `detailSchema` per diagnostic code (JSON Schema ref).
+
+---
+
+## FR-3 — Standard `llmRoute` facet for provider / adapter observability
+
+### Problem
+
+Operators could not tell whether a run intended **OpenAI direct** (`openai/gpt-4.1` → `OPENAI_API_KEY`) vs **OpenRouter** (`openrouter/...` → `OPEN_ROUTER_KEY`). Gateway routing echo (`routing.provider`) arrives only in trace mode.
+
+### Proposed documented type (in logxer docs + optional TS export)
+
+```ts
+type LlmRouteDiagnostics = {
+  rawModel?: string;
+  providerPrefix?: string;
+  modelId?: string;
+  adapterHint?: "openrouter" | "openai-direct" | "anthropic-direct" | "google-direct" | "groq-direct" | "unknown" | "unset";
+  likelyEnvKeys?: string[];
+  routingNote?: string;
+  /** When gateway returned routing diagnostics */
+  actualRouting?: Record<string, unknown>;
+};
+```
+
+Optional helper:
+
+```ts
+import { inferLlmRouteFromModel } from "@x12i/logxer/diagnostics"; // or subpath
+
+const llmRoute = inferLlmRouteFromModel("openai/gpt-4.1");
+logxer.errorCode("PROVIDER_INVOKE_FAILED", {
+  diagnostics: { summary: "…", detail: { llmRoute } },
+});
+```
+
+logxer **does not** call providers — it only standardizes the shape and helper for prefix → adapter → env-key hints.
+
+### Acceptance
+
+- Documented in logxer README.
+- `getJobLogs({ filter: { "detail.llmRoute.adapterHint": "openai-direct" } })` (see FR-4).
+
+---
+
+## FR-4 — Query / filter coded diagnostics by classification and route
+
+### Problem
+
+`getAiTasksJobLogs` / in-process job log viewers cannot answer: “show me **symptoms** for this graph run” or “show failures where `adapterHint=openai-direct`”.
+
+### Proposed API
+
+Extend `GetJobLogsInput.filter`:
+
+```ts
+type GetJobLogsFilter = {
+  jobId?: string;
+  taskId?: string;
+  correlationId?: string;
+  code?: string;
+  level?: LogLevel;
+  failureRole?: FailureRole;
+  failureEvent?: string;
+  /** Dot-path equality on diagnostics.detail, e.g. detail.llmRoute.adapterHint */
+  detail?: Record<string, string | number | boolean>;
+};
+```
+
+### Acceptance
+
+- Filters compose (AND).
+- Document performance note: in-process ring buffer only unless host persists logs.
+
+---
+
+## FR-5 — Causal linking between related diagnostic entries
+
+### Problem
+
+One user-visible failure produces multiple log lines (graph-engine symptom, ai-tasks cause, gateway error). No standard link.
+
+### Proposed fields on coded meta
+
+```ts
+type CausalLink = {
+  /** Log entry id of the diagnostic this entry explains or wraps */
+  causedByLogId?: string;
+  /** Log entry ids this entry supersedes or clarifies */
+  relatedLogIds?: string[];
+  /** Stable upstream error code/name if known */
+  rootCode?: string;
+  rootMessage?: string;
+};
+
+type CodedDiagnosticMeta = {
+  // …existing
+  causal?: CausalLink;
+};
+```
+
+Optional: when `failureClassification.role === "symptom"`, logxer **warns in dev** if `causal.rootMessage` is missing.
+
+### Acceptance
+
+- Job log viewer can render symptom → cause chain.
+- Activix / graph-engine hosts can pass `causal.rootMessage` from wrapped errors without re-logging full stacks.
+
+---
+
+## FR-6 — Catalog entries declare default classification hints
+
+### Problem
+
+`.metadata/log-diagnostics.json` describes codes but not whether a code is typically cause or symptom.
+
+### Proposed catalog extension (optional per entry)
+
+```json
+{
+  "RUN_TASK_EXECUTION_FAILED": {
+    "level": "error",
+    "summaryTemplate": "runTask failed",
+    "defaultClassification": {
+      "role": "cause",
+      "event": "provider_invoke"
+    },
+    "symptomExamples": ["graph_engine_task_run_wrapper"]
+  }
+}
+```
+
+logxer does **not** auto-classify from catalog — catalog is documentation + lint hints for package authors.
+
+### Acceptance
+
+- Catalog linter validates `defaultClassification.role` enum.
+- `logxer validate-catalog` reports entries missing classification docs for `error`-level codes.
+
+---
+
+## FR-7 — `fieldEvidence` helpers for causal and route facets
+
+### Problem
+
+Repeated boilerplate:
+
+```ts
+fieldEvidence("failureRole", "symptom"),
+fieldEvidence("adapterHint", "openai-direct"),
+```
+
+### Proposed helpers
+
+```ts
+fieldEvidenceFailureClassification(c: FailureClassification): EvidenceEntry[];
+fieldEvidenceLlmRoute(route: LlmRouteDiagnostics): EvidenceEntry[];
+fieldEvidenceRootError(err: { name: string; message: string; code?: string }): EvidenceEntry[];
+```
+
+These should produce **consistent field names** matching FR-1/FR-3 JSON paths so evidence aligns with structured fields when both are present.
+
+### Acceptance
+
+- Exported from `@x12i/logxer`.
+- ai-tasks can delete duplicate evidence assembly once adopted.
+
+---
+
+## FR-8 — Symptom deduplication / downgrade when cause already logged
+
+### Problem
+
+Graph-engine logs `TASK_RUN_FAILED` **after** ai-tasks already logged the root cause — noisy consoles, wrong “primary” error.
+
+### Proposed behavior (opt-in per host)
+
+```ts
+createLogxer(config, {
+  causalLogging?: {
+    /** If a cause with same correlationId + rootMessage was logged in-window, emit symptom as debug/warn instead of error */
+    downgradeDuplicateSymptoms?: boolean;
+    windowMs?: number;
+  },
+});
+```
+
+Heuristic (logxer-side, best-effort):
+
+- Same `jobId`/`taskId`/`correlationId`
+- New entry has `failureClassification.role === "symptom"`
+- Recent entry has `role === "cause"` and matching `rootMessage` or `causal.rootCode`
+
+### Acceptance
+
+- Off by default (no behavior change).
+- When enabled, graph-engine wrapper lines can downgrade without losing data.
+
+---
+
+## FR-9 — Preserve wrapped-error context in coded diagnostics
+
+### Problem
+
+Hosts catch errors and return `{ ok: false, error: { message } }` — stack and `cause` chain lost in logs unless manually copied.
+
+### Proposed helper
+
+```ts
+extractErrorDiagnostics(error: unknown): {
+  rootError: { name: string; message: string; code?: string; stack?: string };
+  failureClassification?: FailureClassification; // if present on error
+  llmRoute?: LlmRouteDiagnostics;
+};
+
+logxer.errorFromCaught("TASK_RUN_FAILED", error, { correlation: { jobId, taskId } });
+```
+
+Walks `Error.cause`, reads `failureClassification` / `details` / `observation` when present (duck-typed), classifies if absent.
+
+### Acceptance
+
+- Safe on non-Error throws.
+- Does not mutate the original error.
+
+---
+
+## FR-10 — Cross-package `DebugLogAbstract` value for causal traces
+
+### Problem
+
+ai-tasks uses `DebugLogAbstract.TRACE` for failure logs; graph-engine may use different kinds. Hard to filter “actionable failure diagnostics” vs noise.
+
+### Proposed enum addition
+
+```ts
+enum DebugLogAbstract {
+  // existing…
+  /** Actionable failure: includes classification and/or rootError */
+  FAILURE = "failure",
+  /** Wrapper/surface signal — prefer linked cause entry */
+  FAILURE_SYMPTOM = "failure_symptom",
+}
+```
+
+Document mapping: packages should set `debugKind: FAILURE` for cause, `FAILURE_SYMPTOM` for wrappers when classification is known.
+
+### Acceptance
+
+- Backward compatible (new enum values).
+- Documented in logxer migration notes.
+
+---
+
+## Suggested implementation order
+
+| Priority | FR | Rationale |
+|----------|-----|-----------|
+| P0 | FR-1, FR-2 | Structured classification + detail bag — unlocks everything else |
+| P0 | FR-3 | Provider/route visibility was the main blind spot |
+| P1 | FR-4, FR-7 | Query + helpers — reduces consumer boilerplate |
+| P1 | FR-5, FR-9 | Causal linking across graph-engine ↔ ai-tasks ↔ gateway |
+| P2 | FR-6, FR-8, FR-10 | Catalog, dedup, debugKind polish |
+
+---
+
+## Consumer migration (`@exellix/ai-tasks`)
+
+When logxer ships FR-1–3:
+
+1. Replace `fieldEvidence("failureRole", …)` with `failureClassification` + `fieldEvidenceFailureClassification`.
+2. Move `llmRoute` from custom types to `@x12i/logxer` export; keep ai-tasks re-export for one major version.
+3. Pass `causal.rootMessage` from `RunTaskExecutionError` into graph-engine host adapters.
+4. Delete `src/observability/classifyRunTaskFailure.ts` only if logxer ships equivalent + re-export (or keep as thin wrapper).
+
+---
+
+## References (ai-tasks)
+
+- Interim implementation: `src/observability/classifyRunTaskFailure.ts`, `src/errors/runTaskExecutionError.ts`, `src/observability/logRunTaskFailure.ts`, `src/observability/llmRouteContext.ts`
+- Diagnostic code: `RUN_TASK_EXECUTION_FAILED` in `src/logxer/aiTasksDiagnosticCodes.ts`
+- Related graph-engine behavior: `RealTasksClient` catch → `{ ok: false, error }`; runtime throw `TASK_RUN_FAILED: ${skillKey}`

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@exellix/ai-tasks",
-  "version": "8.1.16",
+  "version": "8.2.0",
   "description": "Task orchestration for the Exellix stack: runTask() with local handlers or LLM-backed execution, task-scoped memory/context enrichment, and executor dispatch via @exellix/ai-skills. ERC-compliant.",
   "type": "module",
   "main": "dist/index.js",
@@ -22,7 +22,7 @@
     "dev": "ts-node src/index.ts",
     "prepublishOnly": "npm run build && node scripts/ensure-nx-content-md-variants.mjs",
     "deps:latest": "npx npm-check-updates -u && npm install",
-    "test": "node scripts/ensure-xynthesis-dist.mjs && node scripts/clean-dist-test.mjs && tsc -p tsconfig.test.json && node scripts/copy-test-fixtures.mjs && node --test --test-force-exit --test-concurrency=1 dist-test/test/narrix/*.js dist-test/test/narrix-then-execute/*.js dist-test/test/aiScoping/*.js dist-test/test/intermediateSteps/*.js dist-test/test/post-steps/*.js dist-test/test/planWebScopeQuestions/*.js dist-test/test/synthesis/*.js dist-test/test/utils/*.js dist-test/test/errors/*.js dist-test/test/validation/*.js dist-test/test/compile/*.js dist-test/test/task-strategies/*.js dist-test/test/observability/*.js dist-test/test/run-task/*.js dist-test/test/internal/*.js dist-test/test/execution-strategies/*.js",
+    "test": "node scripts/ensure-xynthesis-dist.mjs && node scripts/clean-dist-test.mjs && tsc -p tsconfig.test.json && node scripts/copy-test-fixtures.mjs && node --test --test-force-exit --test-concurrency=1 dist-test/test/narrix/*.js dist-test/test/narrix-then-execute/*.js dist-test/test/aiScoping/*.js dist-test/test/intermediateSteps/*.js dist-test/test/post-steps/*.js dist-test/test/planWebScopeQuestions/*.js dist-test/test/synthesis/*.js dist-test/test/utils/*.js dist-test/test/errors/*.js dist-test/test/validation/*.js dist-test/test/compile/*.js dist-test/test/task-strategies/*.js dist-test/test/observability/*.js dist-test/test/run-task/*.js dist-test/test/internal/*.js dist-test/test/execution-strategies/*.js dist-test/test/invocation/*.js",
     "test:with-narrix-ingest": "node scripts/run-npm-test-with-narrix-ingest.mjs",
     "test:e2e:intermediateSteps": "node scripts/run-with-env.mjs RUN_INTERMEDIATE_STEPS_E2E=1 npm run test",
     "test:e2e:synthesis": "node scripts/run-synthesis-e2e.mjs",