replay-self-healing-cli 0.1.0

package/ERROR_CODES.md

@@ -0,0 +1,68 @@
+# Error Codes
+
+All CLI errors are emitted as JSON with:
+
+- `code`
+- `message`
+- optional `path`
+- optional `expected`
+- optional `received`
+- optional `howToFix`
+- optional `example`
+
+## CLI input errors
+
+- `E_UNKNOWN_FLAG` unknown CLI flag
+- `E_TIMEOUT_INVALID` invalid `--timeout-ms`
+- `E_PROMPT_REQUIRED` prompt missing from args/stdin
+- `E_CONTEXT_CONFLICT` both `--context` and `--context-file` set
+- `E_CONTEXT_JSON_INVALID` invalid JSON in context
+- `E_CONTEXT_SHAPE_INVALID` context is not object
+- `E_CONTEXT_FILE_NOT_FOUND` context file missing
+- `E_COMMAND_UNKNOWN` unsupported command
+
+## Runner discovery/execution errors
+
+- `E_RUNNER_NOT_FOUND` no default runner discovered
+- `E_RUNNER_PATH_NOT_FOUND` explicit runner path missing
+- `E_RUNNER_UNSUPPORTED_EXTENSION` unsupported runner extension (e.g. `.ts`)
+- `E_RUNNER_NOT_EXECUTABLE` runner not executable when required
+- `E_RUNNER_START_FAILED` process spawn failure
+- `E_RUNNER_TIMEOUT` runner timed out
+- `E_RUNNER_EXIT_NON_ZERO` runner process exited with non-zero code
+
+## Runner output validation errors
+
+- `E_RUNNER_EMPTY_STDOUT` runner wrote no stdout
+- `E_RUNNER_INVALID_JSON` runner stdout not parseable JSON
+- `E_RUNNER_OUTPUT_SHAPE` output is not JSON object
+- `E_RUNNER_STATUS_INVALID` status is not `ok` or `error`
+- `E_RUNNER_EVENTS_INVALID` events is not array
+- `E_RUNNER_RESPONSE_MISSING` status `ok` but missing response string
+
+## Artifact IO and validation errors
+
+- `E_INPUT_PATH_NOT_FOUND` `--in` target missing
+- `E_INPUT_FILE_NOT_JSON` `--in` file is not `.json`
+- `E_INPUT_NOT_FILE_OR_DIR` `--in` target not file/dir
+- `E_ARTIFACT_JSON_INVALID` invalid JSON artifact file
+- `E_ARTIFACT_SHAPE_INVALID` artifact is not object
+- `E_ARTIFACT_TYPE_INVALID` artifactType mismatch
+- `E_SCHEMA_VERSION_UNSUPPORTED` schemaVersion mismatch
+- `E_ARTIFACT_ID_INVALID` missing/invalid id
+- `E_ARTIFACT_STATUS_INVALID` missing/invalid status
+- `E_ARTIFACT_EVENTS_INVALID` missing/invalid events array
+- `E_ARTIFACT_EVENT_INVALID` event item not object
+- `E_ARTIFACT_EVENT_TYPE_INVALID` event.type invalid
+- `E_ARTIFACT_EVENT_SEQ_INVALID` event.seq invalid
+- `E_ARTIFACT_EVENT_TIMESTAMP_INVALID` event.timestamp invalid
+- `E_HEALED_ARTIFACT_INVALID` healed artifact fails validation
+
+## Heal rule errors
+
+- `E_HEAL_RULES_INVALID_JSON` heal rules file invalid JSON
+- `E_HEAL_RULES_SHAPE_INVALID` heal rules must be object
+
+## Internal fallback
+
+- `E_UNHANDLED` unexpected runtime failure

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,284 @@
+# replay-self-healing-cli
+
+Zero-dependency CLI to capture LLM runs as replay artifacts, auto-heal trace inconsistencies, and produce validation/report output.
+
+## Why this exists
+
+- Prompt-first UX: run one prompt and get replay artifacts immediately.
+- Runner is user-defined: framework choice stays outside this tool.
+- Self-healing: the CLI preserves raw output, repairs common trace issues, and logs every repair action.
+- LLM-friendly errors: failures are always emitted as machine-readable JSON with fix guidance.
+
+## Install
+
+```bash
+npm install -g replay-self-healing-cli
+```
+
+Or run locally from this folder:
+
+```bash
+npm install
+node ./bin/replay.mjs help
+```
+
+This package has **zero dependencies**.
+
+## 30-second quickstart
+
+1. Create a runner at `.replay/runner.mjs`:
+
+```js
+#!/usr/bin/env node
+import process from 'node:process';
+
+let input = '';
+for await (const chunk of process.stdin) {
+  input += chunk;
+}
+
+const request = JSON.parse(input);
+const now = new Date().toISOString();
+
+const output = {
+  status: 'ok',
+  framework: 'custom',
+  response: `Echo: ${request.prompt}`,
+  events: [
+    { seq: 1, timestamp: now, type: 'run_started', runId: request.runId, payload: {} },
+    {
+      seq: 2,
+      timestamp: now,
+      type: 'assistant_message',
+      runId: request.runId,
+      payload: { response: `Echo: ${request.prompt}` }
+    },
+    { seq: 3, timestamp: now, type: 'run_completed', runId: request.runId, payload: {} }
+  ],
+  sources: [],
+  usage: { inputTokens: 0, outputTokens: 0, costUsd: 0 }
+};
+
+process.stdout.write(JSON.stringify(output));
+```
+
+2. Run:
+
+```bash
+replay "What are my top holdings?"
+```
+
+3. Output files are created automatically under:
+
+```text
+.replay/artifacts/YYYY-MM-DD/
+```
+
+## Command overview
+
+### Capture (default)
+
+```bash
+replay "your prompt"
+replay capture "your prompt"
+echo "your prompt" | replay
+```
+
+Optional flags:
+
+- `--runner <path>` override runner discovery
+- `--out <dir>` override output directory
+- `--context '{"k":"v"}'` inline JSON context
+- `--context-file ./context.json` JSON context file
+- `--timeout-ms 120000` runner timeout
+- `--no-heal` skip heal pass
+
+### Validate artifacts
+
+```bash
+replay validate --in .replay/artifacts
+```
+
+### Re-heal existing artifacts
+
+```bash
+replay heal --in .replay/artifacts --out .replay/healed
+```
+
+### Report summary
+
+```bash
+replay report --in .replay/artifacts
+```
+
+## Runner discovery (implied defaults)
+
+If `--runner` is not provided, the CLI checks in this order:
+
+1. `REPLAY_RUNNER` env var
+2. `.replay/runner.mjs`
+3. `.replay/runner.js`
+4. `.replay/runner.cjs`
+5. `.replay/runner`
+6. `package.json` script: `replay:runner`
+
+## Runner input contract
+
+Runner receives one JSON object via `stdin`:
+
+```json
+{
+  "prompt": "What are my top holdings?",
+  "query": "What are my top holdings?",
+  "runId": "run_...",
+  "timestamp": "2026-03-01T12:00:00.000Z",
+  "context": {}
+}
+```
+
+## Runner output contract
+
+Runner must print one JSON object to `stdout`.
+
+Success:
+
+```json
+{
+  "status": "ok",
+  "framework": "langchain",
+  "response": "AAPL is your largest holding.",
+  "events": [
+    {
+      "seq": 1,
+      "timestamp": "2026-03-01T12:00:00.000Z",
+      "type": "run_started",
+      "runId": "run_123",
+      "payload": {}
+    }
+  ],
+  "sources": ["portfolio_db"],
+  "usage": {
+    "inputTokens": 100,
+    "outputTokens": 40,
+    "costUsd": 0.0021
+  }
+}
+```
+
+Failure:
+
+```json
+{
+  "status": "error",
+  "error": {
+    "code": "RUNNER_TOOL_TIMEOUT",
+    "message": "get_portfolio_summary timed out after 15s",
+    "retryable": true
+  },
+  "events": []
+}
+```
+
+Schema references:
+
+- [schemas/runner-output.schema.json](./schemas/runner-output.schema.json)
+- [schemas/replay-artifact.schema.json](./schemas/replay-artifact.schema.json)
+
+## Self-healing behavior
+
+The CLI always writes a raw artifact first, then heals into a replay artifact.
+
+Healing currently includes:
+
+- type alias mapping (`tool_end` -> `tool_completed`, etc.)
+- missing `run_started` insertion
+- missing terminal event insertion (`run_completed`/`run_failed`)
+- non-object event payload normalization to `{}`
+- sequence renumbering to strict monotonic order
+- duplicate event dedupe (exact adjacent duplicates)
+- synthetic `assistant_message` insertion from final response when missing
+
+All applied repairs are written to `*.heal-log.json`.
+
+### Custom heal rules
+
+Optional file: `.replay/heal.rules.json`
+
+```json
+{
+  "version": "v1",
+  "eventAliases": {
+    "tool_end": "tool_completed",
+    "llm_message": "assistant_message"
+  }
+}
+```
+
+## Artifact files
+
+For each run:
+
+- `*.raw.json` raw capture of runner process input/output
+- `*.artifact.json` normalized artifact before healing
+- `*.healed.json` healed replay artifact
+- `*.heal-log.json` structured healing log
+
+## Error model (LLM-friendly)
+
+Errors are always JSON on `stderr`:
+
+```json
+{
+  "error": {
+    "code": "E_RUNNER_EVENTS_INVALID",
+    "message": "Runner output field `events` must be an array.",
+    "path": "$.events",
+    "expected": "array",
+    "received": "object",
+    "howToFix": "Return `events: []` when there are no events."
+  }
+}
+```
+
+See full catalog: [ERROR_CODES.md](./ERROR_CODES.md)
+
+## Publish checklist
+
+1. Authenticate npm on this machine:
+
+```bash
+npm login
+# or: npm adduser
+```
+
+2. Confirm identity and package name availability:
+
+```bash
+npm whoami
+npm view replay-self-healing-cli version
+```
+
+3. Run pre-publish checks:
+
+```bash
+node ./bin/replay.mjs help
+REPLAY_RUNNER=./examples/runner.mjs node ./bin/replay.mjs "test prompt"
+node ./bin/replay.mjs validate --in .replay/artifacts
+npm pack --dry-run
+```
+
+4. Publish:
+
+```bash
+npm publish
+```
+
+5. Verify:
+
+```bash
+npm view replay-self-healing-cli version
+```
+
+## License
+
+MIT

package/bin/replay.mjs

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+import '../src/cli.mjs';

package/package.json

@@ -0,0 +1,43 @@
+{
+  "name": "replay-self-healing-cli",
+  "version": "0.1.0",
+  "description": "Zero-dependency self-healing replay harness CLI for LLM runs",
+  "type": "module",
+  "bin": {
+    "replay": "bin/replay.mjs"
+  },
+  "files": [
+    "bin",
+    "src",
+    "schemas",
+    "README.md",
+    "LICENSE",
+    "ERROR_CODES.md"
+  ],
+  "scripts": {
+    "test": "node --test"
+  },
+  "keywords": [
+    "llm",
+    "evals",
+    "replay",
+    "harness",
+    "cli",
+    "self-healing"
+  ],
+  "license": "MIT",
+  "engines": {
+    "node": ">=18.17.0"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/nathankoerschner/replay-self-healing-cli.git"
+  },
+  "bugs": {
+    "url": "https://github.com/nathankoerschner/replay-self-healing-cli/issues"
+  },
+  "homepage": "https://github.com/nathankoerschner/replay-self-healing-cli#readme",
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/schemas/replay-artifact.schema.json

@@ -0,0 +1,107 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://replay-self-healing-cli.dev/schemas/replay-artifact.schema.json",
+  "title": "Replay Artifact",
+  "type": "object",
+  "required": [
+    "schemaVersion",
+    "artifactType",
+    "id",
+    "timestamp",
+    "prompt",
+    "query",
+    "status",
+    "events"
+  ],
+  "properties": {
+    "schemaVersion": {
+      "type": "string",
+      "const": "1.0.0"
+    },
+    "artifactType": {
+      "type": "string",
+      "const": "replay"
+    },
+    "id": {
+      "type": "string"
+    },
+    "timestamp": {
+      "type": "string",
+      "format": "date-time"
+    },
+    "prompt": {
+      "type": "string"
+    },
+    "query": {
+      "type": "string"
+    },
+    "status": {
+      "type": "string",
+      "enum": ["ok", "error"]
+    },
+    "response": {
+      "type": "string"
+    },
+    "error": {
+      "type": "object"
+    },
+    "events": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "required": ["seq", "timestamp", "type", "runId", "payload"],
+        "properties": {
+          "seq": {
+            "type": "integer",
+            "minimum": 1
+          },
+          "timestamp": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "type": {
+            "type": "string"
+          },
+          "runId": {
+            "type": "string"
+          },
+          "payload": {
+            "type": "object"
+          }
+        }
+      }
+    },
+    "toolCalls": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "required": ["toolName", "status", "durationMs"],
+        "properties": {
+          "toolName": {
+            "type": "string"
+          },
+          "status": {
+            "type": "string",
+            "enum": ["ok", "error"]
+          },
+          "durationMs": {
+            "type": "number",
+            "minimum": 0
+          }
+        }
+      }
+    },
+    "timings": {
+      "type": "object"
+    },
+    "sources": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "usage": {
+      "type": "object"
+    }
+  }
+}

package/schemas/runner-output.schema.json

@@ -0,0 +1,108 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://replay-self-healing-cli.dev/schemas/runner-output.schema.json",
+  "title": "Replay Runner Output",
+  "type": "object",
+  "required": ["status", "events"],
+  "properties": {
+    "status": {
+      "type": "string",
+      "enum": ["ok", "error"]
+    },
+    "framework": {
+      "type": "string"
+    },
+    "response": {
+      "type": "string"
+    },
+    "events": {
+      "type": "array",
+      "items": {
+        "type": "object",
+        "required": ["type"],
+        "properties": {
+          "runId": {
+            "type": "string"
+          },
+          "seq": {
+            "type": "integer",
+            "minimum": 1
+          },
+          "timestamp": {
+            "type": "string",
+            "format": "date-time"
+          },
+          "type": {
+            "type": "string"
+          },
+          "payload": {
+            "type": "object"
+          }
+        }
+      }
+    },
+    "sources": {
+      "type": "array",
+      "items": {
+        "type": "string"
+      }
+    },
+    "usage": {
+      "type": "object",
+      "properties": {
+        "inputTokens": {
+          "type": "number",
+          "minimum": 0
+        },
+        "outputTokens": {
+          "type": "number",
+          "minimum": 0
+        },
+        "costUsd": {
+          "type": "number",
+          "minimum": 0
+        }
+      }
+    },
+    "error": {
+      "type": "object",
+      "properties": {
+        "code": {
+          "type": "string"
+        },
+        "message": {
+          "type": "string"
+        },
+        "retryable": {
+          "type": "boolean"
+        }
+      }
+    }
+  },
+  "allOf": [
+    {
+      "if": {
+        "properties": {
+          "status": {
+            "const": "ok"
+          }
+        }
+      },
+      "then": {
+        "required": ["response"]
+      }
+    },
+    {
+      "if": {
+        "properties": {
+          "status": {
+            "const": "error"
+          }
+        }
+      },
+      "then": {
+        "required": ["error"]
+      }
+    }
+  ]
+}