@node-llm/testing 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,8 @@
+# Changelog
+
+## 0.1.0
+
+- Initial release
+- VCR record/replay
+- Fluent mocking
+- CI-safe defaults

package/README.md

@@ -0,0 +1,541 @@
+# @node-llm/testing 🌑🟢🧪
+
+Deterministic testing infrastructure for NodeLLM-powered AI systems. Built for engineers who prioritize **Boring Solutions**, **Security**, and **High-Fidelity Feedback Loops**.
+
+> 💡 **What is High-Fidelity?**
+> Your tests exercise the same execution path, provider behavior, and tool orchestration as production — without live network calls.
+
+**Framework Support**: ✅ Vitest (native) | ✅ Jest (compatible) | ✅ Any test framework (core APIs)
+
+---
+
+## 🧭 The Philosophy: Two-Tier Testing
+
+We believe AI testing should never be flaky or expensive. We provide two distinct strategies:
+
+### 1. VCR (Integration Testing) 📼
+
+**When to use**: To verify your system works with real LLM responses without paying for every test run.
+
+- **High Fidelity**: Captures the exact raw response from the provider.
+- **Security First**: Automatically scrubs API Keys and sensitive PII from "cassettes".
+- **CI Safe**: Fails-fast in CI if a cassette is missing, preventing accidental live API calls.
+
+### 2. Mocker (Unit Testing) 🎭
+
+**When to use**: To test application logic, edge cases (errors, rate limits), and rare tool-calling paths.
+
+- **Declarative**: Fluent API to define expected prompts and responses.
+- **Multimodal**: Native support for `chat`, `embed`, `paint`, `transcribe`, and `moderate`.
+- **Streaming**: Simulate token-by-token delivery to test real-time UI logic.
+
+---
+
+## 📼 VCR Usage
+
+### Basic Interaction
+
+Wrap your tests in `withVCR` to automatically record interactions the first time they run.
+
+```typescript
+import { withVCR } from "@node-llm/testing";
+
+it(
+  "calculates sentiment correctly",
+  withVCR(async () => {
+    const result = await mySentimentAgent.run("I love NodeLLM!");
+    expect(result.sentiment).toBe("positive");
+  })
+);
+```
+
+### Hierarchical Organization (Rails Mode) 📂
+
+Organize your cassettes into nested subfolders to match your test suite structure.
+
+```typescript
+import { describeVCR, withVCR } from "@node-llm/testing";
+
+describeVCR("Authentication", () => {
+  describeVCR("Login", () => {
+    it(
+      "logs in successfully",
+      withVCR(async () => {
+        // Cassette saved to: .llm-cassettes/authentication/login/logs-in-successfully.json
+      })
+    );
+  });
+});
+```
+
+### Security & Scrubbing 🛡️
+
+The VCR automatically redacts `api_key`, `authorization`, and other sensitive headers. You can add custom redaction:
+
+```typescript
+withVCR({
+  scrub: (data) => data.replace(/SSN: \d+/g, "[REDACTED_SSN]")
+}, async () => { ... });
+```
+
+---
+
+## 🎭 Mocker Usage
+
+### Fluent, Explicit Mocking
+
+Define lightning-fast, zero-network tests for your agents.
+
+```typescript
+import { mockLLM } from "@node-llm/testing";
+
+const mocker = mockLLM();
+
+// Exact match
+mocker.chat("Ping").respond("Pong");
+
+// Regex match
+mocker.chat(/hello/i).respond("Greetings!");
+
+// Simulate a Tool Call
+mocker.chat("What's the weather?").callsTool("get_weather", { city: "London" });
+```
+
+### Streaming Mocks 🌊
+
+Test your streaming logic by simulating token delivery.
+
+```typescript
+mocker.chat("Tell a story").stream(["Once ", "upon ", "a ", "time."]);
+```
+
+### Multimodal Mocks 🎨
+
+```typescript
+mocker.paint(/a cat/i).respond({ url: "https://mock.com/cat.png" });
+mocker.embed("text").respond({ vectors: [[0.1, 0.2, 0.3]] });
+```
+
+---
+
+## 🛣️ Decision Tree: VCR vs Mocker
+
+Choose the right tool for your test:
+
+```
+Does your test need to verify behavior against REAL LLM responses?
+├─ YES → Use VCR (integration testing)
+│   ├─ Do you need to record the first time and replay afterward?
+│   │   └─ YES → Use VCR in "record" or "auto" mode
+│   ├─ Are you testing in CI/CD? (No live API calls allowed)
+│   │   └─ YES → Set VCR_MODE=replay in CI
+│   └─ Need custom scrubbing for sensitive data?
+│       └─ YES → Use withVCR({ scrub: ... })
+│
+└─ NO → Use Mocker (unit testing)
+    ├─ Testing error handling, edge cases, or rare paths?
+    │   └─ YES → Mock the error with mocker.chat(...).respond({ error: ... })
+    ├─ Testing streaming token delivery?
+    │   └─ YES → Use mocker.chat(...).stream([...])
+    └─ Testing tool-calling paths without real tools?
+        └─ YES → Use mocker.chat(...).callsTool(name, params)
+```
+
+**Quick Reference**:
+
+- **VCR**: Database queries, API calls, real provider behavior, network latency
+- **Mocker**: Business logic, UI interactions, error scenarios, tool orchestration
+
+### At-a-Glance Comparison
+
+| Use Case                | VCR               | Mocker |
+| ----------------------- | ----------------- | ------ |
+| Real provider behavior  | ✅                | ❌     |
+| CI-safe (no live calls) | ✅ (after record) | ✅     |
+| Zero network overhead   | ❌ (first run)    | ✅     |
+| Error simulation        | ⚠️ (record real)  | ✅     |
+| Tool orchestration      | ✅                | ✅     |
+| Streaming tokens        | ✅                | ✅     |
+
+---
+
+## ⚙️ Configuration
+
+### Environment Variables
+
+| Env Variable       | Description                                                | Default          |
+| ------------------ | ---------------------------------------------------------- | ---------------- |
+| `VCR_MODE`         | `record`, `replay`, `auto`, or `passthrough`               | `auto`           |
+| `VCR_CASSETTE_DIR` | Base directory for cassettes                               | `test/cassettes` |
+| `CI`               | When true, VCR prevents recording and forces exact matches | (Auto-detected)  |
+
+### Programmatic Configuration
+
+Configure VCR globally for all instances in your test suite:
+
+```typescript
+import { configureVCR, resetVCRConfig } from "@node-llm/testing";
+
+// Before all tests
+beforeAll(() => {
+  configureVCR({
+    // Custom keys to redact in cassettes
+    sensitiveKeys: ["api_key", "bearer_token", "custom_secret"],
+
+    // Custom regex patterns to redact
+    sensitivePatterns: [/api_key=[\w]+/g, /Bearer ([\w.-]+)/g]
+  });
+});
+
+// After all tests
+afterAll(() => {
+  resetVCRConfig();
+});
+```
+
+### Per-Instance Configuration
+
+Override global settings for a specific VCR instance:
+
+```typescript
+withVCR(
+  {
+    mode: "replay",
+    cassettesDir: "./test/fixtures",
+    scrub: (data) => data.replace(/email=\S+@/, "email=[REDACTED]@"),
+    sensitiveKeys: ["session_token"]
+  },
+  async () => {
+    // Test runs here
+  }
+);
+```
+
+});
+
+````
+
+---
+
+## 🧪 Framework Integration
+
+### Vitest (Native Support)
+
+Vitest is the primary test framework with optimized helpers:
+
+```typescript
+import { it, describe } from "vitest";
+import { mockLLM, withVCR, describeVCR } from "@node-llm/testing";
+
+describeVCR("Payments", () => {
+  it(
+    "processes successfully",
+    withVCR(async () => {
+      // ✨ withVCR auto-detects test name ("processes successfully")
+      // ✨ describeVCR auto-manages scopes
+    })
+  );
+});
+```
+
+### Jest Compatibility
+
+All core APIs work with Jest. The only difference: `withVCR()` can't auto-detect test names, so provide it manually:
+
+```typescript
+import { describe, it } from "@jest/globals";
+import { mockLLM, setupVCR, describeVCR } from "@node-llm/testing";
+
+describeVCR("Payments", () => {
+  it("processes successfully", async () => {
+    // ✅ describeVCR works with Jest (framework-agnostic)
+    // ⚠️ withVCR doesn't work here (needs Vitest's expect.getState())
+    // ✅ Use setupVCR instead:
+    const vcr = setupVCR("processes", { mode: "record" });
+
+    const mocker = mockLLM();  // ✅ works with Jest
+    mocker.chat("pay").respond("done");
+
+    // Test logic here
+
+    await vcr.stop();
+  });
+});
+```
+
+### Framework Support Matrix
+
+| API | Vitest | Jest | Any Framework |
+|-----|--------|------|---------------|
+| `mockLLM()` | ✅ | ✅ | ✅ |
+| `describeVCR()` | ✅ | ✅ | ✅ |
+| `setupVCR()` | ✅ | ✅ | ✅ |
+| `withVCR()` | ✅ (auto name) | ⚠️ (manual name) | ⚠️ (manual name) |
+| Mocker class | ✅ | ✅ | ✅ |
+| VCR class | ✅ | ✅ | ✅ |
+
+**Only `withVCR()` is Vitest-specific** because it auto-detects test names. All other APIs are framework-agnostic.
+
+### Any Test Framework
+
+Using raw classes for maximum portability:
+
+```typescript
+import { Mocker, VCR } from "@node-llm/testing";
+
+// Mocker - works everywhere
+const mocker = new Mocker();
+mocker.chat("hello").respond("hi");
+
+// VCR - works everywhere
+const vcr = new VCR("test-name", { mode: "record" });
+// ... run test ...
+await vcr.stop();
+```
+
+---
+
+## 🚨 Error Handling & Debugging
+
+### VCR Common Issues
+
+#### Missing Cassette Error
+
+**Error**: `Error: Cassette file not found`
+
+**Cause**: VCR is in `replay` mode but the cassette doesn't exist yet.
+
+**Solution**:
+```typescript
+// Either: Record it first
+VCR_MODE=record npm test
+
+// Or: Use auto mode (records if missing, replays if exists)
+VCR_MODE=auto npm test
+
+// Or: Explicitly set mode
+withVCR({ mode: "record" }, async () => { ... });
+````
+
+#### Cassette Mismatch Error
+
+**Error**: `AssertionError: No interaction matched the request`
+
+**Cause**: Your code is making a request that doesn't match any recorded interaction.
+
+**Solution**:
+
+```typescript
+// 1. Debug what request was made
+const mocker = mockLLM();
+mocker.onAnyRequest((req) => {
+  console.log("Unexpected request:", req.prompt);
+});
+
+// 2. Re-record the cassette
+rm -rf .llm-cassettes/your-test
+VCR_MODE=record npm test -- your-test
+
+// 3. Commit the updated cassette to git
+```
+
+#### Sensitive Data Not Scrubbed
+
+**Error**: API keys appear in cassette JSON
+
+**Solution**: Add custom scrubbing rules
+
+```typescript
+import { configureVCR } from "@node-llm/testing";
+
+configureVCR({
+  sensitiveKeys: ["x-api-key", "authorization", "custom_token"],
+  sensitivePatterns: [/Bearer ([\w.-]+)/g]
+});
+```
+
+### Mocker Common Issues
+
+#### Strict Mode Enforcement
+
+**Error**: `Error: No mock defined for prompt: "unexpected question"`
+
+**Cause**: Your code asked a question you didn't mock in strict mode.
+
+**Solution**:
+
+```typescript
+// Either: Add the missing mock
+mocker.chat("unexpected question").respond("mocked response");
+
+// Or: Disable strict mode
+const mocker = mockLLM({ strict: false });
+// Now unmocked requests return generic "I don't have a response" message
+
+// Or: Debug what's being asked
+mocker.onAnyRequest((req) => {
+  console.error("Unmatched request:", req.prompt);
+  throw new Error(`Add mock for: mocker.chat("${req.prompt}").respond(...)`);
+});
+```
+
+#### Stream Simulation Issues
+
+**Error**: `TypeError: Cannot read property 'Symbol(Symbol.iterator)' of undefined`
+
+**Cause**: Stream mock not properly yielding tokens.
+
+**Solution**:
+
+```typescript
+// Correct: Array of tokens
+mocker.chat("story").stream(["Once ", "upon ", "a ", "time."]);
+
+// Incorrect: String instead of array
+mocker.chat("story").stream("Once upon a time."); // ❌ Wrong!
+```
+
+### Debug Information
+
+Get detailed insight into what mocks are registered:
+
+```typescript
+const mocker = mockLLM();
+mocker.chat("hello").respond("hi");
+mocker.embed("text").respond({ vectors: [[0.1, 0.2]] });
+
+const debug = mocker.getDebugInfo();
+console.log(debug);
+// Output:
+// {
+//   totalMocks: 2,
+//   methods: ["chat", "embed"]
+// }
+```
+
+---
+
+## 📚 Type Documentation
+
+### VCROptions
+
+```typescript
+interface VCROptions {
+  // Recording/Replay behavior
+  mode?: "record" | "replay" | "auto" | "passthrough";
+  cassettesDir?: string;
+
+  // Security & Scrubbing
+  sensitiveKeys?: string[];
+  sensitivePatterns?: RegExp[];
+  scrub?: (data: string) => string;
+}
+```
+
+### MockerOptions
+
+```typescript
+interface MockerOptions {
+  // Enforce exact matching
+  strict?: boolean;
+
+  // Enable verbose logging
+  debug?: boolean;
+}
+```
+
+### MockResponse
+
+```typescript
+interface MockResponse {
+  // Simple text response
+  content?: string;
+
+  // Tool calling
+  toolName?: string;
+  toolParams?: Record<string, unknown>;
+
+  // Error simulation
+  error?: Error | string;
+
+  // Streaming tokens
+  tokens?: string[];
+
+  // Generation metadata
+  metadata?: {
+    tokensUsed?: number;
+    model?: string;
+  };
+}
+```
+
+### MockerDebugInfo
+
+```typescript
+interface MockerDebugInfo {
+  // Total number of mocks defined
+  totalMocks: number;
+
+  // Array of unique method names used ("chat", "embed", etc.)
+  methods: string[];
+}
+```
+
+---
+
+## 🏛️ Integration with @node-llm/orm
+
+The testing tools operate at the `providerRegistry` level. This means they **automatically** intercept LLM calls made by the ORM layer.
+
+### Pattern: Testing Database Persistence
+
+When using `@node-llm/orm`, you can verify both the database state and the LLM response in a single test.
+
+```typescript
+import { withVCR } from "@node-llm/testing";
+import { createChat } from "@node-llm/orm/prisma";
+
+it(
+  "saves the LLM response to the database",
+  withVCR(async () => {
+    // 1. Setup ORM Chat
+    const chat = await createChat(prisma, llm, { model: "gpt-4" });
+
+    // 2. Interaction (VCR intercepts the LLM call)
+    await chat.ask("Hello ORM!");
+
+    // 3. Verify DB state (standard Prisma/ORM assertions)
+    const messages = await prisma.assistantMessage.findMany({
+      where: { chatId: chat.id }
+    });
+
+    expect(messages).toHaveLength(2); // User + Assistant
+    expect(messages[1].content).toBeDefined();
+  })
+);
+```
+
+### Pattern: Mocking Rare Logic
+
+Use the `Mocker` to test how your application handles complex tool results or errors without setting up a real LLM.
+
+```typescript
+import { mockLLM } from "@node-llm/testing";
+
+it("handles tool errors in ORM sessions", async () => {
+  const mocker = mockLLM();
+  mocker.chat("Search docs").respond({ error: new Error("DB Timeout") });
+
+  const chat = await loadChat(prisma, llm, "existing-id");
+
+  await expect(chat.ask("Search docs")).rejects.toThrow("DB Timeout");
+});
+```
+
+---
+
+## 🏛️ Architecture Contract
+
+- **No Side Effects**: Mocks and VCR interceptors are automatically cleared after each test turn.
+- **Deterministic**: The same input MUST always yield the same output in Replay mode.
+- **Explicit > Implicit**: We prefer explicit mock definitions over complex global state.

package/dist/Mocker.d.ts

@@ -0,0 +1,58 @@
+import { ChatChunk, ToolCall, ModerationResult } from "@node-llm/core";
+export interface MockResponse {
+    content?: string | null;
+    tool_calls?: ToolCall[];
+    usage?: {
+        input_tokens: number;
+        output_tokens: number;
+        total_tokens: number;
+    };
+    error?: Error;
+    finish_reason?: string | null;
+    chunks?: string[] | ChatChunk[];
+    vectors?: number[][];
+    url?: string;
+    data?: string;
+    text?: string;
+    results?: ModerationResult[];
+    revised_prompt?: string;
+    id?: string;
+}
+export type MockMatcher = (request: unknown) => boolean;
+export interface MockDefinition {
+    method: string;
+    match: MockMatcher;
+    response: MockResponse | ((request: unknown) => MockResponse);
+}
+/**
+ * Debug information about defined mocks.
+ */
+export interface MockerDebugInfo {
+    totalMocks: number;
+    methods: string[];
+}
+export declare class Mocker {
+    private mocks;
+    strict: boolean;
+    constructor();
+    chat(query?: string | RegExp): this;
+    stream(chunks: string[] | ChatChunk[]): this;
+    placeholder(query: string | RegExp): this;
+    callsTool(name: string, args?: Record<string, unknown>): this;
+    embed(input?: string | string[]): this;
+    paint(prompt?: string | RegExp): this;
+    transcribe(file?: string | RegExp): this;
+    moderate(input?: string | string[] | RegExp): this;
+    respond(response: string | MockResponse | ((req: unknown) => MockResponse)): this;
+    /**
+     * Returns debug information about defined mocks.
+     * Useful for troubleshooting what mocks are defined.
+     */
+    getDebugInfo(): MockerDebugInfo;
+    clear(): void;
+    private addMock;
+    private getContentString;
+    private setupInterceptor;
+}
+export declare function mockLLM(): Mocker;
+//# sourceMappingURL=Mocker.d.ts.map

package/dist/Mocker.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"Mocker.d.ts","sourceRoot":"","sources":["../src/Mocker.ts"],"names":[],"mappings":"AAAA,OAAO,EAaL,SAAS,EACT,QAAQ,EACR,gBAAgB,EAEjB,MAAM,gBAAgB,CAAC;AAExB,MAAM,WAAW,YAAY;IAC3B,OAAO,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACxB,UAAU,CAAC,EAAE,QAAQ,EAAE,CAAC;IACxB,KAAK,CAAC,EAAE;QACN,YAAY,EAAE,MAAM,CAAC;QACrB,aAAa,EAAE,MAAM,CAAC;QACtB,YAAY,EAAE,MAAM,CAAC;KACtB,CAAC;IACF,KAAK,CAAC,EAAE,KAAK,CAAC;IACd,aAAa,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IAC9B,MAAM,CAAC,EAAE,MAAM,EAAE,GAAG,SAAS,EAAE,CAAC;IAChC,OAAO,CAAC,EAAE,MAAM,EAAE,EAAE,CAAC;IACrB,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,OAAO,CAAC,EAAE,gBAAgB,EAAE,CAAC;IAC7B,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,EAAE,CAAC,EAAE,MAAM,CAAC;CACb;AAED,MAAM,MAAM,WAAW,GAAG,CAAC,OAAO,EAAE,OAAO,KAAK,OAAO,CAAC;AAExD,MAAM,WAAW,cAAc;IAC7B,MAAM,EAAE,MAAM,CAAC;IACf,KAAK,EAAE,WAAW,CAAC;IACnB,QAAQ,EAAE,YAAY,GAAG,CAAC,CAAC,OAAO,EAAE,OAAO,KAAK,YAAY,CAAC,CAAC;CAC/D;AAED;;GAEG;AACH,MAAM,WAAW,eAAe;IAC9B,UAAU,EAAE,MAAM,CAAC;IACnB,OAAO,EAAE,MAAM,EAAE,CAAC;CACnB;AAID,qBAAa,MAAM;IACjB,OAAO,CAAC,KAAK,CAAwB;IAC9B,MAAM,UAAS;;IAMf,IAAI,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI;IAcnC,MAAM,CAAC,MAAM,EAAE,MAAM,EAAE,GAAG,SAAS,EAAE,GAAG,IAAI;IAU5C,WAAW,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI;IAWzC,SAAS,CAAC,IAAI,EAAE,MAAM,EAAE,IAAI,GAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAM,GAAG,IAAI;IAmBjE,KAAK,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,IAAI;IAQtC,KAAK,CAAC,MAAM,CAAC,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI;IASrC,UAAU,CAAC,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,GAAG,IAAI;IASxC,QAAQ,CAAC,KAAK,CAAC,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,MAAM,GAAG,IAAI;IAUlD,OAAO,CAAC,QAAQ,EAAE,MAAM,GAAG,YAAY,GAAG,CAAC,CAAC,GAAG,EAAE,OAAO,KAAK,YAAY,CAAC,GAAG,IAAI;IAWxF;;;OAGG;IACI,YAAY,IAAI,eAAe;IAQ/B,KAAK,IAAI,IAAI;IAKpB,OAAO,CAAC,OAAO;IAKf,OAAO,CAAC,gBAAgB;IASxB,OAAO,CAAC,gBAAgB;CA6GzB;AAED,wBAAgB,OAAO,WAEtB"}