truthguard-ai 0.1.4 → 0.2.0

package/README.md

@@ -1,8 +1,8 @@
 # TruthGuard
 
-**Grounding validation for tool-calling AI agents.**
+**Standardized grounding validation for tool-calling AI agents.**
 
-> Detect when an agent's response contradicts the data returned by the tools it called — without LLM-as-judge overhead.
+> Detect when an agent's response contradicts the data returned by the tools it called — deterministically, without LLM-as-judge overhead.
 
 [![npm version](https://img.shields.io/npm/v/truthguard-ai.svg)](https://www.npmjs.com/package/truthguard-ai)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
@@ -11,21 +11,23 @@
 
 ## The Problem
 
-Most "hallucinations" in tool-calling agents are **grounding failures** — the agent calls a tool, gets accurate data, then ignores it, miscalculates, or fabricates from empty results.
+Most "hallucinations" in tool-calling agents are **grounding failures** — the agent calls a tool, gets accurate data, and then ignores it, miscalculates, or fabricates from empty results. The source of truth is already in the trace.
 
 ## The Solution
 
-TruthGuard extracts factual claims from the agent's response and cross-references them against tool outputs. Deterministic. No LLM calls. Runs in <50ms.
+TruthGuard extracts factual claims from the agent's response, cross-references them against tool outputs, and reports grounding failures with standardized codes — like OBD diagnostic codes for AI.
 
-```bash
+```
 npm install truthguard-ai
 ```
 
+**Zero LLM calls.** 30+ deterministic failure detectors. Runs in <50ms.
+
 ---
 
-## Quick Start
+## Quick Start — 3 Minutes
 
-### Evaluate a trace
+### 1. Evaluate a trace
 
 ```typescript
 import { TraceBuilder, GroundingEngine, generateReport } from 'truthguard-ai';
@@ -37,7 +39,7 @@ const trace = new TraceBuilder({ traceId: 'run-001' })
     { employeeId: 'E01', name: 'Ana Jovic', status: 'on_leave' },
     { employeeId: 'E02', name: 'Ivan Petrovic', status: 'on_leave' },
   ])
-  .addFinalResponse('There are 3 employees on leave today.')
+  .addFinalResponse('There are 3 employees on leave today.')  // ← Bug: says 3, data shows 2
   .build();
 
 const engine = new GroundingEngine();
@@ -45,31 +47,28 @@ const report = engine.evaluate(trace);
 
 console.log(report.groundingScore);       // 0.5
 console.log(report.detectedFailures[0]);  // { type: 'grounding.data_ignored', severity: 'high' }
+
+const { text } = generateReport(report);
+console.log(text);
 ```
 
-### SDK Wrappers (OpenAI & Anthropic)
+### 2. Add a CI quality gate
 
 ```typescript
-import { wrapOpenAI } from 'truthguard-ai';
-import OpenAI from 'openai';
+import { loadDataset, runDataset, evaluateGate, loadGateConfig } from 'truthguard-ai';
 
-const openai = wrapOpenAI(new OpenAI(), {
-  mode: 'block',
-  threshold: 0.85,
-});
-```
-
-```typescript
-import { wrapAnthropic } from 'truthguard-ai';
-import Anthropic from '@anthropic-ai/sdk';
+const entries = loadDataset('./test-cases.jsonl');
+const result = runDataset(entries);
+const gate = loadGateConfig('.ai-rcp-gate.yml');
+const verdict = evaluateGate(result, gate);
 
-const anthropic = wrapAnthropic(new Anthropic(), {
-  mode: 'warn',
-  threshold: 0.80,
-});
+if (!verdict.pass) {
+  console.error(verdict.report);
+  process.exit(1);
+}
 ```
 
-### Production Monitoring (Proxy Mode)
+### 3. Monitor in production (proxy mode)
 
 Works with **any language** — PHP, Python, Go, Java, Ruby, C#:
 
@@ -77,7 +76,7 @@ Works with **any language** — PHP, Python, Go, Java, Ruby, C#:
 npx truthguard-ai observe --port 3001
 ```
 
-Point your AI SDK base URL to the proxy:
+Change your AI base URL:
 ```
 # OpenAI
 OPENAI_BASE_URL=http://localhost:3001/proxy/openai
@@ -86,17 +85,72 @@ OPENAI_BASE_URL=http://localhost:3001/proxy/openai
 ANTHROPIC_BASE_URL=http://localhost:3001/proxy/anthropic
 ```
 
-Your app works identically. TruthGuard evaluates grounding in the background.
+Your app works exactly the same. TruthGuard transparently proxies requests and evaluates grounding in the background.
 
-### CI Quality Gate
+---
 
-```bash
-npx truthguard-ai run test-cases.jsonl --gate .ai-rcp-gate.yml
+## Detection
+
+30+ deterministic failure detectors across grounding, orchestration, and reasoning categories.
+
+Examples:
+- Fabrication from empty tool results
+- Math errors against correct tool data
+- Ignored or altered tool data in the response
+- Entity mismatches and mix-ups
+
+---
+
+## Features
+
+### Diagnostic Advisor
+
+Every detected failure includes actionable diagnostics — root cause identification, evidence, and remediation guidance.
+
+### Policy Engine
+
+Configure per-failure actions — block, warn, or observe:
+
+```typescript
+import { wrapOpenAI, GroundingError } from 'truthguard-ai';
+import OpenAI from 'openai';
+
+const openai = wrapOpenAI(new OpenAI(), {
+  mode: 'block',
+  threshold: 0.85,
+  policy: {
+    rules: {
+      'grounding.empty_fabrication': 'block',
+      'grounding.math_error': 'warn',
+      'reasoning.overconfident_language': 'observe',
+    },
+  },
+});
+```
+
+### Baseline Regression Detection
+
+```typescript
+import { createSnapshot, saveBaseline, loadBaseline, compareToBaseline } from 'truthguard-ai';
+
+// Save after a known-good run
+const snapshot = createSnapshot(result, 'v1.2-main');
+saveBaseline('.ai-rcp-baseline.json', snapshot);
+
+// Compare after changes
+const comparison = compareToBaseline(newResult, snapshot);
+if (!comparison.withinTolerance) {
+  console.error('Regression detected:', comparison.report);
+}
 ```
 
 ### MCP Server (VS Code, Cursor)
 
-Use TruthGuard from your IDE — add to `.vscode/mcp.json`:
+Use TruthGuard directly from your IDE — no terminal needed.
+
+**Setup (one time):**
+1. In VS Code: `Ctrl+Shift+P` → **"MCP: Open User Configuration"**
+2. Add this to `mcp.json`:
 
 ```json
 {
@@ -110,11 +164,23 @@ Use TruthGuard from your IDE — add to `.vscode/mcp.json`:
 }
 ```
 
+3. Restart VS Code
+
+**Usage:** In Copilot Chat, say: *"Call truthguard verify_response with this trace: {...}"*
+
+8 tools available: `verify_response`, `quick_check`, `check_trace_quality`, `list_rules`, `get_failure_info`, `evaluate_with_policy`, `get_live_traces`, `get_trace_report`
+
+The last two tools bridge proxy results to your IDE — ask Copilot *"Call get_live_traces"* to see recent production evaluations.
+
+Full setup guide: [docs/getting-started.md](docs/getting-started.md#ide--mcp-server-vs-code-cursor)
+
 ### Express Middleware
 
 ```typescript
+import express from 'express';
 import { groundingMiddleware, FileStore } from 'truthguard-ai';
 
+const app = express();
 app.post('/api/chat', groundingMiddleware({
   mode: 'warn',
   store: new FileStore('./traces/grounding.jsonl'),
@@ -124,21 +190,68 @@ app.post('/api/chat', groundingMiddleware({
 
 ---
 
-## Key Features
+## CLI
+
+```bash
+npx truthguard-ai debug trace.json              # Evaluate one trace
+npx truthguard-ai run dataset.jsonl             # Batch dataset evaluation
+npx truthguard-ai run dataset.jsonl --gate gate.yml  # CI quality gate
+npx truthguard-ai observe --port 3001           # Start observe server + proxy
+```
+
+---
+
+## CI/CD Integration
+
+### GitHub Actions
+
+```yaml
+# .github/workflows/truthguard-gate.yml
+name: TruthGuard Quality Gate
+on: [push, pull_request]
+
+jobs:
+  grounding-gate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+      - run: npm ci
+      - run: npx truthguard-ai run test-cases.jsonl --gate .ai-rcp-gate.yml
+```
 
-- **30+ deterministic failure detectors** — fabrication from empty results, math errors, ignored tool data, entity mismatches, and more
-- **Policy engine** — per-failure enforcement (block / warn / observe)
-- **Diagnostic reports** — actionable failure reports with severity levels
-- **Baseline regression** — snapshot comparison to catch quality regressions
-- **Multi-language support** — works with responses in 13+ languages
-- **Configurable thresholds** — tune sensitivity to match your use case
+### Gate config (`.ai-rcp-gate.yml`)
+
+```yaml
+name: "Grounding Quality Gate"
+assertions:
+  - metric: grounding_score
+    operator: ">="
+    threshold: 0.90
+  - metric: failure_count
+    operator: "<="
+    threshold: 0
+  - metric: pass_rate
+    operator: ">="
+    threshold: 1.0
+```
 
 ---
 
-## Documentation
+## How It Works
 
-Full docs, guides, and architecture details: [truthguard.dev](https://truthguard.dev)
+1. **Extract** factual claims from the agent's response
+2. **Match** each claim against tool output data
+3. **Detect** failure patterns across grounding, orchestration, and reasoning
+4. **Score** overall grounding quality
+5. **Diagnose** root causes with actionable remediation
+
+**No LLM calls.** 100% deterministic. Configurable tolerances and multi-language support (13+ languages).
+
+---
 
 ## License
 
-MIT
+MIT

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "truthguard-ai",
-  "version": "0.1.4",
+  "version": "0.2.0",
   "description": "TruthGuard — Standardized grounding validation for tool-calling AI agents. Detect, diagnose, and prevent grounding failures.",
   "main": "dist-npm/thin.js",
   "types": "dist-npm/thin.d.ts",
@@ -77,10 +77,13 @@
     "@types/node": "^20.12.7",
     "@typescript-eslint/eslint-plugin": "^7.7.1",
     "@typescript-eslint/parser": "^7.7.1",
+    "bcryptjs": "^3.0.3",
     "cors": "^2.8.6",
     "eslint": "^8.57.0",
     "express": "^5.2.1",
     "jest": "^29.7.0",
+    "js-yaml": "^4.1.1",
+    "jsonwebtoken": "^9.0.3",
     "ts-jest": "^29.1.2",
     "typescript": "^5.4.5"
   },
@@ -98,11 +101,6 @@
       "!src/index.ts"
     ]
   },
-  "dependencies": {
-    "bcryptjs": "^3.0.3",
-    "js-yaml": "^4.1.1",
-    "jsonwebtoken": "^9.0.3"
-  },
   "peerDependencies": {
     "express": ">=4.0.0"
   },