truthguard-ai 0.1.3 → 0.1.5

package/README.md

@@ -1,8 +1,8 @@
 # TruthGuard
 
-**Grounding validation for tool-calling AI agents.**
+**Standardized grounding validation for tool-calling AI agents.**
 
-> Detect when an agent's response contradicts the data returned by the tools it called — without LLM-as-judge overhead.
+> Detect when an agent's response contradicts the data returned by the tools it called — deterministically, without LLM-as-judge overhead.
 
 [![npm version](https://img.shields.io/npm/v/truthguard-ai.svg)](https://www.npmjs.com/package/truthguard-ai)
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
@@ -11,24 +11,26 @@
 
 ## The Problem
 
-Most "hallucinations" in tool-calling agents are **grounding failures** — the agent calls a tool, gets accurate data, then ignores it, miscalculates, or fabricates from empty results.
+Most "hallucinations" in tool-calling agents are **grounding failures** — the agent calls a tool, gets accurate data, and then ignores it, miscalculates, or fabricates from empty results. The source of truth is already in the trace.
 
 ## The Solution
 
-TruthGuard extracts factual claims from the agent's response and cross-references them against tool outputs. Deterministic. No LLM calls. Runs in <50ms.
+TruthGuard extracts factual claims from the agent's response, cross-references them against tool outputs, and reports grounding failures with standardized codes — like OBD diagnostic codes for AI.
 
-```bash
-npm install truthguard-ai
+```
+npm install truthguard
 ```
 
+**Zero LLM calls.** 30+ deterministic failure detectors. Runs in <50ms.
+
 ---
 
-## Quick Start
+## Quick Start — 3 Minutes
 
-### Evaluate a trace
+### 1. Evaluate a trace
 
 ```typescript
-import { TraceBuilder, GroundingEngine, generateReport } from 'truthguard-ai';
+import { TraceBuilder, GroundingEngine, generateReport } from 'truthguard';
 
 const trace = new TraceBuilder({ traceId: 'run-001' })
   .addUserInput('How many employees are on leave today?')
@@ -37,7 +39,7 @@ const trace = new TraceBuilder({ traceId: 'run-001' })
     { employeeId: 'E01', name: 'Ana Jovic', status: 'on_leave' },
     { employeeId: 'E02', name: 'Ivan Petrovic', status: 'on_leave' },
   ])
-  .addFinalResponse('There are 3 employees on leave today.')
+  .addFinalResponse('There are 3 employees on leave today.')  // ← Bug: says 3, data shows 2
   .build();
 
 const engine = new GroundingEngine();
@@ -45,39 +47,36 @@ const report = engine.evaluate(trace);
 
 console.log(report.groundingScore);       // 0.5
 console.log(report.detectedFailures[0]);  // { type: 'grounding.data_ignored', severity: 'high' }
+
+const { text } = generateReport(report);
+console.log(text);
 ```
 
-### SDK Wrappers (OpenAI & Anthropic)
+### 2. Add a CI quality gate
 
 ```typescript
-import { wrapOpenAI } from 'truthguard-ai';
-import OpenAI from 'openai';
+import { loadDataset, runDataset, evaluateGate, loadGateConfig } from 'truthguard';
 
-const openai = wrapOpenAI(new OpenAI(), {
-  mode: 'block',
-  threshold: 0.85,
-});
-```
-
-```typescript
-import { wrapAnthropic } from 'truthguard-ai';
-import Anthropic from '@anthropic-ai/sdk';
+const entries = loadDataset('./test-cases.jsonl');
+const result = runDataset(entries);
+const gate = loadGateConfig('.ai-rcp-gate.yml');
+const verdict = evaluateGate(result, gate);
 
-const anthropic = wrapAnthropic(new Anthropic(), {
-  mode: 'warn',
-  threshold: 0.80,
-});
+if (!verdict.pass) {
+  console.error(verdict.report);
+  process.exit(1);
+}
 ```
 
-### Production Monitoring (Proxy Mode)
+### 3. Monitor in production (proxy mode)
 
 Works with **any language** — PHP, Python, Go, Java, Ruby, C#:
 
 ```bash
-npx truthguard-ai observe --port 3001
+npx truthguard observe --port 3001
 ```
 
-Point your AI SDK base URL to the proxy:
+Change your AI base URL:
 ```
 # OpenAI
 OPENAI_BASE_URL=http://localhost:3001/proxy/openai
@@ -86,17 +85,72 @@ OPENAI_BASE_URL=http://localhost:3001/proxy/openai
 ANTHROPIC_BASE_URL=http://localhost:3001/proxy/anthropic
 ```
 
-Your app works identically. TruthGuard evaluates grounding in the background.
+Your app works exactly the same. TruthGuard transparently proxies requests and evaluates grounding in the background.
 
-### CI Quality Gate
+---
 
-```bash
-npx truthguard-ai run test-cases.jsonl --gate .ai-rcp-gate.yml
+## Detection
+
+30+ deterministic failure detectors across grounding, orchestration, and reasoning categories.
+
+Examples:
+- Fabrication from empty tool results
+- Math errors against correct tool data
+- Ignored or altered tool data in the response
+- Entity mismatches and mix-ups
+
+---
+
+## Features
+
+### Diagnostic Advisor
+
+Every detected failure includes actionable diagnostics — root cause identification, evidence, and remediation guidance.
+
+### Policy Engine
+
+Configure per-failure actions — block, warn, or observe:
+
+```typescript
+import { wrapOpenAI, GroundingError } from 'truthguard';
+import OpenAI from 'openai';
+
+const openai = wrapOpenAI(new OpenAI(), {
+  mode: 'block',
+  threshold: 0.85,
+  policy: {
+    rules: {
+      'grounding.empty_fabrication': 'block',
+      'grounding.math_error': 'warn',
+      'reasoning.overconfident_language': 'observe',
+    },
+  },
+});
+```
+
+### Baseline Regression Detection
+
+```typescript
+import { createSnapshot, saveBaseline, loadBaseline, compareToBaseline } from 'truthguard';
+
+// Save after a known-good run
+const snapshot = createSnapshot(result, 'v1.2-main');
+saveBaseline('.ai-rcp-baseline.json', snapshot);
+
+// Compare after changes
+const comparison = compareToBaseline(newResult, snapshot);
+if (!comparison.withinTolerance) {
+  console.error('Regression detected:', comparison.report);
+}
 ```
 
 ### MCP Server (VS Code, Cursor)
 
-Use TruthGuard from your IDE — add to `.vscode/mcp.json`:
+Use TruthGuard directly from your IDE — no terminal needed.
+
+**Setup (one time):**
+1. In VS Code: `Ctrl+Shift+P` → **"MCP: Open User Configuration"**
+2. Add this to `mcp.json`:
 
 ```json
 {
@@ -104,17 +158,29 @@ Use TruthGuard from your IDE — add to `.vscode/mcp.json`:
     "truthguard": {
       "type": "stdio",
       "command": "npx",
-      "args": ["-y", "truthguard-ai", "mcp"]
+      "args": ["-y", "truthguard", "mcp"]
     }
   }
 }
 ```
 
+3. Restart VS Code
+
+**Usage:** In Copilot Chat, say: *"Call truthguard verify_response with this trace: {...}"*
+
+8 tools available: `verify_response`, `quick_check`, `check_trace_quality`, `list_rules`, `get_failure_info`, `evaluate_with_policy`, `get_live_traces`, `get_trace_report`
+
+The last two tools bridge proxy results to your IDE — ask Copilot *"Call get_live_traces"* to see recent production evaluations.
+
+Full setup guide: [docs/getting-started.md](docs/getting-started.md#ide--mcp-server-vs-code-cursor)
+
 ### Express Middleware
 
 ```typescript
-import { groundingMiddleware, FileStore } from 'truthguard-ai';
+import express from 'express';
+import { groundingMiddleware, FileStore } from 'truthguard';
 
+const app = express();
 app.post('/api/chat', groundingMiddleware({
   mode: 'warn',
   store: new FileStore('./traces/grounding.jsonl'),
@@ -124,21 +190,68 @@ app.post('/api/chat', groundingMiddleware({
 
 ---
 
-## Key Features
+## CLI
+
+```bash
+npx truthguard debug trace.json              # Evaluate one trace
+npx truthguard run dataset.jsonl             # Batch dataset evaluation
+npx truthguard run dataset.jsonl --gate gate.yml  # CI quality gate
+npx truthguard observe --port 3001           # Start observe server + proxy
+```
+
+---
+
+## CI/CD Integration
+
+### GitHub Actions
+
+```yaml
+# .github/workflows/truthguard-gate.yml
+name: TruthGuard Quality Gate
+on: [push, pull_request]
+
+jobs:
+  grounding-gate:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+      - run: npm ci
+      - run: npx truthguard run test-cases.jsonl --gate .ai-rcp-gate.yml
+```
 
-- **30+ deterministic failure detectors** — fabrication from empty results, math errors, ignored tool data, entity mismatches, and more
-- **Policy engine** — per-failure enforcement (block / warn / observe)
-- **Diagnostic reports** — actionable failure reports with severity levels
-- **Baseline regression** — snapshot comparison to catch quality regressions
-- **Multi-language support** — works with responses in 13+ languages
-- **Configurable thresholds** — tune sensitivity to match your use case
+### Gate config (`.ai-rcp-gate.yml`)
+
+```yaml
+name: "Grounding Quality Gate"
+assertions:
+  - metric: grounding_score
+    operator: ">="
+    threshold: 0.90
+  - metric: failure_count
+    operator: "<="
+    threshold: 0
+  - metric: pass_rate
+    operator: ">="
+    threshold: 1.0
+```
 
 ---
 
-## Documentation
+## How It Works
 
-Full docs, guides, and architecture details: [truthguard.dev](https://truthguard.dev)
+1. **Extract** factual claims from the agent's response
+2. **Match** each claim against tool output data
+3. **Detect** failure patterns across grounding, orchestration, and reasoning
+4. **Score** overall grounding quality
+5. **Diagnose** root causes with actionable remediation
+
+**No LLM calls.** 100% deterministic. Configurable tolerances and multi-language support (13+ languages).
+
+---
 
 ## License
 
-MIT
+MIT

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "truthguard-ai",
-  "version": "0.1.3",
+  "version": "0.1.5",
   "description": "TruthGuard — Standardized grounding validation for tool-calling AI agents. Detect, diagnose, and prevent grounding failures.",
   "main": "dist-npm/thin.js",
   "types": "dist-npm/thin.d.ts",

package/README.full.bak

@@ -1,363 +0,0 @@
-# TruthGuard
-
-**Standardized grounding validation for tool-calling AI agents.**
-
-> Detect when an agent's response contradicts the data returned by the tools it called — deterministically, without LLM-as-judge overhead.
-
-[![npm version](https://img.shields.io/npm/v/truthguard.svg)](https://www.npmjs.com/package/truthguard)
-[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
-
----
-
-## The Problem
-
-Most "hallucinations" in tool-calling agents are **grounding failures** — the agent calls a tool, gets accurate data, and then ignores it, miscalculates, or fabricates from empty results. The source of truth is already in the trace.
-
-## The Solution
-
-TruthGuard extracts factual claims from the agent's response, cross-references them against tool outputs, and reports grounding failures with standardized codes — like OBD diagnostic codes for AI.
-
-```
-npm install truthguard
-```
-
-**Zero LLM calls.** Deterministic regex extraction + fuzzy matching. 30 detection rules across 4 categories. Runs in <50ms.
-
----
-
-## Quick Start — 3 Minutes
-
-### 1. Evaluate a trace
-
-```typescript
-import { TraceBuilder, GroundingEngine, generateReport } from 'truthguard';
-
-const trace = new TraceBuilder({ traceId: 'run-001' })
-  .addUserInput('How many employees are on leave today?')
-  .addToolCall('getLeaveRecords', { date: '2024-03-15' })
-  .addToolOutput('getLeaveRecords', [
-    { employeeId: 'E01', name: 'Ana Jovic', status: 'on_leave' },
-    { employeeId: 'E02', name: 'Ivan Petrovic', status: 'on_leave' },
-  ])
-  .addFinalResponse('There are 3 employees on leave today.')  // ← Bug: says 3, data shows 2
-  .build();
-
-const engine = new GroundingEngine();
-const report = engine.evaluate(trace);
-
-console.log(report.groundingScore);       // 0.5
-console.log(report.detectedFailures[0]);  // { type: 'grounding.data_ignored', severity: 'high' }
-
-const { text } = generateReport(report);
-console.log(text);
-```
-
-### 2. Add a CI quality gate
-
-```typescript
-import { loadDataset, runDataset, evaluateGate, loadGateConfig } from 'truthguard';
-
-const entries = loadDataset('./test-cases.jsonl');
-const result = runDataset(entries);
-const gate = loadGateConfig('.ai-rcp-gate.yml');
-const verdict = evaluateGate(result, gate);
-
-if (!verdict.pass) {
-  console.error(verdict.report);
-  process.exit(1);
-}
-```
-
-### 3. Monitor in production (proxy mode)
-
-Works with **any language** — PHP, Python, Go, Java, Ruby, C#:
-
-```bash
-npx truthguard observe --port 3001
-```
-
-Change your AI base URL:
-```php
-// Before: ANTHROPIC_BASE_URL=https://api.anthropic.com
-// After:
-ANTHROPIC_BASE_URL=http://localhost:3001/proxy/anthropic
-```
-
-Your app works exactly the same. TruthGuard transparently proxies requests and evaluates grounding in the background.
-
----
-
-## Detection Rules (30)
-
-### Grounding (16 rules)
-
-| Code | Description |
-|------|-------------|
-| `empty_fabrication` | Tool returned `[]`, agent fabricated results |
-| `no_tool_call` | Factual question answered without calling any tool |
-| `math_error` | Incorrect calculation from correct tool data |
-| `data_ignored` | Tool data altered or ignored in response |
-| `wrong_query` | Tool called with incorrect parameters |
-| `entity_mismatch` | Agent mixed up entities from results |
-| `hallucinated_entity` | Agent invented entity not in tool data |
-| `partial_answer` | Only part of the question answered |
-| `question_not_answered` | Core question not addressed |
-| `selective_omission` | Some tool results selectively excluded |
-| `tool_error_ignored` | Tool error not handled |
-| `stale_knowledge` | Used outdated data instead of tool results |
-| `incomplete_response` | Empty or fallback response despite having data |
-| `irrelevant_context` | Used unrelated data from different context |
-| `contradictory_claims` | Response contains self-contradicting statements |
-| `unverified_value` | Factual values with no tool data to verify against |
-
-### Orchestration (8 rules)
-
-| Code | Description |
-|------|-------------|
-| `malformed_tool_input` | Bad parameter format in tool call |
-| `raw_output_leak` | XML/JSON markup leaked into response |
-| `intermediate_response_leak` | "Let me check..." text shown to user |
-| `excessive_tool_calls` | Redundant repeated tool invocations |
-| `token_limit_truncation` | Response cut off by token limit |
-| `rate_limit_degradation` | Quality degraded due to rate limiting |
-| `quota_exhaustion` | API quota exceeded |
-| `model_fallback` | Unexpected model fallback |
-
-### Reasoning (4) & Safety (2)
-
-`scope_mismatch`, `overconfident_language`, `language_mismatch`, `duplicate_user_input`, `prompt_leak`, `sensitive_data_exposure`
-
----
-
-## Features
-
-### Diagnostic Advisor
-
-Every detected failure includes root cause analysis, evidence from the trace, and two remediation paths:
-
-```typescript
-import { generateAdvisorReport, formatAdvisorReport } from 'truthguard';
-
-const advisor = generateAdvisorReport(report, trace);
-console.log(formatAdvisorReport(advisor));
-// REPAIR ORDER:
-//   1. Fix grounding.no_tool_call (root cause)
-//   2. Fix grounding.unverified_value (likely resolves after #1)
-// PROMPT HINT: "Always call the relevant tool before answering factual questions"
-// CODE GUARD:  if (!trace.hasToolCall()) return forceToolCall(query);
-```
-
-### Policy Engine
-
-Configure per-failure actions — block, warn, or observe:
-
-```typescript
-import { wrapOpenAI, GroundingError } from 'truthguard';
-import OpenAI from 'openai';
-
-const openai = wrapOpenAI(new OpenAI(), {
-  mode: 'block',
-  threshold: 0.85,
-  policy: {
-    rules: {
-      'grounding.empty_fabrication': 'block',
-      'grounding.math_error': 'warn',
-      'reasoning.overconfident_language': 'observe',
-    },
-  },
-});
-```
-
-### Baseline Regression Detection
-
-```typescript
-import { createSnapshot, saveBaseline, loadBaseline, compareToBaseline } from 'truthguard';
-
-// Save after a known-good run
-const snapshot = createSnapshot(result, 'v1.2-main');
-saveBaseline('.ai-rcp-baseline.json', snapshot);
-
-// Compare after changes
-const comparison = compareToBaseline(newResult, snapshot);
-if (!comparison.withinTolerance) {
-  console.error('Regression detected:', comparison.report);
-}
-```
-
-### MCP Server (VS Code, Cursor)
-
-Use TruthGuard directly from your IDE — no terminal needed.
-
-**Setup (one time):**
-1. In VS Code: `Ctrl+Shift+P` → **"MCP: Open User Configuration"**
-2. Add this to `mcp.json`:
-
-```json
-{
-  "servers": {
-    "truthguard": {
-      "type": "stdio",
-      "command": "npx",
-      "args": ["-y", "truthguard", "mcp"]
-    }
-  }
-}
-```
-
-3. Restart VS Code
-
-**Usage:** In Copilot Chat, say: *"Call truthguard verify_response with this trace: {...}"*
-
-8 tools available: `verify_response`, `quick_check`, `check_trace_quality`, `list_rules`, `get_failure_info`, `evaluate_with_policy`, `get_live_traces`, `get_trace_report`
-
-The last two tools bridge proxy results to your IDE — ask Copilot *"Call get_live_traces"* to see recent production evaluations.
-
-Full setup guide: [docs/getting-started.md](docs/getting-started.md#ide--mcp-server-vs-code-cursor)
-
-### Express Middleware
-
-```typescript
-import express from 'express';
-import { groundingMiddleware, FileStore } from 'truthguard';
-
-const app = express();
-app.post('/api/chat', groundingMiddleware({
-  mode: 'warn',
-  store: new FileStore('./traces/grounding.jsonl'),
-  extractTrace: (req, res, body) => body.trace,
-}));
-```
-
----
-
-## CLI
-
-```bash
-npx truthguard debug trace.json              # Evaluate one trace
-npx truthguard run dataset.jsonl             # Batch dataset evaluation
-npx truthguard run dataset.jsonl --gate gate.yml  # CI quality gate
-npx truthguard observe --port 3001           # Start observe server + proxy
-```
-
----
-
-## CI/CD Integration
-
-### GitHub Actions
-
-```yaml
-# .github/workflows/truthguard-gate.yml
-name: TruthGuard Quality Gate
-on: [push, pull_request]
-
-jobs:
-  grounding-gate:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
-        with:
-          node-version: '20'
-      - run: npm ci
-      - run: npx truthguard run test-cases.jsonl --gate .ai-rcp-gate.yml
-```
-
-### Gate config (`.ai-rcp-gate.yml`)
-
-```yaml
-name: "Grounding Quality Gate"
-assertions:
-  - metric: grounding_score
-    operator: ">="
-    threshold: 0.90
-  - metric: failure_count
-    operator: "<="
-    threshold: 0
-  - metric: pass_rate
-    operator: ">="
-    threshold: 1.0
-```
-
----
-
-## How It Works
-
-```
-Agent Response → Claim Extraction → Matcher → Rules → Report
-                  (regex: numbers,   (numeric,  (30      (score,
-                   dates, names,      count,     rules)   failures,
-                   counts)            date,               advisor)
-                                      name)
-```
-
-1. **Extract** factual claims from the agent's text response (numbers, dates, names, counts)
-2. **Match** each claim against values in tool outputs (with configurable tolerances)
-3. **Detect** failure patterns using 30 rules across 4 categories
-4. **Score** grounding quality with severity-weighted formula
-5. **Diagnose** root causes and suggest repair sequence
-
-**No LLM calls.** 100% deterministic. ~55% claim coverage (numbers, dates, names, counts). L2 structured matching (booleans, enums, key-values) extends to ~70-75%.
-
----
-
-## Configurable Tolerances
-
-```yaml
-# .ai-rcp.yml
-tolerances:
-  numeric:
-    relative_tolerance: 0.05    # ±5% for numbers
-    rounding_allowed: true
-  count:
-    exact_match: true
-  date:
-    exact_match: true
-  name:
-    fuzzy_match: true
-    threshold: 0.85             # Jaro-Winkler similarity
-```
-
----
-
-## Language Support
-
-- **Claim extraction:** Numbers, dates (7 formats incl. European DD.MM.YYYY), Serbian months (januar–decembar), relative dates (yesterday/juče, pre N dana)
-- **Unit conversion:** 13 languages (EN, SR, ES, FR, PT, RU, HI, AR, BN, ZH, JA...)
-- **Vague qualifier guard:** English + Serbian (oko, otprilike, negde)
-- **Name matching:** Diacritics-aware (ć→c, š→s) via Jaro-Winkler
-
----
-
-## Architecture
-
-```
-src/
-├── Trace/        TraceBuilder SDK + multi-turn support
-├── Claims/       Claim extraction (regex, multilingual)
-├── Matchers/     Numeric, count, date, name matchers
-├── Rules/        30 detection rules (4 categories)
-├── Grounding/    Engine orchestration + entity-aware grounding
-├── Advisor/      Diagnostic advisor (RCA, repair sequence, hints)
-├── Registry/     Failure registry (severity, suppression graph)
-├── Policy/       Per-failure enforcement (block/warn/observe)
-├── Reports/      JSON + text report generators
-├── Config/       YAML tolerance configuration
-├── Gate/         CI/CD quality gate
-├── Baseline/     Snapshot regression detection
-├── Runner/       JSONL dataset batch evaluation
-├── Mode/         Pipeline (debug/ci/observe/warn/block)
-├── Store/        FileStore + InMemoryStore
-├── Alerting/     Console, Webhook, Callback dispatchers
-├── Middleware/    Express middleware factory
-├── SDK/          OpenAI wrapper (auto trace capture)
-├── Proxy/        Transparent AI API proxy builders
-├── MCP/          MCP Server (8 IDE tools)
-├── L2/           Structured context matching (boolean, enum, key-value)
-└── cli/          CLI commands
-```
-
----
-
-## License
-
-MIT