truthguard-ai 0.1.3 → 0.1.4

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "truthguard-ai",
-  "version": "0.1.3",
+  "version": "0.1.4",
   "description": "TruthGuard — Standardized grounding validation for tool-calling AI agents. Detect, diagnose, and prevent grounding failures.",
   "main": "dist-npm/thin.js",
   "types": "dist-npm/thin.d.ts",

package/README.full.bak

@@ -1,363 +0,0 @@
-# TruthGuard
-
-**Standardized grounding validation for tool-calling AI agents.**
-
-> Detect when an agent's response contradicts the data returned by the tools it called — deterministically, without LLM-as-judge overhead.
-
-[![npm version](https://img.shields.io/npm/v/truthguard.svg)](https://www.npmjs.com/package/truthguard)
-[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
-
----
-
-## The Problem
-
-Most "hallucinations" in tool-calling agents are **grounding failures** — the agent calls a tool, gets accurate data, and then ignores it, miscalculates, or fabricates from empty results. The source of truth is already in the trace.
-
-## The Solution
-
-TruthGuard extracts factual claims from the agent's response, cross-references them against tool outputs, and reports grounding failures with standardized codes — like OBD diagnostic codes for AI.
-
-```
-npm install truthguard
-```
-
-**Zero LLM calls.** Deterministic regex extraction + fuzzy matching. 30 detection rules across 4 categories. Runs in <50ms.
-
----
-
-## Quick Start — 3 Minutes
-
-### 1. Evaluate a trace
-
-```typescript
-import { TraceBuilder, GroundingEngine, generateReport } from 'truthguard';
-
-const trace = new TraceBuilder({ traceId: 'run-001' })
-  .addUserInput('How many employees are on leave today?')
-  .addToolCall('getLeaveRecords', { date: '2024-03-15' })
-  .addToolOutput('getLeaveRecords', [
-    { employeeId: 'E01', name: 'Ana Jovic', status: 'on_leave' },
-    { employeeId: 'E02', name: 'Ivan Petrovic', status: 'on_leave' },
-  ])
-  .addFinalResponse('There are 3 employees on leave today.')  // ← Bug: says 3, data shows 2
-  .build();
-
-const engine = new GroundingEngine();
-const report = engine.evaluate(trace);
-
-console.log(report.groundingScore);       // 0.5
-console.log(report.detectedFailures[0]);  // { type: 'grounding.data_ignored', severity: 'high' }
-
-const { text } = generateReport(report);
-console.log(text);
-```
-
-### 2. Add a CI quality gate
-
-```typescript
-import { loadDataset, runDataset, evaluateGate, loadGateConfig } from 'truthguard';
-
-const entries = loadDataset('./test-cases.jsonl');
-const result = runDataset(entries);
-const gate = loadGateConfig('.ai-rcp-gate.yml');
-const verdict = evaluateGate(result, gate);
-
-if (!verdict.pass) {
-  console.error(verdict.report);
-  process.exit(1);
-}
-```
-
-### 3. Monitor in production (proxy mode)
-
-Works with **any language** — PHP, Python, Go, Java, Ruby, C#:
-
-```bash
-npx truthguard observe --port 3001
-```
-
-Change your AI base URL:
-```php
-// Before: ANTHROPIC_BASE_URL=https://api.anthropic.com
-// After:
-ANTHROPIC_BASE_URL=http://localhost:3001/proxy/anthropic
-```
-
-Your app works exactly the same. TruthGuard transparently proxies requests and evaluates grounding in the background.
-
----
-
-## Detection Rules (30)
-
-### Grounding (16 rules)
-
-| Code | Description |
-|------|-------------|
-| `empty_fabrication` | Tool returned `[]`, agent fabricated results |
-| `no_tool_call` | Factual question answered without calling any tool |
-| `math_error` | Incorrect calculation from correct tool data |
-| `data_ignored` | Tool data altered or ignored in response |
-| `wrong_query` | Tool called with incorrect parameters |
-| `entity_mismatch` | Agent mixed up entities from results |
-| `hallucinated_entity` | Agent invented entity not in tool data |
-| `partial_answer` | Only part of the question answered |
-| `question_not_answered` | Core question not addressed |
-| `selective_omission` | Some tool results selectively excluded |
-| `tool_error_ignored` | Tool error not handled |
-| `stale_knowledge` | Used outdated data instead of tool results |
-| `incomplete_response` | Empty or fallback response despite having data |
-| `irrelevant_context` | Used unrelated data from different context |
-| `contradictory_claims` | Response contains self-contradicting statements |
-| `unverified_value` | Factual values with no tool data to verify against |
-
-### Orchestration (8 rules)
-
-| Code | Description |
-|------|-------------|
-| `malformed_tool_input` | Bad parameter format in tool call |
-| `raw_output_leak` | XML/JSON markup leaked into response |
-| `intermediate_response_leak` | "Let me check..." text shown to user |
-| `excessive_tool_calls` | Redundant repeated tool invocations |
-| `token_limit_truncation` | Response cut off by token limit |
-| `rate_limit_degradation` | Quality degraded due to rate limiting |
-| `quota_exhaustion` | API quota exceeded |
-| `model_fallback` | Unexpected model fallback |
-
-### Reasoning (4) & Safety (2)
-
-`scope_mismatch`, `overconfident_language`, `language_mismatch`, `duplicate_user_input`, `prompt_leak`, `sensitive_data_exposure`
-
----
-
-## Features
-
-### Diagnostic Advisor
-
-Every detected failure includes root cause analysis, evidence from the trace, and two remediation paths:
-
-```typescript
-import { generateAdvisorReport, formatAdvisorReport } from 'truthguard';
-
-const advisor = generateAdvisorReport(report, trace);
-console.log(formatAdvisorReport(advisor));
-// REPAIR ORDER:
-//   1. Fix grounding.no_tool_call (root cause)
-//   2. Fix grounding.unverified_value (likely resolves after #1)
-// PROMPT HINT: "Always call the relevant tool before answering factual questions"
-// CODE GUARD:  if (!trace.hasToolCall()) return forceToolCall(query);
-```
-
-### Policy Engine
-
-Configure per-failure actions — block, warn, or observe:
-
-```typescript
-import { wrapOpenAI, GroundingError } from 'truthguard';
-import OpenAI from 'openai';
-
-const openai = wrapOpenAI(new OpenAI(), {
-  mode: 'block',
-  threshold: 0.85,
-  policy: {
-    rules: {
-      'grounding.empty_fabrication': 'block',
-      'grounding.math_error': 'warn',
-      'reasoning.overconfident_language': 'observe',
-    },
-  },
-});
-```
-
-### Baseline Regression Detection
-
-```typescript
-import { createSnapshot, saveBaseline, loadBaseline, compareToBaseline } from 'truthguard';
-
-// Save after a known-good run
-const snapshot = createSnapshot(result, 'v1.2-main');
-saveBaseline('.ai-rcp-baseline.json', snapshot);
-
-// Compare after changes
-const comparison = compareToBaseline(newResult, snapshot);
-if (!comparison.withinTolerance) {
-  console.error('Regression detected:', comparison.report);
-}
-```
-
-### MCP Server (VS Code, Cursor)
-
-Use TruthGuard directly from your IDE — no terminal needed.
-
-**Setup (one time):**
-1. In VS Code: `Ctrl+Shift+P` → **"MCP: Open User Configuration"**
-2. Add this to `mcp.json`:
-
-```json
-{
-  "servers": {
-    "truthguard": {
-      "type": "stdio",
-      "command": "npx",
-      "args": ["-y", "truthguard", "mcp"]
-    }
-  }
-}
-```
-
-3. Restart VS Code
-
-**Usage:** In Copilot Chat, say: *"Call truthguard verify_response with this trace: {...}"*
-
-8 tools available: `verify_response`, `quick_check`, `check_trace_quality`, `list_rules`, `get_failure_info`, `evaluate_with_policy`, `get_live_traces`, `get_trace_report`
-
-The last two tools bridge proxy results to your IDE — ask Copilot *"Call get_live_traces"* to see recent production evaluations.
-
-Full setup guide: [docs/getting-started.md](docs/getting-started.md#ide--mcp-server-vs-code-cursor)
-
-### Express Middleware
-
-```typescript
-import express from 'express';
-import { groundingMiddleware, FileStore } from 'truthguard';
-
-const app = express();
-app.post('/api/chat', groundingMiddleware({
-  mode: 'warn',
-  store: new FileStore('./traces/grounding.jsonl'),
-  extractTrace: (req, res, body) => body.trace,
-}));
-```
-
----
-
-## CLI
-
-```bash
-npx truthguard debug trace.json              # Evaluate one trace
-npx truthguard run dataset.jsonl             # Batch dataset evaluation
-npx truthguard run dataset.jsonl --gate gate.yml  # CI quality gate
-npx truthguard observe --port 3001           # Start observe server + proxy
-```
-
----
-
-## CI/CD Integration
-
-### GitHub Actions
-
-```yaml
-# .github/workflows/truthguard-gate.yml
-name: TruthGuard Quality Gate
-on: [push, pull_request]
-
-jobs:
-  grounding-gate:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-      - uses: actions/setup-node@v4
-        with:
-          node-version: '20'
-      - run: npm ci
-      - run: npx truthguard run test-cases.jsonl --gate .ai-rcp-gate.yml
-```
-
-### Gate config (`.ai-rcp-gate.yml`)
-
-```yaml
-name: "Grounding Quality Gate"
-assertions:
-  - metric: grounding_score
-    operator: ">="
-    threshold: 0.90
-  - metric: failure_count
-    operator: "<="
-    threshold: 0
-  - metric: pass_rate
-    operator: ">="
-    threshold: 1.0
-```
-
----
-
-## How It Works
-
-```
-Agent Response → Claim Extraction → Matcher → Rules → Report
-                  (regex: numbers,   (numeric,  (30      (score,
-                   dates, names,      count,     rules)   failures,
-                   counts)            date,               advisor)
-                                      name)
-```
-
-1. **Extract** factual claims from the agent's text response (numbers, dates, names, counts)
-2. **Match** each claim against values in tool outputs (with configurable tolerances)
-3. **Detect** failure patterns using 30 rules across 4 categories
-4. **Score** grounding quality with severity-weighted formula
-5. **Diagnose** root causes and suggest repair sequence
-
-**No LLM calls.** 100% deterministic. ~55% claim coverage (numbers, dates, names, counts). L2 structured matching (booleans, enums, key-values) extends to ~70-75%.
-
----
-
-## Configurable Tolerances
-
-```yaml
-# .ai-rcp.yml
-tolerances:
-  numeric:
-    relative_tolerance: 0.05    # ±5% for numbers
-    rounding_allowed: true
-  count:
-    exact_match: true
-  date:
-    exact_match: true
-  name:
-    fuzzy_match: true
-    threshold: 0.85             # Jaro-Winkler similarity
-```
-
----
-
-## Language Support
-
-- **Claim extraction:** Numbers, dates (7 formats incl. European DD.MM.YYYY), Serbian months (januar–decembar), relative dates (yesterday/juče, pre N dana)
-- **Unit conversion:** 13 languages (EN, SR, ES, FR, PT, RU, HI, AR, BN, ZH, JA...)
-- **Vague qualifier guard:** English + Serbian (oko, otprilike, negde)
-- **Name matching:** Diacritics-aware (ć→c, š→s) via Jaro-Winkler
-
----
-
-## Architecture
-
-```
-src/
-├── Trace/        TraceBuilder SDK + multi-turn support
-├── Claims/       Claim extraction (regex, multilingual)
-├── Matchers/     Numeric, count, date, name matchers
-├── Rules/        30 detection rules (4 categories)
-├── Grounding/    Engine orchestration + entity-aware grounding
-├── Advisor/      Diagnostic advisor (RCA, repair sequence, hints)
-├── Registry/     Failure registry (severity, suppression graph)
-├── Policy/       Per-failure enforcement (block/warn/observe)
-├── Reports/      JSON + text report generators
-├── Config/       YAML tolerance configuration
-├── Gate/         CI/CD quality gate
-├── Baseline/     Snapshot regression detection
-├── Runner/       JSONL dataset batch evaluation
-├── Mode/         Pipeline (debug/ci/observe/warn/block)
-├── Store/        FileStore + InMemoryStore
-├── Alerting/     Console, Webhook, Callback dispatchers
-├── Middleware/    Express middleware factory
-├── SDK/          OpenAI wrapper (auto trace capture)
-├── Proxy/        Transparent AI API proxy builders
-├── MCP/          MCP Server (8 IDE tools)
-├── L2/           Structured context matching (boolean, enum, key-value)
-└── cli/          CLI commands
-```
-
----
-
-## License
-
-MIT