llmverify 1.0.2 → 1.4.0

package/CHANGELOG.md

@@ -5,6 +5,139 @@ All notable changes to llmverify will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.4.0] - 2024-12-04
+
+### Added - Enterprise Features
+
+**Enhanced Error Handling:**
+- 20+ standardized error codes (LLMVERIFY_1001 format)
+- Error severity levels (low, medium, high, critical)
+- Error metadata with actionable suggestions
+- Recoverable/non-recoverable classification
+- JSON serialization support
+
+**Logging & Audit System:**
+- Structured logging to `~/.llmverify/logs/*.jsonl`
+- Request ID tracking with UUID
+- Automatic PII sanitization in logs
+- Log rotation (10MB max, keep 10 files)
+- Audit trail to `~/.llmverify/audit/*.jsonl`
+- SHA-256 content hashing
+- Compliance-ready audit exports
+- Log statistics & analytics
+
+**Baseline Drift Detection:**
+- Baseline metrics storage (`~/.llmverify/baseline/baseline.json`)
+- Running averages for latency, content length, risk score
+- Risk distribution tracking
+- Engine score tracking
+- Drift detection with 20% threshold
+- Drift history tracking
+- CLI commands: `baseline:stats`, `baseline:reset`, `baseline:drift`
+
+**Plugin System:**
+- Extensible rule system for custom verification
+- Plugin registry with enable/disable
+- Priority-based execution
+- Category-based filtering
+- Built-in helpers: blacklist, regex, length validator, keyword detector
+- `use()` API for plugin registration
+
+**Security Hardening:**
+- Input validation with size limits
+- Safe regex execution with timeout protection
+- PII sanitization utilities
+- Rate limiter class
+- XSS prevention (HTML escaping)
+- Injection detection
+- URL validation
+
+### Changed
+- `verify()` now integrates logging, audit, baseline tracking, and plugins
+- Enhanced input validation with better error messages
+- Improved error handling throughout codebase
+
+### API Additions
+- `ErrorCode`, `ErrorSeverity`, `getErrorMetadata()`
+- `Logger`, `getLogger()`, `LogLevel`
+- `AuditLogger`, `getAuditLogger()`
+- `BaselineStorage`, `getBaselineStorage()`
+- `Plugin`, `use()`, `createPlugin()`
+- `RateLimiter`, `sanitizeForLogging()`, `safeRegexTest()`
+
+### Documentation
+- Complete implementation of enterprise features
+- All APIs exported and documented
+- CLI commands for baseline management
+
+## [1.3.1] - 2024-12-04
+
+### Added
+- **Complete API Reference Documentation** (`docs/API-REFERENCE.md`)
+  - Comprehensive programmatic API documentation
+  - All functions with parameters, return types, and examples
+  - TypeScript type definitions
+  - Best practices and error handling
+- **JSON Schema for verify() Output** (`schema/verify-result.schema.json`)
+  - Formal JSON Schema (draft-07) for VerifyResult
+  - Complete type definitions and validation rules
+  - Example outputs for reference
+  - Machine-readable schema for validation tools
+- **Enhanced Documentation**
+  - Added schema directory to npm package
+  - Improved API discoverability
+
+### Changed
+- Package now includes `schema/` directory in published files
+- Enhanced type safety with formal JSON schema
+
+### Documentation
+- Complete API reference with all functions documented
+- JSON schema for programmatic validation
+- TypeScript type definitions reference
+- Best practices guide
+
+## [1.3.0] - 2024-12-04
+
+### Added
+- **HTTP Server Mode**: New `llmverify-serve` command starts a long-running HTTP API server
+  - Default port 9009, configurable via `--port` flag
+  - RESTful endpoints: `/verify`, `/check-input`, `/check-pii`, `/classify`, `/health`
+  - Full CORS support for local development
+  - Graceful shutdown handling
+- **IDE Integration**: Comprehensive guide for Windsurf, Cursor, VS Code, and custom IDEs
+  - Example code for TypeScript, JavaScript, Python
+  - System prompt templates for AI assistants
+  - Production deployment guidelines
+- **Server Endpoints**:
+  - `POST /verify` - Main verification endpoint (accepts `text` or `content`)
+  - `POST /check-input` - Input safety check for prompt injection
+  - `POST /check-pii` - PII detection and redaction
+  - `POST /classify` - Output classification with intent and hallucination risk
+  - `GET /health` - Health check with version info
+  - `GET /` - API documentation endpoint
+- **Enhanced CLI**:
+  - Improved `--output json` mode for scripting
+  - Better error messages and validation
+  - Exit codes for CI/CD integration (0=low, 1=moderate, 2=high/critical)
+
+### Changed
+- Updated package.json to include Express.js dependency
+- Added `bin/llmverify-serve.js` executable
+- Enhanced README with server mode documentation and IDE integration examples
+- Improved API response format with consistent structure across all endpoints
+
+### Fixed
+- CLI now properly handles `--file` and `--json` flags
+- Better error handling for missing or invalid input
+
+### Documentation
+- Added comprehensive server mode section to README
+- Added IDE integration guide with examples for multiple languages
+- Added production deployment best practices
+- Added API response format documentation
+- Updated CLI usage examples
+
 ## [1.0.0] - 2025-12-02
 
 ### Added

package/README.md

@@ -110,6 +110,339 @@ console.log(response.llmverify.health); // 'stable' | 'degraded' | 'unstable'
 
 ---
 
+## 🆕 Enterprise Features (v1.4.0)
+
+**NEW in v1.4.0**: Production-grade monitoring, logging, and extensibility.
+
+### Enhanced Error Handling
+```typescript
+import { verify, ErrorCode } from 'llmverify';
+
+try {
+  const result = await verify({ content });
+} catch (error) {
+  console.log(error.code); // LLMVERIFY_1003
+  console.log(error.metadata.suggestion); // Actionable fix
+}
+```
+
+### Logging & Audit Trails
+```typescript
+import { getLogger, getAuditLogger } from 'llmverify';
+
+const logger = getLogger({ level: 'info' });
+const requestId = logger.startRequest();
+logger.info('Processing request', { userId: '123' });
+
+// Compliance-ready audit trail
+const auditLogger = getAuditLogger();
+// Automatically logs all verifications to ~/.llmverify/audit/
+```
+
+### Baseline Drift Detection
+```typescript
+import { getBaselineStorage } from 'llmverify';
+
+const storage = getBaselineStorage();
+const stats = storage.getStatistics();
+console.log(`Baseline: ${stats.sampleCount} samples`);
+
+// Automatic drift detection (20% threshold)
+// CLI: npx llmverify baseline:stats
+```
+
+### Plugin System
+```typescript
+import { use, createPlugin } from 'llmverify';
+
+const customRule = createPlugin({
+  id: 'my-rule',
+  name: 'Custom Verification Rule',
+  execute: async (context) => ({
+    findings: [],
+    score: 0
+  })
+});
+
+use(customRule);
+// Now all verify() calls include your custom rule
+```
+
+### Security Utilities
+```typescript
+import { RateLimiter, sanitizeForLogging, safeRegexTest } from 'llmverify';
+
+const limiter = new RateLimiter(100, 60000); // 100 req/min
+if (!limiter.isAllowed(userId)) {
+  throw new Error('Rate limit exceeded');
+}
+
+// PII-safe logging
+const safe = sanitizeForLogging(content); // Removes emails, phones, SSNs
+```
+
+**See [CHANGELOG.md](CHANGELOG.md) for complete v1.4.0 feature list.**
+
+---
+
+## 📛 Show Your Badge
+
+Display the "Built with llmverify" badge on your project to show you're using AI verification!
+
+### Generate Your Badge
+
+```bash
+npx llmverify badge --name "My Project" --url "https://myproject.com"
+```
+
+### Add to Your README
+
+```markdown
+[![Built with llmverify](https://img.shields.io/badge/Built_with-llmverify-blue)](https://github.com/subodhkc/llmverify-npm)
+```
+
+### Programmatic Badge Generation
+
+```typescript
+import { generateBadgeForProject } from 'llmverify';
+
+const { markdown, html } = generateBadgeForProject('My Project', 'https://myproject.com');
+console.log(markdown); // Copy to README.md
+```
+
+**Badge Features:**
+- ✅ Verified signature for authenticity
+- ✅ Markdown and HTML formats
+- ✅ Customizable project name and URL
+- ✅ Downloadable badge image in `/assets/badge.svg`
+
+---
+
+## Server Mode — Run llmverify in Your IDE
+
+**NEW in v1.3.0**: Start a long-running HTTP server for seamless IDE integration.
+
+### Quick Start
+
+```bash
+# Start the server (default port 9009)
+npx llmverify-serve
+
+# Or specify a custom port
+npx llmverify-serve --port=8080
+```
+
+The server will start at `http://localhost:9009` with the following endpoints:
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/health` | GET | Health check |
+| `/verify` | POST | Verify AI output (main endpoint) |
+| `/check-input` | POST | Check input for prompt injection |
+| `/check-pii` | POST | Detect and redact PII |
+| `/classify` | POST | Classify output intent and hallucination risk |
+
+### API Usage Examples
+
+#### Verify AI Output
+
+```bash
+curl -X POST http://localhost:9009/verify \
+  -H "Content-Type: application/json" \
+  -d '{"text": "Your AI output here"}'
+```
+
+```javascript
+// Node.js / JavaScript
+const response = await fetch('http://localhost:9009/verify', {
+  method: 'POST',
+  headers: { 'Content-Type': 'application/json' },
+  body: JSON.stringify({ text: 'Your AI output here' })
+});
+const result = await response.json();
+console.log(result.result.risk.level); // "low" | "moderate" | "high" | "critical"
+```
+
+```python
+# Python
+import requests
+
+response = requests.post('http://localhost:9009/verify', json={
+    'text': 'Your AI output here'
+})
+result = response.json()
+print(result['result']['risk']['level'])
+```
+
+#### Check Input Safety
+
+```bash
+curl -X POST http://localhost:9009/check-input \
+  -H "Content-Type: application/json" \
+  -d '{"text": "User input to check"}'
+```
+
+#### Detect PII
+
+```bash
+curl -X POST http://localhost:9009/check-pii \
+  -H "Content-Type: application/json" \
+  -d '{"text": "Contact me at john@example.com"}'
+```
+
+### IDE Integration Guide
+
+#### Windsurf / Cursor / VS Code
+
+Add this to your AI assistant's system prompt or rules:
+
+```markdown
+## AI Output Verification
+
+Before responding to the user, verify your output using the local llmverify server:
+
+1. Start the server: `npx llmverify-serve` (runs on http://localhost:9009)
+2. POST your response to http://localhost:9009/verify
+3. Check the risk level in the response
+4. If risk is "high" or "critical", revise your response
+
+Example verification code:
+```javascript
+const verifyOutput = async (text) => {
+  const response = await fetch('http://localhost:9009/verify', {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify({ text })
+  });
+  return await response.json();
+};
+
+// Before sending response to user
+const verification = await verifyOutput(myResponse);
+if (verification.result.risk.level === 'critical') {
+  // Revise response or alert user
+}
+```
+```
+
+#### Custom IDE Integration
+
+```typescript
+// Add to your IDE extension or AI agent
+import fetch from 'node-fetch';
+
+async function verifyAIOutput(text: string) {
+  try {
+    const response = await fetch('http://localhost:9009/verify', {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ text })
+    });
+    
+    const result = await response.json();
+    
+    if (!result.success) {
+      console.error('Verification failed:', result.error);
+      return null;
+    }
+    
+    return {
+      riskLevel: result.result.risk.level,
+      action: result.result.risk.action,
+      findings: result.result.findings,
+      safe: result.result.risk.level === 'low'
+    };
+  } catch (error) {
+    console.error('Failed to connect to llmverify server:', error);
+    return null;
+  }
+}
+
+// Usage in your AI workflow
+const aiResponse = await generateAIResponse(userPrompt);
+const verification = await verifyAIOutput(aiResponse);
+
+if (verification && !verification.safe) {
+  console.warn(`AI output has ${verification.riskLevel} risk`);
+  // Handle accordingly - revise, flag, or block
+}
+```
+
+#### GitHub Copilot / AI Assistants
+
+For AI assistants that support custom tools or MCP servers, you can integrate llmverify as a verification step:
+
+```json
+{
+  "tools": [
+    {
+      "name": "verify_output",
+      "description": "Verify AI output for safety, PII, and hallucinations",
+      "endpoint": "http://localhost:9009/verify",
+      "method": "POST",
+      "required": ["text"]
+    }
+  ]
+}
+```
+
+### Server Response Format
+
+All endpoints return JSON with this structure:
+
+```typescript
+{
+  success: boolean;
+  result?: {
+    risk: {
+      level: "low" | "moderate" | "high" | "critical";
+      action: "allow" | "review" | "block";
+      score: number; // 0-1
+    };
+    findings: Array<{
+      category: string;
+      severity: string;
+      message: string;
+    }>;
+    // ... additional fields
+  };
+  error?: string;
+  version: string;
+}
+```
+
+### Production Deployment
+
+For production use, consider:
+
+1. **Authentication**: Add API key middleware
+2. **Rate Limiting**: Use `express-rate-limit`
+3. **HTTPS**: Deploy behind a reverse proxy (nginx, Caddy)
+4. **Monitoring**: Add logging and health checks
+5. **Scaling**: Run multiple instances with load balancing
+
+Example with authentication:
+
+```typescript
+import express from 'express';
+import { startServer } from 'llmverify/dist/server';
+
+const app = express();
+
+// Add API key middleware
+app.use((req, res, next) => {
+  const apiKey = req.headers['x-api-key'];
+  if (apiKey !== process.env.LLMVERIFY_API_KEY) {
+    return res.status(401).json({ error: 'Unauthorized' });
+  }
+  next();
+});
+
+startServer(9009);
+```
+
+---
+
 ## Why llmverify?
 
 | Problem | Solution |
@@ -423,6 +756,10 @@ export default async function handler(req, res) {
 ### Quick Start Commands
 
 ```bash
+# ★ Start HTTP server for IDE integration (NEW in v1.3.0)
+npx llmverify-serve                                  # Default port 9009
+npx llmverify-serve --port=8080                      # Custom port
+
 # ★ Interactive setup wizard (first-time users)
 npx llmverify wizard
 
@@ -438,6 +775,9 @@ npx llmverify verify "Your AI output here"
 # From file
 npx llmverify verify --file output.txt
 
+# JSON output (for scripting)
+npx llmverify verify "Your AI output" --output json
+
 # JSON validation
 npx llmverify verify --json '{"status": "ok"}'
 ```
@@ -889,8 +1229,11 @@ MIT License. See [LICENSE](LICENSE) for details.
 ## Documentation
 
 - [GETTING-STARTED.md](docs/GETTING-STARTED.md) - Beginner-friendly guide for students
+- [API-REFERENCE.md](docs/API-REFERENCE.md) - Complete programmatic API documentation
+- [SERVER-MODE.md](docs/SERVER-MODE.md) - HTTP server mode guide for IDE integration
 - [ALGORITHMS.md](docs/ALGORITHMS.md) - How each engine computes scores
 - [LIMITATIONS.md](docs/LIMITATIONS.md) - What llmverify can and cannot do
+- [JSON Schema](schema/verify-result.schema.json) - Formal schema for verify() output
 
 ## Links
 

package/bin/llmverify-serve.js

@@ -0,0 +1,8 @@
+#!/usr/bin/env node
+
+/**
+ * llmverify-serve CLI entry point
+ * Starts the HTTP server for IDE integration
+ */
+
+require('../dist/server.js');

package/dist/badge/generator.d.ts

@@ -0,0 +1,58 @@
+/**
+ * Badge Generator and Verification
+ *
+ * Generate "Built with llmverify" badges for verified applications
+ *
+ * @module badge/generator
+ */
+/**
+ * Badge configuration
+ */
+export interface BadgeConfig {
+    projectName: string;
+    projectUrl?: string;
+    verifiedDate: string;
+    version: string;
+}
+/**
+ * Badge verification data
+ */
+export interface BadgeVerification {
+    projectName: string;
+    verifiedDate: string;
+    version: string;
+    signature: string;
+    valid: boolean;
+}
+/**
+ * Generate badge verification signature
+ */
+export declare function generateBadgeSignature(config: BadgeConfig): string;
+/**
+ * Verify badge signature
+ */
+export declare function verifyBadgeSignature(projectName: string, verifiedDate: string, version: string, signature: string): boolean;
+/**
+ * Generate badge markdown
+ */
+export declare function generateBadgeMarkdown(config: BadgeConfig): string;
+/**
+ * Generate badge HTML
+ */
+export declare function generateBadgeHTML(config: BadgeConfig): string;
+/**
+ * Extract badge verification from markdown/HTML
+ */
+export declare function extractBadgeVerification(content: string): BadgeVerification | null;
+/**
+ * CLI helper to generate badge
+ */
+export declare function generateBadgeForProject(projectName: string, projectUrl?: string, version?: string): {
+    markdown: string;
+    html: string;
+    signature: string;
+};
+/**
+ * Save badge to file
+ */
+export declare function saveBadgeToFile(outputPath: string, projectName: string, projectUrl?: string): void;