@thinkhive/sdk 3.1.1 → 3.3.0

package/README.md

@@ -1,16 +1,16 @@
-# ThinkHive SDK
+# ThinkHive SDK v3.3.0
 
 The official JavaScript/TypeScript SDK for [ThinkHive](https://thinkhive.ai) - AI Agent Observability Platform.
 
 ## Features
 
-- **25 Trace Format Support**: Automatic detection and normalization for LangSmith, Langfuse, Helicone, CrewAI, Opik, Braintrust, HoneyHive, Datadog, MLflow, AgentOps, Portkey, TruLens, Lunary, LangWatch, OpenLIT, Maxim AI, Galileo, PostHog, Keywords AI, Agenta, and more
-- **Trace Analysis**: Analyze AI agent traces with detailed explainability
-- **RAG Evaluation**: 8 quality metrics for RAG systems (groundedness, faithfulness, etc.)
-- **Hallucination Detection**: 9 types of hallucination detection
-- **Business Impact**: Industry-specific ROI calculations
+- **OpenTelemetry-Based Tracing**: Built on OTLP for seamless integration with existing observability tools
+- **Run-Centric Architecture**: Atomic unit of work tracking with claims, calibration, and linking
+- **Facts vs Inferences**: Claims API for separating verified facts from inferences
+- **Deterministic Ticket Linking**: 7 methods for linking runs to support tickets
+- **Calibrated Predictions**: Brier scores for prediction accuracy
 - **Auto-Instrumentation**: Works with LangChain, OpenAI, Anthropic, and more
-- **OpenTelemetry**: Built on OTLP for seamless integration
+- **Multi-Format Support**: Normalizes traces from 25+ observability platforms
 
 ## Installation
 
@@ -20,190 +20,341 @@ npm install @thinkhive/sdk
 
 ## Quick Start
 
-### Basic Usage
+### Basic Initialization
 
 ```typescript
-import { ThinkHive } from '@thinkhive/sdk';
+import { init, runs, traceLLM, shutdown } from '@thinkhive/sdk';
 
-// Initialize client
-const client = new ThinkHive({
-  apiKey: 'your_api_key',
-  baseUrl: 'https://api.thinkhive.ai'
+// Initialize the SDK
+init({
+  apiKey: 'th_your_api_key',
+  serviceName: 'my-ai-agent',
+  autoInstrument: true,
+  frameworks: ['langchain', 'openai'],
 });
 
-// Send a trace
-const result = await client.trace({
-  userMessage: 'What is the weather in San Francisco?',
-  agentResponse: 'The weather in San Francisco is currently 65°F and sunny.',
-  agentId: 'weather-agent'
+// Create a run (atomic unit of work)
+const run = await runs.create({
+  agentId: 'weather-agent',
+  conversation: [
+    { role: 'user', content: 'What is the weather in San Francisco?' },
+    { role: 'assistant', content: 'The weather in San Francisco is currently 65°F and sunny.' }
+  ],
+  outcome: 'success',
 });
 
-console.log(`Trace ID: ${result.traceId}`);
-if (result.analysis) {
-  console.log(`Outcome: ${result.analysis.outcome.verdict}`);
-  console.log(`Impact Score: ${result.analysis.businessImpact.impactScore}`);
-}
+console.log(`Run ID: ${run.id}`);
+
+// Shutdown when done
+await shutdown();
 ```
 
-### With Business Context
+### Manual Tracing
 
 ```typescript
-const result = await client.trace({
-  userMessage: 'I want to cancel my order #12345',
-  agentResponse: 'I understand you want to cancel order #12345...',
-  agentId: 'support-agent',
-  businessContext: {
-    customerId: 'cust_abc123',
-    transactionValue: 150.00,
-    priority: 'high',
-    industry: 'ecommerce'
-  }
+import { init, traceLLM, traceRetrieval, traceTool, traceChain } from '@thinkhive/sdk';
+
+init({ apiKey: 'th_your_api_key', serviceName: 'my-agent' });
+
+// Trace an LLM call
+const response = await traceLLM({
+  name: 'generate-response',
+  modelName: 'gpt-4',
+  provider: 'openai',
+  input: { prompt: 'Hello!' }
+}, async () => {
+  // Your LLM call here
+  return await openai.chat.completions.create({...});
 });
 
-// Access ROI metrics
-if (result.analysis?.businessImpact?.roi) {
-  const roi = result.analysis.businessImpact.roi;
-  console.log(`Estimated Revenue Loss: $${roi.estimatedRevenueLoss}`);
-  console.log(`Churn Probability: ${roi.churnProbability}%`);
-}
+// Trace a retrieval operation
+const docs = await traceRetrieval({
+  name: 'search-knowledge-base',
+  query: 'refund policy',
+  topK: 5
+}, async () => {
+  return await vectorStore.similaritySearch(query, 5);
+});
+
+// Trace a tool call
+const result = await traceTool({
+  name: 'lookup-order',
+  toolName: 'order_lookup',
+  parameters: { orderId: '12345' }
+}, async () => {
+  return await lookupOrder('12345');
+});
 ```
 
-### Explainer API
+### Analyzer API (User-Selected Analysis)
 
 ```typescript
-// Full trace analysis with RAG evaluation
-const analysis = await client.explainer.analyze({
-  userMessage: 'What is your return policy?',
-  agentResponse: 'Items can be returned within 30 days...',
-  retrievedContexts: ['Return Policy: 30 day returns...'],
-  outcome: 'success'
-}, {
-  tier: 'full_llm',
-  includeRagEvaluation: true,
-  includeHallucinationDetection: true
+import { analyzer } from '@thinkhive/sdk';
+
+// Estimate cost before running analysis
+const estimate = await analyzer.estimateCost({
+  traceIds: ['trace-1', 'trace-2', 'trace-3'],
+  tier: 'standard',
+});
+console.log(`Estimated cost: $${estimate.estimatedCost}`);
+
+// Analyze specific traces
+const analysis = await analyzer.analyze({
+  traceIds: ['trace-1', 'trace-2'],
+  tier: 'standard',
+  includeRootCause: true,
+  includeLayers: true,
 });
 
-console.log(`Summary: ${analysis.summary}`);
-console.log(`Groundedness: ${analysis.ragEvaluation?.groundedness}`);
+// Analyze traces by time window with smart sampling
+const windowAnalysis = await analyzer.analyzeWindow({
+  agentId: 'support-agent',
+  startDate: new Date('2024-01-01'),
+  endDate: new Date('2024-01-31'),
+  filters: { outcomes: ['failure'], minSeverity: 'medium' },
+  sampling: { strategy: 'smart', samplePercent: 10 },
+});
+
+// Get aggregated insights
+const summary = await analyzer.summarize({
+  agentId: 'support-agent',
+  startDate: new Date('2024-01-01'),
+  endDate: new Date('2024-01-31'),
+});
+```
 
-// Batch analysis
-const batchResult = await client.explainer.analyzeBatch([
-  { userMessage: '...', agentResponse: '...' },
-  { userMessage: '...', agentResponse: '...' }
-], { tier: 'fast_llm' });
+### Issues API (Clustered Failure Patterns)
 
-// Semantic search
-const searchResults = await client.explainer.search({
-  query: 'refund complaints',
-  filters: { outcome: 'failure' },
-  limit: 10
+```typescript
+import { issues } from '@thinkhive/sdk';
+
+// List issues for an agent
+const issueList = await issues.list('support-agent', {
+  status: 'open',
+  limit: 10,
 });
+
+// Get a specific issue
+const issue = await issues.get('issue-123');
+
+// Get fixes for an issue
+const fixes = await issues.getFixes('issue-123');
 ```
 
-### Quality Metrics
+### API Key Management
 
 ```typescript
-// Get RAG scores
-const ragScores = await client.quality.getRagScores('trace-123');
-console.log(`Groundedness: ${ragScores.groundedness}`);
-console.log(`Faithfulness: ${ragScores.faithfulness}`);
-
-// Get hallucination report
-const report = await client.quality.getHallucinationReport('trace-123');
-if (report.hasHallucinations) {
-  for (const detection of report.detectedTypes) {
-    console.log(`- ${detection.type}: ${detection.description}`);
-  }
+import { apiKeys, hasPermission, canAccessAgent } from '@thinkhive/sdk';
+
+// Create a scoped API key
+const result = await apiKeys.create({
+  name: 'CI Pipeline Key',
+  permissions: {
+    read: true,
+    write: true,
+    delete: false
+  },
+  scopeType: 'agent', // Restrict to specific agents
+  allowedAgentIds: ['agent-prod-001'],
+  environment: 'production',
+  expiresAt: new Date(Date.now() + 90 * 24 * 60 * 60 * 1000) // 90 days
+});
+
+console.log(`Key created: ${result.name} (${result.keyPrefix}...)`);
+
+// Check permissions
+if (hasPermission(key, 'write')) {
+  // Can write data
 }
 
-// Evaluate RAG for custom input
-const evaluation = await client.quality.evaluateRag({
-  query: 'What is the return policy?',
-  response: 'Items can be returned within 30 days.',
-  contexts: [{ content: 'Return Policy: 30 day returns...' }]
-});
+// Check agent access
+if (canAccessAgent(key, 'agent-123')) {
+  // Can access this agent
+}
 ```
 
-### ROI Analytics
+### Claims API (Facts vs Inferences)
 
 ```typescript
-// Get ROI summary
-const summary = await client.analytics.getRoiSummary();
-console.log(`Revenue Saved: $${summary.totalRevenueSaved}`);
-
-// Get per-agent ROI
-const agentRoi = await client.analytics.getRoiByAgent('support-agent');
-console.log(`Success Rate: ${agentRoi.successRate}%`);
-
-// Get correlation analysis
-const correlations = await client.analytics.getCorrelations();
-for (const corr of correlations.correlations) {
-  console.log(`${corr.type}: ${corr.actionableInsight}`);
+import { claims, isFact, isInference, getHighConfidenceClaims } from '@thinkhive/sdk';
+
+// List claims for a run
+const claimList = await claims.list(runId);
+
+// Filter by type
+const facts = claimList.filter(isFact);
+const inferences = claimList.filter(isInference);
+
+// Get high confidence claims
+const confident = getHighConfidenceClaims(claimList, 0.9);
+```
+
+### Calibration API (Prediction Accuracy)
+
+```typescript
+import { calibration, calculateBrierScore, isWellCalibrated } from '@thinkhive/sdk';
+
+// Get calibration status
+const status = await calibration.getStatus(agentId);
+
+// Calculate Brier score for predictions
+const brierScore = calculateBrierScore(predictions, outcomes);
+
+// Check if well calibrated
+if (isWellCalibrated(status)) {
+  console.log('Agent predictions are well calibrated');
 }
 ```
 
-### Providing Feedback
+### Business Metrics API
 
 ```typescript
-// After receiving user feedback
-await client.feedback({
-  traceId: result.traceId,
-  rating: 5,
-  wasHelpful: true,
-  comment: 'Very accurate response!'
+import {
+  businessMetrics,
+  isMetricReady,
+  needsMoreTraces,
+  getStatusMessage
+} from '@thinkhive/sdk';
+
+// Get current metric value with status
+const metric = await businessMetrics.current('agent-123', 'Deflection Rate');
+console.log(`${metric.metricName}: ${metric.valueFormatted}`);
+
+if (metric.status === 'insufficient_data') {
+  console.log(`Need ${metric.minTraceThreshold - metric.traceCount} more traces`);
+}
+
+// Get historical data for graphing
+const history = await businessMetrics.history('agent-123', 'Deflection Rate', {
+  startDate: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000),
+  endDate: new Date(),
+  granularity: 'daily',
 });
 
-// When response was incorrect
-await client.feedback({
-  traceId: result.traceId,
-  rating: 2,
-  wasHelpful: false,
-  hadIssues: ['incorrect_info', 'too_long'],
-  correctedResponse: 'The correct answer is...'
+console.log(`${history.dataPoints.length} data points`);
+console.log(`Change: ${history.summary.changePercent}%`);
+
+// Record external metric values (from CRM, surveys, etc.)
+await businessMetrics.record('agent-123', {
+  metricName: 'CSAT/NPS',
+  value: 4.5,
+  unit: 'score',
+  periodStart: '2024-01-01T00:00:00Z',
+  periodEnd: '2024-01-07T23:59:59Z',
+  source: 'survey_system',
+  sourceDetails: { surveyId: 'survey_456', responseCount: 150 },
 });
 ```
 
+#### Metric Status Types
+
+| Status | Description |
+|--------|-------------|
+| `ready` | Metric calculated and ready to display |
+| `insufficient_data` | Need more traces before calculation |
+| `awaiting_external` | External data source not connected |
+| `stale` | Data is older than expected |
+
+### Ticket Linking (Zendesk Integration)
+
+```typescript
+import {
+  linking,
+  generateZendeskMarker,
+  linkRunToZendeskTicket
+} from '@thinkhive/sdk';
+
+// Generate a marker to embed in ticket
+const marker = generateZendeskMarker(runId);
+// Returns: <!-- thinkhive:run:abc123 -->
+
+// Link a run to a ticket
+await linkRunToZendeskTicket(runId, ticketId);
+
+// Get best linking method
+import { getBestLinkMethod } from '@thinkhive/sdk';
+const method = getBestLinkMethod(runData);
+// Returns: 'conversation_id' | 'subject_hash' | 'marker' | etc.
+```
+
 ### Auto-Instrumentation
 
 ```typescript
-import { init, autoInstrument } from '@thinkhive/sdk';
+import { init } from '@thinkhive/sdk';
 
-// Initialize SDK
+// Initialize with auto-instrumentation
 init({
-  apiKey: 'your_api_key',
+  apiKey: 'th_your_api_key',
   serviceName: 'my-ai-agent',
   autoInstrument: true,
-  frameworks: ['langchain', 'openai']
-});
-
-// Or manually instrument
-autoInstrument(client, {
-  frameworks: ['langchain', 'openai'],
-  capturePrompts: true,
-  captureResponses: true,
-  businessContext: { industry: 'saas' }
+  frameworks: ['langchain', 'openai', 'anthropic']
 });
 
-// Now all LangChain and OpenAI calls are automatically traced!
+// Now all LangChain, OpenAI, and Anthropic calls are automatically traced!
 ```
 
 ## Analysis Tiers
 
-| Tier | Description | Latency | Cost |
-|------|-------------|---------|------|
-| `rule_based` | Pattern matching, keyword extraction | ~50ms | Free |
-| `fast_llm` | Quick LLM analysis (GPT-3.5) | ~500ms | Low |
-| `full_llm` | Complete analysis (GPT-4o) | ~3s | Standard |
-| `deep` | Multi-pass with validation | ~15s | Premium |
+| Tier | Description | Use Case |
+|------|-------------|----------|
+| `fast` | Quick pattern-based analysis | High-volume, low-latency needs |
+| `standard` | LLM-powered analysis | Default for most use cases |
+| `deep` | Multi-pass with validation | Critical traces, root cause analysis |
 
 ## Environment Variables
 
 | Variable | Description |
 |----------|-------------|
 | `THINKHIVE_API_KEY` | Your ThinkHive API key |
-| `THINKHIVE_ENDPOINT` | Custom API endpoint (optional) |
+| `THINKHIVE_ENDPOINT` | Custom API endpoint (default: https://demo.thinkhive.ai) |
 | `THINKHIVE_SERVICE_NAME` | Service name for traces (optional) |
 
+## V3 Architecture
+
+### Key Concepts
+
+**Run-Centric Model**: The atomic unit of work is a "Run" (not a trace). A run captures:
+- Conversation messages
+- Retrieved contexts
+- Tool calls
+- Outcome and metadata
+
+**Facts vs Inferences**: Claims API separates:
+- **Facts**: Verified information from retrieval or tool calls
+- **Inferences**: LLM-generated conclusions
+- **Computed**: Derived values from rules
+
+**Calibrated Predictions**: Track prediction accuracy using:
+- Brier scores for overall calibration
+- ECE (Expected Calibration Error) for bucketed analysis
+
+### API Structure
+
+| API | Description |
+|-----|-------------|
+| `runs` | Create and manage runs (atomic work units) |
+| `claims` | Manage facts/inferences for runs |
+| `calibration` | Track prediction accuracy |
+| `analyzer` | User-selected trace analysis |
+| `issues` | Clustered failure patterns |
+| `linking` | Connect runs to support tickets |
+| `customerContext` | Time-series customer snapshots |
+| `apiKeys` | API key management |
+| `businessMetrics` | Industry-driven metrics with historical tracking |
+| `roiAnalytics` | Business ROI and financial impact analysis |
+| `qualityMetrics` | RAG evaluation and hallucination detection |
+
+### New Evaluation APIs (v3.0)
+
+| API | Description |
+|-----|-------------|
+| `humanReview` | Human-in-the-loop review queues |
+| `nondeterminism` | Multi-sample reliability testing |
+| `evalHealth` | Evaluation metric health monitoring |
+| `deterministicGraders` | Rule-based evaluation |
+| `conversationEval` | Multi-turn conversation evaluation |
+| `transcriptPatterns` | Pattern detection in transcripts |
+
 ## API Reference
 
 See [API Documentation](https://docs.thinkhive.ai/sdk/javascript/reference) for complete type definitions.