@peakinfer/cli 1.0.133

package/docs/events-format.md

@@ -0,0 +1,295 @@
+# Runtime Events Format
+
+PeakInfer correlates your code with runtime behavior. This document describes how to format runtime event data for drift detection.
+
+## Quick Start
+
+Export your LLM inference events as JSONL (newline-delimited JSON):
+
+```jsonl
+{"id":"evt_1","ts":"2024-12-21T10:00:00Z","provider":"openai","model":"gpt-4","input_tokens":150,"output_tokens":50,"latency_ms":1200}
+{"id":"evt_2","ts":"2024-12-21T10:00:01Z","provider":"anthropic","model":"claude-3-opus","input_tokens":200,"output_tokens":100,"latency_ms":2500}
+```
+
+Save to a file (e.g., `events.jsonl`) and pass to PeakInfer:
+
+```bash
+# CLI
+peakinfer analyze ./src --events events.jsonl
+
+# GitHub Action
+- uses: kalmantic/peakinfer@v1
+  with:
+    path: ./src
+    events: ./events.jsonl
+```
+
+---
+
+## InferenceEvent Schema
+
+Each event represents one LLM inference call:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `id` | string | Yes | Unique event identifier |
+| `ts` | string | Yes | ISO 8601 timestamp |
+| `provider` | string | Yes | Provider name (see below) |
+| `model` | string | Yes | Model name |
+| `input_tokens` | number | Yes | Input token count |
+| `output_tokens` | number | Yes | Output token count |
+| `latency_ms` | number | Yes | Total latency in milliseconds |
+| `intent` | string | No | Business intent (e.g., "summarize", "translate") |
+| `callsite_id` | string | No | Link to code location (improves correlation) |
+| `streaming` | boolean | No | Was this a streaming request? |
+| `ttft_ms` | number | No | Time to first token (streaming only) |
+| `batch_size` | number | No | Batch size if batched |
+| `batch_id` | string | No | Batch group identifier |
+| `cached` | boolean | No | Was response cached? |
+| `retry_count` | number | No | Number of retries |
+| `fallback_used` | boolean | No | Was fallback triggered? |
+| `original_model` | string | No | Original model if fallback used |
+
+---
+
+## Provider Names
+
+Use lowercase provider names:
+
+| Provider | Value |
+|----------|-------|
+| OpenAI | `openai` |
+| Anthropic | `anthropic` |
+| Google | `google` |
+| Azure OpenAI | `azure_openai` |
+| AWS Bedrock | `bedrock` |
+| Cohere | `cohere` |
+| Mistral | `mistral` |
+| Together AI | `together` |
+| Fireworks | `fireworks` |
+| Groq | `groq` |
+| Replicate | `replicate` |
+| Perplexity | `perplexity` |
+| vLLM | `vllm` |
+| SGLang | `sglang` |
+| TGI | `tgi` |
+| Ollama | `ollama` |
+| llama.cpp | `llamacpp` |
+
+---
+
+## Supported Formats
+
+PeakInfer auto-detects the following formats:
+
+### Direct Parse (No LLM Needed)
+
+| Format | Extension | Description |
+|--------|-----------|-------------|
+| JSONL | `.jsonl` | Newline-delimited JSON (preferred) |
+| JSON Array | `.json` | Array of event objects |
+| CSV | `.csv` | Comma-separated values |
+| TSV | `.tsv` | Tab-separated values |
+
+### Agent-Normalized (Requires API Key)
+
+PeakInfer can parse exports from observability platforms:
+
+| Platform | Notes |
+|----------|-------|
+| OpenTelemetry | OTLP traces/spans |
+| Jaeger | Distributed tracing format |
+| Zipkin | Tracing format |
+| LangSmith | Trace exports |
+| Helicone | Proxy logs |
+| LiteLLM | Proxy event logs |
+| Portkey | Gateway logs |
+
+For these formats, provide your Anthropic API key:
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+peakinfer analyze ./src --events otel-traces.json
+```
+
+---
+
+## CSV/TSV Column Names
+
+For CSV/TSV files, use these column names:
+
+```csv
+id,ts,provider,model,input_tokens,output_tokens,latency_ms,streaming
+evt_1,2024-12-21T10:00:00Z,openai,gpt-4,150,50,1200,false
+evt_2,2024-12-21T10:00:01Z,anthropic,claude-3-opus,200,100,2500,true
+```
+
+Alternative column names are supported:
+- `timestamp` → `ts`
+- `provider_name` → `provider`
+- `model_name` → `model`
+- `tokens_in` → `input_tokens`
+- `tokens_out` → `output_tokens`
+- `latency`, `duration_ms` → `latency_ms`
+
+---
+
+## Linking Events to Code
+
+For better correlation, add `callsite_id` to events matching your code locations:
+
+```jsonl
+{"id":"evt_1","ts":"2024-12-21T10:00:00Z","provider":"openai","model":"gpt-4","input_tokens":150,"output_tokens":50,"latency_ms":1200,"callsite_id":"src/services/chat.ts:42"}
+```
+
+PeakInfer will match this to the inference point at `src/services/chat.ts:42`.
+
+---
+
+## Drift Detection
+
+When runtime data is provided, PeakInfer detects drift between code and runtime:
+
+| Drift Type | Description |
+|------------|-------------|
+| `codeOnly` | Inference point in code but never called |
+| `runtimeOnly` | Runtime calls with no matching code location |
+| `mismatch` | Model/provider differs between code and runtime |
+| `patternDrift` | Pattern mismatch (e.g., streaming in code, blocking in runtime) |
+
+Example PR comment:
+
+```
+🔒 RUNTIME CORRELATION
+
+| Location | Code | Runtime | Drift |
+|----------|------|---------|-------|
+| chat.ts:42 | streaming: true | streaming: 0% | patternDrift |
+| api.ts:15 | gpt-4 | gpt-4-turbo | mismatch |
+```
+
+---
+
+## GitHub Action Integration
+
+### From File in Repository
+
+```yaml
+- uses: kalmantic/peakinfer@v1
+  with:
+    path: ./src
+    events: ./traces/events.jsonl
+```
+
+### From URL
+
+Fetch events from your observability platform:
+
+```yaml
+- uses: kalmantic/peakinfer@v1
+  with:
+    path: ./src
+    events-url: ${{ secrets.OBSERVABILITY_URL }}
+```
+
+### From GitHub Artifact
+
+If events are produced by a previous job:
+
+```yaml
+- uses: actions/download-artifact@v4
+  with:
+    name: inference-events
+
+- uses: kalmantic/peakinfer@v1
+  with:
+    path: ./src
+    events: ./inference-events/events.jsonl
+```
+
+---
+
+## Generating Events
+
+### OpenAI SDK (Python)
+
+```python
+import json
+import time
+from openai import OpenAI
+
+client = OpenAI()
+events = []
+
+start = time.time()
+response = client.chat.completions.create(
+    model="gpt-4",
+    messages=[{"role": "user", "content": "Hello"}]
+)
+latency = (time.time() - start) * 1000
+
+events.append({
+    "id": response.id,
+    "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+    "provider": "openai",
+    "model": response.model,
+    "input_tokens": response.usage.prompt_tokens,
+    "output_tokens": response.usage.completion_tokens,
+    "latency_ms": int(latency),
+    "streaming": False,
+})
+
+# Write to JSONL
+with open("events.jsonl", "a") as f:
+    f.write(json.dumps(events[-1]) + "\n")
+```
+
+### Anthropic SDK (Python)
+
+```python
+import json
+import time
+import anthropic
+
+client = anthropic.Anthropic()
+events = []
+
+start = time.time()
+response = client.messages.create(
+    model="claude-3-opus-20240229",
+    max_tokens=1024,
+    messages=[{"role": "user", "content": "Hello"}]
+)
+latency = (time.time() - start) * 1000
+
+events.append({
+    "id": response.id,
+    "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+    "provider": "anthropic",
+    "model": response.model,
+    "input_tokens": response.usage.input_tokens,
+    "output_tokens": response.usage.output_tokens,
+    "latency_ms": int(latency),
+    "streaming": False,
+})
+
+with open("events.jsonl", "a") as f:
+    f.write(json.dumps(events[-1]) + "\n")
+```
+
+---
+
+## Best Practices
+
+1. **Export regularly**: Run event exports as part of your CI/CD pipeline
+2. **Include callsite_id**: Improves code-to-runtime correlation accuracy
+3. **Keep events recent**: Use last 24-48 hours of data for meaningful drift detection
+4. **Use streaming field**: Critical for detecting streaming drift
+5. **Track retries**: Helps identify reliability issues
+
+---
+
+## Related
+
+- [InferenceMap Spec](inferencemap-spec.md) — Output schema for analysis results
+- [README](../README.md) — Quick start guide

package/docs/inferencemap-spec.md

@@ -0,0 +1,344 @@
+# InferenceMap v0.1 Specification
+
+The InferenceMap is PeakInfer's structured output format for static analysis results. It provides a machine-readable representation of all LLM inference points in a codebase.
+
+## Overview
+
+```json
+{
+  "version": "0.1",
+  "root": "./src",
+  "generatedAt": "2024-12-21T10:00:00Z",
+  "metadata": { ... },
+  "summary": { ... },
+  "callsites": [ ... ]
+}
+```
+
+---
+
+## Top-Level Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `version` | string | Schema version (currently `"0.1"`) |
+| `root` | string | Analyzed path (relative or absolute) |
+| `generatedAt` | string | ISO 8601 timestamp |
+| `metadata` | object | Analysis context (optional) |
+| `summary` | object | Aggregate statistics |
+| `callsites` | array | List of inference points |
+
+---
+
+## Metadata Object
+
+Optional context about how the analysis was performed:
+
+```json
+{
+  "metadata": {
+    "absolutePath": "/Users/dev/project/src",
+    "promptId": "unified-analyzer",
+    "promptVersion": "1.6.0",
+    "templatesVersion": "1.0.0",
+    "llmProvider": "anthropic",
+    "llmModel": "claude-sonnet-4-20250514"
+  }
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `absolutePath` | string | Full absolute path analyzed |
+| `promptId` | string | Analysis prompt identifier |
+| `promptVersion` | string | Analysis prompt version |
+| `templatesVersion` | string | peakinfer-templates version |
+| `llmProvider` | string | LLM provider used (`anthropic`, `none`) |
+| `llmModel` | string | LLM model used for analysis |
+
+---
+
+## Summary Object
+
+Aggregate statistics for quick overview:
+
+```json
+{
+  "summary": {
+    "totalCallsites": 7,
+    "providers": ["openai", "anthropic"],
+    "models": ["gpt-4", "claude-3-opus"],
+    "patterns": {
+      "streaming": 3,
+      "batching": 0,
+      "retries": 5,
+      "caching": 1,
+      "fallback": 2
+    }
+  }
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `totalCallsites` | number | Total inference points detected |
+| `providers` | string[] | Unique providers found |
+| `models` | string[] | Unique models found |
+| `patterns` | object | Count of each pattern detected |
+
+---
+
+## Callsite Object
+
+Each inference point (callsite) has this structure:
+
+```json
+{
+  "id": "src/services/chat.ts:42",
+  "file": "src/services/chat.ts",
+  "line": 42,
+  "provider": "openai",
+  "model": "gpt-4",
+  "framework": "langchain",
+  "runtime": null,
+  "patterns": {
+    "streaming": true,
+    "batching": false,
+    "retries": true,
+    "caching": false,
+    "fallback": true
+  },
+  "confidence": 0.95
+}
+```
+
+### Required Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `id` | string | Unique identifier (typically `file:line`) |
+| `file` | string | File path (relative to root) |
+| `line` | number | Line number |
+| `provider` | string \| null | Provider name |
+| `model` | string \| null | Model name |
+| `framework` | string \| null | Framework (langchain, llamaindex, etc.) |
+| `runtime` | string \| null | Runtime (vllm, tgi, etc.) |
+| `patterns` | object | Detected code patterns |
+| `confidence` | number | Detection confidence (0-1) |
+
+### Patterns Object
+
+| Pattern | Type | Description |
+|---------|------|-------------|
+| `streaming` | boolean | Streaming enabled in code |
+| `batching` | boolean | Batch processing detected |
+| `retries` | boolean | Retry logic present |
+| `caching` | boolean | Caching implemented |
+| `fallback` | boolean | Fallback logic present |
+
+---
+
+## Provider Values
+
+Valid provider values:
+
+```typescript
+type Provider =
+  | 'openai'
+  | 'anthropic'
+  | 'google'
+  | 'cohere'
+  | 'mistral'
+  | 'bedrock'
+  | 'azure_openai'
+  | 'together'
+  | 'fireworks'
+  | 'groq'
+  | 'replicate'
+  | 'perplexity'
+  | 'vllm'
+  | 'sglang'
+  | 'tgi'
+  | 'ollama'
+  | 'llamacpp'
+  | 'unknown';
+```
+
+---
+
+## Complete Example
+
+```json
+{
+  "version": "0.1",
+  "root": "./src",
+  "generatedAt": "2024-12-21T10:00:00.000Z",
+  "metadata": {
+    "absolutePath": "/Users/dev/project/src",
+    "llmProvider": "anthropic",
+    "llmModel": "claude-sonnet-4-20250514"
+  },
+  "summary": {
+    "totalCallsites": 3,
+    "providers": ["openai", "anthropic"],
+    "models": ["gpt-4", "gpt-4-turbo", "claude-3-opus"],
+    "patterns": {
+      "streaming": 2,
+      "batching": 0,
+      "retries": 2,
+      "caching": 1,
+      "fallback": 1
+    }
+  },
+  "callsites": [
+    {
+      "id": "src/services/chat.ts:42",
+      "file": "src/services/chat.ts",
+      "line": 42,
+      "provider": "openai",
+      "model": "gpt-4",
+      "framework": null,
+      "runtime": null,
+      "patterns": {
+        "streaming": true,
+        "batching": false,
+        "retries": true,
+        "caching": false,
+        "fallback": false
+      },
+      "confidence": 0.95
+    },
+    {
+      "id": "src/services/summarize.ts:15",
+      "file": "src/services/summarize.ts",
+      "line": 15,
+      "provider": "openai",
+      "model": "gpt-4-turbo",
+      "framework": "langchain",
+      "runtime": null,
+      "patterns": {
+        "streaming": false,
+        "batching": false,
+        "retries": true,
+        "caching": true,
+        "fallback": true
+      },
+      "confidence": 0.88
+    },
+    {
+      "id": "src/api/translate.ts:78",
+      "file": "src/api/translate.ts",
+      "line": 78,
+      "provider": "anthropic",
+      "model": "claude-3-opus",
+      "framework": null,
+      "runtime": null,
+      "patterns": {
+        "streaming": true,
+        "batching": false,
+        "retries": false,
+        "caching": false,
+        "fallback": false
+      },
+      "confidence": 0.92
+    }
+  ]
+}
+```
+
+---
+
+## Usage
+
+### CLI Output
+
+Generate InferenceMap with the CLI:
+
+```bash
+# Save to file
+peakinfer analyze ./src --json > inference-map.json
+
+# Or use built-in artifact saving
+peakinfer analyze ./src --save
+# Creates .peakinfer/inference-map.json
+```
+
+### Programmatic Access (TypeScript)
+
+```typescript
+import { InferenceMap } from '@kalmantic/peakinfer';
+import fs from 'fs';
+
+const map: InferenceMap = JSON.parse(
+  fs.readFileSync('.peakinfer/inference-map.json', 'utf-8')
+);
+
+console.log(`Found ${map.summary.totalCallsites} inference points`);
+
+for (const callsite of map.callsites) {
+  if (!callsite.patterns.retries) {
+    console.log(`Missing retries: ${callsite.id}`);
+  }
+}
+```
+
+### Zod Schema Validation
+
+```typescript
+import { InferenceMap as InferenceMapSchema } from '@kalmantic/peakinfer';
+
+// Validate JSON against schema
+const result = InferenceMapSchema.safeParse(jsonData);
+
+if (!result.success) {
+  console.error('Invalid InferenceMap:', result.error);
+}
+```
+
+---
+
+## Versioning
+
+The `version` field indicates the schema version:
+
+| Version | Description |
+|---------|-------------|
+| `0.1` | Initial specification (current) |
+
+Future versions will maintain backward compatibility where possible.
+
+---
+
+## TypeScript Definition
+
+The full Zod schema is available in `src/types.ts`:
+
+```typescript
+export const InferenceMap = z.object({
+  version: z.string(),
+  root: z.string(),
+  generatedAt: z.string(),
+  metadata: z.object({
+    absolutePath: z.string(),
+    promptId: z.string().optional(),
+    promptVersion: z.string().optional(),
+    templatesVersion: z.string().optional(),
+    llmProvider: z.string().optional(),
+    llmModel: z.string().optional(),
+  }).optional(),
+  summary: z.object({
+    totalCallsites: z.number(),
+    providers: z.array(z.string()),
+    models: z.array(z.string()),
+    patterns: z.record(z.number()),
+  }),
+  callsites: z.array(Callsite),
+});
+```
+
+---
+
+## Related
+
+- [Runtime Events Format](events-format.md) — Input schema for runtime correlation
+- [README](../README.md) — Quick start guide