@fallom/trace 0.1.0

package/README.md

@@ -0,0 +1,275 @@
+# @fallom/trace
+
+Model A/B testing and tracing for LLM applications. Zero latency, production-ready.
+
+## Installation
+
+```bash
+npm install @fallom/trace
+
+# With auto-instrumentation for your LLM provider:
+npm install @fallom/trace @traceloop/node-server-sdk
+```
+
+## Quick Start
+
+```typescript
+import fallom from '@fallom/trace';
+import OpenAI from 'openai';
+
+// Initialize FIRST - before importing your LLM libraries
+fallom.init({ apiKey: 'your-api-key' });
+
+// Set default session context for tracing
+fallom.trace.setSession('my-agent', sessionId);
+
+// All LLM calls are now automatically traced!
+const openai = new OpenAI();
+const response = await openai.chat.completions.create({
+  model: 'gpt-4o',
+  messages: [{ role: 'user', content: 'Hello!' }],
+});
+```
+
+## Model A/B Testing
+
+Run A/B tests on models with zero latency. Same session always gets same model (sticky assignment).
+
+```typescript
+import { models } from '@fallom/trace';
+
+// Get assigned model for this session
+const model = await models.get('summarizer-config', sessionId);
+// Returns: "gpt-4o" or "claude-3-5-sonnet" based on your config weights
+
+const agent = new Agent({ model });
+await agent.run(message);
+```
+
+### Version Pinning
+
+Pin to a specific config version, or use latest (default):
+
+```typescript
+// Use latest version (default)
+const model = await models.get('my-config', sessionId);
+
+// Pin to specific version
+const model = await models.get('my-config', sessionId, { version: 2 });
+```
+
+### Fallback for Resilience
+
+Always provide a fallback so your app works even if Fallom is down:
+
+```typescript
+const model = await models.get('my-config', sessionId, {
+  fallback: 'gpt-4o-mini', // Used if config not found or Fallom unreachable
+});
+```
+
+**Resilience guarantees:**
+- Short timeouts (1-2 seconds max)
+- Background config sync (never blocks your requests)
+- Graceful degradation (returns fallback on any error)
+- Your app is never impacted by Fallom being down
+
+## Tracing
+
+Auto-capture all LLM calls with OpenTelemetry instrumentation.
+
+> ⚠️ **Important:** Auto-tracing only works with supported LLM SDKs (OpenAI, Anthropic, etc.) - not raw HTTP requests. If you're using an OpenAI-compatible API like OpenRouter, LiteLLM, or a self-hosted model, use the OpenAI SDK with a custom `baseURL`:
+>
+> ```typescript
+> import OpenAI from 'openai';
+> 
+> // OpenRouter, LiteLLM, vLLM, etc.
+> const client = new OpenAI({
+>   baseURL: 'https://openrouter.ai/api/v1', // or your provider's URL
+>   apiKey: 'your-provider-key',
+> });
+> 
+> // Now this call will be auto-traced!
+> const response = await client.chat.completions.create({
+>   model: 'gpt-4o',
+>   messages: [...],
+> });
+> ```
+
+### Automatic Tracing
+
+```typescript
+import fallom from '@fallom/trace';
+
+// Initialize before making LLM calls
+fallom.init();
+
+// Set session context
+fallom.trace.setSession('my-agent', sessionId);
+
+// All LLM calls automatically traced with:
+// - Model, tokens, latency
+// - Prompts and completions
+// - Your config_key and session_id
+const response = await openai.chat.completions.create({
+  model: 'gpt-4o',
+  messages: [...],
+});
+```
+
+### Async Context Propagation
+
+For proper session context across async boundaries, use `runWithSession`:
+
+```typescript
+import { trace } from '@fallom/trace';
+
+await trace.runWithSession('my-agent', sessionId, async () => {
+  // All LLM calls in here have session context
+  await agent.run(message);
+  await anotherAsyncOperation();
+});
+```
+
+### Custom Metrics
+
+Record business metrics that OTEL can't capture automatically:
+
+```typescript
+import { trace } from '@fallom/trace';
+
+// Record custom metrics for this session
+trace.span({
+  outlier_score: 0.8,
+  user_satisfaction: 4,
+  conversion: true,
+});
+
+// Or explicitly specify session (for batch jobs)
+trace.span(
+  { outlier_score: 0.8 },
+  { configKey: 'my-agent', sessionId: 'user123-convo456' }
+);
+```
+
+## Configuration
+
+### Environment Variables
+
+```bash
+FALLOM_API_KEY=your-api-key
+FALLOM_BASE_URL=https://spans.fallom.com  # or http://localhost:8001 for local dev
+FALLOM_CAPTURE_CONTENT=true  # set to "false" for privacy mode
+```
+
+### Initialization Options
+
+```typescript
+fallom.init({
+  apiKey: 'your-api-key',           // Or use FALLOM_API_KEY env var
+  baseUrl: 'https://spans.fallom.com', // Or use FALLOM_BASE_URL env var
+  captureContent: true,              // Set false for privacy mode
+});
+```
+
+### Privacy Mode
+
+For companies with strict data policies, disable prompt/completion capture:
+
+```typescript
+// Via parameter
+fallom.init({ captureContent: false });
+
+// Or via environment variable
+// FALLOM_CAPTURE_CONTENT=false
+```
+
+In privacy mode, Fallom still tracks:
+- ✅ Model used
+- ✅ Token counts
+- ✅ Latency
+- ✅ Session/config context
+- ❌ Prompt content (not captured)
+- ❌ Completion content (not captured)
+
+## API Reference
+
+### `fallom.init(options?)`
+
+Initialize the SDK. Call this before making LLM calls for auto-instrumentation.
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `apiKey` | `string` | `FALLOM_API_KEY` env | Your Fallom API key |
+| `baseUrl` | `string` | `https://spans.fallom.com` | API base URL |
+| `captureContent` | `boolean` | `true` | Capture prompt/completion text |
+
+### `fallom.models.get(configKey, sessionId, options?)`
+
+Get model assignment for a session.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `configKey` | `string` | Your config name from the dashboard |
+| `sessionId` | `string` | Unique session/conversation ID (sticky assignment) |
+| `options.version` | `number` | Pin to specific version (default: latest) |
+| `options.fallback` | `string` | Model to return if anything fails |
+| `options.debug` | `boolean` | Enable debug logging |
+
+Returns: `Promise<string>` - The assigned model name
+
+### `fallom.trace.setSession(configKey, sessionId)`
+
+Set trace context. All subsequent LLM calls will be tagged with this session.
+
+### `fallom.trace.runWithSession(configKey, sessionId, fn)`
+
+Run a function with session context that propagates across async boundaries.
+
+### `fallom.trace.clearSession()`
+
+Clear trace context.
+
+### `fallom.trace.span(data, options?)`
+
+Record custom business metrics.
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `data` | `Record<string, unknown>` | Metrics to record |
+| `options.configKey` | `string` | Optional if `setSession()` was called |
+| `options.sessionId` | `string` | Optional if `setSession()` was called |
+
+### `fallom.trace.shutdown()`
+
+Gracefully shutdown the tracing SDK. Call this on process exit.
+
+## Supported LLM Providers
+
+Auto-instrumentation available for:
+- OpenAI (+ OpenAI-compatible APIs: OpenRouter, LiteLLM, vLLM, Ollama, etc.)
+- Anthropic
+- Cohere
+- AWS Bedrock
+- Google Generative AI
+- Azure OpenAI
+- LangChain
+- And more via Traceloop
+
+Install `@traceloop/node-server-sdk` for comprehensive LLM instrumentation.
+
+**Note:** You must use the official SDK for your provider. Raw HTTP requests (e.g., `fetch()`) will not be traced. For OpenAI-compatible APIs, use the OpenAI SDK with a custom `baseURL`.
+
+## Examples
+
+See the `../examples/` folder for complete examples:
+- `random-fact/` - Simple A/B testing with Hono server
+
+## Requirements
+
+- Node.js >= 18.0.0
+
+## License
+
+MIT
+

package/dist/index.d.mts

@@ -0,0 +1,245 @@
+/**
+ * Fallom tracing module.
+ *
+ * Auto-instruments all LLM calls via OTEL and groups them by session.
+ * Also supports custom spans for business metrics.
+ */
+interface SessionContext {
+    configKey: string;
+    sessionId: string;
+}
+/**
+ * Initialize Fallom tracing. Auto-instruments all LLM calls.
+ *
+ * @param options - Configuration options
+ * @param options.apiKey - Your Fallom API key. Defaults to FALLOM_API_KEY env var.
+ * @param options.baseUrl - API base URL. Defaults to FALLOM_BASE_URL env var, or https://spans.fallom.com
+ * @param options.captureContent - Whether to capture prompt/completion content in traces.
+ *                                 Set to false for privacy/compliance. Defaults to true.
+ *                                 Also respects FALLOM_CAPTURE_CONTENT env var ("true"/"false").
+ *
+ * @example
+ * ```typescript
+ * import fallom from 'fallom';
+ *
+ * // Normal usage (captures everything)
+ * fallom.trace.init();
+ *
+ * // Privacy mode (no prompts/completions stored)
+ * fallom.trace.init({ captureContent: false });
+ *
+ * fallom.trace.setSession("my-agent", sessionId);
+ * await agent.run(message); // Automatically traced
+ * ```
+ */
+declare function init$2(options?: {
+    apiKey?: string;
+    baseUrl?: string;
+    captureContent?: boolean;
+}): void;
+/**
+ * Set the current session context.
+ *
+ * All subsequent LLM calls in this async context will be
+ * automatically tagged with this configKey and sessionId.
+ *
+ * @param configKey - Your config name (e.g., "linkedin-agent")
+ * @param sessionId - Your session/conversation ID
+ *
+ * @example
+ * ```typescript
+ * trace.setSession("linkedin-agent", sessionId);
+ * await agent.run(message); // Automatically traced with session
+ * ```
+ */
+declare function setSession(configKey: string, sessionId: string): void;
+/**
+ * Run a function with session context.
+ * Use this to ensure session context propagates across async boundaries.
+ *
+ * @param configKey - Your config name
+ * @param sessionId - Your session ID
+ * @param fn - Function to run with session context
+ *
+ * @example
+ * ```typescript
+ * await trace.runWithSession("my-agent", sessionId, async () => {
+ *   await agent.run(message); // Has session context
+ * });
+ * ```
+ */
+declare function runWithSession<T>(configKey: string, sessionId: string, fn: () => T): T;
+/**
+ * Get current session context, if any.
+ */
+declare function getSession(): SessionContext | undefined;
+/**
+ * Clear session context.
+ */
+declare function clearSession(): void;
+/**
+ * Record custom business metrics. Latest value per field wins.
+ *
+ * Use this for metrics that OTEL can't capture automatically:
+ * - Outlier scores
+ * - Engagement metrics
+ * - Conversion rates
+ * - Any business-specific outcome
+ *
+ * @param data - Dict of metrics to record
+ * @param options - Optional session identifiers
+ * @param options.configKey - Config name (optional if setSession was called)
+ * @param options.sessionId - Session ID (optional if setSession was called)
+ *
+ * @example
+ * ```typescript
+ * // If session context is set:
+ * trace.span({ outlier_score: 0.8, engagement: 42 });
+ *
+ * // Or explicitly:
+ * trace.span(
+ *   { outlier_score: 0.8 },
+ *   { configKey: "linkedin-agent", sessionId: "user123-convo456" }
+ * );
+ * ```
+ */
+declare function span(data: Record<string, unknown>, options?: {
+    configKey?: string;
+    sessionId?: string;
+}): void;
+/**
+ * Shutdown the tracing SDK gracefully.
+ */
+declare function shutdown(): Promise<void>;
+
+declare const trace_clearSession: typeof clearSession;
+declare const trace_getSession: typeof getSession;
+declare const trace_runWithSession: typeof runWithSession;
+declare const trace_setSession: typeof setSession;
+declare const trace_shutdown: typeof shutdown;
+declare const trace_span: typeof span;
+declare namespace trace {
+  export { trace_clearSession as clearSession, trace_getSession as getSession, init$2 as init, trace_runWithSession as runWithSession, trace_setSession as setSession, trace_shutdown as shutdown, trace_span as span };
+}
+
+/**
+ * Fallom models module.
+ *
+ * Provides model A/B testing with versioned configs.
+ * Zero latency on get() - uses local hash + cached config.
+ *
+ * Design principles:
+ * - Never block user's app if Fallom is down
+ * - Very short timeouts (1-2 seconds max)
+ * - Always return a usable model (fallback if needed)
+ * - Background sync keeps configs fresh
+ */
+/**
+ * Initialize Fallom models.
+ *
+ * This is optional - get() will auto-init if needed.
+ * Non-blocking: starts background config fetch immediately.
+ */
+declare function init$1(options?: {
+    apiKey?: string;
+    baseUrl?: string;
+}): void;
+/**
+ * Get model assignment for a session.
+ *
+ * This is zero latency - uses local hash computation + cached config.
+ * No network call on the hot path.
+ *
+ * Same session_id always returns same model (sticky assignment).
+ *
+ * Also automatically sets trace context, so all subsequent LLM calls
+ * are tagged with this session.
+ *
+ * @param configKey - Your config name (e.g., "linkedin-agent")
+ * @param sessionId - Your session/conversation ID (must be consistent)
+ * @param options - Optional settings
+ * @param options.version - Pin to specific version (1, 2, etc). undefined = latest
+ * @param options.fallback - Model to return if config not found or Fallom is down
+ * @param options.debug - Enable debug logging
+ * @returns Model string (e.g., "claude-opus", "gpt-4o")
+ * @throws Error if config not found AND no fallback provided
+ */
+declare function get(configKey: string, sessionId: string, options?: {
+    version?: number;
+    fallback?: string;
+    debug?: boolean;
+}): Promise<string>;
+
+declare const models_get: typeof get;
+declare namespace models {
+  export { models_get as get, init$1 as init };
+}
+
+/**
+ * Combined initialization for both trace and models.
+ */
+interface InitOptions {
+    apiKey?: string;
+    baseUrl?: string;
+    captureContent?: boolean;
+}
+/**
+ * Initialize both trace and models at once.
+ *
+ * @param options - Configuration options
+ * @param options.apiKey - Your Fallom API key. Defaults to FALLOM_API_KEY env var.
+ * @param options.baseUrl - API base URL. Defaults to FALLOM_BASE_URL or https://spans.fallom.com
+ * @param options.captureContent - Whether to capture prompt/completion content (default: true)
+ *
+ * @example
+ * ```typescript
+ * import fallom from 'fallom';
+ *
+ * // Basic initialization
+ * fallom.init({ apiKey: "your-api-key" });
+ *
+ * // Local development
+ * fallom.init({ baseUrl: "http://localhost:8001" });
+ *
+ * // Privacy mode
+ * fallom.init({ captureContent: false });
+ * ```
+ */
+declare function init(options?: InitOptions): void;
+
+/**
+ * Fallom - Model A/B testing and tracing for LLM applications.
+ *
+ * @example
+ * ```typescript
+ * import fallom from 'fallom';
+ *
+ * // Initialize (call this early, before LLM imports if possible)
+ * fallom.init({ apiKey: "your-api-key" });
+ *
+ * // Set session context for tracing
+ * fallom.trace.setSession("my-agent", sessionId);
+ *
+ * // Get A/B tested model
+ * const model = await fallom.models.get("my-config", sessionId, {
+ *   fallback: "gpt-4o-mini"
+ * });
+ *
+ * // Use with OpenAI
+ * const response = await openai.chat.completions.create({
+ *   model,
+ *   messages: [...]
+ * });
+ *
+ * // Record custom metrics
+ * fallom.trace.span({ user_satisfaction: 5 });
+ * ```
+ */
+
+declare const _default: {
+    init: typeof init;
+    trace: typeof trace;
+    models: typeof models;
+};
+
+export { type InitOptions, _default as default, init, models, trace };