lupislabs 1.0.0 → 1.0.2

package/README.md

@@ -1,476 +1,345 @@
-# Lupis JavaScript SDK
+# Lupis Labs JavaScript SDK
 
-[![npm version](https://badge.fury.io/js/%40lupislabs%2Fjs-sdk.svg)](https://badge.fury.io/js/%40lupislabs%2Fjs-sdk)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+JavaScript SDK for LupisLabs with OpenTelemetry tracing and custom event tracking.
 
-A pure OpenTelemetry-based SDK for tracing HTTP requests in JavaScript applications with automatic instrumentation.
+## Installation
 
-## ✨ Features
+```bash
+npm install lupislabs
+```
 
-- 🔭 **OpenTelemetry Native** - Uses official OpenTelemetry auto-instrumentation
-- 🚀 **Zero Manual Patching** - Automatic HTTP/fetch interception
-- 📊 **OTLP Export** - Standard protocol for any observability backend
-- 💬 **Conversation Grouping** - Group related requests with chatId
-- 🎯 **Smart Provider Detection** - Automatically detects AI providers (OpenAI, Claude, etc.)
-- 📝 **Full Request/Response Capture** - Automatically captures request & response bodies, headers, and status (extends OpenTelemetry's capabilities)
-- 🔄 **No Conflicts** - Works alongside other OpenTelemetry instrumentation
-- ⚡ **Lightweight** - Clean, minimal codebase
+### Development-only install
 
-## 📦 Installation
+If you only need Lupis for local debugging, install it as a dev dependency:
 
 ```bash
-npm install @lupislabs/js-sdk
+npm install --save-dev lupislabs
 ```
 
-## 🚀 Quick Start
+When the SDK is not explicitly enabled it stays idle, so keeping it in `devDependencies` will not impact your production runtime.
+
+## Features
+
+- 🔍 **Automatic HTTP Tracing**: Captures AI API calls automatically
+- 💬 **Chat ID Support**: Group traces by conversation/chat ID
+- 🎯 **TypeScript Support**: Full TypeScript definitions included
+- 🔒 **Privacy-First**: Never collects request/response bodies, only analytics data
+
+## Quick Start
 
 ```javascript
-import LupisSDK from '@lupislabs/js-sdk';
+import LupisSDK from 'lupislabs';
 
-const sdk = new LupisSDK({
+const lupis = LupisSDK.init({
   projectId: 'your-project-id',
-  otlpEndpoint: 'http://localhost:3010/v1/traces',
+  enabled: true, // Explicitly enable the SDK
 });
 
-await sdk.run(async () => {
-  const response = await fetch('https://api.openai.com/v1/chat/completions', {
-    method: 'POST',
-    headers: {
-      'Authorization': `Bearer ${API_KEY}`,
-      'Content-Type': 'application/json',
-    },
-    body: JSON.stringify({
-      model: 'gpt-4',
-      messages: [{ role: 'user', content: 'Hello!' }],
-    }),
-  });
-  
-  return await response.json();
-}, { chatId: 'conversation-123' });
 ```
 
-## 📝 Request & Response Capture
+The SDK caches the configured instance globally. Subsequent calls to `LupisSDK.init(...)` return the same instance, and you can fetch it anywhere with `LupisSDK.getInstance()`.
 
-Unlike standard OpenTelemetry implementations that only capture metadata, Lupis SDK automatically captures full request and response bodies, headers, and status codes. This is achieved through a custom `ResponseCaptureProcessor` that extends OpenTelemetry's capabilities.
-
-### What Gets Captured
-
-**Request Data:**
-
-- Request body (JSON, text, etc.)
-- Request headers
-- HTTP method and URL
-
-**Response Data:**
-
-- Response body (full content)
-- Response headers
-- HTTP status code
-- Response timing
+```typescript
+// In another module
+import LupisSDK from 'lupislabs';
 
-### How It Works
+const lupis = LupisSDK.getInstance();
+if (lupis?.isEnabled()) {
+  lupis.trackEvent('feature_used', { feature: 'export_data' });
+}
+```
 
-The SDK uses a custom span processor that:
+## Checking the active state
 
-1. Intercepts the fetch API at the earliest point
-2. Captures request data before the request is sent
-3. Clones and reads the response body without affecting the original response
-4. Attaches all data to the OpenTelemetry span as attributes
+You can verify whether the SDK activated:
 
-### Example
+```typescript
+const lupis = LupisSDK.init({ projectId: 'your-project-id' });
 
-```javascript
-await sdk.run(async () => {
-  const response = await fetch('https://api.openai.com/v1/chat/completions', {
-    method: 'POST',
-    headers: {
-      'Content-Type': 'application/json',
-      'Authorization': `Bearer ${API_KEY}`,
-    },
-    body: JSON.stringify({
-      model: 'gpt-4',
-      messages: [{ role: 'user', content: 'Hello!' }],
-    }),
-  });
-  
-  const data = await response.json();
-}, { chatId: 'my-conversation' });
+if (!lupis.isEnabled()) {
+  console.log('Lupis SDK is currently disabled.');
+}
 ```
 
-The span will automatically include:
-
-- `http.request.body` - The full request payload
-- `http.request.headers` - All request headers (as JSON string)
-- `http.response.body` - The complete response body
-- `http.response.headers` - All response headers (as JSON string)
-- `http.response.status` - HTTP status code (e.g., 200, 404, 500)
-
-See `examples/response-capture-example.js` for more detailed examples.
-
-## 📖 Configuration
+## Configuration
 
 ```typescript
 interface LupisConfig {
-  projectId: string;           // Required: Your project identifier
-  enabled?: boolean;           // Default: true
-  otlpEndpoint?: string;       // Default: 'http://localhost:4318/v1/traces'
-  serviceName?: string;        // Default: 'lupis-sdk'
-  serviceVersion?: string;     // Default: '1.0.0'
+  projectId: string;
+  enabled?: boolean;
+  serviceName?: string;
+  serviceVersion?: string;
+  filterSensitiveData?: boolean;
+  sensitiveDataPatterns?: string[];
+  redactionMode?: 'mask' | 'remove' | 'hash';
 }
-
-const sdk = new LupisSDK(config);
 ```
 
-## 🔭 OpenTelemetry Integration
-
-This SDK uses **pure OpenTelemetry** with automatic instrumentation:
+- `projectId` (required): Your project identifier
+- `enabled` (optional): Enable/disable tracking. When omitted, the SDK checks `LUPIS_SDK_ENABLED` first, then enables automatically when `NODE_ENV` is defined and not `'production'`. Without those hints it stays disabled (ideal for devDependency usage).
+- `serviceName` (optional): Service name for traces (default: 'lupis-sdk')
+- `serviceVersion` (optional): Service version (default: '1.0.0')
+- `filterSensitiveData` (optional): Enable sensitive data filtering (default: true)
+- `sensitiveDataPatterns` (optional): Custom regex patterns to filter (default: common API keys, tokens, etc.)
+- `redactionMode` (optional): How to redact sensitive data: 'mask', 'remove', or 'hash' (default: 'mask')
 
-- **Browser**: `@opentelemetry/instrumentation-fetch` auto-instruments `fetch()`
-- **Node.js**: `@opentelemetry/instrumentation-http` auto-instruments `http` and `https`
+All telemetry is sent to `http://127.0.0.1:9009` by default:
 
-### Features
+- Traces → `http://127.0.0.1:9009/api/traces`
 
-✅ **Automatic span creation** for all HTTP requests  
-✅ **Semantic conventions** (http.method, http.url, http.status_code)  
-✅ **W3C Trace Context** propagation  
-✅ **OTLP export** to any backend (Jaeger, Tempo, Datadog, etc.)  
-✅ **Custom attributes** (projectId, chatId, provider)  
+## Conversation Grouping
 
-### Architecture
-
-```
-HTTP Request (fetch/http)
-    ↓
-Custom Fetch Patch (captures request/response data)
-    ↓
-OpenTelemetry Auto-Instrumentation
-    ├─ FetchInstrumentation (browser)
-    └─ HttpInstrumentation (Node.js)
-    ↓
-Span Created with Attributes
-    ↓
-TracerProvider
-    ├─ ChatIdSpanProcessor (adds chatId)
-    ├─ ResponseCaptureProcessor (adds request/response bodies)
-    └─ BatchSpanProcessor + OTLPExporter
-        ↓
-    Observability Backend
-```
-
-## 📋 Usage Examples
-
-### Basic HTTP Tracing
+Group traces by conversation/thread using `chatId`:
 
 ```javascript
-await sdk.run(async () => {
-  const response = await fetch('https://api.example.com/data');
-  return await response.json();
-}, { chatId: 'my-conversation' });
-```
+// Set global chat ID for all subsequent traces
+lupis.setChatId('conversation_123');
 
-### Multiple Requests
+// Or set per-operation chat ID
+lupis.run(async () => {
+  // Your AI conversation code here
+}, { chatId: 'conversation_123' });
 
-```javascript
-await sdk.run(async () => {
-  const user = await fetch('/api/user').then(r => r.json());
-  const posts = await fetch('/api/posts').then(r => r.json());
-  return { user, posts };
-}, { chatId: 'user-session-123' });
+lupis.clearChatId();
 ```
 
-### AI Provider Requests
+## Metadata Types
 
-```javascript
-// OpenAI
-await sdk.run(async () => {
-  const response = await fetch('https://api.openai.com/v1/chat/completions', {
-    method: 'POST',
-    headers: { 'Authorization': `Bearer ${API_KEY}` },
-    body: JSON.stringify({
-      model: 'gpt-4',
-      messages: [{ role: 'user', content: 'Hello!' }],
-    }),
-  });
-}, { chatId: 'openai-conversation' });
-
-// Claude
-await sdk.run(async () => {
-  const response = await fetch('https://api.anthropic.com/v1/messages', {
-    method: 'POST',
-    headers: { 'x-api-key': API_KEY },
-    body: JSON.stringify({
-      model: 'claude-3-sonnet-20240229',
-      messages: [{ role: 'user', content: 'Hello!' }],
-    }),
-  });
-}, { chatId: 'claude-conversation' });
-```
-
-### Custom Spans
+### sessionId vs chatId
 
-```javascript
-import { otel } from '@lupislabs/js-sdk';
-
-const span = sdk.createSpan('data-processing', {
-  'processing.type': 'batch',
-  'batch.size': 100,
-}, otel.SpanKind.INTERNAL);
-
-try {
-  // Your processing logic
-  span.end();
-} catch (error) {
-  span.recordException(error);
-  span.setStatus({ 
-    code: otel.SpanStatusCode.ERROR, 
-    message: error.message 
-  });
-  span.end();
-}
-```
+- **`sessionId`**: Browser/app session identifier that persists across conversations
+  - Used for analytics and user journey tracking
+  - Example: `'browser_session_abc123'`
+  
+- **`chatId`**: Individual conversation/thread identifier
+  - Used for grouping related traces within a conversation
+  - Changes for each new conversation
+  - Example: `'chat_thread_xyz789'`
 
-### Using OpenTelemetry API Directly
+### Example Usage
 
 ```javascript
-const tracer = sdk.getTracer();
-
-const span = tracer.startSpan('custom-operation', {
-  attributes: {
-    'operation.type': 'ai-inference',
-  },
+// Set user context (persists across conversations)
+lupis.setMetadata({
+  userId: 'user_123',
+  organizationId: 'org_456',
+  sessionId: 'browser_session_abc123', // Same across conversations
 });
 
-// Your code
-span.end();
-```
+// Start a new conversation
+lupis.setChatId('conversation_1');
 
-## 🔌 Export to Observability Backends
+await lupis.run(async () => {
+  // AI conversation code
+}, { chatId: 'conversation_1' });
 
-### Jaeger
+// Start another conversation (same session, different chat)
+lupis.setChatId('conversation_2');
 
-```javascript
-const sdk = new LupisSDK({
-  projectId: 'my-project',
-  otlpEndpoint: 'http://localhost:4318/v1/traces',
-});
+await lupis.run(async () => {
+  // Another AI conversation code
+}, { chatId: 'conversation_2' });
 ```
 
-### Grafana Tempo
+## OpenTelemetry Integration
+
+The SDK automatically instruments HTTP requests and creates traces. Access the tracer:
 
 ```javascript
-const sdk = new LupisSDK({
-  projectId: 'my-project',
-  otlpEndpoint: 'https://tempo.example.com/v1/traces',
+const tracer = lupis.getTracer();
+
+const span = lupis.createSpan('custom-operation', {
+  'custom.attribute': 'value',
 });
-```
 
-### Datadog
+// Your code here
 
-```javascript
-const sdk = new LupisSDK({
-  projectId: 'my-project',
-  otlpEndpoint: 'https://http-intake.logs.datadoghq.com/v1/traces',
-});
+span.end();
 ```
 
-### New Relic
+## Data Collection
 
-```javascript
-const sdk = new LupisSDK({
-  projectId: 'my-project',
-  otlpEndpoint: 'https://otlp.nr-data.net/v1/traces',
-});
-```
+The SDK collects only analytics-focused data while protecting sensitive information:
 
-## 💬 Conversation Grouping
+### ✅ **Collected Data**
 
-Group related requests with `chatId`:
+- **HTTP Metadata**: URL, method, status code, duration, headers (filtered)
+- **Token Usage**: Input/output/cache tokens from AI providers
+- **Cost Analytics**: Calculated costs based on token usage and provider pricing
+- **Model Information**: AI model used for requests
+- **User Context**: User ID, organization ID, session ID, chat ID
+- **Performance Metrics**: Response times, error rates, success/failure status
 
-```javascript
-// All requests in this block will have the same chatId
-await sdk.run(async () => {
-  await fetch('/api/chat', { 
-    method: 'POST',
-    body: JSON.stringify({ message: 'Hello' })
-  });
-  await fetch('/api/chat', { 
-    method: 'POST',
-    body: JSON.stringify({ message: 'How are you?' })
-  });
-}, { chatId: 'conversation-123' });
-
-// Or set/clear chatId manually
-sdk.setChatId('conversation-123');
-// Make requests...
-sdk.clearChatId();
-```
+### ❌ **Never Collected**
 
-## 🏷️ Span Attributes
+- **Request Bodies**: Full request payloads are never captured
+- **Response Bodies**: Full response content is never captured
+- **Sensitive Data**: API keys, tokens, passwords (filtered by default)
+- **Personal Information**: PII is not collected by default
 
-All HTTP spans automatically include:
+### 🔒 **Privacy Protection**
 
-**Standard OpenTelemetry:**
+- Sensitive data filtering enabled by default
+- Request/response bodies skipped to reduce span size
+- Focus on analytics and cost tracking only
+- User-controlled data collection
 
-- `http.method` - HTTP method (GET, POST, etc.)
-- `http.url` - Full URL
-- `http.status_code` - Response status code
+## Sensitive Data Filtering
 
-**Custom Lupis Attributes:**
+The SDK automatically filters sensitive data in production to protect API keys, tokens, and other sensitive information. This feature is **enabled by default** for security.
 
-- `lupis.project.id` - Your project ID
-- `lupis.chat.id` - Conversation ID (when set)
-- `http.provider` - Detected provider (openai, claude, cohere, huggingface, google)
+### Default Filtering
 
-**Request/Response Capture Attributes (Custom Extension):**
+The SDK automatically filters these common sensitive patterns:
 
-- `http.request.body` - Full request body content
-- `http.request.headers` - Request headers as JSON string
-- `http.response.body` - Full response body content
-- `http.response.headers` - Response headers as JSON string
-- `http.response.status` - HTTP response status code
+#### API Keys & Tokens
 
-## 🔧 API Reference
+- `sk-[a-zA-Z0-9]{20,}` - OpenAI API keys
+- `pk_[a-zA-Z0-9]{20,}` - Paddle API keys  
+- `ak-[a-zA-Z0-9]{20,}` - Anthropic API keys
+- `Bearer [a-zA-Z0-9._-]+` - Bearer tokens
+- `x-api-key`, `authorization` - API key headers
 
-### LupisSDK
+#### Authentication
 
-```typescript
-class LupisSDK {
-  constructor(config: LupisConfig)
-  
-  async run<T>(fn: () => Promise<T> | T, options?: { chatId?: string }): Promise<T>
-  
-  setChatId(chatId: string): void
-  clearChatId(): void
-  
-  getTracer(): Tracer
-  createSpan(name: string, attributes?: Attributes, spanKind?: SpanKind): Span
-  
-  async shutdown(): Promise<void>
-}
-```
+- `password`, `passwd`, `pwd` - Password fields
+- `token`, `access_token`, `refresh_token`, `session_token` - Various tokens
+- `secret`, `private_key`, `api_secret` - Secret fields
 
-### Methods
+#### Personal Data
 
-#### `sdk.run(fn, options)`
+- `ssn`, `social_security` - Social Security Numbers
+- `credit_card`, `card_number` - Credit card numbers
+- `cvv`, `cvc` - Security codes
 
-Execute a function with automatic HTTP tracing:
+### Redaction Modes
 
-```typescript
-await sdk.run(async () => {
-  // Your code
-}, { chatId: 'optional-chat-id' });
-```
+Choose how sensitive data is replaced when you initialize the SDK:
 
-#### `sdk.setChatId(chatId)` / `sdk.clearChatId()`
+#### Mask Mode (Default)
 
-Manually set/clear the chatId for subsequent requests:
+```javascript
+const lupis = LupisSDK.init({
+  projectId: 'your-project-id',
+  redactionMode: 'mask', // Default
+});
 
-```typescript
-sdk.setChatId('conversation-123');
-// All requests will have this chatId
-sdk.clearChatId();
+// Examples:
+// sk-1234567890abcdef1234567890abcdef12345678 → sk-1***5678
+// Bearer sk-1234567890abcdef1234567890abcdef12345678 → Bear***5678
+// password: 'secret-password' → password: '***'
 ```
 
-#### `sdk.getTracer()`
+#### Remove Mode
 
-Get the OpenTelemetry tracer for advanced usage:
+```javascript
+const lupis = LupisSDK.init({
+  projectId: 'your-project-id',
+  redactionMode: 'remove',
+});
 
-```typescript
-const tracer = sdk.getTracer();
-const span = tracer.startSpan('my-operation');
+// Examples:
+// sk-1234567890abcdef1234567890abcdef12345678 → [REDACTED]
+// password: 'secret-password' → password: [REDACTED]
 ```
 
-#### `sdk.createSpan(name, attributes, spanKind)`
+#### Hash Mode
 
-Create a custom span:
+```javascript
+const lupis = LupisSDK.init({
+  projectId: 'your-project-id',
+  redactionMode: 'hash',
+});
 
-```typescript
-const span = sdk.createSpan('operation-name', {
-  'custom.attribute': 'value',
-}, SpanKind.INTERNAL);
+// Examples:
+// sk-1234567890abcdef1234567890abcdef12345678 → [HASH:2dd0e9d5]
+// password: 'secret-password' → password: [HASHED]
 ```
 
-#### `sdk.shutdown()`
+### Custom Patterns
 
-Gracefully shutdown and flush all spans:
+Add your own sensitive data patterns during initialization:
 
-```typescript
-await sdk.shutdown();
+```javascript
+const lupis = LupisSDK.init({
+  projectId: 'your-project-id',
+  filterSensitiveData: true,
+  sensitiveDataPatterns: [
+    'sk-[a-zA-Z0-9]{20,}', // OpenAI API keys
+    'Bearer [a-zA-Z0-9._-]+', // Bearer tokens
+    'custom_secret', // Your custom field
+    'my_api_key', // Your custom field
+    'email', // Email addresses
+  ],
+  redactionMode: 'mask',
+});
 ```
 
-## 🎯 Provider Detection
+### What Gets Filtered
 
-The SDK automatically detects AI providers based on URL patterns:
+The SDK filters sensitive data in:
 
-| Provider | URL Pattern | Attribute Value |
-|----------|-------------|-----------------|
-| OpenAI | `api.openai.com` | `openai` |
-| Anthropic (Claude) | `api.anthropic.com` | `claude` |
-| Cohere | `api.cohere.ai` | `cohere` |
-| HuggingFace | `api.huggingface.co` | `huggingface` |
-| Google AI | `generativelanguage.googleapis.com` | `google` |
-| Others | - | `unknown` |
+- **Request Headers**: Authorization, API keys, tokens
+- **Span Attributes**: All OpenTelemetry span attributes
 
-## ⚙️ Best Practices
+**Note**: Request and response bodies are never collected, so no filtering is needed for them.
 
-### 1. Meaningful ChatIds
+### Disable Filtering (Development Only)
 
-```javascript
-// Good: Descriptive and unique
-{ chatId: 'user-login-flow-2024-03-15' }
-{ chatId: 'document-analysis-task-456' }
-
-// Avoid: Generic or unclear
-{ chatId: 'test' }
-{ chatId: '1' }
-```
-
-### 2. Always Shutdown
+⚠️ **Warning**: Only disable filtering in development environments:
 
 ```javascript
-process.on('SIGTERM', async () => {
-  await sdk.shutdown();
-  process.exit(0);
+const lupis = LupisSDK.init({
+  projectId: 'your-project-id',
+  filterSensitiveData: false, // ⚠️ Sensitive data will be exposed!
 });
 ```
 
-### 3. Error Handling
+### Production Security
 
-```javascript
-await sdk.run(async () => {
-  try {
-    const response = await fetch('/api/data');
-    if (!response.ok) {
-      throw new Error(`HTTP error! status: ${response.status}`);
-    }
-    return await response.json();
-  } catch (error) {
-    console.error('Request failed:', error);
-    throw error;
-  }
-}, { chatId: 'error-handling-example' });
-```
+- ✅ **Enabled by default** - No configuration needed
+- ✅ **Comprehensive coverage** - Common sensitive patterns included
+- ✅ **Configurable** - Add custom patterns as needed
+- ✅ **Performance optimized** - Minimal impact when enabled
+- ✅ **Debugging friendly** - Mask mode preserves partial data for debugging
+
+## Shutdown
 
-## 📚 Documentation
+The SDK automatically flushes buffered telemetry on an interval and registers Node.js signal hooks so traces are shipped during normal shutdown—no manual shutdown call is required for normal apps. Only call `await lupis.shutdown()` when you need to force an immediate flush (for example, in short-lived scripts or tests).
 
-- [ARCHITECTURE.md](./ARCHITECTURE.md) - Detailed architecture
-- [OPENTELEMETRY.md](./OPENTELEMETRY.md) - OpenTelemetry integration guide
-- [examples/](./examples/) - Code examples
+## Testing
 
-## 🤝 Contributing
+The SDK includes comprehensive tests for sensitive data filtering:
+
+```bash
+# Run unit tests
+npm run test:sensitive:unit
+
+# Run integration tests
+npm run test:sensitive:detailed
+
+# Run all tests
+npm run test:all
+```
 
-Contributions are welcome! Please feel free to submit a Pull Request.
+### Test Coverage
 
-## 📄 License
+- ✅ **Unit Tests**: Filter utility with all redaction modes
+- ✅ **Integration Tests**: SDK with HTTP interception
+- ✅ **Custom Patterns**: User-defined sensitive data patterns
+- ✅ **Disabled Filtering**: Development mode verification
+- ✅ **Request/Response**: Headers, bodies, and span attributes
 
-This project is licensed under the MIT License.
+## Examples
 
-## 🆘 Support
+See the `examples/` directory for more usage examples:
 
-If you encounter any issues:
-1. Check the [Issues](https://github.com/lupislabs/lupis/issues) page
-2. Create a new issue with detailed information
+- `event-tracking-example.js` - Custom event tracking
+- `anthropic-example.js` - Anthropic API integration
+- `openai-example.js` - OpenAI API integration
+- `langchain-example.js` - LangChain integration
+- `streaming-example.js` - Streaming responses
 
----
+## License
 
 **Made with ❤️ by the Lupis team**