lupislabs 1.0.0 → 1.0.1

package/README.md

@@ -1,476 +1,338 @@
-# Lupis JavaScript SDK
+# Lupis Labs JavaScript SDK
 
-[![npm version](https://badge.fury.io/js/%40lupislabs%2Fjs-sdk.svg)](https://badge.fury.io/js/%40lupislabs%2Fjs-sdk)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+JavaScript SDK for LupisLabs with OpenTelemetry tracing and custom event tracking.
 
-A pure OpenTelemetry-based SDK for tracing HTTP requests in JavaScript applications with automatic instrumentation.
-
-## ✨ Features
-
-- 🔭 **OpenTelemetry Native** - Uses official OpenTelemetry auto-instrumentation
-- 🚀 **Zero Manual Patching** - Automatic HTTP/fetch interception
-- 📊 **OTLP Export** - Standard protocol for any observability backend
-- 💬 **Conversation Grouping** - Group related requests with chatId
-- 🎯 **Smart Provider Detection** - Automatically detects AI providers (OpenAI, Claude, etc.)
-- 📝 **Full Request/Response Capture** - Automatically captures request & response bodies, headers, and status (extends OpenTelemetry's capabilities)
-- 🔄 **No Conflicts** - Works alongside other OpenTelemetry instrumentation
-- ⚡ **Lightweight** - Clean, minimal codebase
-
-## 📦 Installation
+## Installation
 
 ```bash
-npm install @lupislabs/js-sdk
+npm install lupislabs
 ```
 
-## 🚀 Quick Start
+### Development-only install
 
-```javascript
-import LupisSDK from '@lupislabs/js-sdk';
-
-const sdk = new LupisSDK({
-  projectId: 'your-project-id',
-  otlpEndpoint: 'http://localhost:3010/v1/traces',
-});
+If you only need Lupis for local debugging, install it as a dev dependency:
 
-await sdk.run(async () => {
-  const response = await fetch('https://api.openai.com/v1/chat/completions', {
-    method: 'POST',
-    headers: {
-      'Authorization': `Bearer ${API_KEY}`,
-      'Content-Type': 'application/json',
-    },
-    body: JSON.stringify({
-      model: 'gpt-4',
-      messages: [{ role: 'user', content: 'Hello!' }],
-    }),
-  });
-  
-  return await response.json();
-}, { chatId: 'conversation-123' });
+```bash
+npm install --save-dev lupislabs
 ```
 
-## 📝 Request & Response Capture
+When the SDK is not explicitly enabled it stays idle, so keeping it in `devDependencies` will not impact your production runtime.
 
-Unlike standard OpenTelemetry implementations that only capture metadata, Lupis SDK automatically captures full request and response bodies, headers, and status codes. This is achieved through a custom `ResponseCaptureProcessor` that extends OpenTelemetry's capabilities.
+## Features
 
-### What Gets Captured
+- 🔍 **Automatic HTTP Tracing**: Captures AI API calls automatically
+- 💬 **Chat ID Support**: Group traces by conversation/chat ID
+- 🎯 **TypeScript Support**: Full TypeScript definitions included
+- 🔒 **Privacy-First**: Never collects request/response bodies, only analytics data
 
-**Request Data:**
+## Quick Start
 
-- Request body (JSON, text, etc.)
-- Request headers
-- HTTP method and URL
-
-**Response Data:**
+```javascript
+import LupisSDK from 'lupislabs';
 
-- Response body (full content)
-- Response headers
-- HTTP status code
-- Response timing
+const lupis = new LupisSDK({
+  projectId: 'your-project-id',
+  enabled: true, // Explicitly enable the SDK
+});
 
-### How It Works
+await lupis.shutdown();
+```
 
-The SDK uses a custom span processor that:
+## Checking the active state
 
-1. Intercepts the fetch API at the earliest point
-2. Captures request data before the request is sent
-3. Clones and reads the response body without affecting the original response
-4. Attaches all data to the OpenTelemetry span as attributes
+You can verify whether the SDK activated:
 
-### Example
+```typescript
+const lupis = new LupisSDK({ projectId: 'your-project-id' });
 
-```javascript
-await sdk.run(async () => {
-  const response = await fetch('https://api.openai.com/v1/chat/completions', {
-    method: 'POST',
-    headers: {
-      'Content-Type': 'application/json',
-      'Authorization': `Bearer ${API_KEY}`,
-    },
-    body: JSON.stringify({
-      model: 'gpt-4',
-      messages: [{ role: 'user', content: 'Hello!' }],
-    }),
-  });
-  
-  const data = await response.json();
-}, { chatId: 'my-conversation' });
+if (!lupis.isEnabled()) {
+  console.log('Lupis SDK is currently disabled.');
+}
 ```
 
-The span will automatically include:
-
-- `http.request.body` - The full request payload
-- `http.request.headers` - All request headers (as JSON string)
-- `http.response.body` - The complete response body
-- `http.response.headers` - All response headers (as JSON string)
-- `http.response.status` - HTTP status code (e.g., 200, 404, 500)
-
-See `examples/response-capture-example.js` for more detailed examples.
-
-## 📖 Configuration
+## Configuration
 
 ```typescript
 interface LupisConfig {
-  projectId: string;           // Required: Your project identifier
-  enabled?: boolean;           // Default: true
-  otlpEndpoint?: string;       // Default: 'http://localhost:4318/v1/traces'
-  serviceName?: string;        // Default: 'lupis-sdk'
-  serviceVersion?: string;     // Default: '1.0.0'
+  projectId: string;
+  enabled?: boolean;
+  serviceName?: string;
+  serviceVersion?: string;
+  filterSensitiveData?: boolean;
+  sensitiveDataPatterns?: string[];
+  redactionMode?: 'mask' | 'remove' | 'hash';
 }
-
-const sdk = new LupisSDK(config);
 ```
 
-## 🔭 OpenTelemetry Integration
-
-This SDK uses **pure OpenTelemetry** with automatic instrumentation:
+- `projectId` (required): Your project identifier
+- `enabled` (optional): Enable/disable tracking. When omitted, the SDK checks `LUPIS_SDK_ENABLED` first, then enables automatically when `NODE_ENV` is defined and not `'production'`. Without those hints it stays disabled (ideal for devDependency usage).
+- `serviceName` (optional): Service name for traces (default: 'lupis-sdk')
+- `serviceVersion` (optional): Service version (default: '1.0.0')
+- `filterSensitiveData` (optional): Enable sensitive data filtering (default: true)
+- `sensitiveDataPatterns` (optional): Custom regex patterns to filter (default: common API keys, tokens, etc.)
+- `redactionMode` (optional): How to redact sensitive data: 'mask', 'remove', or 'hash' (default: 'mask')
 
-- **Browser**: `@opentelemetry/instrumentation-fetch` auto-instruments `fetch()`
-- **Node.js**: `@opentelemetry/instrumentation-http` auto-instruments `http` and `https`
+All telemetry is sent to `http://127.0.0.1:9009` by default:
 
-### Features
+- Traces → `http://127.0.0.1:9009/api/traces`
 
-✅ **Automatic span creation** for all HTTP requests  
-✅ **Semantic conventions** (http.method, http.url, http.status_code)  
-✅ **W3C Trace Context** propagation  
-✅ **OTLP export** to any backend (Jaeger, Tempo, Datadog, etc.)  
-✅ **Custom attributes** (projectId, chatId, provider)  
+## Conversation Grouping
 
-### Architecture
-
-```
-HTTP Request (fetch/http)
-    ↓
-Custom Fetch Patch (captures request/response data)
-    ↓
-OpenTelemetry Auto-Instrumentation
-    ├─ FetchInstrumentation (browser)
-    └─ HttpInstrumentation (Node.js)
-    ↓
-Span Created with Attributes
-    ↓
-TracerProvider
-    ├─ ChatIdSpanProcessor (adds chatId)
-    ├─ ResponseCaptureProcessor (adds request/response bodies)
-    └─ BatchSpanProcessor + OTLPExporter
-        ↓
-    Observability Backend
-```
-
-## 📋 Usage Examples
-
-### Basic HTTP Tracing
+Group traces by conversation/thread using `chatId`:
 
 ```javascript
-await sdk.run(async () => {
-  const response = await fetch('https://api.example.com/data');
-  return await response.json();
-}, { chatId: 'my-conversation' });
-```
+// Set global chat ID for all subsequent traces
+lupis.setChatId('conversation_123');
 
-### Multiple Requests
+// Or set per-operation chat ID
+lupis.run(async () => {
+  // Your AI conversation code here
+}, { chatId: 'conversation_123' });
 
-```javascript
-await sdk.run(async () => {
-  const user = await fetch('/api/user').then(r => r.json());
-  const posts = await fetch('/api/posts').then(r => r.json());
-  return { user, posts };
-}, { chatId: 'user-session-123' });
+lupis.clearChatId();
 ```
 
-### AI Provider Requests
-
-```javascript
-// OpenAI
-await sdk.run(async () => {
-  const response = await fetch('https://api.openai.com/v1/chat/completions', {
-    method: 'POST',
-    headers: { 'Authorization': `Bearer ${API_KEY}` },
-    body: JSON.stringify({
-      model: 'gpt-4',
-      messages: [{ role: 'user', content: 'Hello!' }],
-    }),
-  });
-}, { chatId: 'openai-conversation' });
-
-// Claude
-await sdk.run(async () => {
-  const response = await fetch('https://api.anthropic.com/v1/messages', {
-    method: 'POST',
-    headers: { 'x-api-key': API_KEY },
-    body: JSON.stringify({
-      model: 'claude-3-sonnet-20240229',
-      messages: [{ role: 'user', content: 'Hello!' }],
-    }),
-  });
-}, { chatId: 'claude-conversation' });
-```
+## Metadata Types
 
-### Custom Spans
+### sessionId vs chatId
 
-```javascript
-import { otel } from '@lupislabs/js-sdk';
-
-const span = sdk.createSpan('data-processing', {
-  'processing.type': 'batch',
-  'batch.size': 100,
-}, otel.SpanKind.INTERNAL);
-
-try {
-  // Your processing logic
-  span.end();
-} catch (error) {
-  span.recordException(error);
-  span.setStatus({ 
-    code: otel.SpanStatusCode.ERROR, 
-    message: error.message 
-  });
-  span.end();
-}
-```
+- **`sessionId`**: Browser/app session identifier that persists across conversations
+  - Used for analytics and user journey tracking
+  - Example: `'browser_session_abc123'`
+  
+- **`chatId`**: Individual conversation/thread identifier
+  - Used for grouping related traces within a conversation
+  - Changes for each new conversation
+  - Example: `'chat_thread_xyz789'`
 
-### Using OpenTelemetry API Directly
+### Example Usage
 
 ```javascript
-const tracer = sdk.getTracer();
-
-const span = tracer.startSpan('custom-operation', {
-  attributes: {
-    'operation.type': 'ai-inference',
-  },
+// Set user context (persists across conversations)
+lupis.setMetadata({
+  userId: 'user_123',
+  organizationId: 'org_456',
+  sessionId: 'browser_session_abc123', // Same across conversations
 });
 
-// Your code
-span.end();
-```
+// Start a new conversation
+lupis.setChatId('conversation_1');
 
-## 🔌 Export to Observability Backends
+await lupis.run(async () => {
+  // AI conversation code
+}, { chatId: 'conversation_1' });
 
-### Jaeger
+// Start another conversation (same session, different chat)
+lupis.setChatId('conversation_2');
 
-```javascript
-const sdk = new LupisSDK({
-  projectId: 'my-project',
-  otlpEndpoint: 'http://localhost:4318/v1/traces',
-});
+await lupis.run(async () => {
+  // Another AI conversation code
+}, { chatId: 'conversation_2' });
 ```
 
-### Grafana Tempo
+## OpenTelemetry Integration
+
+The SDK automatically instruments HTTP requests and creates traces. Access the tracer:
 
 ```javascript
-const sdk = new LupisSDK({
-  projectId: 'my-project',
-  otlpEndpoint: 'https://tempo.example.com/v1/traces',
+const tracer = lupis.getTracer();
+
+const span = lupis.createSpan('custom-operation', {
+  'custom.attribute': 'value',
 });
-```
 
-### Datadog
+// Your code here
 
-```javascript
-const sdk = new LupisSDK({
-  projectId: 'my-project',
-  otlpEndpoint: 'https://http-intake.logs.datadoghq.com/v1/traces',
-});
+span.end();
 ```
 
-### New Relic
+## Data Collection
 
-```javascript
-const sdk = new LupisSDK({
-  projectId: 'my-project',
-  otlpEndpoint: 'https://otlp.nr-data.net/v1/traces',
-});
-```
+The SDK collects only analytics-focused data while protecting sensitive information:
 
-## 💬 Conversation Grouping
+### ✅ **Collected Data**
 
-Group related requests with `chatId`:
+- **HTTP Metadata**: URL, method, status code, duration, headers (filtered)
+- **Token Usage**: Input/output/cache tokens from AI providers
+- **Cost Analytics**: Calculated costs based on token usage and provider pricing
+- **Model Information**: AI model used for requests
+- **User Context**: User ID, organization ID, session ID, chat ID
+- **Performance Metrics**: Response times, error rates, success/failure status
 
-```javascript
-// All requests in this block will have the same chatId
-await sdk.run(async () => {
-  await fetch('/api/chat', { 
-    method: 'POST',
-    body: JSON.stringify({ message: 'Hello' })
-  });
-  await fetch('/api/chat', { 
-    method: 'POST',
-    body: JSON.stringify({ message: 'How are you?' })
-  });
-}, { chatId: 'conversation-123' });
-
-// Or set/clear chatId manually
-sdk.setChatId('conversation-123');
-// Make requests...
-sdk.clearChatId();
-```
+### ❌ **Never Collected**
 
-## 🏷️ Span Attributes
+- **Request Bodies**: Full request payloads are never captured
+- **Response Bodies**: Full response content is never captured
+- **Sensitive Data**: API keys, tokens, passwords (filtered by default)
+- **Personal Information**: PII is not collected by default
 
-All HTTP spans automatically include:
+### 🔒 **Privacy Protection**
 
-**Standard OpenTelemetry:**
+- Sensitive data filtering enabled by default
+- Request/response bodies skipped to reduce span size
+- Focus on analytics and cost tracking only
+- User-controlled data collection
 
-- `http.method` - HTTP method (GET, POST, etc.)
-- `http.url` - Full URL
-- `http.status_code` - Response status code
+## Sensitive Data Filtering
 
-**Custom Lupis Attributes:**
+The SDK automatically filters sensitive data in production to protect API keys, tokens, and other sensitive information. This feature is **enabled by default** for security.
 
-- `lupis.project.id` - Your project ID
-- `lupis.chat.id` - Conversation ID (when set)
-- `http.provider` - Detected provider (openai, claude, cohere, huggingface, google)
+### Default Filtering
 
-**Request/Response Capture Attributes (Custom Extension):**
+The SDK automatically filters these common sensitive patterns:
 
-- `http.request.body` - Full request body content
-- `http.request.headers` - Request headers as JSON string
-- `http.response.body` - Full response body content
-- `http.response.headers` - Response headers as JSON string
-- `http.response.status` - HTTP response status code
+#### API Keys & Tokens
 
-## 🔧 API Reference
+- `sk-[a-zA-Z0-9]{20,}` - OpenAI API keys
+- `pk_[a-zA-Z0-9]{20,}` - Paddle API keys  
+- `ak-[a-zA-Z0-9]{20,}` - Anthropic API keys
+- `Bearer [a-zA-Z0-9._-]+` - Bearer tokens
+- `x-api-key`, `authorization` - API key headers
 
-### LupisSDK
+#### Authentication
 
-```typescript
-class LupisSDK {
-  constructor(config: LupisConfig)
-  
-  async run<T>(fn: () => Promise<T> | T, options?: { chatId?: string }): Promise<T>
-  
-  setChatId(chatId: string): void
-  clearChatId(): void
-  
-  getTracer(): Tracer
-  createSpan(name: string, attributes?: Attributes, spanKind?: SpanKind): Span
-  
-  async shutdown(): Promise<void>
-}
-```
+- `password`, `passwd`, `pwd` - Password fields
+- `token`, `access_token`, `refresh_token`, `session_token` - Various tokens
+- `secret`, `private_key`, `api_secret` - Secret fields
 
-### Methods
+#### Personal Data
 
-#### `sdk.run(fn, options)`
+- `ssn`, `social_security` - Social Security Numbers
+- `credit_card`, `card_number` - Credit card numbers
+- `cvv`, `cvc` - Security codes
 
-Execute a function with automatic HTTP tracing:
+### Redaction Modes
 
-```typescript
-await sdk.run(async () => {
-  // Your code
-}, { chatId: 'optional-chat-id' });
-```
+Choose how sensitive data is replaced:
 
-#### `sdk.setChatId(chatId)` / `sdk.clearChatId()`
+#### Mask Mode (Default)
 
-Manually set/clear the chatId for subsequent requests:
+```javascript
+const lupis = new LupisSDK({
+  projectId: 'your-project-id',
+  redactionMode: 'mask', // Default
+});
 
-```typescript
-sdk.setChatId('conversation-123');
-// All requests will have this chatId
-sdk.clearChatId();
+// Examples:
+// sk-1234567890abcdef1234567890abcdef12345678 → sk-1***5678
+// Bearer sk-1234567890abcdef1234567890abcdef12345678 → Bear***5678
+// password: 'secret-password' → password: '***'
 ```
 
-#### `sdk.getTracer()`
+#### Remove Mode
 
-Get the OpenTelemetry tracer for advanced usage:
+```javascript
+const lupis = new LupisSDK({
+  projectId: 'your-project-id',
+  redactionMode: 'remove',
+});
 
-```typescript
-const tracer = sdk.getTracer();
-const span = tracer.startSpan('my-operation');
+// Examples:
+// sk-1234567890abcdef1234567890abcdef12345678 → [REDACTED]
+// password: 'secret-password' → password: [REDACTED]
 ```
 
-#### `sdk.createSpan(name, attributes, spanKind)`
+#### Hash Mode
 
-Create a custom span:
+```javascript
+const lupis = new LupisSDK({
+  projectId: 'your-project-id',
+  redactionMode: 'hash',
+});
 
-```typescript
-const span = sdk.createSpan('operation-name', {
-  'custom.attribute': 'value',
-}, SpanKind.INTERNAL);
+// Examples:
+// sk-1234567890abcdef1234567890abcdef12345678 → [HASH:2dd0e9d5]
+// password: 'secret-password' → password: [HASHED]
 ```
 
-#### `sdk.shutdown()`
+### Custom Patterns
 
-Gracefully shutdown and flush all spans:
+Add your own sensitive data patterns:
 
-```typescript
-await sdk.shutdown();
+```javascript
+const lupis = new LupisSDK({
+  projectId: 'your-project-id',
+  filterSensitiveData: true,
+  sensitiveDataPatterns: [
+    'sk-[a-zA-Z0-9]{20,}', // OpenAI API keys
+    'Bearer [a-zA-Z0-9._-]+', // Bearer tokens
+    'custom_secret', // Your custom field
+    'my_api_key', // Your custom field
+    'email', // Email addresses
+  ],
+  redactionMode: 'mask',
+});
 ```
 
-## 🎯 Provider Detection
+### What Gets Filtered
 
-The SDK automatically detects AI providers based on URL patterns:
+The SDK filters sensitive data in:
 
-| Provider | URL Pattern | Attribute Value |
-|----------|-------------|-----------------|
-| OpenAI | `api.openai.com` | `openai` |
-| Anthropic (Claude) | `api.anthropic.com` | `claude` |
-| Cohere | `api.cohere.ai` | `cohere` |
-| HuggingFace | `api.huggingface.co` | `huggingface` |
-| Google AI | `generativelanguage.googleapis.com` | `google` |
-| Others | - | `unknown` |
+- **Request Headers**: Authorization, API keys, tokens
+- **Span Attributes**: All OpenTelemetry span attributes
 
-## ⚙️ Best Practices
+**Note**: Request and response bodies are never collected, so no filtering is needed for them.
 
-### 1. Meaningful ChatIds
+### Disable Filtering (Development Only)
 
-```javascript
-// Good: Descriptive and unique
-{ chatId: 'user-login-flow-2024-03-15' }
-{ chatId: 'document-analysis-task-456' }
-
-// Avoid: Generic or unclear
-{ chatId: 'test' }
-{ chatId: '1' }
-```
-
-### 2. Always Shutdown
+⚠️ **Warning**: Only disable filtering in development environments:
 
 ```javascript
-process.on('SIGTERM', async () => {
-  await sdk.shutdown();
-  process.exit(0);
+const lupis = new LupisSDK({
+  projectId: 'your-project-id',
+  filterSensitiveData: false, // ⚠️ Sensitive data will be exposed!
 });
 ```
 
-### 3. Error Handling
+### Production Security
+
+- ✅ **Enabled by default** - No configuration needed
+- ✅ **Comprehensive coverage** - Common sensitive patterns included
+- ✅ **Configurable** - Add custom patterns as needed
+- ✅ **Performance optimized** - Minimal impact when enabled
+- ✅ **Debugging friendly** - Mask mode preserves partial data for debugging
+
+## Shutdown
+
+Always call `shutdown()` to stop interception and flush pending traces:
 
 ```javascript
-await sdk.run(async () => {
-  try {
-    const response = await fetch('/api/data');
-    if (!response.ok) {
-      throw new Error(`HTTP error! status: ${response.status}`);
-    }
-    return await response.json();
-  } catch (error) {
-    console.error('Request failed:', error);
-    throw error;
-  }
-}, { chatId: 'error-handling-example' });
+await lupis.shutdown();
 ```
 
-## 📚 Documentation
+## Testing
+
+The SDK includes comprehensive tests for sensitive data filtering:
 
-- [ARCHITECTURE.md](./ARCHITECTURE.md) - Detailed architecture
-- [OPENTELEMETRY.md](./OPENTELEMETRY.md) - OpenTelemetry integration guide
-- [examples/](./examples/) - Code examples
+```bash
+# Run unit tests
+npm run test:sensitive:unit
 
-## 🤝 Contributing
+# Run integration tests
+npm run test:sensitive:detailed
+
+# Run all tests
+npm run test:all
+```
 
-Contributions are welcome! Please feel free to submit a Pull Request.
+### Test Coverage
 
-## 📄 License
+- ✅ **Unit Tests**: Filter utility with all redaction modes
+- ✅ **Integration Tests**: SDK with HTTP interception
+- ✅ **Custom Patterns**: User-defined sensitive data patterns
+- ✅ **Disabled Filtering**: Development mode verification
+- ✅ **Request/Response**: Headers, bodies, and span attributes
 
-This project is licensed under the MIT License.
+## Examples
 
-## 🆘 Support
+See the `examples/` directory for more usage examples:
 
-If you encounter any issues:
-1. Check the [Issues](https://github.com/lupislabs/lupis/issues) page
-2. Create a new issue with detailed information
+- `event-tracking-example.js` - Custom event tracking
+- `anthropic-example.js` - Anthropic API integration
+- `openai-example.js` - OpenAI API integration
+- `langchain-example.js` - LangChain integration
+- `streaming-example.js` - Streaming responses
 
----
+## License
 
 **Made with ❤️ by the Lupis team**