ml-cache 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Nicolas Mondain
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,599 @@
+# ml-cache
+
+**Store your business data today. Train your AI models tomorrow.**
+
+[![npm version](https://img.shields.io/npm/v/ml-cache.svg)](https://www.npmjs.com/package/ml-cache)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![TypeScript](https://img.shields.io/badge/TypeScript-5.0+-blue.svg)](https://www.typescriptlang.org/)
+[![Node.js](https://img.shields.io/badge/Node.js-18+-green.svg)](https://nodejs.org/)
+
+---
+
+## The Problem
+
+Machine learning is transforming every industry, but there's a catch: **you need massive amounts of quality data to train effective models**. Companies that start collecting data today will have a significant competitive advantage when:
+
+- ML training costs continue to drop exponentially
+- Your business grows and you need personalized AI features
+- You want to build recommendation engines, fraud detection, or predictive analytics
+- Custom models become essential for differentiation
+
+**The data you're generating right now is invaluable for future AI/ML applications. Don't let it slip away.**
+
+## The Solution
+
+`ml-cache` is a lightweight TypeScript SDK that captures your business events and stores them in Amazon S3 Glacier — the most cost-effective cold storage solution available. It's designed with a simple philosophy:
+
+> **Collect everything now. Pay almost nothing. Train models when ready.**
+
+### Why Cold Storage?
+
+| Storage Type | Cost per TB/month | Retrieval |
+|--------------|-------------------|-----------|
+| S3 Standard | ~$23 | Instant |
+| S3 Glacier | ~$4 | Minutes to hours |
+| S3 Glacier Deep Archive | ~$1 | 12-48 hours |
+
+For ML training data that you'll access months or years from now, cold storage is **20x cheaper** than standard storage.
+
+---
+
+## Features
+
+- **Simple, Familiar API** — Inspired by analytics SDKs like RudderStack and Segment
+- **Automatic Batching** — Efficiently groups events to minimize API calls
+- **Smart Retry Logic** — Exponential backoff ensures no data loss
+- **Type-Safe** — Full TypeScript support with comprehensive type definitions
+- **Flexible Storage** — S3 Standard, Glacier, or Glacier Deep Archive
+- **Rich Context** — Capture user, device, page, and campaign data
+- **Zero Dependencies on Analytics** — Direct AWS integration, no middlemen
+- **Production Ready** — Battle-tested error handling and graceful shutdown
+
+---
+
+## Installation
+
+```bash
+npm install ml-cache
+```
+
+```bash
+yarn add ml-cache
+```
+
+```bash
+pnpm add ml-cache
+```
+
+---
+
+## Quick Start
+
+```typescript
+import { MLCacheClient } from 'ml-cache';
+
+// Initialize the client
+const mlCache = new MLCacheClient({
+  credentials: {
+    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
+    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
+  },
+  s3: {
+    bucket: 'my-ml-data-lake',
+    region: 'us-east-1',
+    storageClass: 'GLACIER', // Cost-effective cold storage
+  },
+  storageMode: 'S3',
+  sourceApp: 'my-webapp',
+  environment: 'production',
+});
+
+// Track business events
+await mlCache.track({
+  eventType: 'purchase',
+  properties: {
+    productId: 'SKU-12345',
+    productName: 'Premium Widget',
+    price: 99.99,
+    currency: 'USD',
+    quantity: 2,
+  },
+  context: {
+    user: {
+      userId: 'user-789',
+      traits: {
+        plan: 'premium',
+        signupDate: '2024-01-15',
+      },
+    },
+  },
+});
+
+// Identify users
+await mlCache.identify('user-789', {
+  email: 'user@example.com',
+  name: 'Jane Doe',
+  company: 'Acme Corp',
+});
+
+// Track page views
+await mlCache.page('Product Details', {
+  url: '/products/SKU-12345',
+  referrer: 'google.com',
+});
+
+// Graceful shutdown (flushes remaining events)
+await mlCache.shutdown();
+```
+
+---
+
+## Configuration
+
+### Full Configuration Options
+
+```typescript
+import { MLCacheClient, type MLCacheConfig } from 'ml-cache';
+
+const config: MLCacheConfig = {
+  // Required: AWS Credentials
+  credentials: {
+    accessKeyId: 'AKIA...',
+    secretAccessKey: '...',
+    sessionToken: '...', // Optional: for temporary credentials
+  },
+
+  // S3 Configuration (required for S3 or S3_TO_GLACIER mode)
+  s3: {
+    bucket: 'my-ml-data-bucket',
+    region: 'us-east-1',
+    prefix: 'events/', // Optional: folder prefix for objects
+    storageClass: 'GLACIER', // STANDARD, GLACIER, DEEP_ARCHIVE, etc.
+  },
+
+  // Glacier Configuration (required for GLACIER mode)
+  glacier: {
+    vaultName: 'my-ml-vault',
+    region: 'us-east-1',
+    accountId: '-', // Optional: defaults to current account
+  },
+
+  // Storage mode
+  storageMode: 'S3', // 'S3' | 'GLACIER' | 'S3_TO_GLACIER'
+
+  // Batching configuration
+  batch: {
+    enabled: true, // Enable event batching
+    maxSize: 100, // Max events per batch
+    maxWaitMs: 30000, // Flush every 30 seconds
+  },
+
+  // Retry configuration
+  retry: {
+    maxRetries: 3,
+    initialDelayMs: 1000,
+    maxDelayMs: 30000,
+    exponentialBackoff: true,
+  },
+
+  // Logging configuration
+  log: {
+    level: 'info', // 'debug' | 'info' | 'warn' | 'error' | 'silent'
+    enabled: true,
+    customLogger: (level, message, data) => {
+      // Your custom logging logic
+    },
+  },
+
+  // Metadata
+  sourceApp: 'my-application',
+  environment: 'production',
+  debug: false,
+};
+
+const client = new MLCacheClient(config);
+```
+
+### Storage Classes
+
+Choose the right storage class for your needs:
+
+| Storage Class | Use Case | Retrieval Time |
+|--------------|----------|----------------|
+| `STANDARD` | Frequent access, testing | Instant |
+| `STANDARD_IA` | Infrequent access | Instant |
+| `GLACIER` | **Recommended for ML data** | 1-5 minutes |
+| `DEEP_ARCHIVE` | Rarely accessed, lowest cost | 12-48 hours |
+
+---
+
+## Event Types
+
+### Track Events
+
+Capture any business event with rich properties:
+
+```typescript
+await mlCache.track({
+  eventType: 'checkout_completed',
+  properties: {
+    orderId: 'ORD-123456',
+    total: 299.99,
+    items: [
+      { sku: 'WIDGET-A', quantity: 2, price: 49.99 },
+      { sku: 'WIDGET-B', quantity: 1, price: 199.99 },
+    ],
+    paymentMethod: 'credit_card',
+    shippingMethod: 'express',
+  },
+  context: {
+    user: { userId: 'user-123' },
+    campaign: {
+      source: 'google',
+      medium: 'cpc',
+      name: 'summer_sale',
+    },
+  },
+});
+```
+
+### Identify Users
+
+Build user profiles for personalization:
+
+```typescript
+await mlCache.identify('user-123', {
+  email: 'user@example.com',
+  name: 'John Doe',
+  plan: 'enterprise',
+  company: 'Acme Corp',
+  employeeCount: 500,
+  industry: 'technology',
+});
+```
+
+### Page Views
+
+Track navigation patterns:
+
+```typescript
+await mlCache.page('Pricing', {
+  url: 'https://example.com/pricing',
+  path: '/pricing',
+  title: 'Pricing Plans - Example',
+  referrer: 'https://google.com',
+});
+```
+
+---
+
+## Event Context
+
+Enrich events with contextual data:
+
+```typescript
+await mlCache.track({
+  eventType: 'feature_used',
+  properties: { feature: 'dark_mode' },
+  context: {
+    // User context
+    user: {
+      userId: 'user-123',
+      anonymousId: 'anon-456',
+      traits: {
+        plan: 'pro',
+        role: 'admin',
+      },
+    },
+
+    // Device context
+    device: {
+      userAgent: navigator.userAgent,
+      deviceType: 'desktop',
+      os: 'macOS',
+      browser: 'Chrome',
+      screenResolution: '1920x1080',
+      locale: 'en-US',
+      timezone: 'America/New_York',
+    },
+
+    // Page context
+    page: {
+      url: window.location.href,
+      path: window.location.pathname,
+      title: document.title,
+      referrer: document.referrer,
+    },
+
+    // Campaign/UTM context
+    campaign: {
+      source: 'newsletter',
+      medium: 'email',
+      name: 'weekly_digest',
+      content: 'cta_button',
+    },
+
+    // App context
+    app: {
+      name: 'MyApp',
+      version: '2.1.0',
+      build: '456',
+    },
+
+    // Custom context
+    custom: {
+      experimentId: 'exp-123',
+      variant: 'B',
+    },
+  },
+});
+```
+
+---
+
+## Callbacks & Monitoring
+
+```typescript
+// Monitor all tracked events
+mlCache.onEvent((event) => {
+  console.log('Event tracked:', event.eventType, event.eventId);
+});
+
+// Handle errors
+mlCache.onError((error, event) => {
+  console.error('Failed to store event:', error.message);
+  // Optionally: send to error tracking service
+});
+
+// Monitor flushes
+mlCache.onFlush((result) => {
+  console.log(`Flushed ${result.eventCount} events`);
+  if (result.failedEventIds.length > 0) {
+    console.warn('Failed events:', result.failedEventIds);
+  }
+});
+
+// Health check
+const health = await mlCache.getHealth();
+console.log('SDK Health:', health);
+// {
+//   healthy: true,
+//   s3Connected: true,
+//   glacierConnected: false,
+//   queueSize: 5,
+//   lastFlush: '2024-01-15T10:30:00.000Z',
+// }
+```
+
+---
+
+## Data Format
+
+Events are stored in NDJSON (Newline Delimited JSON) format, perfect for:
+
+- **Apache Spark** — Native NDJSON support
+- **AWS Athena** — Query directly with SQL
+- **Pandas** — `pd.read_json(file, lines=True)`
+- **Any ML pipeline** — Simple line-by-line parsing
+
+### S3 Object Structure
+
+```
+s3://my-bucket/ml-cache-events/
+├── 2024/
+│   ├── 01/
+│   │   ├── 15/
+│   │   │   ├── 10/
+│   │   │   │   ├── batch_1705312200_a1b2c3d4.ndjson
+│   │   │   │   └── batch_1705312500_e5f6g7h8.ndjson
+```
+
+### Event Schema
+
+```json
+{
+  "eventId": "550e8400-e29b-41d4-a716-446655440000",
+  "timestamp": "2024-01-15T10:30:00.000Z",
+  "eventType": "purchase",
+  "properties": {
+    "productId": "SKU-123",
+    "amount": 99.99
+  },
+  "context": {
+    "user": { "userId": "user-456" }
+  },
+  "metadata": {
+    "sdkVersion": "1.0.0",
+    "sourceApp": "my-app",
+    "environment": "production",
+    "batchId": "batch_1705312200_a1b2c3d4"
+  }
+}
+```
+
+---
+
+## AWS Setup
+
+### IAM Policy
+
+Create an IAM policy with minimal required permissions:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "s3:PutObject",
+        "s3:GetBucketLocation"
+      ],
+      "Resource": [
+        "arn:aws:s3:::your-bucket-name",
+        "arn:aws:s3:::your-bucket-name/*"
+      ]
+    }
+  ]
+}
+```
+
+For Glacier mode, add:
+
+```json
+{
+  "Effect": "Allow",
+  "Action": [
+    "glacier:UploadArchive",
+    "glacier:DescribeVault"
+  ],
+  "Resource": "arn:aws:glacier:*:*:vaults/your-vault-name"
+}
+```
+
+### S3 Lifecycle Policy (Optional)
+
+Automatically transition data to deeper cold storage:
+
+```json
+{
+  "Rules": [
+    {
+      "ID": "MLDataLifecycle",
+      "Status": "Enabled",
+      "Prefix": "ml-cache-events/",
+      "Transitions": [
+        {
+          "Days": 90,
+          "StorageClass": "GLACIER"
+        },
+        {
+          "Days": 365,
+          "StorageClass": "DEEP_ARCHIVE"
+        }
+      ]
+    }
+  ]
+}
+```
+
+---
+
+## Best Practices
+
+### 1. Capture Rich Context
+
+The more context you capture now, the better your models will be:
+
+```typescript
+// Good: Rich context for future ML
+await mlCache.track({
+  eventType: 'product_viewed',
+  properties: {
+    productId: 'SKU-123',
+    category: 'electronics',
+    price: 299.99,
+    inStock: true,
+    viewDuration: 45,
+    scrollDepth: 0.8,
+  },
+  context: {
+    user: { userId: 'user-456', traits: { segment: 'high-value' } },
+    page: { referrer: 'google.com', searchQuery: 'best headphones' },
+    device: { deviceType: 'mobile', os: 'iOS' },
+  },
+});
+```
+
+### 2. Use Consistent Event Names
+
+Establish naming conventions early:
+
+```typescript
+// Use snake_case for event types
+'user_signed_up'    // Good
+'userSignedUp'      // Avoid
+'User Signed Up'    // Avoid
+
+// Be specific
+'checkout_started'   // Good
+'checkout'          // Too vague
+```
+
+### 3. Graceful Shutdown
+
+Always flush events before application exit:
+
+```typescript
+process.on('SIGTERM', async () => {
+  await mlCache.shutdown();
+  process.exit(0);
+});
+```
+
+### 4. Monitor Queue Size
+
+Prevent memory issues in high-traffic scenarios:
+
+```typescript
+setInterval(() => {
+  const queueSize = mlCache.getQueueSize();
+  if (queueSize > 5000) {
+    console.warn(`Queue size high: ${queueSize}`);
+  }
+}, 60000);
+```
+
+---
+
+## Future ML Use Cases
+
+The data you collect today can power tomorrow's AI features:
+
+| Event Type | Future ML Application |
+|------------|----------------------|
+| `purchase` | Recommendation engine, demand forecasting |
+| `page_view` | Content personalization, A/B test analysis |
+| `search` | Search ranking, query understanding |
+| `support_ticket` | Automated responses, sentiment analysis |
+| `user_behavior` | Churn prediction, engagement scoring |
+| `product_interaction` | Dynamic pricing, inventory optimization |
+
+---
+
+## API Reference
+
+### MLCacheClient
+
+| Method | Description |
+|--------|-------------|
+| `track(event)` | Track a custom event |
+| `identify(userId, traits?)` | Identify a user with traits |
+| `page(name, properties?)` | Track a page view |
+| `flush()` | Manually flush the event queue |
+| `getHealth()` | Get SDK health status |
+| `getQueueSize()` | Get current queue size |
+| `getVersion()` | Get SDK version |
+| `shutdown()` | Gracefully shutdown the client |
+| `onEvent(callback)` | Register event callback |
+| `onError(callback)` | Register error callback |
+| `onFlush(callback)` | Register flush callback |
+
+---
+
+## License
+
+MIT
+
+---
+
+## Contributing
+
+Contributions are welcome! Please read our contributing guidelines and submit pull requests to the GitHub repository.
+
+---
+
+<p align="center">
+  <strong>Start collecting your ML training data today.</strong><br>
+  <em>The best time to start was yesterday. The second best time is now.</em>
+</p>