@small-ltsc/ml 0.1.0 → 0.2.4

package/README.md

@@ -1,5 +1,8 @@
 # @small-ltsc/ml
 
+[![npm](https://img.shields.io/npm/v/@small-ltsc/ml)](https://www.npmjs.com/package/@small-ltsc/ml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](../../LICENSE)
+
 Optional ML features for **Small LTSC** - Pattern importance scoring, quality prediction, and adaptive region detection.
 
 ## Installation
@@ -8,13 +11,13 @@ Optional ML features for **Small LTSC** - Pattern importance scoring, quality pr
 npm install @small-ltsc/ml @small-ltsc/sdk
 ```
 
-Note: `@small-ltsc/sdk` is a peer dependency.
+Note: `@small-ltsc/sdk` is a required peer dependency.
 
 ## Features
 
-- **Pattern Importance Scoring** - Determine which patterns are semantically important
-- **Quality Prediction** - Predict if compression will degrade model performance
-- **Region Detection** - Identify system prompts, user input, and context for adaptive compression
+- **Pattern Importance Scoring** - Determine which patterns are semantically important and should be preserved
+- **Quality Prediction** - Predict if compression will degrade model performance before applying it
+- **Region Detection** - Identify system prompts, user input, and context for adaptive compression strategies
 
 ## Pattern Importance
 
@@ -28,13 +31,13 @@ const scorer = new PositionalImportanceScorer({ decayRate: 2.0 });
 const patterns = await discoverPatterns(tokens);
 const scores = await scorer.scorePatterns(tokens, patterns);
 
-// Filter out high-importance patterns (preserve them)
+// Filter out high-importance patterns (preserve them from compression)
 const safeToCompress = filterByImportance(patterns, scores, 0.8);
 ```
 
 ### Embedding-Based Scoring
 
-For more accurate importance scoring with an embedding model:
+For more accurate importance scoring using an embedding model:
 
 ```typescript
 import { EmbeddingImportanceScorer } from '@small-ltsc/ml';
@@ -48,7 +51,7 @@ const scores = await scorer.scorePatterns(tokens, patterns);
 
 ## Quality Prediction
 
-Predict if compressed output will maintain quality:
+Predict if compressed output will maintain quality before committing:
 
 ```typescript
 import { createQualityPredictor } from '@small-ltsc/ml';
@@ -79,17 +82,17 @@ console.log(prediction.features);
 
 ## Region Detection
 
-Detect semantic regions for adaptive compression:
+Detect semantic regions for adaptive compression strategies:
 
 ```typescript
 import { detectRegions, RegionType, filterPatternsByRegion } from '@small-ltsc/ml';
 
 const regions = detectRegions(tokens, {
-  systemMarkers: [[58, 71905, 60]], // [SYSTEM] tokens
+  systemMarkers: [[58, 71905, 60]], // [SYSTEM] token sequence
   retentionTargets: {
-    [RegionType.SYSTEM]: 0.98,  // Minimal compression
-    [RegionType.USER]: 0.85,    // Moderate
-    [RegionType.CONTEXT]: 0.6,  // Aggressive
+    [RegionType.SYSTEM]: 0.98,  // Minimal compression (preserve instructions)
+    [RegionType.USER]: 0.85,    // Moderate compression
+    [RegionType.CONTEXT]: 0.6,  // Aggressive compression (RAG content)
   },
 });
 
@@ -99,11 +102,13 @@ const filtered = filterPatternsByRegion(patterns, regions, tokens);
 
 ### Region Types
 
-- `SYSTEM` - System instructions (high retention)
-- `USER` - User input (moderate retention)
-- `CONTEXT` - Injected context/documents (low retention)
-- `CODE` - Code blocks (moderate retention)
-- `UNKNOWN` - Default region
+| Region | Description | Default Retention |
+|--------|-------------|-------------------|
+| `SYSTEM` | System instructions | 98% (minimal compression) |
+| `USER` | User input | 85% (moderate) |
+| `CONTEXT` | Injected context/documents | 60% (aggressive) |
+| `CODE` | Code blocks | 80% (moderate) |
+| `UNKNOWN` | Default region | 75% |
 
 ## Custom Embedding Provider
 
@@ -134,25 +139,40 @@ const scorer = new EmbeddingImportanceScorer(new OpenAIEmbeddings());
 
 ### Importance Scoring
 
-- `PositionalImportanceScorer` - Score by position (early = important)
-- `EmbeddingImportanceScorer` - Score by context diversity
-- `CombinedImportanceScorer` - Combine positional and embedding scoring
-- `adjustPrioritiesByImportance(patterns, scores, threshold)` - Adjust pattern priorities
-- `filterByImportance(patterns, scores, threshold)` - Filter high-importance patterns
+| Export | Description |
+|--------|-------------|
+| `PositionalImportanceScorer` | Score patterns by position (earlier = more important) |
+| `EmbeddingImportanceScorer` | Score patterns by contextual diversity |
+| `CombinedImportanceScorer` | Combine positional and embedding scoring |
+| `adjustPrioritiesByImportance()` | Adjust pattern priorities based on scores |
+| `filterByImportance()` | Filter out high-importance patterns |
 
 ### Quality Prediction
 
-- `HeuristicQualityPredictor` - Rule-based quality prediction
-- `EmbeddingQualityPredictor` - Enhanced with embedding similarity
-- `createQualityPredictor(provider?, config?)` - Factory function
+| Export | Description |
+|--------|-------------|
+| `HeuristicQualityPredictor` | Rule-based quality prediction |
+| `EmbeddingQualityPredictor` | Enhanced prediction with embedding similarity |
+| `createQualityPredictor()` | Factory function for creating predictors |
 
 ### Region Detection
 
-- `detectRegions(tokens, config?)` - Detect semantic regions
-- `detectRegionsHeuristic(tokens)` - Simple heuristic detection
-- `filterPatternsByRegion(patterns, regions, tokens)` - Filter by region
-- `getRegionCompressionSettings(regionType)` - Get settings for region
+| Export | Description |
+|--------|-------------|
+| `detectRegions()` | Detect semantic regions in token sequence |
+| `detectRegionsHeuristic()` | Simple heuristic-based detection |
+| `filterPatternsByRegion()` | Filter patterns based on region constraints |
+| `getRegionCompressionSettings()` | Get default settings for a region type |
+| `RegionType` | Enum of available region types |
 
 ## License
 
-MIT
+MIT License - see [LICENSE](../../LICENSE) for details.
+
+## Contributors
+
+Built by [Triage Sec](https://triage-sec.com) - an applied team of researchers and engineers working towards building resiliency for AI systems.
+
+- Nikhil Srivastava (University of California, Berkeley)
+- Omansh Bainsla (Georgia Tech)
+- Sahil Chatiwala (Georgia Tech)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@small-ltsc/ml",
-  "version": "0.1.0",
+"version": "0.2.4",
   "description": "ML features for Small LTSC - Pattern importance scoring and quality prediction",
   "type": "module",
   "main": "./dist/esm/index.js",
@@ -17,12 +17,13 @@
   ],
   "sideEffects": false,
   "scripts": {
+    "clean": "rm -rf dist",
     "build": "npm run build:esm && npm run build:types",
     "build:esm": "tsc -p tsconfig.esm.json",
     "build:types": "tsc -p tsconfig.types.json",
     "test": "vitest",
     "lint": "eslint src --ext .ts",
-    "prepublishOnly": "npm run build"
+    "prepublishOnly": "npm run clean && npm run build"
   },
   "keywords": [
     "compression",
@@ -32,14 +33,23 @@
     "ml",
     "embeddings"
   ],
-  "author": "",
+  "author": "Triage Sec <nicks@triage-sec.com>",
+  "contributors": [
+    "Nikhil Srivastava",
+    "Omansh Bainsla",
+    "Sahil Chatiwala"
+  ],
   "license": "MIT",
   "repository": {
     "type": "git",
-    "url": "https://github.com/triage-sec/small"
+    "url": "git+https://github.com/triage-sec/small.git"
+  },
+  "bugs": {
+    "url": "https://github.com/triage-sec/small/issues"
   },
+  "homepage": "https://github.com/triage-sec/small#readme",
   "peerDependencies": {
-    "@small-ltsc/sdk": "^0.1.0"
+"@small-ltsc/sdk": "^0.2.4"
   },
   "devDependencies": {
     "@types/node": "^20.10.0",