@memberjunction/ai-vectors 5.21.0 → 5.22.0

package/README.md

@@ -1,24 +1,44 @@
 # @memberjunction/ai-vectors
 
-The core foundation package for vector operations in MemberJunction. Provides base classes, interfaces, and type definitions that all other vector packages build upon.
+Core foundation package for vector operations in MemberJunction. Provides text processing utilities (chunking, extraction), base classes for vectorization pipelines, and interfaces for embedding providers and vector databases.
+
+## Installation
+
+```bash
+npm install @memberjunction/ai-vectors
+```
+
+## What's Included
+
+| Export | Type | Purpose |
+|---|---|---|
+| `TextChunker` | Class | Token-aware text splitting with sentence, paragraph, and fixed strategies |
+| `TextExtractor` | Class | HTML stripping, entity decoding, MIME-type routing, token truncation |
+| `VectorBase` | Class | Base class providing RunView, Metadata, AIEngine integration for subclasses |
+| `IEmbedding` | Interface | Contract for single and batch text embedding generation |
+| `IVectorDatabase` | Interface | Contract for vector database management (create/delete/list indexes) |
+| `IVectorIndex` | Interface | Contract for CRUD operations on vector records within an index |
+| `ChunkTextParams` | Type | Configuration for `TextChunker.ChunkText()` |
+| `TextChunk` | Type | Output chunk with text, offsets, token count, and index |
+| `PageRecordsParams` | Type | Paginated entity record retrieval configuration |
 
 ## Architecture
 
 ```mermaid
 graph TD
     subgraph Core["@memberjunction/ai-vectors"]
+        TC["TextChunker"]
+        TE["TextExtractor"]
         VB["VectorBase"]
         IE["IEmbedding"]
         IVD["IVectorDatabase"]
         IVI["IVectorIndex"]
-        PT["PageRecordsParams"]
     end
 
     subgraph MJCore["MemberJunction Core"]
         MD["Metadata"]
         RV["RunView"]
         BE["BaseEntity"]
-        UI["UserInfo"]
     end
 
     subgraph AIEngine["AI Engine"]
@@ -39,6 +59,8 @@ graph TD
     AIM --> MOD
     AIM --> VDB
     SYNC --> VB
+    SYNC --> TC
+    SYNC --> TE
     DUPE --> VB
 
     style Core fill:#2d6a9f,stroke:#1a4971,color:#fff
@@ -47,167 +69,253 @@ graph TD
     style Consumers fill:#7c5295,stroke:#563a6b,color:#fff
 ```
 
-## Installation
+## TextChunker
 
-```bash
-npm install @memberjunction/ai-vectors
+Token-aware text splitting that respects natural language boundaries. All methods are static.
+
+### Strategies
+
+| Strategy | Splits On | Best For |
+|---|---|---|
+| `sentence` | Sentence-ending punctuation (`.` `!` `?`) | Prose, articles, descriptions |
+| `paragraph` | Double newlines (`\n\n`) | Structured documents, Markdown, reports |
+| `fixed` | Whitespace boundaries at the character limit | Logs, code, unstructured data |
+
+### Basic Usage
+
+```typescript
+import { TextChunker, ChunkTextParams, TextChunk } from '@memberjunction/ai-vectors';
+
+const article = `Machine learning models require training data.
+The quality of training data directly impacts model performance.
+Data preprocessing is a critical step in any ML pipeline.
+
+Feature engineering transforms raw data into meaningful representations.
+Good features can dramatically improve model accuracy.`;
+
+// Sentence strategy (default)
+const chunks: TextChunk[] = TextChunker.ChunkText({
+    Text: article,
+    MaxChunkTokens: 128,
+    Strategy: 'sentence'
+});
+
+for (const chunk of chunks) {
+    console.log(`Chunk ${chunk.Index}: ${chunk.TokenCount} tokens, offset ${chunk.StartOffset}-${chunk.EndOffset}`);
+    console.log(chunk.Text);
+}
 ```
 
-## Overview
+### Paragraph Strategy
 
-This package serves as the shared foundation for all vector-related operations in MemberJunction. It provides:
+```typescript
+const markdownDoc = `## Introduction
 
-- **VectorBase** -- a base class that integrates with MemberJunction metadata, AI models, vector databases, and the entity system
-- **Interfaces** -- contracts for embedding generation (`IEmbedding`), vector database management (`IVectorDatabase`), and vector index CRUD (`IVectorIndex`)
-- **Type definitions** -- `PageRecordsParams` for paginated data retrieval across entities
+This document covers the architecture of our data pipeline.
+It handles ingestion, transformation, and storage.
 
-All higher-level vector packages (`ai-vector-sync`, `ai-vector-dupe`) extend `VectorBase` to inherit its data access, model lookup, and user context capabilities.
+## Processing
 
-## Core Components
+Records are validated against schema constraints.
+Invalid records are routed to a dead-letter queue.
 
-### VectorBase Class
+## Storage
 
-The primary class that downstream packages extend. It wraps MemberJunction's `Metadata`, `RunView`, and `AIEngine` systems into a unified base for vector operations.
+Processed data is stored in both relational and vector databases.
+Vector embeddings enable semantic search across all records.`;
 
-```mermaid
-classDiagram
-    class VectorBase {
-        +Metadata : Metadata
-        +RunView : RunView
-        +CurrentUser : UserInfo
-        #GetRecordsByEntityID(entityID, recordIDs?) BaseEntity[]
-        #PageRecordsByEntityID~T~(params) T[]
-        #GetAIModel(id?) AIModelEntityExtended
-        #GetVectorDatabase(id?) VectorDatabaseEntity
-        #RunViewForSingleValue~T~(entityName, filter) T
-        #SaveEntity(entity) boolean
-        #BuildExtraFilter(compositeKeys) string
-    }
+const chunks = TextChunker.ChunkText({
+    Text: markdownDoc,
+    MaxChunkTokens: 256,
+    Strategy: 'paragraph'
+});
+// Each paragraph becomes a chunk (or paragraphs merge if they fit together)
+```
 
-    class EntityVectorSyncer {
-        +VectorizeEntity(params, user) VectorizeEntityResponse
-        +Config(forceRefresh, user) void
-    }
+### Fixed Strategy
 
-    class DuplicateRecordDetector {
-        +getDuplicateRecords(params, user) PotentialDuplicateResponse
-    }
+```typescript
+const logData = `2024-01-15T10:00:00Z INFO Server started on port 4000
+2024-01-15T10:00:01Z INFO Connected to database
+2024-01-15T10:00:02Z WARN High memory usage detected: 85%
+2024-01-15T10:00:03Z ERROR Connection timeout after 30000ms`;
+
+const chunks = TextChunker.ChunkText({
+    Text: logData,
+    MaxChunkTokens: 64,
+    Strategy: 'fixed'
+});
+```
 
-    VectorBase <|-- EntityVectorSyncer
-    VectorBase <|-- DuplicateRecordDetector
+### Configuring Overlap
 
-    style VectorBase fill:#2d6a9f,stroke:#1a4971,color:#fff
-    style EntityVectorSyncer fill:#7c5295,stroke:#563a6b,color:#fff
-    style DuplicateRecordDetector fill:#7c5295,stroke:#563a6b,color:#fff
+Overlap repeats trailing content from the previous chunk at the start of the next chunk, preserving context across chunk boundaries. Defaults to 10% of `MaxChunkTokens`.
+
+```typescript
+// Explicit overlap: 50 tokens of shared context between chunks
+const chunks = TextChunker.ChunkText({
+    Text: longDocument,
+    MaxChunkTokens: 512,
+    OverlapTokens: 50,
+    Strategy: 'sentence'
+});
+
+// No overlap
+const chunks = TextChunker.ChunkText({
+    Text: longDocument,
+    MaxChunkTokens: 512,
+    OverlapTokens: 0,
+    Strategy: 'sentence'
+});
 ```
 
-**Key capabilities:**
+### Token Estimation
 
-| Method | Purpose |
-|---|---|
-| `GetRecordsByEntityID` | Load all entity records, optionally filtered by composite keys |
-| `PageRecordsByEntityID` | Paginated retrieval with configurable page size, result type, and filter |
-| `GetAIModel` | Locate an embedding model from `AIEngine.Instance.Models` by ID or get the first available |
-| `GetVectorDatabase` | Locate a vector database from `AIEngine.Instance.VectorDatabases` by ID or get the first available |
-| `RunViewForSingleValue` | Query for a single entity record matching a filter |
-| `SaveEntity` | Save a `BaseEntity` with the current user context automatically applied |
-| `BuildExtraFilter` | Convert an array of `CompositeKey` objects into a SQL filter string |
+`EstimateTokenCount` provides a fast approximation using the ~4 characters per token heuristic for English text. This is suitable for chunking where exact counts are not critical.
 
-### Interfaces
+```typescript
+const tokens = TextChunker.EstimateTokenCount('This is a sample sentence.');
+// Returns: 7 (26 characters / 4)
+
+// For production accuracy with specific models, use tiktoken directly
+// and pass the result to MaxChunkTokens for precise control
+```
 
-#### IEmbedding
+### TextChunk Output Shape
 
-Contract for text embedding providers:
+Each chunk includes full position metadata for traceability back to the source:
 
 ```typescript
-interface IEmbedding {
-    createEmbedding(text: string, options?: unknown): unknown;
-    createBatchEmbedding(text: string[], options?: unknown): unknown;
+interface TextChunk {
+    Text: string;        // The chunk text content
+    StartOffset: number; // Start character offset in original text
+    EndOffset: number;   // End character offset (exclusive)
+    TokenCount: number;  // Approximate token count
+    Index: number;       // 0-based chunk index
 }
 ```
 
-#### IVectorDatabase
+## TextExtractor
+
+Static utilities for extracting clean plain text from various content formats. Dependency-light (regex-based, no DOM parser required).
 
-Contract for vector database management operations:
+### HTML Extraction
 
 ```typescript
-interface IVectorDatabase {
-    listIndexes(options?: unknown): unknown;
-    createIndex(options: unknown): unknown;
-    deleteIndex(indexID: unknown, options?: unknown): unknown;
-    editIndex(indexID: unknown, options?: unknown): unknown;
-}
+import { TextExtractor } from '@memberjunction/ai-vectors';
+
+const html = `
+<html>
+<head><style>body { color: red; }</style></head>
+<body>
+  <h1>Welcome</h1>
+  <p>This is a <strong>formatted</strong> paragraph with &amp; entities.</p>
+  <script>alert('removed');</script>
+  <ul>
+    <li>Item one</li>
+    <li>Item two</li>
+  </ul>
+</body>
+</html>`;
+
+const text = TextExtractor.ExtractFromHTML(html);
+// "Welcome\nThis is a formatted paragraph with & entities.\nItem one\nItem two"
 ```
 
-#### IVectorIndex
+What it does:
+- Removes `<script>` and `<style>` elements entirely
+- Converts block-level elements (`<p>`, `<div>`, `<h1>`-`<h6>`, `<li>`, `<br>`, etc.) to newlines
+- Strips all remaining HTML tags
+- Decodes named entities (`&amp;`, `&lt;`, `&gt;`, `&quot;`, `&nbsp;`, `&mdash;`, `&hellip;`, etc.)
+- Decodes numeric entities (decimal `&#169;` and hex `&#xA9;`)
+- Normalizes whitespace (collapses runs of spaces, limits consecutive newlines to 2)
 
-Contract for CRUD operations on vector records within an index:
+### Plain Text Normalization
 
 ```typescript
-interface IVectorIndex {
-    createRecord(record: unknown, options?: unknown): unknown;
-    createRecords(records: unknown[], options?: unknown): unknown;
-    getRecord(recordID: unknown, options?: unknown): unknown;
-    getRecords(recordIDs: unknown[], options?: unknown): unknown;
-    updateRecord(record: unknown, options?: unknown): unknown;
-    updateRecords(records: unknown[], options?: unknown): unknown;
-    deleteRecord(recordID: unknown, options?: unknown): unknown;
-    deleteRecords(recordIDs: unknown[], options?: unknown): unknown;
-}
+const raw = "  Some text\x00with\x07control\x1Fcharacters\n\n\n\n\nand  extra   spaces  ";
+const clean = TextExtractor.ExtractFromPlainText(raw);
+// "Some textwithcontrolcharacters\n\nand extra spaces"
 ```
 
-### PageRecordsParams Type
+Removes control characters (`\x00`-`\x1F` except `\n` and `\t`), normalizes whitespace, trims.
 
-Configuration for paginated entity record retrieval:
+### MIME-Type Routing
 
 ```typescript
-type PageRecordsParams = {
-    EntityID: string | number;
-    PageNumber: number;
-    PageSize: number;
-    ResultType: "entity_object" | "simple" | "count_only";
-    Filter?: string;
-};
+// Automatically selects the right extraction method
+const fromHTML = TextExtractor.ExtractByMimeType(htmlContent, 'text/html');
+const fromPlain = TextExtractor.ExtractByMimeType(plainContent, 'text/plain');
+const fromCSV = TextExtractor.ExtractByMimeType(csvContent, 'text/csv');  // Falls back to plain text
+
+// For binary formats (PDF, DOCX), extract text with a dedicated library first,
+// then pass through ExtractFromPlainText for normalization:
+// const pdfText = await pdfParse(buffer);
+// const clean = TextExtractor.ExtractFromPlainText(pdfText);
 ```
 
-## Usage
+### Token Truncation
 
-### Extending VectorBase
+```typescript
+// Truncate text to fit within a model's context window
+const truncated = TextExtractor.TruncateToTokenLimit(veryLongText, 8192);
+// Truncates at the last whitespace boundary before the estimated character limit
+```
+
+## VectorBase
+
+Abstract base class that downstream vector packages extend. Provides integrated access to MemberJunction's Metadata, RunView, and AIEngine systems.
+
+### Class Diagram
+
+```mermaid
+classDiagram
+    class VectorBase {
+        +Metadata : Metadata
+        +RunView : RunView
+        +CurrentUser : UserInfo
+        #GetRecordsByEntityID(entityID, recordIDs?) BaseEntity[]
+        #PageRecordsByEntityID~T~(params) T[]
+        #GetAIModel(id?) MJAIModelEntityExtended
+        #GetVectorDatabase(id?) MJVectorDatabaseEntity
+        #RunViewForSingleValue~T~(entityName, filter) T | null
+        #SaveEntity(entity) boolean
+        #BuildExtraFilter(compositeKeys) string
+    }
+```
 
-Create specialized vector processing classes by extending `VectorBase`:
+### Extending VectorBase
 
 ```typescript
 import { VectorBase, PageRecordsParams } from '@memberjunction/ai-vectors';
 import { BaseEntity } from '@memberjunction/core';
 
 export class MyVectorProcessor extends VectorBase {
-    async processEntity(entityId: string): Promise<void> {
-        // Retrieve all records for an entity
+    async ProcessEntity(entityId: string): Promise<void> {
+        // Load all records for an entity
         const records = await this.GetRecordsByEntityID(entityId);
 
         // Access configured AI models and vector databases
-        const model = this.GetAIModel();
-        const vectorDb = this.GetVectorDatabase();
+        const model = this.GetAIModel();       // First available embedding model
+        const vectorDb = this.GetVectorDatabase(); // First available vector DB
 
         for (const record of records) {
-            // Process each record with the embedding model and vector DB
+            // Generate embeddings, upsert into vector DB
         }
     }
 
-    async processInPages(entityId: string): Promise<void> {
+    async ProcessInPages(entityId: string): Promise<void> {
         let page = 1;
         let hasMore = true;
 
         while (hasMore) {
-            const params: PageRecordsParams = {
+            const records = await this.PageRecordsByEntityID<Record<string, unknown>>({
                 EntityID: entityId,
                 PageNumber: page,
                 PageSize: 100,
                 ResultType: 'simple',
                 Filter: "Status = 'Active'"
-            };
-
-            const records = await this.PageRecordsByEntityID<Record<string, unknown>>(params);
+            });
             hasMore = records.length === 100;
             page++;
         }
@@ -222,7 +330,7 @@ import { VectorBase } from '@memberjunction/ai-vectors';
 import { CompositeKey } from '@memberjunction/core';
 
 class FilteredProcessor extends VectorBase {
-    async getSpecificRecords(entityId: string): Promise<void> {
+    async GetSpecificRecords(entityId: string): Promise<void> {
         const keys: CompositeKey[] = [
             { KeyValuePairs: [{ FieldName: 'ID', Value: 'abc-123' }] },
             { KeyValuePairs: [{ FieldName: 'ID', Value: 'def-456' }] }
@@ -234,9 +342,45 @@ class FilteredProcessor extends VectorBase {
 }
 ```
 
-## Package Ecosystem
+## API Reference
+
+### TextChunker (Static Methods)
+
+| Method | Parameters | Returns | Description |
+|---|---|---|---|
+| `ChunkText` | `params: ChunkTextParams` | `TextChunk[]` | Split text into token-bounded chunks using the specified strategy |
+| `EstimateTokenCount` | `text: string` | `number` | Fast token count approximation (~4 chars/token) |
+
+### TextExtractor (Static Methods)
 
-This package sits at the base of the MemberJunction vector package hierarchy:
+| Method | Parameters | Returns | Description |
+|---|---|---|---|
+| `ExtractFromHTML` | `html: string` | `string` | Strip tags, decode entities, normalize whitespace |
+| `ExtractFromPlainText` | `text: string` | `string` | Remove control characters, normalize whitespace |
+| `ExtractByMimeType` | `content: string, mimeType: string` | `string` | Route to the appropriate extraction method by MIME type |
+| `TruncateToTokenLimit` | `text: string, maxTokens: number` | `string` | Truncate at whitespace boundary within the token budget |
+
+### VectorBase (Protected Methods for Subclasses)
+
+| Method | Returns | Description |
+|---|---|---|
+| `GetRecordsByEntityID(entityID, recordIDs?)` | `Promise<BaseEntity[]>` | Load entity records, optionally filtered by composite keys |
+| `PageRecordsByEntityID<T>(params)` | `Promise<T[]>` | Paginated retrieval with configurable page size and filter |
+| `GetAIModel(id?)` | `MJAIModelEntityExtended` | Locate an embedding model by ID or get the first available |
+| `GetVectorDatabase(id?)` | `MJVectorDatabaseEntity` | Locate a vector database by ID or get the first available |
+| `RunViewForSingleValue<T>(entityName, filter)` | `Promise<T \| null>` | Query for a single entity record matching a filter |
+| `SaveEntity(entity)` | `Promise<boolean>` | Save a BaseEntity with CurrentUser context applied |
+| `BuildExtraFilter(compositeKeys)` | `string` | Convert CompositeKey array to a SQL filter string |
+
+### Interfaces
+
+| Interface | Methods | Purpose |
+|---|---|---|
+| `IEmbedding` | `createEmbedding`, `createBatchEmbedding` | Text embedding generation |
+| `IVectorDatabase` | `listIndexes`, `createIndex`, `deleteIndex`, `editIndex` | Vector database management |
+| `IVectorIndex` | `createRecord(s)`, `getRecord(s)`, `updateRecord(s)`, `deleteRecord(s)` | Vector record CRUD |
+
+## Package Ecosystem
 
 | Package | Depends On Core | Purpose |
 |---|---|---|
@@ -244,19 +388,11 @@ This package sits at the base of the MemberJunction vector package hierarchy:
 | `@memberjunction/ai-vector-sync` | Yes | Entity-to-vector synchronization |
 | `@memberjunction/ai-vector-dupe` | Yes | Duplicate detection via vector similarity |
 | `@memberjunction/ai-vectors-memory` | No | In-memory vector search and clustering |
-| `@memberjunction/ai-vectors-pinecone` | No | Pinecone implementation of `VectorDBBase` |
+| `@memberjunction/ai-vectors-pinecone` | No | Pinecone implementation of VectorDBBase |
 
-## Dependencies
+## Further Reading
 
-| Package | Purpose |
-|---|---|
-| `@memberjunction/core` | Metadata, RunView, BaseEntity, UserInfo |
-| `@memberjunction/global` | MJGlobal class factory |
-| `@memberjunction/core-entities` | VectorDatabaseEntity and other entity types |
-| `@memberjunction/aiengine` | AIEngine singleton for model and database discovery |
-| `@memberjunction/ai` | Base AI abstractions |
-| `@memberjunction/ai-core-plus` | AIModelEntityExtended |
-| `@memberjunction/ai-vectordb` | VectorDBBase and related types |
+- [Text Processing Guide](docs/TEXT_PROCESSING_GUIDE.md) -- in-depth guide on chunking strategies, overlap tuning, HTML edge cases, and integration with vectorization/autotagging pipelines
 
 ## Development
 
@@ -264,8 +400,11 @@ This package sits at the base of the MemberJunction vector package hierarchy:
 # Build
 npm run build
 
-# Development mode
-npm run start
+# Run tests
+npm run test
+
+# Watch mode
+npm run test:watch
 ```
 
 ## License

package/dist/generic/IEmbedding.d.ts

@@ -1,5 +1,24 @@
+/**
+ * Interface for embedding providers.
+ * @deprecated Use BaseEmbeddings from @memberjunction/ai instead, which provides
+ * a more complete API with proper typing (EmbedTextResult, EmbedTextsResult).
+ */
 export interface IEmbedding {
-    createEmbedding(text: string, options?: any): any;
-    createBatchEmbedding(text: string[], options?: any): any;
+    createEmbedding(text: string, options?: EmbeddingOptions): Promise<EmbeddingResult>;
+    createBatchEmbedding(text: string[], options?: EmbeddingOptions): Promise<BatchEmbeddingResult>;
+}
+export interface EmbeddingOptions {
+    /** The model to use for embedding generation */
+    model?: string;
+    /** Number of dimensions for the output vectors */
+    dimensions?: number;
+}
+export interface EmbeddingResult {
+    vector: number[];
+    tokenCount?: number;
+}
+export interface BatchEmbeddingResult {
+    vectors: number[][];
+    tokenCounts?: number[];
 }
 //# sourceMappingURL=IEmbedding.d.ts.map

package/dist/generic/IEmbedding.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"IEmbedding.d.ts","sourceRoot":"","sources":["../../src/generic/IEmbedding.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,UAAU;IACvB,eAAe,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;IAClD,oBAAoB,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;CAC5D"}
+{"version":3,"file":"IEmbedding.d.ts","sourceRoot":"","sources":["../../src/generic/IEmbedding.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AACH,MAAM,WAAW,UAAU;IACvB,eAAe,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,eAAe,CAAC,CAAC;IACpF,oBAAoB,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,oBAAoB,CAAC,CAAC;CACnG;AAED,MAAM,WAAW,gBAAgB;IAC7B,gDAAgD;IAChD,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,kDAAkD;IAClD,UAAU,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,MAAM,WAAW,eAAe;IAC5B,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,UAAU,CAAC,EAAE,MAAM,CAAC;CACvB;AAED,MAAM,WAAW,oBAAoB;IACjC,OAAO,EAAE,MAAM,EAAE,EAAE,CAAC;IACpB,WAAW,CAAC,EAAE,MAAM,EAAE,CAAC;CAC1B"}

package/dist/generic/IVectorDatabase.d.ts

@@ -1,7 +1,40 @@
+/**
+ * Interface for vector database providers.
+ * @deprecated Use VectorDBBase from @memberjunction/ai-vectordb instead, which provides
+ * a complete abstract class with proper typing (IndexList, BaseResponse, CreateIndexParams).
+ */
 export interface IVectorDatabase {
-    listIndexes(options?: any): any;
-    createIndex(options: any): any;
-    deleteIndex(indexID: any, options?: any): any;
-    editIndex(indexID: any, options?: any): any;
+    listIndexes(options?: VectorDatabaseOptions): Promise<VectorIndexInfo[]>;
+    createIndex(options: CreateVectorIndexOptions): Promise<VectorDatabaseResult>;
+    deleteIndex(indexID: string, options?: VectorDatabaseOptions): Promise<VectorDatabaseResult>;
+    editIndex(indexID: string, options?: EditVectorIndexOptions): Promise<VectorDatabaseResult>;
+}
+export interface VectorDatabaseOptions {
+    /** Timeout in milliseconds for the operation */
+    timeoutMs?: number;
+}
+export interface CreateVectorIndexOptions extends VectorDatabaseOptions {
+    /** Name of the index to create */
+    name: string;
+    /** Number of dimensions for vectors in this index */
+    dimension: number;
+    /** Distance metric: cosine, euclidean, or dotproduct */
+    metric?: 'cosine' | 'euclidean' | 'dotproduct';
+}
+export interface EditVectorIndexOptions extends VectorDatabaseOptions {
+    /** New name for the index */
+    name?: string;
+}
+export interface VectorIndexInfo {
+    /** Name of the index */
+    name: string;
+    /** Number of dimensions for vectors in this index */
+    dimension: number;
+    /** Distance metric used */
+    metric: string;
+}
+export interface VectorDatabaseResult {
+    success: boolean;
+    message: string;
 }
 //# sourceMappingURL=IVectorDatabase.d.ts.map

package/dist/generic/IVectorDatabase.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"IVectorDatabase.d.ts","sourceRoot":"","sources":["../../src/generic/IVectorDatabase.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,eAAe;IAC5B,WAAW,CAAC,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;IAChC,WAAW,CAAC,OAAO,EAAE,GAAG,GAAG,GAAG,CAAC;IAC/B,WAAW,CAAC,OAAO,EAAE,GAAG,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;IAC9C,SAAS,CAAC,OAAO,EAAE,GAAG,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;CAC/C"}
+{"version":3,"file":"IVectorDatabase.d.ts","sourceRoot":"","sources":["../../src/generic/IVectorDatabase.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AACH,MAAM,WAAW,eAAe;IAC5B,WAAW,CAAC,OAAO,CAAC,EAAE,qBAAqB,GAAG,OAAO,CAAC,eAAe,EAAE,CAAC,CAAC;IACzE,WAAW,CAAC,OAAO,EAAE,wBAAwB,GAAG,OAAO,CAAC,oBAAoB,CAAC,CAAC;IAC9E,WAAW,CAAC,OAAO,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,qBAAqB,GAAG,OAAO,CAAC,oBAAoB,CAAC,CAAC;IAC7F,SAAS,CAAC,OAAO,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,sBAAsB,GAAG,OAAO,CAAC,oBAAoB,CAAC,CAAC;CAC/F;AAED,MAAM,WAAW,qBAAqB;IAClC,gDAAgD;IAChD,SAAS,CAAC,EAAE,MAAM,CAAC;CACtB;AAED,MAAM,WAAW,wBAAyB,SAAQ,qBAAqB;IACnE,kCAAkC;IAClC,IAAI,EAAE,MAAM,CAAC;IACb,qDAAqD;IACrD,SAAS,EAAE,MAAM,CAAC;IAClB,wDAAwD;IACxD,MAAM,CAAC,EAAE,QAAQ,GAAG,WAAW,GAAG,YAAY,CAAC;CAClD;AAED,MAAM,WAAW,sBAAuB,SAAQ,qBAAqB;IACjE,6BAA6B;IAC7B,IAAI,CAAC,EAAE,MAAM,CAAC;CACjB;AAED,MAAM,WAAW,eAAe;IAC5B,wBAAwB;IACxB,IAAI,EAAE,MAAM,CAAC;IACb,qDAAqD;IACrD,SAAS,EAAE,MAAM,CAAC;IAClB,2BAA2B;IAC3B,MAAM,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,oBAAoB;IACjC,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE,MAAM,CAAC;CACnB"}

package/dist/generic/IVectorIndex.d.ts

@@ -1,11 +1,40 @@
+/**
+ * Interface for vector index operations (CRUD on individual vectors).
+ * @deprecated Use VectorDBBase from @memberjunction/ai-vectordb instead, which provides
+ * a complete abstract class with proper typing (VectorRecord, BaseResponse, UpdateOptions).
+ */
 export interface IVectorIndex {
-    createRecord(record: any, options?: any): any;
-    createRecords(records: any[], options?: any): any;
-    getRecord(recordID: any, options?: any): any;
-    getRecords(recordIDs: any[], options?: any): any;
-    updateRecord(record: any, options?: any): any;
-    updateRecords(records: any[], options?: any): any;
-    deleteRecord(recordID: any, options?: any): any;
-    deleteRecords(recordIDs: any[], options?: any): any;
+    createRecord(record: VectorIndexRecord, options?: VectorIndexOptions): Promise<VectorIndexResult>;
+    createRecords(records: VectorIndexRecord[], options?: VectorIndexOptions): Promise<VectorIndexResult>;
+    getRecord(recordID: string, options?: VectorIndexOptions): Promise<VectorIndexRecordResult>;
+    getRecords(recordIDs: string[], options?: VectorIndexOptions): Promise<VectorIndexRecordsResult>;
+    updateRecord(record: VectorIndexRecord, options?: VectorIndexOptions): Promise<VectorIndexResult>;
+    updateRecords(records: VectorIndexRecord[], options?: VectorIndexOptions): Promise<VectorIndexResult>;
+    deleteRecord(recordID: string, options?: VectorIndexOptions): Promise<VectorIndexResult>;
+    deleteRecords(recordIDs: string[], options?: VectorIndexOptions): Promise<VectorIndexResult>;
+}
+export interface VectorIndexOptions {
+    /** Namespace or partition for the records */
+    namespace?: string;
+    /** Timeout in milliseconds */
+    timeoutMs?: number;
+}
+export interface VectorIndexRecord {
+    /** Unique identifier for the vector record */
+    id: string;
+    /** The vector embedding values */
+    values: number[];
+    /** Optional metadata associated with the record */
+    metadata?: Record<string, string | boolean | number | string[]>;
+}
+export interface VectorIndexResult {
+    success: boolean;
+    message: string;
+}
+export interface VectorIndexRecordResult extends VectorIndexResult {
+    record?: VectorIndexRecord;
+}
+export interface VectorIndexRecordsResult extends VectorIndexResult {
+    records?: VectorIndexRecord[];
 }
 //# sourceMappingURL=IVectorIndex.d.ts.map

package/dist/generic/IVectorIndex.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"IVectorIndex.d.ts","sourceRoot":"","sources":["../../src/generic/IVectorIndex.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,YAAY;IACzB,YAAY,CAAC,MAAM,EAAE,GAAG,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;IAC9C,aAAa,CAAC,OAAO,EAAE,GAAG,EAAE,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;IAClD,SAAS,CAAC,QAAQ,EAAE,GAAG,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;IAC7C,UAAU,CAAC,SAAS,EAAE,GAAG,EAAE,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;IACjD,YAAY,CAAC,MAAM,EAAE,GAAG,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;IAC9C,aAAa,CAAC,OAAO,EAAE,GAAG,EAAE,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;IAClD,YAAY,CAAC,QAAQ,EAAE,GAAG,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;IAChD,aAAa,CAAC,SAAS,EAAE,GAAG,EAAE,EAAE,OAAO,CAAC,EAAE,GAAG,GAAG,GAAG,CAAC;CACvD"}
+{"version":3,"file":"IVectorIndex.d.ts","sourceRoot":"","sources":["../../src/generic/IVectorIndex.ts"],"names":[],"mappings":"AAAA;;;;GAIG;AACH,MAAM,WAAW,YAAY;IACzB,YAAY,CAAC,MAAM,EAAE,iBAAiB,EAAE,OAAO,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAAC;IAClG,aAAa,CAAC,OAAO,EAAE,iBAAiB,EAAE,EAAE,OAAO,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAAC;IACtG,SAAS,CAAC,QAAQ,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC,uBAAuB,CAAC,CAAC;IAC5F,UAAU,CAAC,SAAS,EAAE,MAAM,EAAE,EAAE,OAAO,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC,wBAAwB,CAAC,CAAC;IACjG,YAAY,CAAC,MAAM,EAAE,iBAAiB,EAAE,OAAO,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAAC;IAClG,aAAa,CAAC,OAAO,EAAE,iBAAiB,EAAE,EAAE,OAAO,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAAC;IACtG,YAAY,CAAC,QAAQ,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAAC;IACzF,aAAa,CAAC,SAAS,EAAE,MAAM,EAAE,EAAE,OAAO,CAAC,EAAE,kBAAkB,GAAG,OAAO,CAAC,iBAAiB,CAAC,CAAC;CAChG;AAED,MAAM,WAAW,kBAAkB;IAC/B,6CAA6C;IAC7C,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,8BAA8B;IAC9B,SAAS,CAAC,EAAE,MAAM,CAAC;CACtB;AAED,MAAM,WAAW,iBAAiB;IAC9B,8CAA8C;IAC9C,EAAE,EAAE,MAAM,CAAC;IACX,kCAAkC;IAClC,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,mDAAmD;IACnD,QAAQ,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,GAAG,MAAM,GAAG,MAAM,EAAE,CAAC,CAAC;CACnE;AAED,MAAM,WAAW,iBAAiB;IAC9B,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,uBAAwB,SAAQ,iBAAiB;IAC9D,MAAM,CAAC,EAAE,iBAAiB,CAAC;CAC9B;AAED,MAAM,WAAW,wBAAyB,SAAQ,iBAAiB;IAC/D,OAAO,CAAC,EAAE,iBAAiB,EAAE,CAAC;CACjC"}