voctar 0.1.0

package/docs/API.md

@@ -0,0 +1,361 @@
+# Voctar API Reference
+
+This document covers the full public API exported by `voctar`.
+
+## Package Exports
+
+```typescript
+import {
+  Voctar,
+  Vector, // alias of Voctar
+  chunking,
+  ChunkingService,
+  // types
+  type VectorConfig,
+  type EmbedOptions,
+  type SearchOptions,
+  type SearchResult,
+  type VectorDocument,
+  type CollectionConfig,
+  type EmbeddingProvider,
+  type VectorStoreProvider,
+  // errors
+  VectorEmbeddingError,
+  VectorSearchError,
+  VectorStoreError,
+} from 'voctar';
+```
+
+## `class Voctar` (aka `Vector`)
+
+### Constructor
+
+```typescript
+new Voctar(config?: VectorConfig)
+```
+
+Creates a Voctar client with embedding and store providers.
+
+Notes:
+
+- If `store` is omitted, Voctar defaults to SQLite at `./vector.db`.
+- `autoChunk` defaults to `true`.
+- `defaultChunkStrategy` defaults to `'recursive'`.
+- `defaultChunkSize` defaults to `1000`.
+- `defaultChunkOverlap` defaults to `200`.
+
+### `embed(collection, text, options?)`
+
+```typescript
+embed(
+  collection: string,
+  text: string,
+  options?: EmbedOptions
+): Promise<{ documentId: string; chunkIds: string[] }>
+```
+
+Embeds one document into a collection.
+
+Behavior:
+
+- Validates collection name and text.
+- Ensures collection exists.
+- Auto-chunks when needed (based on token limits and chunk settings).
+- If `options.documentId` is provided and already exists, old chunks are removed first.
+
+Returns:
+
+- `documentId`: parent document id
+- `chunkIds`: final stored ids
+
+### `embedBatch(collection, documents, user_id?)`
+
+```typescript
+embedBatch(
+  collection: string,
+  documents: VectorDocument[],
+  user_id?: string
+): Promise<string[]>
+```
+
+Embeds multiple documents in one batch.
+
+Behavior:
+
+- Ensures collection exists.
+- Re-chunks oversized documents when they exceed model token limits.
+- Upserts all vectors to the configured store.
+
+Returns:
+
+- Array of stored point ids.
+
+### `search(collection, query, options?)`
+
+```typescript
+search(
+  collection: string,
+  query: string,
+  options?: SearchOptions
+): Promise<SearchResult[]>
+```
+
+Performs semantic similarity search.
+
+Behavior:
+
+- Embeds the query.
+- Searches configured vector store.
+- Returns normalized results with ISO `createdAt`.
+- Includes internal system metadata only when `includeSystem` is `true`.
+
+### `upsert(collection, documentId, text, options?)`
+
+```typescript
+upsert(
+  collection: string,
+  documentId: string,
+  text: string,
+  options?: EmbedOptions
+): Promise<void>
+```
+
+Replaces a document:
+
+1. deletes previous document/chunks by `documentId`,
+2. re-embeds the new text with the same `documentId`.
+
+### `delete(collection, documentId | documentIds)`
+
+```typescript
+delete(collection: string, documentId: string | string[]): Promise<void>
+```
+
+Deletes one or many documents and their chunks.
+
+Behavior:
+
+- Deletes direct ids.
+- Also resolves chunk ids through `system._documentId` filter and deletes them.
+
+### `deleteCollection(collection)`
+
+```typescript
+deleteCollection(collection: string): Promise<void>
+```
+
+Deletes an entire collection from the configured store.
+
+### `ensureCollection(collection, config?)`
+
+```typescript
+ensureCollection(
+  collection: string,
+  config?: CollectionConfig
+): Promise<void>
+```
+
+Ensures collection exists with optional collection configuration.
+
+### `getEmbeddingProvider()`
+
+```typescript
+getEmbeddingProvider(): EmbeddingProvider
+```
+
+Returns the active embedding provider instance.
+
+### `getVectorStoreProvider()`
+
+```typescript
+getVectorStoreProvider(): VectorStoreProvider
+```
+
+Returns the active vector store provider instance.
+
+### Static Helpers
+
+#### `Voctar.getChunkId(documentId, chunkIndex)`
+
+```typescript
+static getChunkId(documentId: string, chunkIndex: number): string
+```
+
+Builds chunk id using `documentId#chunkIndex`.
+
+#### `Voctar.parseChunkId(chunkId)`
+
+```typescript
+static parseChunkId(
+  chunkId: string
+): { documentId: string; chunkIndex: number } | null
+```
+
+Parses chunk id into parent id and index, or returns `null`.
+
+#### `Voctar.isChunkId(id)`
+
+```typescript
+static isChunkId(id: string): boolean
+```
+
+Checks whether an id contains the chunk separator format.
+
+## Core Config and Types
+
+### `VectorConfig`
+
+```typescript
+interface VectorConfig {
+  embedding?: RuntimeEmbeddingConfig;
+  store?: RuntimeStoreConfig;
+  embeddingProvider?: EmbeddingProvider; // deprecated
+  vectorStoreProvider?: VectorStoreProvider; // deprecated
+  defaultChunkSize?: number;
+  defaultChunkStrategy?: 'fixed' | 'recursive' | 'semantic' | 'sentence' | 'paragraph';
+  defaultChunkOverlap?: number;
+  autoChunk?: boolean;
+}
+```
+
+### `EmbedOptions`
+
+```typescript
+interface EmbedOptions {
+  documentId?: string;
+  metadata?: Record<string, any>;
+  chunkSize?: number;
+  chunkStrategy?: 'fixed' | 'recursive' | 'semantic' | 'sentence' | 'paragraph';
+  chunkOverlap?: number;
+  autoChunk?: boolean;
+  user_id?: string;
+}
+```
+
+### `SearchOptions`
+
+```typescript
+interface SearchOptions {
+  limit?: number;
+  scoreThreshold?: number;
+  filter?: Record<string, any>;
+  includeSystem?: boolean;
+}
+```
+
+Filter behavior:
+
+- keys without `.` are matched under `metadata.*`
+- keys with `.` are used as-is
+- scalar = equality
+- array = OR match for that field
+
+### `SearchResult`
+
+```typescript
+interface SearchResult {
+  id: string;
+  text: string;
+  score: number;
+  createdAt: string; // ISO 8601
+  metadata?: Record<string, any>;
+  system?: Record<string, any>; // only when includeSystem=true
+}
+```
+
+### `RuntimeEmbeddingConfig`
+
+```typescript
+type RuntimeEmbeddingConfig =
+  | {
+      type: 'openai';
+      apiKey: string;
+      model?: string;
+      dimension?: number;
+      maxRetries?: number;
+    }
+  | {
+      type: 'custom';
+      provider: EmbeddingProvider;
+    };
+```
+
+### `RuntimeStoreConfig`
+
+```typescript
+type RuntimeStoreConfig =
+  | { type: 'sqlite'; path?: string; inMemory?: boolean }
+  | { type: 'qdrant'; url: string; port?: number; apiKey?: string; timeout?: number; checkCompatibility?: boolean }
+  | { type: 'memory' }
+  | { type: 'custom'; provider: VectorStoreProvider };
+```
+
+### `CollectionConfig`
+
+```typescript
+interface CollectionConfig {
+  dimension?: number;
+  distance?: 'cosine' | 'euclidean' | 'dot';
+}
+```
+
+### `VectorDocument`
+
+```typescript
+interface VectorDocument {
+  id: string;
+  text: string;
+  metadata?: Record<string, any>;
+}
+```
+
+## Provider Interfaces
+
+### `EmbeddingProvider`
+
+```typescript
+interface EmbeddingProvider {
+  embed(text: string): Promise<number[]>;
+  embedBatch(texts: string[]): Promise<number[][]>;
+  getDimension(): number;
+  getModelName(): string;
+  getTokenLimit(): number;
+}
+```
+
+### `VectorStoreProvider`
+
+```typescript
+interface VectorStoreProvider {
+  ensureCollection(name: string, dimension: number, config?: CollectionConfig): Promise<void>;
+  upsert(collection: string, points: VectorPoint[]): Promise<void>;
+  search(collection: string, vector: number[], options: SearchOptions): Promise<SearchResult[]>;
+  delete(collection: string, ids: string[]): Promise<void>;
+  deleteCollection(collection: string): Promise<void>;
+  getIdsByFilter(collection: string, filter: Record<string, any>, limit?: number): Promise<string[]>;
+}
+```
+
+## Error Types
+
+### `VectorEmbeddingError`
+
+- Raised on embedding/chunking/query-embedding failures.
+- Includes optional `cause` and optional `data`.
+
+### `VectorSearchError`
+
+- Raised by store-level search providers.
+
+### `VectorStoreError`
+
+- Raised on store operations (collection creation, delete, upsert, etc.).
+
+## Chunking API
+
+Voctar also exports chunking utilities:
+
+- `chunking` singleton service
+- `ChunkingService` class
+
+For full chunking-specific API and behavior, see [`CHUNKING.md`](./CHUNKING.md).

package/docs/CHUNKING.md

@@ -0,0 +1,280 @@
+# Chunking Guide
+
+This is the single source of truth for chunking in Voctar, including:
+
+- chunk ID format and metadata model,
+- strategy behavior and options,
+- semantic chunking behavior,
+- usage examples.
+
+## Chunk Strategies
+
+Different strategies for different content:
+
+```typescript
+// Code - fixed size
+await vector.embed('code', sourceCode, {
+  chunkStrategy: 'fixed',
+  chunkSize: 2000,
+});
+
+// Articles - recursive (default, splits on natural boundaries)
+await vector.embed('articles', article, {
+  chunkStrategy: 'recursive',
+  chunkSize: 1000,
+});
+
+// Narrative - sentence-based
+await vector.embed('stories', story, {
+  chunkStrategy: 'sentence',
+  chunkSize: 800,
+});
+
+// Structured docs - paragraph-based
+await vector.embed('docs', documentation, {
+  chunkStrategy: 'paragraph',
+  chunkSize: 1500,
+});
+```
+
+## How Chunks Are Stored
+
+### Chunk ID Format
+
+Chunks use a predictable ID format: `documentId#chunkIndex`
+
+```typescript
+// Document ID: "user-manual-v2"
+// Chunks get IDs like:
+"user-manual-v2#0"  // First chunk
+"user-manual-v2#1"  // Second chunk
+"user-manual-v2#2"  // Third chunk
+// ... etc
+```
+
+### Benefits
+
+1. **Easy Identification**: You can tell which document a chunk belongs to
+2. **Ordered Retrieval**: Chunk index preserves the order
+3. **Simple Deletion**: Delete all chunks by ID pattern
+4. **No External Tracking**: No need for separate mapping tables
+
+### Metadata Structure
+
+Every chunk stores rich metadata:
+
+```typescript
+{
+  // User metadata
+  title: "User Manual",
+  author: "Tech Team",
+  version: "2.0",
+  
+  // Chunk position info
+  documentId: "user-manual-v2",
+  chunkIndex: 0,           // 0-based index
+  totalChunks: 15,         // Total chunks in document
+  startChar: 0,            // Start position in original text
+  endChar: 1000,           // End position in original text
+  
+  // System metadata
+  _isChunk: true,          // Indicates this is a chunk
+  _documentId: "user-manual-v2",  // Parent document ID
+  _chunkId: "user-manual-v2#0",   // Full chunk ID
+  
+  // Original text
+  text: "Chapter 1: Introduction..."
+}
+```
+
+## Examples
+
+### Basic Chunking
+
+```typescript
+import { Voctar } from '@libs/voctar';
+
+const vector = new Voctar({
+  embedding: {
+    type: 'openai',
+    apiKey: '<your-api-key>',
+  },
+  store: {
+    type: 'sqlite',
+    path: 'data/vector.db',
+  },
+});
+
+const longText = '...10,000 characters...';
+
+// Embed with auto-chunking
+const result = await vector.embed('docs', longText, {
+  documentId: 'article-123',
+  metadata: { author: 'Alice' }
+});
+
+console.log(result.documentId);  // "article-123"
+console.log(result.chunkIds);    // ["article-123#0", "article-123#1", ...]
+```
+
+### Parsing Chunk IDs
+
+```typescript
+import { Voctar } from '@libs/voctar';
+
+const chunkId = "article-123#5";
+
+// Parse chunk ID
+const parsed = Voctar.parseChunkId(chunkId);
+console.log(parsed);
+// { documentId: "article-123", chunkIndex: 5 }
+
+// Check if ID is a chunk
+console.log(Voctar.isChunkId(chunkId));  // true
+console.log(Voctar.isChunkId("article-123"));  // false
+
+// Generate chunk ID
+const id = Voctar.getChunkId("article-123", 5);
+console.log(id);  // "article-123#5"
+```
+
+### Search with Chunk Context
+
+```typescript
+const results = await vector.search('docs', 'installation steps');
+
+results.forEach(result => {
+  // Check if result is from a chunked document
+  if (Voctar.isChunkId(result.id)) {
+    const { documentId, chunkIndex } = Voctar.parseChunkId(result.id)!;
+    
+    console.log(`Found in document: ${documentId}`);
+    console.log(`Chunk ${chunkIndex + 1} of ${result.metadata.totalChunks}`);
+    console.log(`Text position: ${result.metadata.startChar}-${result.metadata.endChar}`);
+  } else {
+    console.log(`Non-chunked document: ${result.id}`);
+  }
+  
+  console.log(result.text);
+});
+```
+
+### Reconstructing Original Document
+
+```typescript
+// Search for all chunks of a document
+const results = await vector.search('docs', 'anything', {
+  filter: { _documentId: 'article-123' },
+  limit: 1000,
+});
+
+// Sort chunks by index
+const sortedChunks = results
+  .map(r => ({
+    ...r,
+    ...Voctar.parseChunkId(r.id)!
+  }))
+  .sort((a, b) => a.chunkIndex - b.chunkIndex);
+
+// Reconstruct (approximately - note overlap)
+const reconstructed = sortedChunks
+  .map(chunk => chunk.text)
+  .join('\n\n');
+```
+
+### Deletion
+
+```typescript
+// Delete document and all its chunks
+await vector.delete('docs', 'article-123');
+
+// This deletes:
+// - article-123 (if it wasn't chunked)
+// - article-123#0, article-123#1, article-123#2, ... (all chunks)
+```
+
+### Upsert (Update)
+
+```typescript
+// Original document
+await vector.embed('docs', originalText, {
+  documentId: 'article-123',
+  metadata: { version: 1 }
+});
+
+// Update the entire document
+await vector.upsert('docs', 'article-123', updatedText, {
+  metadata: { version: 2, updated: Date.now() }
+});
+
+// All old chunks are deleted
+// New chunks are created with same documentId
+// Chunk IDs reset: article-123#0, article-123#1, ...
+```
+
+## Implementation Notes
+
+### Max Chunks Per Document
+
+Currently limited to 1000 chunks per document for efficient deletion.
+
+If your documents might exceed this:
+
+```typescript
+// Increase chunk size
+await vector.embed('docs', veryLongText, {
+  chunkSize: 2000,  // Larger chunks = fewer chunks
+});
+
+// Or split into multiple documents
+const parts = splitIntoSections(veryLongText);
+for (const [index, part] of parts.entries()) {
+  await vector.embed('docs', part, {
+    documentId: `article-123-part${index}`,
+  });
+}
+```
+
+### Chunk ID Character Limit
+
+Keep document IDs reasonable (< 100 chars) to avoid hitting ID length limits in vector stores.
+
+### Metadata Filtering
+
+Filter searches by chunk metadata:
+
+```typescript
+// Find only first chunks
+const results = await vector.search('docs', 'query', {
+  filter: { chunkIndex: 0 }
+});
+
+// Find chunks from specific document
+const results = await vector.search('docs', 'query', {
+  filter: { _documentId: 'article-123' }
+});
+
+// Combine filters
+const results = await vector.search('docs', 'query', {
+  filter: {
+    _documentId: 'article-123',
+    author: 'Alice',
+  }
+});
+```
+
+## Best Practices
+
+1. **Use meaningful document IDs**: They become part of chunk IDs
+2. **Add rich metadata**: Makes filtering and retrieval easier
+3. **Choose appropriate chunk size**: Balance between context and precision
+4. **Use chunk overlap**: Prevents losing context at boundaries
+5. **Track versions**: Include version info in metadata for updates
+
+## Future Improvements
+
+- [ ] Metadata-based deletion (query by `_documentId` and delete all matches)
+- [ ] Chunk merging for adjacent results
+- [ ] Automatic chunk retrieval by proximity
+- [ ] Chunk caching for faster document reconstruction
+