@mastra/rag 2.1.1 → 2.1.2

package/dist/docs/references/reference-rag-chunk.md

@@ -0,0 +1,221 @@
+# .chunk()
+
+The `.chunk()` function splits documents into smaller segments using various strategies and options.
+
+## Example
+
+```typescript
+import { MDocument } from '@mastra/rag'
+
+const doc = MDocument.fromMarkdown(`
+# Introduction
+This is a sample document that we want to split into chunks.
+
+## Section 1
+Here is the first section with some content.
+
+## Section 2
+Here is another section with different content.
+`)
+
+// Basic chunking with defaults
+const chunks = await doc.chunk()
+
+// Markdown-specific chunking with header extraction
+const chunksWithMetadata = await doc.chunk({
+  strategy: 'markdown',
+  headers: [
+    ['#', 'title'],
+    ['##', 'section'],
+  ],
+  extract: {
+    summary: true, // Extract summaries with default settings
+    keywords: true, // Extract keywords with default settings
+  },
+})
+```
+
+## Parameters
+
+The following parameters are available for all chunking strategies. **Important:** Each strategy will only utilize a subset of these parameters relevant to its specific use case.
+
+**strategy?:** (`'recursive' | 'character' | 'token' | 'markdown' | 'semantic-markdown' | 'html' | 'json' | 'latex' | 'sentence'`): The chunking strategy to use. If not specified, defaults based on document type. Depending on the chunking strategy, there are additional optionals. Defaults: .md files → 'markdown', .html/.htm → 'html', .json → 'json', .tex → 'latex', others → 'recursive'
+
+**maxSize?:** (`number`): Maximum size of each chunk. \*\*Note:\*\* Some strategy configurations (markdown with headers, HTML with headers) ignore this parameter. (Default: `4000`)
+
+**overlap?:** (`number`): Number of characters/tokens that overlap between chunks. (Default: `50`)
+
+**lengthFunction?:** (`(text: string) => number`): Function to calculate text length. Defaults to character count.
+
+**separatorPosition?:** (`'start' | 'end'`): Where to position the separator in chunks. 'start' attaches to beginning of next chunk, 'end' attaches to end of current chunk. If not specified, separators are discarded.
+
+**addStartIndex?:** (`boolean`): Whether to add start index metadata to chunks. (Default: `false`)
+
+**stripWhitespace?:** (`boolean`): Whether to strip whitespace from chunks. (Default: `true`)
+
+**extract?:** (`ExtractParams`): Metadata extraction configuration.
+
+See [ExtractParams reference](https://mastra.ai/reference/rag/extract-params) for details on the `extract` parameter.
+
+## Strategy-Specific Options
+
+Strategy-specific options are passed as top-level parameters alongside the strategy parameter. For example:
+
+```typescript
+// Character strategy example
+const chunks = await doc.chunk({
+  strategy: 'character',
+  separator: '.', // Character-specific option
+  isSeparatorRegex: false, // Character-specific option
+  maxSize: 300, // general option
+})
+
+// Recursive strategy example
+const chunks = await doc.chunk({
+  strategy: 'recursive',
+  separators: ['\n\n', '\n', ' '], // Recursive-specific option
+  language: 'markdown', // Recursive-specific option
+  maxSize: 500, // general option
+})
+
+// Sentence strategy example
+const chunks = await doc.chunk({
+  strategy: 'sentence',
+  maxSize: 450, // Required for sentence strategy
+  minSize: 50, // Sentence-specific option
+  sentenceEnders: ['.'], // Sentence-specific option
+  fallbackToCharacters: false, // Sentence-specific option
+})
+
+// HTML strategy example
+const chunks = await doc.chunk({
+  strategy: 'html',
+  headers: [
+    ['h1', 'title'],
+    ['h2', 'subtitle'],
+  ], // HTML-specific option
+})
+
+// Markdown strategy example
+const chunks = await doc.chunk({
+  strategy: 'markdown',
+  headers: [
+    ['#', 'title'],
+    ['##', 'section'],
+  ], // Markdown-specific option
+  stripHeaders: true, // Markdown-specific option
+})
+
+// Semantic Markdown strategy example
+const chunks = await doc.chunk({
+  strategy: 'semantic-markdown',
+  joinThreshold: 500, // Semantic Markdown-specific option
+  modelName: 'gpt-3.5-turbo', // Semantic Markdown-specific option
+})
+
+// Token strategy example
+const chunks = await doc.chunk({
+  strategy: 'token',
+  encodingName: 'gpt2', // Token-specific option
+  modelName: 'gpt-3.5-turbo', // Token-specific option
+  maxSize: 1000, // general option
+})
+```
+
+The options documented below are passed directly at the top level of the configuration object, not nested within a separate options object.
+
+### Character
+
+**separators?:** (`string[]`): Array of separators to try in order of preference. The strategy will attempt to split on the first separator, then fall back to subsequent ones.
+
+**isSeparatorRegex?:** (`boolean`): Whether the separator is a regex pattern (Default: `false`)
+
+### Recursive
+
+**separators?:** (`string[]`): Array of separators to try in order of preference. The strategy will attempt to split on the first separator, then fall back to subsequent ones.
+
+**isSeparatorRegex?:** (`boolean`): Whether the separators are regex patterns (Default: `false`)
+
+**language?:** (`Language`): Programming or markup language for language-specific splitting behavior. See Language enum for supported values.
+
+### Sentence
+
+**maxSize:** (`number`): Maximum size of each chunk (required for sentence strategy)
+
+**minSize?:** (`number`): Minimum size of each chunk. Chunks smaller than this will be merged with adjacent chunks when possible. (Default: `50`)
+
+**targetSize?:** (`number`): Preferred target size for chunks. Defaults to 80% of maxSize. The strategy will try to create chunks close to this size.
+
+**sentenceEnders?:** (`string[]`): Array of characters that mark sentence endings for splitting boundaries. (Default: `['.', '!', '?']`)
+
+**fallbackToWords?:** (`boolean`): Whether to fall back to word-level splitting for sentences that exceed maxSize. (Default: `true`)
+
+**fallbackToCharacters?:** (`boolean`): Whether to fall back to character-level splitting for words that exceed maxSize. Only applies if fallbackToWords is enabled. (Default: `true`)
+
+### HTML
+
+**headers:** (`Array<[string, string]>`): Array of \[selector, metadata key] pairs for header-based splitting
+
+**sections:** (`Array<[string, string]>`): Array of \[selector, metadata key] pairs for section-based splitting
+
+**returnEachLine?:** (`boolean`): Whether to return each line as a separate chunk
+
+**Important:** When using the HTML strategy, all general options are ignored. Use `headers` for header-based splitting or `sections` for section-based splitting. If used together, `sections` will be ignored.
+
+### Markdown
+
+**headers?:** (`Array<[string, string]>`): Array of \[header level, metadata key] pairs
+
+**stripHeaders?:** (`boolean`): Whether to remove headers from the output
+
+**returnEachLine?:** (`boolean`): Whether to return each line as a separate chunk
+
+**Important:** When using the `headers` option, the markdown strategy ignores all general options and content is split based on the markdown header structure. To use size-based chunking with markdown, omit the `headers` parameter.
+
+### Semantic Markdown
+
+**joinThreshold?:** (`number`): Maximum token count for merging related sections. Sections exceeding this limit individually are left intact, but smaller sections are merged with siblings or parents if the combined size stays under this threshold. (Default: `500`)
+
+**modelName?:** (`string`): Name of the model for tokenization. If provided, the model's underlying tokenization \`encodingName\` will be used.
+
+**encodingName?:** (`string`): Name of the token encoding to use. Derived from \`modelName\` if available. (Default: `cl100k_base`)
+
+**allowedSpecial?:** (`Set<string> | 'all'`): Set of special tokens allowed during tokenization, or 'all' to allow all special tokens
+
+**disallowedSpecial?:** (`Set<string> | 'all'`): Set of special tokens to disallow during tokenization, or 'all' to disallow all special tokens (Default: `all`)
+
+### Token
+
+**encodingName?:** (`string`): Name of the token encoding to use
+
+**modelName?:** (`string`): Name of the model for tokenization
+
+**allowedSpecial?:** (`Set<string> | 'all'`): Set of special tokens allowed during tokenization, or 'all' to allow all special tokens
+
+**disallowedSpecial?:** (`Set<string> | 'all'`): Set of special tokens to disallow during tokenization, or 'all' to disallow all special tokens
+
+### JSON
+
+**maxSize:** (`number`): Maximum size of each chunk
+
+**minSize?:** (`number`): Minimum size of each chunk
+
+**ensureAscii?:** (`boolean`): Whether to ensure ASCII encoding
+
+**convertLists?:** (`boolean`): Whether to convert lists in the JSON
+
+### Latex
+
+The Latex strategy uses only the general chunking options listed above. It provides LaTeX-aware splitting optimized for mathematical and academic documents.
+
+## Return Value
+
+Returns a `MDocument` instance containing the chunked documents. Each chunk includes:
+
+```typescript
+interface DocumentNode {
+  text: string
+  metadata: Record<string, any>
+  embedding?: number[]
+}
+```

package/dist/docs/references/reference-rag-database-config.md

@@ -0,0 +1,261 @@
+# DatabaseConfig
+
+The `DatabaseConfig` type allows you to specify database-specific configurations when using vector query tools. These configurations enable you to leverage unique features and optimizations offered by different vector stores.
+
+## Type Definition
+
+```typescript
+export type DatabaseConfig = {
+  pinecone?: PineconeConfig
+  pgvector?: PgVectorConfig
+  chroma?: ChromaConfig
+  [key: string]: any // Extensible for future databases
+}
+```
+
+## Database-Specific Types
+
+### PineconeConfig
+
+Configuration options specific to Pinecone vector store.
+
+**namespace?:** (`string`): Pinecone namespace for organizing and isolating vectors within the same index. Useful for multi-tenancy or environment separation.
+
+**sparseVector?:** (`{ indices: number[]; values: number[]; }`): objectindices:number\[]Array of indices for sparse vector componentsvalues:number\[]Array of values corresponding to the indices
+
+**Use Cases:**
+
+- Multi-tenant applications (separate namespaces per tenant)
+- Environment isolation (dev/staging/prod namespaces)
+- Hybrid search combining semantic and keyword matching
+
+### PgVectorConfig
+
+Configuration options specific to PostgreSQL with pgvector extension.
+
+**minScore?:** (`number`): Minimum similarity score threshold for results. Only vectors with similarity scores above this value will be returned.
+
+**ef?:** (`number`): HNSW search parameter that controls the size of the dynamic candidate list during search. Higher values improve accuracy at the cost of speed. Typically set between topK and 200.
+
+**probes?:** (`number`): IVFFlat probe parameter that specifies the number of index cells to visit during search. Higher values improve recall at the cost of speed.
+
+**Performance Guidelines:**
+
+- **ef**: Start with 2-4x your topK value, increase for better accuracy
+- **probes**: Start with 1-10, increase for better recall
+- **minScore**: Use values between 0.5-0.9 depending on your quality requirements
+
+**Use Cases:**
+
+- Performance optimization for high-load scenarios
+- Quality filtering to remove irrelevant results
+- Fine-tuning search accuracy vs speed tradeoffs
+
+### ChromaConfig
+
+Configuration options specific to Chroma vector store.
+
+**where?:** (`Record<string, any>`): Metadata filtering conditions using MongoDB-style query syntax. Filters results based on metadata fields.
+
+**whereDocument?:** (`Record<string, any>`): Document content filtering conditions. Allows filtering based on the actual document text content.
+
+**Filter Syntax Examples:**
+
+```typescript
+// Simple equality
+where: { "category": "technical" }
+
+// Operators
+where: { "price": { "$gt": 100 } }
+
+// Multiple conditions
+where: {
+  "category": "electronics",
+  "inStock": true
+}
+
+// Document content filtering
+whereDocument: { "$contains": "API documentation" }
+```
+
+**Use Cases:**
+
+- Advanced metadata filtering
+- Content-based document filtering
+- Complex query combinations
+
+## Usage Examples
+
+**Basic Usage**:
+
+### Basic Database Configuration
+
+```typescript
+import { createVectorQueryTool } from '@mastra/rag'
+
+const vectorTool = createVectorQueryTool({
+  vectorStoreName: 'pinecone',
+  indexName: 'documents',
+  model: embedModel,
+  databaseConfig: {
+    pinecone: {
+      namespace: 'production',
+    },
+  },
+})
+```
+
+**Runtime Override**:
+
+### Runtime Configuration Override
+
+```typescript
+import { RequestContext } from '@mastra/core/request-context'
+
+// Initial configuration
+const vectorTool = createVectorQueryTool({
+  vectorStoreName: 'pinecone',
+  indexName: 'documents',
+  model: embedModel,
+  databaseConfig: {
+    pinecone: {
+      namespace: 'development',
+    },
+  },
+})
+
+// Override at runtime
+const requestContext = new RequestContext()
+requestContext.set('databaseConfig', {
+  pinecone: {
+    namespace: 'production',
+  },
+})
+
+await vectorTool.execute({ queryText: 'search query' }, { mastra, requestContext })
+```
+
+**Multi-Database**:
+
+### Multi-Database Configuration
+
+```typescript
+const vectorTool = createVectorQueryTool({
+  vectorStoreName: 'dynamic', // Will be determined at runtime
+  indexName: 'documents',
+  model: embedModel,
+  databaseConfig: {
+    pinecone: {
+      namespace: 'default',
+    },
+    pgvector: {
+      minScore: 0.8,
+      ef: 150,
+    },
+    chroma: {
+      where: { type: 'documentation' },
+    },
+  },
+})
+```
+
+> **Note:** **Multi-Database Support**: When you configure multiple databases, only the configuration matching the actual vector store being used will be applied.
+
+**Performance Tuning**:
+
+### Performance Tuning
+
+```typescript
+// High accuracy configuration
+const highAccuracyTool = createVectorQueryTool({
+  vectorStoreName: 'postgres',
+  indexName: 'embeddings',
+  model: embedModel,
+  databaseConfig: {
+    pgvector: {
+      ef: 400, // High accuracy
+      probes: 20, // High recall
+      minScore: 0.85, // High quality threshold
+    },
+  },
+})
+
+// High speed configuration
+const highSpeedTool = createVectorQueryTool({
+  vectorStoreName: 'postgres',
+  indexName: 'embeddings',
+  model: embedModel,
+  databaseConfig: {
+    pgvector: {
+      ef: 50, // Lower accuracy, faster
+      probes: 3, // Lower recall, faster
+      minScore: 0.6, // Lower quality threshold
+    },
+  },
+})
+```
+
+## Extensibility
+
+The `DatabaseConfig` type is designed to be extensible. To add support for a new vector database:
+
+```typescript
+// 1. Define the configuration interface
+export interface NewDatabaseConfig {
+  customParam1?: string
+  customParam2?: number
+}
+
+// 2. Extend DatabaseConfig type
+export type DatabaseConfig = {
+  pinecone?: PineconeConfig
+  pgvector?: PgVectorConfig
+  chroma?: ChromaConfig
+  newdatabase?: NewDatabaseConfig
+  [key: string]: any
+}
+
+// 3. Use in vector query tool
+const vectorTool = createVectorQueryTool({
+  vectorStoreName: 'newdatabase',
+  indexName: 'documents',
+  model: embedModel,
+  databaseConfig: {
+    newdatabase: {
+      customParam1: 'value',
+      customParam2: 42,
+    },
+  },
+})
+```
+
+## Best Practices
+
+1. **Environment Configuration**: Use different namespaces or configurations for different environments
+2. **Performance Tuning**: Start with default values and adjust based on your specific needs
+3. **Quality Filtering**: Use minScore to filter out low-quality results
+4. **Runtime Flexibility**: Override configurations at runtime for dynamic scenarios
+5. **Documentation**: Document your specific configuration choices for team members
+
+## Migration Guide
+
+Existing vector query tools continue to work without changes. To add database configurations:
+
+```diff
+const vectorTool = createVectorQueryTool({
+  vectorStoreName: 'pinecone',
+  indexName: 'documents',
+  model: embedModel,
++ databaseConfig: {
++   pinecone: {
++     namespace: 'production'
++   }
++ }
+});
+```
+
+## Related
+
+- [createVectorQueryTool()](https://mastra.ai/reference/tools/vector-query-tool)
+- [Hybrid Vector Search](https://mastra.ai/docs/rag/retrieval)
+- [Metadata Filters](https://mastra.ai/reference/rag/metadata-filters)

package/dist/docs/references/reference-rag-document.md

@@ -0,0 +1,114 @@
+# MDocument
+
+The MDocument class processes documents for RAG applications. The main methods are `.chunk()` and `.extractMetadata()`.
+
+## Constructor
+
+**docs:** (`Array<{ text: string, metadata?: Record<string, any> }>`): Array of document chunks with their text content and optional metadata
+
+**type:** (`'text' | 'html' | 'markdown' | 'json' | 'latex'`): Type of document content
+
+## Static Methods
+
+### fromText()
+
+Creates a document from plain text content.
+
+```typescript
+static fromText(text: string, metadata?: Record<string, any>): MDocument
+```
+
+### fromHTML()
+
+Creates a document from HTML content.
+
+```typescript
+static fromHTML(html: string, metadata?: Record<string, any>): MDocument
+```
+
+### fromMarkdown()
+
+Creates a document from Markdown content.
+
+```typescript
+static fromMarkdown(markdown: string, metadata?: Record<string, any>): MDocument
+```
+
+### fromJSON()
+
+Creates a document from JSON content.
+
+```typescript
+static fromJSON(json: string, metadata?: Record<string, any>): MDocument
+```
+
+## Instance Methods
+
+### chunk()
+
+Splits document into chunks and optionally extracts metadata.
+
+```typescript
+async chunk(params?: ChunkParams): Promise<Chunk[]>
+```
+
+See [chunk() reference](https://mastra.ai/reference/rag/chunk) for detailed options.
+
+### getDocs()
+
+Returns array of processed document chunks.
+
+```typescript
+getDocs(): Chunk[]
+```
+
+### getText()
+
+Returns array of text strings from chunks.
+
+```typescript
+getText(): string[]
+```
+
+### getMetadata()
+
+Returns array of metadata objects from chunks.
+
+```typescript
+getMetadata(): Record<string, any>[]
+```
+
+### extractMetadata()
+
+Extracts metadata using specified extractors. See [ExtractParams reference](https://mastra.ai/reference/rag/extract-params) for details.
+
+```typescript
+async extractMetadata(params: ExtractParams): Promise<MDocument>
+```
+
+## Examples
+
+```typescript
+import { MDocument } from '@mastra/rag'
+
+// Create document from text
+const doc = MDocument.fromText('Your content here')
+
+// Split into chunks with metadata extraction
+const chunks = await doc.chunk({
+  strategy: 'markdown',
+  headers: [
+    ['#', 'title'],
+    ['##', 'section'],
+  ],
+  extract: {
+    summary: true, // Extract summaries with default settings
+    keywords: true, // Extract keywords with default settings
+  },
+})
+
+// Get processed chunks
+const docs = doc.getDocs()
+const texts = doc.getText()
+const metadata = doc.getMetadata()
+```