@mastra/rag 2.1.0 → 2.1.1

package/dist/docs/rag/05-reference.md

@@ -1,868 +0,0 @@
-# Rag API Reference
-
-> API reference for rag - 7 entries
-
-
----
-
-## Reference: GraphRAG
-
-> Documentation for the GraphRAG class in Mastra, which implements a graph-based approach to retrieval augmented generation.
-
-The `GraphRAG` class implements a graph-based approach to retrieval augmented generation. It creates a knowledge graph from document chunks where nodes represent documents and edges represent semantic relationships, enabling both direct similarity matching and discovery of related content through graph traversal.
-
-## Basic Usage
-
-```typescript
-import { GraphRAG } from "@mastra/rag";
-
-const graphRag = new GraphRAG({
-  dimension: 1536,
-  threshold: 0.7,
-});
-
-// Create the graph from chunks and embeddings
-graphRag.createGraph(documentChunks, embeddings);
-
-// Query the graph with embedding
-const results = await graphRag.query({
-  query: queryEmbedding,
-  topK: 10,
-  randomWalkSteps: 100,
-  restartProb: 0.15,
-});
-```
-
-## Constructor Parameters
-
-## Methods
-
-### createGraph
-
-Creates a knowledge graph from document chunks and their embeddings.
-
-```typescript
-createGraph(chunks: GraphChunk[], embeddings: GraphEmbedding[]): void
-```
-
-#### Parameters
-
-### query
-
-Performs a graph-based search combining vector similarity and graph traversal.
-
-```typescript
-query({
-  query,
-  topK = 10,
-  randomWalkSteps = 100,
-  restartProb = 0.15
-}: {
-  query: number[];
-  topK?: number;
-  randomWalkSteps?: number;
-  restartProb?: number;
-}): RankedNode[]
-```
-
-#### Parameters
-
-#### Returns
-
-Returns an array of `RankedNode` objects, where each node contains:
-
-## Advanced Example
-
-```typescript
-const graphRag = new GraphRAG({
-  dimension: 1536,
-  threshold: 0.8, // Stricter similarity threshold
-});
-
-// Create graph from chunks and embeddings
-graphRag.createGraph(documentChunks, embeddings);
-
-// Query with custom parameters
-const results = await graphRag.query({
-  query: queryEmbedding,
-  topK: 5,
-  randomWalkSteps: 200,
-  restartProb: 0.2,
-});
-```
-
-## Related
-
-- [createGraphRAGTool](../tools/graph-rag-tool)
-
----
-
-## Reference: .chunk()
-
-> Documentation for the chunk function in Mastra, which splits documents into smaller segments using various strategies.
-
-The `.chunk()` function splits documents into smaller segments using various strategies and options.
-
-## Example
-
-```typescript
-import { MDocument } from "@mastra/rag";
-
-const doc = MDocument.fromMarkdown(`
-# Introduction
-This is a sample document that we want to split into chunks.
-
-## Section 1
-Here is the first section with some content.
-
-## Section 2 
-Here is another section with different content.
-`);
-
-// Basic chunking with defaults
-const chunks = await doc.chunk();
-
-// Markdown-specific chunking with header extraction
-const chunksWithMetadata = await doc.chunk({
-  strategy: "markdown",
-  headers: [
-    ["#", "title"],
-    ["##", "section"],
-  ],
-  extract: {
-    summary: true, // Extract summaries with default settings
-    keywords: true, // Extract keywords with default settings
-  },
-});
-```
-
-## Parameters
-
-The following parameters are available for all chunking strategies.
-**Important:** Each strategy will only utilize a subset of these parameters relevant to its specific use case.
-
-See [ExtractParams reference](https://mastra.ai/reference/rag/extract-params) for details on the `extract` parameter.
-
-## Strategy-Specific Options
-
-Strategy-specific options are passed as top-level parameters alongside the strategy parameter. For example:
-
-```typescript
-// Character strategy example
-const chunks = await doc.chunk({
-  strategy: "character",
-  separator: ".", // Character-specific option
-  isSeparatorRegex: false, // Character-specific option
-  maxSize: 300, // general option
-});
-
-// Recursive strategy example
-const chunks = await doc.chunk({
-  strategy: "recursive",
-  separators: ["\n\n", "\n", " "], // Recursive-specific option
-  language: "markdown", // Recursive-specific option
-  maxSize: 500, // general option
-});
-
-// Sentence strategy example
-const chunks = await doc.chunk({
-  strategy: "sentence",
-  maxSize: 450, // Required for sentence strategy
-  minSize: 50, // Sentence-specific option
-  sentenceEnders: ["."], // Sentence-specific option
-  fallbackToCharacters: false, // Sentence-specific option
-});
-
-// HTML strategy example
-const chunks = await doc.chunk({
-  strategy: "html",
-  headers: [
-    ["h1", "title"],
-    ["h2", "subtitle"],
-  ], // HTML-specific option
-});
-
-// Markdown strategy example
-const chunks = await doc.chunk({
-  strategy: "markdown",
-  headers: [
-    ["#", "title"],
-    ["##", "section"],
-  ], // Markdown-specific option
-  stripHeaders: true, // Markdown-specific option
-});
-
-// Semantic Markdown strategy example
-const chunks = await doc.chunk({
-  strategy: "semantic-markdown",
-  joinThreshold: 500, // Semantic Markdown-specific option
-  modelName: "gpt-3.5-turbo", // Semantic Markdown-specific option
-});
-
-// Token strategy example
-const chunks = await doc.chunk({
-  strategy: "token",
-  encodingName: "gpt2", // Token-specific option
-  modelName: "gpt-3.5-turbo", // Token-specific option
-  maxSize: 1000, // general option
-});
-```
-
-The options documented below are passed directly at the top level of the configuration object, not nested within a separate options object.
-
-### Character
-
-### Recursive
-
-### Sentence
-
-### HTML
-
-**Important:** When using the HTML strategy, all general options are ignored. Use `headers` for header-based splitting or `sections` for section-based splitting. If used together, `sections` will be ignored.
-
-### Markdown
-
-**Important:** When using the `headers` option, the markdown strategy ignores all general options and content is split based on the markdown header structure. To use size-based chunking with markdown, omit the `headers` parameter.
-
-### Semantic Markdown
-
-### Token
-
-### JSON
-
-### Latex
-
-The Latex strategy uses only the general chunking options listed above. It provides LaTeX-aware splitting optimized for mathematical and academic documents.
-
-## Return Value
-
-Returns a `MDocument` instance containing the chunked documents. Each chunk includes:
-
-```typescript
-interface DocumentNode {
-  text: string;
-  metadata: Record<string, any>;
-  embedding?: number[];
-}
-```
-
----
-
-## Reference: DatabaseConfig
-
-> API reference for database-specific configuration types used with vector query tools in Mastra RAG systems.
-
-The `DatabaseConfig` type allows you to specify database-specific configurations when using vector query tools. These configurations enable you to leverage unique features and optimizations offered by different vector stores.
-
-## Type Definition
-
-```typescript
-export type DatabaseConfig = {
-  pinecone?: PineconeConfig;
-  pgvector?: PgVectorConfig;
-  chroma?: ChromaConfig;
-  [key: string]: any; // Extensible for future databases
-};
-```
-
-## Database-Specific Types
-
-### PineconeConfig
-
-Configuration options specific to Pinecone vector store.
-
-**Use Cases:**
-
-- Multi-tenant applications (separate namespaces per tenant)
-- Environment isolation (dev/staging/prod namespaces)
-- Hybrid search combining semantic and keyword matching
-
-### PgVectorConfig
-
-Configuration options specific to PostgreSQL with pgvector extension.
-
-**Performance Guidelines:**
-
-- **ef**: Start with 2-4x your topK value, increase for better accuracy
-- **probes**: Start with 1-10, increase for better recall
-- **minScore**: Use values between 0.5-0.9 depending on your quality requirements
-
-**Use Cases:**
-
-- Performance optimization for high-load scenarios
-- Quality filtering to remove irrelevant results
-- Fine-tuning search accuracy vs speed tradeoffs
-
-### ChromaConfig
-
-Configuration options specific to Chroma vector store.
-
-**Filter Syntax Examples:**
-
-```typescript
-// Simple equality
-where: { "category": "technical" }
-
-// Operators
-where: { "price": { "$gt": 100 } }
-
-// Multiple conditions
-where: {
-  "category": "electronics",
-  "inStock": true
-}
-
-// Document content filtering
-whereDocument: { "$contains": "API documentation" }
-```
-
-**Use Cases:**
-
-- Advanced metadata filtering
-- Content-based document filtering
-- Complex query combinations
-
-## Usage Examples
-
-  **basic-usage:**
-
-    ### Basic Database Configuration
-
-    ```typescript
-    import { createVectorQueryTool } from '@mastra/rag';
-
-    const vectorTool = createVectorQueryTool({
-      vectorStoreName: 'pinecone',
-      indexName: 'documents',
-      model: embedModel,
-      databaseConfig: {
-        pinecone: {
-          namespace: 'production'
-        }
-      }
-    });
-    ```
-
-  
-
-  **runtime-override:**
-
-    ### Runtime Configuration Override
-
-    ```typescript
-    import { RequestContext } from '@mastra/core/request-context';
-
-    // Initial configuration
-    const vectorTool = createVectorQueryTool({
-      vectorStoreName: 'pinecone',
-      indexName: 'documents',
-      model: embedModel,
-      databaseConfig: {
-        pinecone: {
-          namespace: 'development'
-        }
-      }
-    });
-
-    // Override at runtime
-    const requestContext = new RequestContext();
-    requestContext.set('databaseConfig', {
-      pinecone: {
-        namespace: 'production'
-      }
-    });
-
-    await vectorTool.execute(
-      { queryText: 'search query' },
-      { mastra, requestContext }
-    );
-    ```
-
-  
-
-  **multi-database:**
-
-    ### Multi-Database Configuration
-
-    ```typescript
-    const vectorTool = createVectorQueryTool({
-      vectorStoreName: 'dynamic', // Will be determined at runtime
-      indexName: 'documents',
-      model: embedModel,
-      databaseConfig: {
-        pinecone: {
-          namespace: 'default'
-        },
-        pgvector: {
-          minScore: 0.8,
-          ef: 150
-        },
-        chroma: {
-          where: { 'type': 'documentation' }
-        }
-      }
-    });
-    ```
-
-    > **Note:**
-
-**Multi-Database Support**: When you configure multiple databases, only the configuration matching the actual vector store being used will be applied.
-
-  
-
-  **performance-tuning:**
-
-    ### Performance Tuning
-
-    ```typescript
-    // High accuracy configuration
-    const highAccuracyTool = createVectorQueryTool({
-      vectorStoreName: 'postgres',
-      indexName: 'embeddings',
-      model: embedModel,
-      databaseConfig: {
-        pgvector: {
-          ef: 400,        // High accuracy
-          probes: 20,     // High recall
-          minScore: 0.85  // High quality threshold
-        }
-      }
-    });
-
-    // High speed configuration
-    const highSpeedTool = createVectorQueryTool({
-      vectorStoreName: 'postgres',
-      indexName: 'embeddings',
-      model: embedModel,
-      databaseConfig: {
-        pgvector: {
-          ef: 50,         // Lower accuracy, faster
-          probes: 3,      // Lower recall, faster
-          minScore: 0.6   // Lower quality threshold
-        }
-      }
-    });
-    ```
-
-  
-
-## Extensibility
-
-The `DatabaseConfig` type is designed to be extensible. To add support for a new vector database:
-
-```typescript
-// 1. Define the configuration interface
-export interface NewDatabaseConfig {
-  customParam1?: string;
-  customParam2?: number;
-}
-
-// 2. Extend DatabaseConfig type
-export type DatabaseConfig = {
-  pinecone?: PineconeConfig;
-  pgvector?: PgVectorConfig;
-  chroma?: ChromaConfig;
-  newdatabase?: NewDatabaseConfig;
-  [key: string]: any;
-};
-
-// 3. Use in vector query tool
-const vectorTool = createVectorQueryTool({
-  vectorStoreName: "newdatabase",
-  indexName: "documents",
-  model: embedModel,
-  databaseConfig: {
-    newdatabase: {
-      customParam1: "value",
-      customParam2: 42,
-    },
-  },
-});
-```
-
-## Best Practices
-
-1. **Environment Configuration**: Use different namespaces or configurations for different environments
-2. **Performance Tuning**: Start with default values and adjust based on your specific needs
-3. **Quality Filtering**: Use minScore to filter out low-quality results
-4. **Runtime Flexibility**: Override configurations at runtime for dynamic scenarios
-5. **Documentation**: Document your specific configuration choices for team members
-
-## Migration Guide
-
-Existing vector query tools continue to work without changes. To add database configurations:
-
-```diff
-const vectorTool = createVectorQueryTool({
-  vectorStoreName: 'pinecone',
-  indexName: 'documents',
-  model: embedModel,
-+ databaseConfig: {
-+   pinecone: {
-+     namespace: 'production'
-+   }
-+ }
-});
-```
-
-## Related
-
-- [createVectorQueryTool()](https://mastra.ai/reference/tools/vector-query-tool)
-- [Hybrid Vector Search](https://mastra.ai/docs/rag/retrieval#metadata-filtering)
-- [Metadata Filters](https://mastra.ai/reference/rag/metadata-filters)
-
----
-
-## Reference: MDocument
-
-> Documentation for the MDocument class in Mastra, which handles document processing and chunking.
-
-The MDocument class processes documents for RAG applications. The main methods are `.chunk()` and `.extractMetadata()`.
-
-## Constructor
-
-## Static Methods
-
-### fromText()
-
-Creates a document from plain text content.
-
-```typescript
-static fromText(text: string, metadata?: Record<string, any>): MDocument
-```
-
-### fromHTML()
-
-Creates a document from HTML content.
-
-```typescript
-static fromHTML(html: string, metadata?: Record<string, any>): MDocument
-```
-
-### fromMarkdown()
-
-Creates a document from Markdown content.
-
-```typescript
-static fromMarkdown(markdown: string, metadata?: Record<string, any>): MDocument
-```
-
-### fromJSON()
-
-Creates a document from JSON content.
-
-```typescript
-static fromJSON(json: string, metadata?: Record<string, any>): MDocument
-```
-
-## Instance Methods
-
-### chunk()
-
-Splits document into chunks and optionally extracts metadata.
-
-```typescript
-async chunk(params?: ChunkParams): Promise<Chunk[]>
-```
-
-See [chunk() reference](./chunk) for detailed options.
-
-### getDocs()
-
-Returns array of processed document chunks.
-
-```typescript
-getDocs(): Chunk[]
-```
-
-### getText()
-
-Returns array of text strings from chunks.
-
-```typescript
-getText(): string[]
-```
-
-### getMetadata()
-
-Returns array of metadata objects from chunks.
-
-```typescript
-getMetadata(): Record<string, any>[]
-```
-
-### extractMetadata()
-
-Extracts metadata using specified extractors. See [ExtractParams reference](./extract-params) for details.
-
-```typescript
-async extractMetadata(params: ExtractParams): Promise<MDocument>
-```
-
-## Examples
-
-```typescript
-import { MDocument } from "@mastra/rag";
-
-// Create document from text
-const doc = MDocument.fromText("Your content here");
-
-// Split into chunks with metadata extraction
-const chunks = await doc.chunk({
-  strategy: "markdown",
-  headers: [
-    ["#", "title"],
-    ["##", "section"],
-  ],
-  extract: {
-    summary: true, // Extract summaries with default settings
-    keywords: true, // Extract keywords with default settings
-  },
-});
-
-// Get processed chunks
-const docs = doc.getDocs();
-const texts = doc.getText();
-const metadata = doc.getMetadata();
-```
-
----
-
-## Reference: ExtractParams
-
-> Documentation for metadata extraction configuration in Mastra.
-
-ExtractParams configures metadata extraction from document chunks using LLM analysis.
-
-## Example
-
-```typescript
-import { MDocument } from "@mastra/rag";
-
-const doc = MDocument.fromText(text);
-const chunks = await doc.chunk({
-  extract: {
-    title: true, // Extract titles using default settings
-    summary: true, // Generate summaries using default settings
-    keywords: true, // Extract keywords using default settings
-  },
-});
-
-// Example output:
-// chunks[0].metadata = {
-//   documentTitle: "AI Systems Overview",
-//   sectionSummary: "Overview of artificial intelligence concepts and applications",
-//   excerptKeywords: "KEYWORDS: AI, machine learning, algorithms"
-// }
-```
-
-## Parameters
-
-The `extract` parameter accepts the following fields:
-
-## Extractor Arguments
-
-### TitleExtractorsArgs
-
-### SummaryExtractArgs
-
-### QuestionAnswerExtractArgs
-
-### KeywordExtractArgs
-
-### SchemaExtractArgs
-
-## Advanced Example
-
-```typescript
-import { MDocument } from "@mastra/rag";
-
-const doc = MDocument.fromText(text);
-const chunks = await doc.chunk({
-  extract: {
-    // Title extraction with custom settings
-    title: {
-      nodes: 2, // Extract 2 title nodes
-      nodeTemplate: "Generate a title for this: {context}",
-      combineTemplate: "Combine these titles: {context}",
-    },
-
-    // Summary extraction with custom settings
-    summary: {
-      summaries: ["self"], // Generate summaries for current chunk
-      promptTemplate: "Summarize this: {context}",
-    },
-
-    // Question generation with custom settings
-    questions: {
-      questions: 3, // Generate 3 questions
-      promptTemplate: "Generate {numQuestions} questions about: {context}",
-      embeddingOnly: false,
-    },
-
-    // Keyword extraction with custom settings
-    keywords: {
-      keywords: 5, // Extract 5 keywords
-      promptTemplate: "Extract {maxKeywords} key terms from: {context}",
-    },
-
-    // Schema extraction with Zod
-    schema: {
-        schema: z.object({
-            productName: z.string(),
-            category: z.enum(["electronics", "clothing"]),
-        }),
-        instructions: "Extract product information.",
-        metadataKey: "product",
-    },
-  },
-});
-
-// Example output:
-// chunks[0].metadata = {
-//   documentTitle: "AI in Modern Computing",
-//   sectionSummary: "Overview of AI concepts and their applications in computing",
-//   questionsThisExcerptCanAnswer: "1. What is machine learning?\n2. How do neural networks work?",
-//   excerptKeywords: "1. Machine learning\n2. Neural networks\n3. Training data",
-//   product: {
-//     productName: "Neural Net 2000",
-//     category: "electronics"
-//   }
-// }
-```
-
-## Document Grouping for Title Extraction
-
-When using the `TitleExtractor`, you can group multiple chunks together for title extraction by specifying a shared `docId` in the `metadata` field of each chunk. All chunks with the same `docId` will receive the same extracted title. If no `docId` is set, each chunk is treated as its own document for title extraction.
-
-**Example:**
-
-```ts
-import { MDocument } from "@mastra/rag";
-
-const doc = new MDocument({
-  docs: [
-    { text: "chunk 1", metadata: { docId: "docA" } },
-    { text: "chunk 2", metadata: { docId: "docA" } },
-    { text: "chunk 3", metadata: { docId: "docB" } },
-  ],
-  type: "text",
-});
-
-await doc.extractMetadata({ title: true });
-// The first two chunks will share a title, while the third chunk will be assigned a separate title.
-```
-
----
-
-## Reference: rerank()
-
-> Documentation for the rerank function in Mastra, which provides advanced reranking capabilities for vector search results.
-
-The `rerank()` function provides advanced reranking capabilities for vector search results by combining semantic relevance, vector similarity, and position-based scoring.
-
-```typescript
-function rerank(
-  results: QueryResult[],
-  query: string,
-  modelConfig: ModelConfig,
-  options?: RerankerFunctionOptions,
-): Promise<RerankResult[]>;
-```
-
-## Usage Example
-
-```typescript
-import { rerank } from "@mastra/rag";
-
-const model = "openai/gpt-5.1";
-
-const rerankedResults = await rerank(
-  vectorSearchResults,
-  "How do I deploy to production?",
-  model,
-  {
-    weights: {
-      semantic: 0.5,
-      vector: 0.3,
-      position: 0.2,
-    },
-    topK: 3,
-  },
-);
-```
-
-## Parameters
-
-The rerank function accepts any LanguageModel from the Vercel AI SDK. When using the Cohere model `rerank-v3.5`, it will automatically use Cohere's reranking capabilities.
-
-> **Note:** For semantic scoring to work properly during re-ranking, each result must include the text content in its `metadata.text` field.
-
-### RerankerFunctionOptions
-
-## Returns
-
-The function returns an array of `RerankResult` objects:
-
-### ScoringDetails
-
-## Related
-
-- [createVectorQueryTool](../tools/vector-query-tool)
-
----
-
-## Reference: rerankWithScorer()
-
-> Documentation for the rerank function in Mastra, which provides advanced reranking capabilities for vector search results.
-
-The `rerankWithScorer()` function provides advanced reranking capabilities for vector search results by combining semantic relevance, vector similarity, and position-based scoring.
-
-```typescript
-function rerankWithScorer({
-  results: QueryResult[],
-  query: string,
-  scorer: RelevanceScoreProvider,
-  options?: RerankerFunctionOptions,
-}): Promise<RerankResult[]>;
-```
-
-## Usage Example
-
-```typescript
-import { rerankWithScorer as rerank, CohereRelevanceScorer } from "@mastra/rag";
-
-const scorer = new CohereRelevanceScorer("rerank-v3.5");
-
-const rerankedResults = await rerank({
-  results: vectorSearchResults,
-  query: "How do I deploy to production?",
-  scorer,
-  options: {
-    weights: {
-      semantic: 0.5,
-      vector: 0.3,
-      position: 0.2,
-    },
-    topK: 3,
-  },
-});
-```
-
-## Parameters
-
-The `rerankWithScorer` function accepts any `RelevanceScoreProvider` from @mastra/rag.
-
-> **Note:** For semantic scoring to work properly during re-ranking, each result must include the text content in its `metadata.text` field.
-
-### RerankerFunctionOptions
-
-## Returns
-
-The function returns an array of `RerankResult` objects:
-
-### ScoringDetails
-
-## Related
-
-- [createVectorQueryTool](../tools/vector-query-tool)