@mastra/mcp-docs-server 1.1.5 → 1.1.6

package/.docs/docs/observability/tracing/processors/sensitive-data-filter.md

@@ -0,0 +1,300 @@
+# Sensitive Data Filter
+
+The Sensitive Data Filter is a span processor that redacts sensitive information from your traces during the processing pipeline before export. This ensures that passwords, API keys, tokens, and other confidential data never leave your application or get stored in observability platforms.
+
+## Default Configuration
+
+The Sensitive Data Filter is included in the recommended observability configuration:
+
+```ts
+import {
+  Observability,
+  DefaultExporter,
+  CloudExporter,
+  SensitiveDataFilter,
+} from '@mastra/observability'
+
+export const mastra = new Mastra({
+  observability: new Observability({
+    configs: {
+      default: {
+        serviceName: 'mastra',
+        exporters: [new DefaultExporter(), new CloudExporter()],
+        spanOutputProcessors: [
+          new SensitiveDataFilter(), // Redacts sensitive fields before export
+        ],
+      },
+    },
+  }),
+  storage: new LibSQLStore({
+    id: 'mastra-storage',
+    url: 'file:./mastra.db',
+  }),
+})
+```
+
+With the default configuration, the filter redacts these common sensitive field names:
+
+- `password`
+- `token`
+- `secret`
+- `key`
+- `apikey`
+- `auth`
+- `authorization`
+- `bearer`
+- `bearertoken`
+- `jwt`
+- `credential`
+- `clientsecret`
+- `privatekey`
+- `refresh`
+- `ssn`
+
+> **Note:** Field matching is case-insensitive and normalizes separators. For example, `api-key`, `api_key`, and `Api Key` are all treated as `apikey`.
+
+## How It Works
+
+The Sensitive Data Filter processes spans before they're sent to exporters, scanning through:
+
+- **Attributes** - Span metadata and properties
+- **Metadata** - Custom metadata attached to spans
+- **Input** - Data sent to agents, tools, and LLMs
+- **Output** - Responses and results
+- **Error Information** - Stack traces and error details
+
+When a sensitive field is detected, its value is replaced with `[REDACTED]` by default. The filter handles nested objects, arrays, and circular references safely.
+
+## Custom Configuration
+
+You can customize which fields are redacted and how redaction appears:
+
+```ts
+import { SensitiveDataFilter, DefaultExporter, Observability } from '@mastra/observability'
+
+export const mastra = new Mastra({
+  observability: new Observability({
+    configs: {
+      production: {
+        serviceName: 'my-service',
+        exporters: [new DefaultExporter()],
+        spanOutputProcessors: [
+          new SensitiveDataFilter({
+            // Add custom sensitive fields
+            sensitiveFields: [
+              // Default fields
+              'password',
+              'token',
+              'secret',
+              'key',
+              'apikey',
+              // Custom fields for your application
+              'creditCard',
+              'bankAccount',
+              'routingNumber',
+              'email',
+              'phoneNumber',
+              'dateOfBirth',
+            ],
+            // Custom redaction token
+            redactionToken: '***SENSITIVE***',
+            // Redaction style
+            redactionStyle: 'full', // or 'partial'
+          }),
+        ],
+      },
+    },
+  }),
+})
+```
+
+## Redaction Styles
+
+The filter supports two redaction styles:
+
+### Full Redaction (Default)
+
+Replaces the entire value with a fixed token:
+
+```jsonc
+// Before
+{
+  "apiKey": "sk-abc123xyz789def456",
+  "userId": "user_12345"
+}
+
+// After
+{
+  "apiKey": "[REDACTED]",
+  "userId": "user_12345"
+}
+```
+
+### Partial Redaction
+
+Shows the first and last 3 characters, useful for debugging without exposing full values:
+
+```ts
+new SensitiveDataFilter({
+  redactionStyle: 'partial',
+})
+```
+
+```json
+// Before
+{
+  "apiKey": "sk-abc123xyz789def456",
+  "creditCard": "4111111111111111"
+}
+
+// After
+{
+  "apiKey": "sk-…456",
+  "creditCard": "411…111"
+}
+```
+
+Values shorter than 7 characters are fully redacted to prevent information leakage.
+
+## Field Matching Rules
+
+The filter uses intelligent field matching:
+
+1. **Case-Insensitive**: `APIKey`, `apikey`, and `ApiKey` are all matched
+
+2. **Separator-Agnostic**: `api-key`, `api_key`, and `apiKey` are treated identically
+
+3. **Exact Matching**: After normalization, fields must match exactly
+
+   - `token` matches `token`, `Token`, `TOKEN`
+   - `token` does NOT match `promptTokens` or `tokenCount`
+
+## Nested Object Handling
+
+The filter recursively processes nested structures:
+
+```json
+// Before
+{
+  "user": {
+    "id": "12345",
+    "credentials": {
+      "password": "SuperSecret123!",
+      "apiKey": "sk-production-key"
+    }
+  },
+  "config": {
+    "auth": {
+      "jwt": "eyJhbGciOiJIUzI1NiIs..."
+    }
+  }
+}
+
+// After
+{
+  "user": {
+    "id": "12345",
+    "credentials": {
+      "password": "[REDACTED]",
+      "apiKey": "[REDACTED]"
+    }
+  },
+  "config": {
+    "auth": {
+      "jwt": "[REDACTED]"
+    }
+  }
+}
+```
+
+## Performance Considerations
+
+The Sensitive Data Filter is designed to be lightweight and efficient:
+
+- **Synchronous Processing**: No async operations, minimal latency impact
+- **Circular Reference Handling**: Safely handles complex object graphs
+- **Error Recovery**: If filtering fails, the field is replaced with an error marker rather than crashing
+
+## Disabling the Filter
+
+If you need to disable sensitive data filtering (not recommended for production):
+
+```ts
+export const mastra = new Mastra({
+  observability: new Observability({
+    configs: {
+      debug: {
+        serviceName: 'debug-service',
+        spanOutputProcessors: [], // No processors, including no SensitiveDataFilter
+        exporters: [new DefaultExporter()],
+      },
+    },
+  }),
+})
+```
+
+> **Warning:** Only disable sensitive data filtering in controlled environments. Never disable it when sending traces to external services or shared storage.
+
+## Common Use Cases
+
+### Healthcare Applications
+
+```ts
+new SensitiveDataFilter({
+  sensitiveFields: [
+    // HIPAA-related fields
+    'ssn',
+    'socialSecurityNumber',
+    'medicalRecordNumber',
+    'mrn',
+    'healthInsuranceNumber',
+    'diagnosisCode',
+    'icd10',
+    'prescription',
+    'medication',
+  ],
+})
+```
+
+### Financial Services
+
+```ts
+new SensitiveDataFilter({
+  sensitiveFields: [
+    // PCI compliance fields
+    'creditCard',
+    'ccNumber',
+    'cardNumber',
+    'cvv',
+    'cvc',
+    'securityCode',
+    'expirationDate',
+    'expiry',
+    'bankAccount',
+    'accountNumber',
+    'routingNumber',
+    'iban',
+    'swift',
+  ],
+})
+```
+
+## Error Handling
+
+If the filter encounters an error while processing a field, it replaces the field with a safe error marker:
+
+```json
+{
+  "problematicField": {
+    "error": {
+      "processor": "sensitive-data-filter"
+    }
+  }
+}
+```
+
+This ensures that processing errors don't prevent traces from being exported or cause application crashes.
+
+## Related
+
+- [SensitiveDataFilter API](https://mastra.ai/reference/observability/tracing/processors/sensitive-data-filter)

package/.docs/docs/rag/chunking-and-embedding.md

@@ -0,0 +1,183 @@
+# Chunking and Embedding Documents
+
+Before processing, create a MDocument instance from your content. You can initialize it from various formats:
+
+```ts
+const docFromText = MDocument.fromText('Your plain text content...')
+const docFromHTML = MDocument.fromHTML('<html>Your HTML content...</html>')
+const docFromMarkdown = MDocument.fromMarkdown('# Your Markdown content...')
+const docFromJSON = MDocument.fromJSON(`{ "key": "value" }`)
+```
+
+## Step 1: Document Processing
+
+Use `chunk` to split documents into manageable pieces. Mastra supports multiple chunking strategies optimized for different document types:
+
+- `recursive`: Smart splitting based on content structure
+- `character`: Simple character-based splits
+- `token`: Token-aware splitting
+- `markdown`: Markdown-aware splitting
+- `semantic-markdown`: Markdown splitting based on related header families
+- `html`: HTML structure-aware splitting
+- `json`: JSON structure-aware splitting
+- `latex`: LaTeX structure-aware splitting
+- `sentence`: Sentence-aware splitting
+
+> **Note:** Each strategy accepts different parameters optimized for its chunking approach.
+
+Here's an example of how to use the `recursive` strategy:
+
+```ts
+const chunks = await doc.chunk({
+  strategy: 'recursive',
+  maxSize: 512,
+  overlap: 50,
+  separators: ['\n'],
+  extract: {
+    metadata: true, // Optionally extract metadata
+  },
+})
+```
+
+For text where preserving sentence structure is important, here's an example of how to use the `sentence` strategy:
+
+```ts
+const chunks = await doc.chunk({
+  strategy: 'sentence',
+  maxSize: 450,
+  minSize: 50,
+  overlap: 0,
+  sentenceEnders: ['.'],
+})
+```
+
+For markdown documents where preserving the semantic relationships between sections is important, here's an example of how to use the `semantic-markdown` strategy:
+
+```ts
+const chunks = await doc.chunk({
+  strategy: 'semantic-markdown',
+  joinThreshold: 500,
+  modelName: 'gpt-3.5-turbo',
+})
+```
+
+> **Note:** Metadata extraction may use LLM calls, so ensure your API key is set.
+
+We go deeper into chunking strategies in our [`chunk()` reference documentation](https://mastra.ai/reference/rag/chunk).
+
+## Step 2: Embedding Generation
+
+Transform chunks into embeddings using your preferred provider. Mastra supports embedding models through the model router.
+
+### Using the Model Router
+
+The simplest way is to use Mastra's model router with `provider/model` strings:
+
+```ts
+import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
+import { embedMany } from 'ai'
+
+const { embeddings } = await embedMany({
+  model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
+  values: chunks.map(chunk => chunk.text),
+})
+```
+
+Mastra supports OpenAI and Google embedding models. For a complete list of supported embedding models, see the [embeddings reference](https://mastra.ai/reference/rag/embeddings).
+
+The model router automatically handles API key detection from environment variables.
+
+The embedding functions return vectors, arrays of numbers representing the semantic meaning of your text, ready for similarity searches in your vector database.
+
+### Configuring Embedding Dimensions
+
+Embedding models typically output vectors with a fixed number of dimensions (e.g., 1536 for OpenAI's `text-embedding-3-small`). Some models support reducing this dimensionality, which can help:
+
+- Decrease storage requirements in vector databases
+- Reduce computational costs for similarity searches
+
+Here are some supported models:
+
+OpenAI (text-embedding-3 models):
+
+```ts
+import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
+
+const { embeddings } = await embedMany({
+  model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
+  options: {
+    dimensions: 256, // Only supported in text-embedding-3 and later
+  },
+  values: chunks.map(chunk => chunk.text),
+})
+```
+
+Google (text-embedding-001):
+
+```ts
+const { embeddings } = await embedMany({
+  model: google('gemini-embedding-001', {
+    outputDimensionality: 256, // Truncates excessive values from the end
+  }),
+  values: chunks.map(chunk => chunk.text),
+})
+```
+
+> **Vector Database Compatibility:** When storing embeddings, the vector database index must be configured to match the output size of your embedding model. If the dimensions do not match, you may get errors or data corruption.
+
+## Example: Complete Pipeline
+
+Here's an example showing document processing and embedding generation with both providers:
+
+```ts
+import { embedMany } from 'ai'
+
+import { MDocument } from '@mastra/rag'
+
+// Initialize document
+const doc = MDocument.fromText(`
+  Climate change poses significant challenges to global agriculture.
+  Rising temperatures and changing precipitation patterns affect crop yields.
+`)
+
+// Create chunks
+const chunks = await doc.chunk({
+  strategy: 'recursive',
+  maxSize: 256,
+  overlap: 50,
+})
+
+// Generate embeddings with OpenAI
+import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
+
+const { embeddings } = await embedMany({
+  model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
+  values: chunks.map(chunk => chunk.text),
+})
+
+// OR
+
+// Generate embeddings with Cohere
+const { embeddings } = await embedMany({
+  model: 'cohere/embed-english-v3.0',
+  values: chunks.map(chunk => chunk.text),
+})
+
+// Store embeddings in your vector database
+await vectorStore.upsert({
+  indexName: 'embeddings',
+  vectors: embeddings,
+})
+```
+
+##
+
+For more examples of different chunking strategies and embedding configurations, see:
+
+- [Chunk Reference](https://mastra.ai/reference/rag/chunk)
+- [Embeddings Reference](https://mastra.ai/reference/rag/embeddings)
+
+For more details on vector databases and embeddings, see:
+
+- [Vector Databases](https://mastra.ai/docs/rag/vector-databases)
+- [Embedding API Reference](https://mastra.ai/reference/rag/embeddings)

package/.docs/docs/rag/graph-rag.md

@@ -0,0 +1,215 @@
+# GraphRAG
+
+Graph-based retrieval enhances traditional vector search by following relationships between chunks of information. This approach is useful when information is spread across multiple documents or when documents reference each other.
+
+## When to use GraphRAG
+
+GraphRAG is particularly effective when:
+
+- Information is spread across multiple documents
+- Documents reference each other
+- You need to traverse relationships to find complete answers
+- Understanding connections between concepts is important
+- Simple vector similarity misses important contextual relationships
+
+For straightforward semantic search without relationship traversal, use [standard retrieval methods](https://mastra.ai/docs/rag/retrieval).
+
+## How GraphRAG works
+
+GraphRAG combines vector similarity with knowledge graph traversal:
+
+1. Initial vector search retrieves relevant chunks based on semantic similarity
+2. A knowledge graph is constructed from the retrieved chunks
+3. The graph is traversed to find connected information
+4. Results include both directly relevant chunks and related content
+
+This process helps surface information that might not be semantically similar to the query but is contextually relevant through connections.
+
+## Creating a graph query tool
+
+The Graph Query Tool provides agents with the ability to perform graph-based retrieval:
+
+```ts
+import { createGraphRAGTool } from '@mastra/rag'
+import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
+
+const graphQueryTool = createGraphRAGTool({
+  vectorStoreName: 'pgVector',
+  indexName: 'embeddings',
+  model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
+  graphOptions: {
+    threshold: 0.7,
+  },
+})
+```
+
+### Configuration options
+
+The `graphOptions` parameter controls how the knowledge graph is built and traversed:
+
+- `threshold`: Similarity threshold (0-1) for determining which chunks are related. Higher values create sparser graphs with stronger connections; lower values create denser graphs with more potential relationships.
+- `dimension`: Vector embedding dimension. Must match the embedding model's output dimension (e.g., 1536 for OpenAI's text-embedding-3-small).
+
+```ts
+const graphQueryTool = createGraphRAGTool({
+  vectorStoreName: 'pgVector',
+  indexName: 'embeddings',
+  model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
+  graphOptions: {
+    dimension: 1536,
+    threshold: 0.7,
+  },
+})
+```
+
+## Using GraphRAG with agents
+
+Integrate the graph query tool with an agent to enable graph-based retrieval:
+
+```ts
+import { Agent } from '@mastra/core/agent'
+
+const ragAgent = new Agent({
+  id: 'rag-agent',
+  name: 'GraphRAG Agent',
+  instructions: `You are a helpful assistant that answers questions based on the provided context.
+When answering questions, use the graph query tool to find relevant information and relationships.
+Base your answers on the context provided by the tool, and clearly state if the context doesn't contain enough information.`,
+  model: 'openai/gpt-5.1',
+  tools: {
+    graphQueryTool,
+  },
+})
+```
+
+## Document processing and storage
+
+Before using graph-based retrieval, process documents into chunks and store their embeddings:
+
+```ts
+import { MDocument } from '@mastra/rag'
+import { embedMany } from 'ai'
+import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
+
+// Create and chunk document
+const doc = MDocument.fromText('Your document content here...')
+
+const chunks = await doc.chunk({
+  strategy: 'recursive',
+  size: 512,
+  overlap: 50,
+  separator: '\n',
+})
+
+// Generate embeddings
+const { embeddings } = await embedMany({
+  model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
+  values: chunks.map(chunk => chunk.text),
+})
+
+// Store in vector database
+const vectorStore = mastra.getVector('pgVector')
+await vectorStore.createIndex({
+  indexName: 'embeddings',
+  dimension: 1536,
+})
+await vectorStore.upsert({
+  indexName: 'embeddings',
+  vectors: embeddings,
+  metadata: chunks?.map(chunk => ({ text: chunk.text })),
+})
+```
+
+## Querying with GraphRAG
+
+Once configured, the agent can perform graph-based queries:
+
+```ts
+const query = 'What are the effects of infrastructure changes on local businesses?'
+const response = await ragAgent.generate(query)
+console.log(response.text)
+```
+
+The agent uses the graph query tool to:
+
+1. Convert the query to an embedding
+2. Find semantically similar chunks in the vector store
+3. Build a knowledge graph from related chunks
+4. Traverse the graph to find connected information
+5. Return comprehensive context for generating the response
+
+## Choosing the right threshold
+
+The threshold parameter significantly impacts retrieval quality:
+
+- **High threshold (0.8-0.9)**: Strict connections, fewer relationships, more precise but potentially incomplete results
+- **Medium threshold (0.6-0.8)**: Balanced approach, good for most use cases
+- **Low threshold (0.4-0.6)**: More connections, broader context, risk of including less relevant information
+
+Start with 0.7 and adjust based on your specific use case:
+
+```ts
+// Strict connections for precise answers
+const strictGraphTool = createGraphRAGTool({
+  vectorStoreName: 'pgVector',
+  indexName: 'embeddings',
+  model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
+  graphOptions: {
+    threshold: 0.85,
+  },
+})
+
+// Broader connections for exploratory queries
+const broadGraphTool = createGraphRAGTool({
+  vectorStoreName: 'pgVector',
+  indexName: 'embeddings',
+  model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
+  graphOptions: {
+    threshold: 0.5,
+  },
+})
+```
+
+## Combining with other retrieval methods
+
+GraphRAG can be used alongside other retrieval approaches:
+
+```ts
+import { createVectorQueryTool } from '@mastra/rag'
+
+const vectorQueryTool = createVectorQueryTool({
+  vectorStoreName: 'pgVector',
+  indexName: 'embeddings',
+  model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
+})
+
+const graphQueryTool = createGraphRAGTool({
+  vectorStoreName: 'pgVector',
+  indexName: 'embeddings',
+  model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
+  graphOptions: {
+    threshold: 0.7,
+  },
+})
+
+const agent = new Agent({
+  id: 'rag-agent',
+  name: 'RAG Agent',
+  instructions: `Use vector search for simple fact-finding queries.
+Use graph search when you need to understand relationships or find connected information.`,
+  model: 'openai/gpt-5.1',
+  tools: {
+    vectorQueryTool,
+    graphQueryTool,
+  },
+})
+```
+
+This gives the agent flexibility to choose the appropriate retrieval method based on the query.
+
+## Reference
+
+For detailed API documentation, see:
+
+- [GraphRAG Class](https://mastra.ai/reference/rag/graph-rag)
+- [createGraphRAGTool()](https://mastra.ai/reference/tools/graph-rag-tool)