@mastra/mcp-docs-server 0.0.3 → 0.0.4-alpha.1

package/.docs/raw/guides/04-research-assistant.mdx

@@ -0,0 +1,273 @@
+---
+title: "Building a Research Paper Assistant | Mastra RAG Guides"
+description: Guide on creating an AI research assistant that can analyze and answer questions about academic papers using RAG.
+---
+
+import { Steps } from "nextra/components";
+
+# Building a Research Paper Assistant with RAG
+
+In this guide, we'll create an AI research assistant that can analyze academic papers and answer specific questions about their content using Retrieval Augmented Generation (RAG).
+
+We'll use the foundational Transformer paper [Attention Is All You Need](https://arxiv.org/html/1706.03762) as our example.
+
+## Understanding RAG Components
+
+Let's understand how RAG works and how we'll implement each component:
+
+1. Knowledge Store/Index
+   - Converting text into vector representations
+   - Creating numerical representations of content
+   - Implementation: We'll use OpenAI's text-embedding-3-small to create embeddings and store them in PgVector
+
+2. Retriever
+   - Finding relevant content via similarity search
+   - Matching query embeddings with stored vectors
+   - Implementation: We'll use PgVector to perform similarity searches on our stored embeddings
+
+3. Generator
+   - Processing retrieved content with an LLM
+   - Creating contextually informed responses
+   - Implementation: We'll use GPT-4o-mini to generate answers based on retrieved content
+
+Our implementation will:
+1. Process the Transformer paper into embeddings
+2. Store them in PgVector for quick retrieval
+3. Use similarity search to find relevant sections
+4. Generate accurate responses using retrieved context
+
+## Project Structure
+
+```
+research-assistant/
+├── src/
+│   ├── agents/
+│   │   └── researchAgent.ts
+│   └── index.ts
+├── package.json
+└── .env
+```
+
+<Steps>
+### Initialize Project and Install Dependencies
+
+First, create a new directory for your project and navigate into it:
+
+```bash
+mkdir research-assistant
+cd research-assistant
+```
+
+Initialize a new Node.js project and install the required dependencies:
+
+```bash
+npm init -y
+npm install @mastra/core @mastra/rag @mastra/pg @ai-sdk/openai zod
+```
+
+Set up environment variables for API access and database connection:
+
+```bash filename=".env" copy
+OPENAI_API_KEY=your_openai_api_key
+POSTGRES_CONNECTION_STRING=your_connection_string
+```
+
+Create the necessary files for our project:
+
+```bash
+mkdir -p src/agents
+touch src/agents/researchAgent.ts src/index.ts
+```
+
+### Create the Research Assistant Agent
+
+Now we'll create our RAG-enabled research assistant. The agent uses:
+- A [Vector Query Tool](/docs/reference/tools/vector-query-tool) for performing semantic search over our vector store to find relevant content in our papers.
+- GPT-4o-mini for understanding queries and generating responses
+- Custom instructions that guide the agent on how to analyze papers, use retrieved content effectively, and acknowledge limitations
+
+```typescript copy showLineNumbers filename="src/agents/researchAgent.ts"
+import { Agent } from '@mastra/core/agent';
+import { openai } from '@ai-sdk/openai';
+import { createVectorQueryTool } from '@mastra/rag';
+
+// Create a tool for semantic search over our paper embeddings
+const vectorQueryTool = createVectorQueryTool({
+  vectorStoreName: 'pgVector',
+  indexName: 'papers',
+  model: openai.embedding('text-embedding-3-small'),
+});
+
+export const researchAgent = new Agent({
+  name: 'Research Assistant',
+  instructions: 
+    `You are a helpful research assistant that analyzes academic papers and technical documents.
+    Use the provided vector query tool to find relevant information from your knowledge base, 
+    and provide accurate, well-supported answers based on the retrieved content.
+    Focus on the specific content available in the tool and acknowledge if you cannot find sufficient information to answer a question.
+    Base your responses only on the content provided, not on general knowledge.`,
+  model: openai('gpt-4o-mini'),
+  tools: {
+    vectorQueryTool,
+  },
+});
+```
+
+### Set Up the Mastra Instance and Vector Store
+
+```typescript copy showLineNumbers filename="src/index.ts"
+import { MDocument } from '@mastra/rag';
+import { Mastra } from '@mastra/core';
+import { PgVector } from '@mastra/pg';
+import { embedMany } from 'ai';
+
+import { researchAgent } from './agents/researchAgent';
+
+const pgVector = new PgVector(process.env.POSTGRES_CONNECTION_STRING!);
+export const mastra = new Mastra({
+  agents: { researchAgent },
+  vectors: { pgVector },
+});
+```
+
+### Load and Process the Paper
+
+This step handles the initial document processing. We:
+1. Fetch the research paper from its URL
+2. Convert it into a document object
+3. Split it into smaller, manageable chunks for better processing
+
+```typescript copy showLineNumbers{14} filename="src/index.ts"
+// Load the paper
+const paperUrl = "https://arxiv.org/html/1706.03762";
+const response = await fetch(paperUrl);
+const paperText = await response.text();
+
+// Create document and chunk it
+const doc = MDocument.fromText(paperText);
+const chunks = await doc.chunk({
+  strategy: 'recursive',
+  size: 512,
+  overlap: 50,
+  separator: '\n',
+});
+```
+
+### Create and Store Embeddings
+
+Finally, we'll prepare our content for RAG by:
+1. Generating embeddings for each chunk of text
+2. Creating a vector store index to hold our embeddings
+3. Storing both the embeddings and metadata (original text and source information) in our vector database
+
+> **Note**: This metadata is crucial as it allows us to return the actual content when the vector store finds relevant matches.
+
+This allows our agent to efficiently search and retrieve relevant information.
+
+```typescript copy showLineNumbers{28} filename="src/index.ts"
+// Generate embeddings
+const { embeddings } = await embedMany({
+  model: openai.embedding('text-embedding-3-small'),
+  values: chunks.map(chunk => chunk.text),
+});
+
+// Get the vector store instance from Mastra
+const vectorStore = mastra.getVector('pgVector');
+
+// Create an index for our paper chunks
+await vectorStore.createIndex({
+  indexName: 'papers',
+  dimension: 1536,
+});
+
+// Store embeddings
+await vectorStore.upsert({
+  indexName: 'papers',
+  vectors: embeddings,
+  metadata: chunks.map(chunk => ({
+    text: chunk.text,
+    source: 'transformer-paper'
+  })),
+});
+```
+
+This will:
+1. Load the paper from the URL
+2. Split it into manageable chunks
+3. Generate embeddings for each chunk
+4. Store both the embeddings and text in our vector database
+
+### Test the Assistant
+
+Let's test our research assistant with different types of queries:
+
+```typescript filename="src/index.ts" showLineNumbers{52} copy
+const agent = mastra.getAgent('researchAgent');
+
+// Basic query about concepts
+const query1 = "What problems does sequence modeling face with neural networks?";
+const response1 = await agent.generate(query1);
+console.log("\nQuery:", query1);
+console.log("Response:", response1.text);
+```
+
+You should see output like:
+```
+Query: What problems does sequence modeling face with neural networks?
+Response: Sequence modeling with neural networks faces several key challenges:
+1. Vanishing and exploding gradients during training, especially with long sequences
+2. Difficulty handling long-term dependencies in the input
+3. Limited computational efficiency due to sequential processing
+4. Challenges in parallelizing computations, resulting in longer training times
+```
+
+Let's try another question:
+```typescript filename="src/index.ts" showLineNumbers{60} copy
+// Query about specific findings
+const query2 = "What improvements were achieved in translation quality?";
+const response2 = await agent.generate(query2);
+console.log("\nQuery:", query2);
+console.log("Response:", response2.text);
+```
+
+Output:
+```
+Query: What improvements were achieved in translation quality?
+Response: The model showed significant improvements in translation quality, achieving more than 2.0 
+BLEU points improvement over previously reported models on the WMT 2014 English-to-German translation 
+task, while also reducing training costs.
+```
+
+### Serve the Application
+
+Start the Mastra server to expose your research assistant via API:
+
+```bash
+mastra dev --dir src
+```
+
+Your research assistant will be available at:
+```
+http://localhost:4111/api/agents/researchAgent/generate
+```
+
+Test with curl:
+
+```bash
+curl -X POST http://localhost:4111/api/agents/researchAgent/generate \
+  -H "Content-Type: application/json" \
+  -d '{
+    "messages": [
+      { "role": "user", "content": "What were the main findings about model parallelization?" }
+    ]
+  }'
+```
+</Steps>
+
+## Advanced RAG Examples
+
+Explore these examples for more advanced RAG techniques:
+- [Filter RAG](/examples/rag/usage/filter-rag) for filtering results using metadata
+- [Cleanup RAG](/examples/rag/usage/cleanup-rag) for optimizing information density
+- [Chain of Thought RAG](/examples/rag/usage/cot-rag) for complex reasoning queries using workflows
+- [Rerank RAG](/examples/rag/usage/rerank-rag) for improved result relevance

package/.docs/raw/local-dev/mastra-dev.mdx

@@ -74,7 +74,7 @@ The Tools playground allows you to test your custom tools in isolation:
 
 ## REST API Endpoints
 
-`mastra dev` also spins up REST API routes for your agents and workflows via the local [Mastra Server](/docs/deployment/server). This allows you to test your API endpoints before deployment. See [Mastra Dev reference](http://localhost:3000/docs/reference/cli/dev#routes) for more details about all endpoints.
+`mastra dev` also spins up REST API routes for your agents and workflows via the local [Mastra Server](/docs/deployment/server). This allows you to test your API endpoints before deployment. See [Mastra Dev reference](/docs/reference/cli/dev#routes) for more details about all endpoints.
 
 You can then leverage the [Mastra Client](/docs/deployment/client) SDK to interact with your served REST API routes seamlessly.
 
@@ -105,4 +105,4 @@ This architecture allows you to start developing immediately without setting up
 
 `mastra dev` makes it easy to develop, debug, and iterate on your AI logic in a self-contained environment before deploying to production.
 
-- [Mastra Dev reference](../reference/cli/dev.mdx)
+- [Mastra Dev reference](../reference/cli/dev.mdx)

package/.docs/raw/observability/logging.mdx

@@ -0,0 +1,38 @@
+---
+title: "Logging | Mastra Observability Documentation"
+description: Documentation on effective logging in Mastra, crucial for understanding application behavior and improving AI accuracy.
+---
+
+import Image from "next/image";
+
+# Logging
+
+In Mastra, logs can detail when certain functions run, what input data they receive, and how they respond.
+
+## Basic Setup
+
+Here's a minimal example that sets up a **console logger** at the `INFO` level. This will print out informational messages and above (i.e., `DEBUG`, `INFO`, `WARN`, `ERROR`) to the console.
+
+```typescript filename="mastra.config.ts" showLineNumbers copy
+import { Mastra } from "@mastra/core";
+import { createLogger } from "@mastra/core/logger";
+
+export const mastra = new Mastra({
+  // Other Mastra configuration...
+  logger: createLogger({
+    name: "Mastra",
+    level: "info",
+  }),
+});
+```
+
+In this configuration:
+
+- `name: "Mastra"` specifies the name to group logs under.
+- `level: "info"` sets the minimum severity of logs to record.
+
+## Configuration
+
+- For more details on the options you can pass to `createLogger()`, see the [createLogger reference documentation](/docs/reference/observability/create-logger.mdx).
+- Once you have a `Logger` instance, you can call its methods (e.g., `.info()`, `.warn()`, `.error()`) in the [Logger instance reference documentation](/docs/reference/observability/logger.mdx).
+- If you want to send your logs to an external service for centralized collection, analysis, or storage, you can configure other logger types such as Upstash Redis. Consult the [createLogger reference documentation](/docs/reference/observability/create-logger.mdx) for details on parameters like `url`, `token`, and `key` when using the `UPSTASH` logger type.

package/.docs/raw/observability/nextjs-tracing.mdx

@@ -0,0 +1,102 @@
+---
+title: "Next.js Tracing | Mastra Observability Documentation"
+description: "Set up OpenTelemetry tracing for Next.js applications"
+---
+
+# Next.js Tracing
+
+If you're using Next.js, you have two options for setting up OpenTelemetry instrumentation:
+
+### Option 1: Using Vercel's OTEL Setup
+
+If you're deploying to Vercel, you can use their built-in OpenTelemetry setup:
+
+1. Install the required dependencies:
+
+```bash copy
+npm install @opentelemetry/api @vercel/otel
+```
+
+2. Create an instrumentation file at the root of your project (or in the src folder if using one):
+
+```ts filename="instrumentation.ts" copy
+import { registerOTel } from '@vercel/otel'
+
+export function register() {
+  registerOTel({ serviceName: 'your-project-name' })
+}
+```
+
+### Option 2: Using Custom Exporters
+
+If you're using other observability tools (like Langfuse), you can configure a custom exporter:
+
+1. Install the required dependencies (example using Langfuse):
+
+```bash copy
+npm install @opentelemetry/api langfuse-vercel
+```
+
+2. Create an instrumentation file:
+
+```ts filename="instrumentation.ts" copy
+import {
+  NodeSDK,
+  ATTR_SERVICE_NAME,
+  Resource,
+} from '@mastra/core/telemetry/otel-vendor';
+import { LangfuseExporter } from 'langfuse-vercel';
+
+export function register() {
+  const exporter = new LangfuseExporter({
+    // ... Langfuse config
+  })
+
+  const sdk = new NodeSDK({
+    resource: new Resource({
+      [ATTR_SERVICE_NAME]: 'ai',
+    }),
+    traceExporter: exporter,
+  });
+
+  sdk.start();
+}
+```
+
+### Next.js Configuration
+
+For either option, enable the instrumentation hook in your Next.js config:
+
+```ts filename="next.config.ts" showLineNumbers copy
+import type { NextConfig } from "next";
+
+const nextConfig: NextConfig = {
+  experimental: {
+    instrumentationHook: true // Not required in Next.js 15+
+  }
+};
+
+export default nextConfig;
+```
+
+### Mastra Configuration
+
+Configure your Mastra instance:
+
+```typescript filename="mastra.config.ts" copy
+import { Mastra } from "@mastra/core";
+
+export const mastra = new Mastra({
+  // ... other config
+  telemetry: {
+    serviceName: "your-project-name",
+    enabled: true
+  }
+});
+```
+
+This setup will enable OpenTelemetry tracing for your Next.js application and Mastra operations.
+
+For more details, see the documentation for:
+- [Next.js Instrumentation](https://nextjs.org/docs/app/building-your-application/optimizing/instrumentation)
+- [Vercel OpenTelemetry](https://vercel.com/docs/observability/otel-overview/quickstart)

package/.docs/raw/observability/tracing.mdx

@@ -0,0 +1,110 @@
+---
+title: "Tracing | Mastra Observability Documentation"
+description: "Set up OpenTelemetry tracing for Mastra applications"
+---
+
+import Image from "next/image";
+
+# Tracing
+
+Mastra supports the OpenTelemetry Protocol (OTLP) for tracing and monitoring your application. When telemetry is enabled, Mastra automatically traces all core primitives including agent operations, LLM interactions, tool executions, integration calls, workflow runs, and database operations. Your telemetry data can then be exported to any OTEL collector.
+
+### Basic Configuration
+
+Here's a simple example of enabling telemetry:
+
+```ts filename="mastra.config.ts" showLineNumbers copy
+export const mastra = new Mastra({
+  // ... other config
+  telemetry: {
+    serviceName: "my-app",
+    enabled: true,
+    sampling: {
+      type: "always_on",
+    },
+    export: {
+      type: "otlp",
+      endpoint: "http://localhost:4318", // SigNoz local endpoint
+    },
+  },
+});
+```
+
+### Configuration Options
+
+The telemetry config accepts these properties:
+
+```ts
+type OtelConfig = {
+  // Name to identify your service in traces (optional)
+  serviceName?: string;
+
+  // Enable/disable telemetry (defaults to true)
+  enabled?: boolean;
+
+  // Control how many traces are sampled
+  sampling?: {
+    type: "ratio" | "always_on" | "always_off" | "parent_based";
+    probability?: number; // For ratio sampling
+    root?: {
+      probability: number; // For parent_based sampling
+    };
+  };
+
+  // Where to send telemetry data
+  export?: {
+    type: "otlp" | "console";
+    endpoint?: string;
+    headers?: Record<string, string>;
+  };
+};
+```
+
+See the [OtelConfig reference documentation](/docs/reference/observability/otel-config.mdx) for more details.
+
+### Environment Variables
+
+You can configure the OTLP endpoint and headers through environment variables:
+
+```env filename=".env" copy
+OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
+OTEL_EXPORTER_OTLP_HEADERS=x-api-key=your-api-key
+```
+
+Then in your config:
+
+```ts filename="mastra.config.ts" showLineNumbers copy
+export const mastra = new Mastra({
+  // ... other config
+  telemetry: {
+    serviceName: "my-app",
+    enabled: true,
+    export: {
+      type: "otlp",
+      // endpoint and headers will be picked up from env vars
+    },
+  },
+});
+```
+
+### Example: SigNoz Integration
+
+Here's what a traced agent interaction looks like in [SigNoz](https://signoz.io):
+
+<img
+  src="/docs/signoz-telemetry-demo.png"
+  alt="Agent interaction trace showing spans, LLM calls, and tool executions"
+  style={{ maxWidth: "800px", width: "100%", margin: "8px 0" }}
+  className="nextra-image rounded-md"
+  data-zoom
+  width={800}
+  height={400}
+/>
+
+### Other Supported Providers
+
+For a complete list of supported observability providers and their configuration details, see the [Observability Providers reference](../reference/observability/providers/).
+
+### Framework-Specific Setup
+
+For Next.js applications, see our [Next.js Tracing](/docs/observability/nextjs-tracing) guide.

package/.docs/raw/rag/overview.mdx

@@ -35,9 +35,9 @@ const chunks = await doc.chunk({
   overlap: 50,
 });
 
-// 3. Generate embeddings
+// 3. Generate embeddings; we need to pass the text of each chunk
 const { embeddings } = await embedMany({
-  values: chunks,
+  values: chunks.map(chunk => chunk.text),
   model: openai.embedding("text-embedding-3-small"),
 });
 
@@ -81,5 +81,5 @@ See the [OTel Configuration](../reference/observability/otel-config.mdx) page fo
 
 ## More resources
 
-- [Chain of Thought RAG Example](../../examples/rag/cot-rag.mdx)
+- [Chain of Thought RAG Example](../../examples/rag/usage/cot-rag.mdx)
 - [All RAG Examples](../../examples/) (including different chunking strategies, embedding models, and vector stores)

package/.docs/raw/rag/retrieval.mdx

@@ -42,6 +42,9 @@ const results = await pgVector.query({
   queryVector: embedding,
   topK: 10,
 });
+
+// Display results
+console.log(results);
 ```
 
 Results include both the text content and a similarity score:
@@ -62,7 +65,7 @@ Results include both the text content and a similarity score:
 ]
 ```
 
-For an example of how to use the basic retrieval method, see the [Retrieve Results](../../examples/rag/retrieve-results.mdx) example.
+For an example of how to use the basic retrieval method, see the [Retrieve Results](../../examples/rag/query/retrieve-results.mdx) example.
 
 ## Advanced Retrieval options
 
@@ -143,7 +146,7 @@ Common use cases for metadata filtering:
 - Combine multiple conditions for precise querying
 - Filter by document attributes (e.g., language, author)
 
-For an example of how to use metadata filtering, see the [Hybrid Vector Search](../../examples/rag/hybrid-vector-search.mdx) example.
+For an example of how to use metadata filtering, see the [Hybrid Vector Search](../../examples/rag/query/hybrid-vector-search.mdx) example.
 
 ### Vector Query Tool
 
@@ -334,7 +337,7 @@ The re-ranked results combine vector similarity with semantic understanding to i
 
 For more details about re-ranking, see the [rerank()](/docs/reference/rag/rerank) method.
 
-For an example of how to use the re-ranking method, see the [Re-ranking Results](../../examples/rag/rerank.mdx) example.
+For an example of how to use the re-ranking method, see the [Re-ranking Results](../../examples/rag/rerank/rerank.mdx) example.
 
 ### Graph-based Retrieval
 
@@ -359,4 +362,4 @@ const graphQueryTool = createGraphQueryTool({
 
 For more details about graph-based retrieval, see the [GraphRAG](/docs/reference/rag/graph-rag) class and the [createGraphQueryTool()](/docs/reference/tools/graph-rag-tool) function.
 
-For an example of how to use the graph-based retrieval method, see the [Graph-based Retrieval](../../examples/rag/graph-rag.mdx) example.
+For an example of how to use the graph-based retrieval method, see the [Graph-based Retrieval](../../examples/rag/usage/graph-rag.mdx) example.