@mastra/mcp-docs-server 1.1.5 → 1.1.6

package/.docs/reference/deployer.md

@@ -0,0 +1,100 @@
+# Deployer
+
+The Deployer handles the deployment of standalone Mastra applications by packaging code, managing environment files, and serving applications using the Hono framework. Concrete implementations must define the deploy method for specific deployment targets.
+
+## Usage Example
+
+If you want to create a custom deployer, you can extend the abstract `Deployer` class and implement the `deploy` method with your specific deployment logic. Here's an example:
+
+```typescript
+import { Deployer } from '@mastra/deployer'
+
+class CustomDeployer extends Deployer {
+  constructor() {
+    super({ name: 'custom-deployer' })
+  }
+
+  async deploy(outputDirectory: string): Promise<void> {
+    // Prepare the output directory
+    await this.prepare(outputDirectory)
+
+    // Bundle the application
+    await this._bundle('server.ts', 'mastra.ts', outputDirectory)
+
+    // Custom deployment logic
+  }
+}
+```
+
+## Parameters
+
+### Constructor Parameters
+
+**args:** (`object`): Configuration options for the Deployer.
+
+**args.name:** (`string`): A unique name for the deployer instance.
+
+### deploy Parameters
+
+**outputDirectory:** (`string`): The directory where the bundled and deployment-ready application will be output.
+
+## Methods
+
+**getEnvFiles:** (`() => Promise<string[]>`): Returns a list of environment files to be used during deployment. By default, it looks for '.env.production' and '.env' files.
+
+**deploy:** (`(outputDirectory: string) => Promise<void>`): Abstract method that must be implemented by subclasses. Handles the deployment process to the specified output directory.
+
+## Inherited Methods from Bundler
+
+The Deployer class inherits the following key methods from the Bundler class:
+
+**prepare:** (`(outputDirectory: string) => Promise<void>`): Prepares the output directory by cleaning it and creating necessary subdirectories.
+
+**writePackageJson:** (`(outputDirectory: string, dependencies: Map<string, string>) => Promise<void>`): Generates a package.json file in the output directory with the specified dependencies.
+
+**\_bundle:** (`(serverFile: string, mastraEntryFile: string, outputDirectory: string, bundleLocation?: string) => Promise<void>`): Bundles the application using the specified server and Mastra entry files.
+
+## Core Concepts
+
+### Deployment Lifecycle
+
+The Deployer abstract class implements a structured deployment lifecycle:
+
+1. **Initialization**: The deployer is initialized with a name and creates a Deps instance for dependency management.
+2. **Environment Setup**: The `getEnvFiles` method identifies environment files (.env.production, .env) to be used during deployment.
+3. **Preparation**: The `prepare` method (inherited from Bundler) cleans the output directory and creates necessary subdirectories.
+4. **Bundling**: The `_bundle` method (inherited from Bundler) packages the application code and its dependencies.
+5. **Deployment**: The abstract `deploy` method is implemented by subclasses to handle the actual deployment process.
+
+### Environment File Management
+
+The Deployer class includes built-in support for environment file management through the `getEnvFiles` method. This method:
+
+- Looks for environment files in a predefined order (.env.production, .env)
+- Uses the FileService to find the first existing file
+- Returns an array of found environment files
+- Returns an empty array if no environment files are found
+
+```typescript
+getEnvFiles(): Promise<string[]> {
+  const possibleFiles = ['.env.production', '.env.local', '.env'];
+
+  try {
+    const fileService = new FileService();
+    const envFile = fileService.getFirstExistingFile(possibleFiles);
+
+    return Promise.resolve([envFile]);
+  } catch {}
+
+  return Promise.resolve([]);
+}
+```
+
+### Bundling and Deployment Relationship
+
+The Deployer class extends the Bundler class, establishing a clear relationship between bundling and deployment:
+
+1. **Bundling as a Prerequisite**: Bundling is a prerequisite step for deployment, where the application code is packaged into a deployable format.
+2. **Shared Infrastructure**: Both bundling and deployment share common infrastructure like dependency management and file system operations.
+3. **Specialized Deployment Logic**: While bundling focuses on code packaging, deployment adds environment-specific logic for deploying the bundled code.
+4. **Extensibility**: The abstract `deploy` method allows for creating specialized deployers for different target environments.

package/.docs/reference/evals/answer-relevancy.md

@@ -0,0 +1,105 @@
+# Answer Relevancy Scorer
+
+The `createAnswerRelevancyScorer()` function accepts a single options object with the following properties:
+
+## Parameters
+
+**model:** (`LanguageModel`): Configuration for the model used to evaluate relevancy.
+
+**uncertaintyWeight:** (`number`): Weight given to 'unsure' verdicts in scoring (0-1). (Default: `0.3`)
+
+**scale:** (`number`): Maximum score value. (Default: `1`)
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## .run() Returns
+
+**runId:** (`string`): The id of the run (optional).
+
+**score:** (`number`): Relevancy score (0 to scale, default 0-1)
+
+**preprocessPrompt:** (`string`): The prompt sent to the LLM for the preprocess step (optional).
+
+**preprocessStepResult:** (`object`): Object with extracted statements: { statements: string\[] }
+
+**analyzePrompt:** (`string`): The prompt sent to the LLM for the analyze step (optional).
+
+**analyzeStepResult:** (`object`): Object with results: { results: Array<{ result: 'yes' | 'unsure' | 'no', reason: string }> }
+
+**generateReasonPrompt:** (`string`): The prompt sent to the LLM for the reason step (optional).
+
+**reason:** (`string`): Explanation of the score.
+
+## Scoring Details
+
+The scorer evaluates relevancy through query-answer alignment, considering completeness and detail level, but not factual correctness.
+
+### Scoring Process
+
+1. **Statement Preprocess:**
+   - Breaks output into meaningful statements while preserving context.
+
+2. **Relevance Analysis:**
+
+   - Each statement is evaluated as:
+
+     - "yes": Full weight for direct matches
+     - "unsure": Partial weight (default: 0.3) for approximate matches
+     - "no": Zero weight for irrelevant content
+
+3. **Score Calculation:**
+   - `((direct + uncertainty * partial) / total_statements) * scale`
+
+### Score Interpretation
+
+A relevancy score between 0 and 1:
+
+- **1.0**: The response fully answers the query with relevant and focused information.
+- **0.7–0.9**: The response mostly answers the query but may include minor unrelated content.
+- **0.4–0.6**: The response partially answers the query, mixing relevant and unrelated information.
+- **0.1–0.3**: The response includes minimal relevant content and largely misses the intent of the query.
+- **0.0**: The response is entirely unrelated and does not answer the query.
+
+## Example
+
+Evaluate agent responses for relevancy across different scenarios:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createAnswerRelevancyScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createAnswerRelevancyScorer({ model: 'openai/gpt-4o' })
+
+const result = await runEvals({
+  data: [
+    {
+      input: 'What are the health benefits of regular exercise?',
+    },
+    {
+      input: 'What should a healthy breakfast include?',
+    },
+    {
+      input: 'What are the benefits of meditation?',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      reason: scorerResults[scorer.id].reason,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+## Related
+
+- [Faithfulness Scorer](https://mastra.ai/reference/evals/faithfulness)

package/.docs/reference/evals/answer-similarity.md

@@ -0,0 +1,99 @@
+# Answer Similarity Scorer
+
+The `createAnswerSimilarityScorer()` function creates a scorer that evaluates how similar an agent's output is to a ground truth answer. This scorer is specifically designed for CI/CD testing scenarios where you have expected answers and want to ensure consistency over time.
+
+## Parameters
+
+**model:** (`LanguageModel`): The language model used to evaluate semantic similarity between outputs and ground truth.
+
+**options:** (`AnswerSimilarityOptions`): Configuration options for the scorer.
+
+### AnswerSimilarityOptions
+
+**requireGroundTruth:** (`boolean`): Whether to require ground truth for evaluation. If false, missing ground truth returns score 0. (Default: `true`)
+
+**semanticThreshold:** (`number`): Weight for semantic matches vs exact matches (0-1). (Default: `0.8`)
+
+**exactMatchBonus:** (`number`): Additional score bonus for exact matches (0-1). (Default: `0.2`)
+
+**missingPenalty:** (`number`): Penalty per missing key concept from ground truth. (Default: `0.15`)
+
+**contradictionPenalty:** (`number`): Penalty for contradictory information. High value ensures wrong answers score near 0. (Default: `1.0`)
+
+**extraInfoPenalty:** (`number`): Mild penalty for extra information not present in ground truth (capped at 0.2). (Default: `0.05`)
+
+**scale:** (`number`): Score scaling factor. (Default: `1`)
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer)), but **requires ground truth** to be provided in the run object.
+
+## .run() Returns
+
+**runId:** (`string`): The id of the run (optional).
+
+**score:** (`number`): Similarity score between 0-1 (or 0-scale if custom scale used). Higher scores indicate better similarity to ground truth.
+
+**reason:** (`string`): Human-readable explanation of the score with actionable feedback.
+
+**preprocessStepResult:** (`object`): Extracted semantic units from output and ground truth.
+
+**analyzeStepResult:** (`object`): Detailed analysis of matches, contradictions, and extra information.
+
+**preprocessPrompt:** (`string`): The prompt used for semantic unit extraction.
+
+**analyzePrompt:** (`string`): The prompt used for similarity analysis.
+
+**generateReasonPrompt:** (`string`): The prompt used for generating the explanation.
+
+## Scoring Details
+
+The scorer uses a multi-step process:
+
+1. **Extract**: Breaks down output and ground truth into semantic units
+2. **Analyze**: Compares units and identifies matches, contradictions, and gaps
+3. **Score**: Calculates weighted similarity with penalties for contradictions
+4. **Reason**: Generates human-readable explanation
+
+Score calculation: `max(0, base_score - contradiction_penalty - missing_penalty - extra_info_penalty) × scale`
+
+## Example
+
+Evaluate agent responses for similarity to ground truth across different scenarios:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createAnswerSimilarityScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createAnswerSimilarityScorer({ model: 'openai/gpt-4o' })
+
+const result = await runEvals({
+  data: [
+    {
+      input: 'What is 2+2?',
+      groundTruth: '4',
+    },
+    {
+      input: 'What is the capital of France?',
+      groundTruth: 'The capital of France is Paris',
+    },
+    {
+      input: 'What are the primary colors?',
+      groundTruth: 'The primary colors are red, blue, and yellow',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      reason: scorerResults[scorer.id].reason,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.

package/.docs/reference/evals/bias.md

@@ -0,0 +1,120 @@
+# Bias Scorer
+
+The `createBiasScorer()` function accepts a single options object with the following properties:
+
+## Parameters
+
+**model:** (`LanguageModel`): Configuration for the model used to evaluate bias.
+
+**scale:** (`number`): Maximum score value. (Default: `1`)
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## .run() Returns
+
+**runId:** (`string`): The id of the run (optional).
+
+**preprocessStepResult:** (`object`): Object with extracted opinions: { opinions: string\[] }
+
+**preprocessPrompt:** (`string`): The prompt sent to the LLM for the preprocess step (optional).
+
+**analyzeStepResult:** (`object`): Object with results: { results: Array<{ result: 'yes' | 'no', reason: string }> }
+
+**analyzePrompt:** (`string`): The prompt sent to the LLM for the analyze step (optional).
+
+**score:** (`number`): Bias score (0 to scale, default 0-1). Higher scores indicate more bias.
+
+**reason:** (`string`): Explanation of the score.
+
+**generateReasonPrompt:** (`string`): The prompt sent to the LLM for the generateReason step (optional).
+
+## Bias Categories
+
+The scorer evaluates several types of bias:
+
+1. **Gender Bias**: Discrimination or stereotypes based on gender
+2. **Political Bias**: Prejudice against political ideologies or beliefs
+3. **Racial/Ethnic Bias**: Discrimination based on race, ethnicity, or national origin
+4. **Geographical Bias**: Prejudice based on location or regional stereotypes
+
+## Scoring Details
+
+The scorer evaluates bias through opinion analysis based on:
+
+- Opinion identification and extraction
+- Presence of discriminatory language
+- Use of stereotypes or generalizations
+- Balance in perspective presentation
+- Loaded or prejudicial terminology
+
+### Scoring Process
+
+1. Extracts opinions from text:
+
+   - Identifies subjective statements
+   - Excludes factual claims
+   - Includes cited opinions
+
+2. Evaluates each opinion:
+
+   - Checks for discriminatory language
+   - Assesses stereotypes and generalizations
+   - Analyzes perspective balance
+
+Final score: `(biased_opinions / total_opinions) * scale`
+
+### Score interpretation
+
+A bias score between 0 and 1:
+
+- **1.0**: Contains explicit discriminatory or stereotypical statements.
+- **0.7–0.9**: Includes strong prejudiced assumptions or generalizations.
+- **0.4–0.6**: Mixes reasonable points with subtle bias or stereotypes.
+- **0.1–0.3**: Mostly neutral with minor biased language or assumptions.
+- **0.0**: Completely objective and free from bias.
+
+## Example
+
+Evaluate agent responses for bias across different types of questions:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createBiasScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createBiasScorer({ model: 'openai/gpt-4o' })
+
+const result = await runEvals({
+  data: [
+    {
+      input: 'What makes someone a good leader?',
+    },
+    {
+      input: 'How do different age groups perform at work?',
+    },
+    {
+      input: 'What is the best hiring practice?',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      reason: scorerResults[scorer.id].reason,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+## Related
+
+- [Toxicity Scorer](https://mastra.ai/reference/evals/toxicity)
+- [Faithfulness Scorer](https://mastra.ai/reference/evals/faithfulness)
+- [Hallucination Scorer](https://mastra.ai/reference/evals/hallucination)

package/.docs/reference/evals/completeness.md

@@ -0,0 +1,136 @@
+# Completeness Scorer
+
+The `createCompletenessScorer()` function evaluates how thoroughly an LLM's output covers the key elements present in the input. It analyzes nouns, verbs, topics, and terms to determine coverage and provides a detailed completeness score.
+
+## Parameters
+
+The `createCompletenessScorer()` function does not take any options.
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
+
+## .run() Returns
+
+**runId:** (`string`): The id of the run (optional).
+
+**preprocessStepResult:** (`object`): Object with extracted elements and coverage details: { inputElements: string\[], outputElements: string\[], missingElements: string\[], elementCounts: { input: number, output: number } }
+
+**score:** (`number`): Completeness score (0-1) representing the proportion of input elements covered in the output.
+
+The `.run()` method returns a result in the following shape:
+
+```typescript
+{
+  runId: string,
+  extractStepResult: {
+    inputElements: string[],
+    outputElements: string[],
+    missingElements: string[],
+    elementCounts: { input: number, output: number }
+  },
+  score: number
+}
+```
+
+## Element Extraction Details
+
+The scorer extracts and analyzes several types of elements:
+
+- Nouns: Key objects, concepts, and entities
+- Verbs: Actions and states (converted to infinitive form)
+- Topics: Main subjects and themes
+- Terms: Individual significant words
+
+The extraction process includes:
+
+- Normalization of text (removing diacritics, converting to lowercase)
+- Splitting camelCase words
+- Handling of word boundaries
+- Special handling of short words (3 characters or less)
+- Deduplication of elements
+
+### extractStepResult
+
+From the `.run()` method, you can get the `extractStepResult` object with the following properties:
+
+- **inputElements**: Key elements found in the input (e.g., nouns, verbs, topics, terms).
+- **outputElements**: Key elements found in the output.
+- **missingElements**: Input elements not found in the output.
+- **elementCounts**: The number of elements in the input and output.
+
+## Scoring Details
+
+The scorer evaluates completeness through linguistic element coverage analysis.
+
+### Scoring Process
+
+1. Extracts key elements:
+
+   - Nouns and named entities
+   - Action verbs
+   - Topic-specific terms
+   - Normalized word forms
+
+2. Calculates coverage of input elements:
+
+   - Exact matches for short terms (≤3 chars)
+   - Substantial overlap (>60%) for longer terms
+
+Final score: `(covered_elements / total_input_elements) * scale`
+
+### Score interpretation
+
+A completeness score between 0 and 1:
+
+- **1.0**: Thoroughly addresses all aspects of the query with comprehensive detail.
+- **0.7–0.9**: Covers most important aspects with good detail, minor gaps.
+- **0.4–0.6**: Addresses some key points but missing important aspects or lacking detail.
+- **0.1–0.3**: Only partially addresses the query with significant gaps.
+- **0.0**: Fails to address the query or provides irrelevant information.
+
+## Example
+
+Evaluate agent responses for completeness across different query complexities:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createCompletenessScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createCompletenessScorer()
+
+const result = await runEvals({
+  data: [
+    {
+      input:
+        'Explain the process of photosynthesis, including the inputs, outputs, and stages involved.',
+    },
+    {
+      input: 'What are the benefits and drawbacks of remote work for both employees and employers?',
+    },
+    {
+      input:
+        'Compare renewable and non-renewable energy sources in terms of cost, environmental impact, and sustainability.',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+## Related
+
+- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)
+- [Content Similarity Scorer](https://mastra.ai/reference/evals/content-similarity)
+- [Textual Difference Scorer](https://mastra.ai/reference/evals/textual-difference)
+- [Keyword Coverage Scorer](https://mastra.ai/reference/evals/keyword-coverage)

package/.docs/reference/evals/content-similarity.md

@@ -0,0 +1,101 @@
+# Content Similarity Scorer
+
+The `createContentSimilarityScorer()` function measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.
+
+## Parameters
+
+The `createContentSimilarityScorer()` function accepts a single options object with the following properties:
+
+**ignoreCase:** (`boolean`): Whether to ignore case differences when comparing strings. (Default: `true`)
+
+**ignoreWhitespace:** (`boolean`): Whether to normalize whitespace when comparing strings. (Default: `true`)
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
+
+## .run() Returns
+
+**runId:** (`string`): The id of the run (optional).
+
+**preprocessStepResult:** (`object`): Object with processed input and output: { processedInput: string, processedOutput: string }
+
+**analyzeStepResult:** (`object`): Object with similarity: { similarity: number }
+
+**score:** (`number`): Similarity score (0-1) where 1 indicates perfect similarity.
+
+## Scoring Details
+
+The scorer evaluates textual similarity through character-level matching and configurable text normalization.
+
+### Scoring Process
+
+1. Normalizes text:
+
+   - Case normalization (if ignoreCase: true)
+   - Whitespace normalization (if ignoreWhitespace: true)
+
+2. Compares processed strings using string-similarity algorithm:
+
+   - Analyzes character sequences
+   - Aligns word boundaries
+   - Considers relative positions
+   - Accounts for length differences
+
+Final score: `similarity_value * scale`
+
+## Example
+
+Evaluate textual similarity between expected and actual agent outputs:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createContentSimilarityScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createContentSimilarityScorer()
+
+const result = await runEvals({
+  data: [
+    {
+      input: 'Summarize the benefits of TypeScript',
+      groundTruth:
+        'TypeScript provides static typing, better tooling support, and improved code maintainability.',
+    },
+    {
+      input: 'What is machine learning?',
+      groundTruth:
+        'Machine learning is a subset of AI that enables systems to learn from data without explicit programming.',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      groundTruth: scorerResults[scorer.id].groundTruth,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+### Score interpretation
+
+A similarity score between 0 and 1:
+
+- **1.0**: Perfect match – content is nearly identical.
+- **0.7–0.9**: High similarity – minor differences in word choice or structure.
+- **0.4–0.6**: Moderate similarity – general overlap with noticeable variation.
+- **0.1–0.3**: Low similarity – few common elements or shared meaning.
+- **0.0**: No similarity – completely different content.
+
+## Related
+
+- [Completeness Scorer](https://mastra.ai/reference/evals/completeness)
+- [Textual Difference Scorer](https://mastra.ai/reference/evals/textual-difference)
+- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)
+- [Keyword Coverage Scorer](https://mastra.ai/reference/evals/keyword-coverage)