@mastra/mcp-docs-server 0.13.7-alpha.1 → 0.13.7-alpha.2

package/.docs/raw/scorers/off-the-shelf-scorers.mdx

@@ -0,0 +1,30 @@
+---
+title: "Built-in Scorers"
+description: "Overview of Mastra's ready-to-use scorers for evaluating AI outputs across quality, safety, and performance dimensions."
+---
+
+# Built-in Scorers
+
+Mastra provides a comprehensive set of built-in scorers for evaluating AI outputs. These scorers are optimized for common evaluation scenarios and are ready to use in your agents and workflows.
+
+## Available Scorers
+
+### Accuracy and Reliability
+
+These scorers evaluate how correct, truthful, and complete your agent's answers are:
+
+- [`answer-relevancy`](/reference/scorers/answer-relevancy): Evaluates how well responses address the input query (`0-1`, higher is better)
+- [`faithfulness`](/reference/scorers/faithfulness): Measures how accurately responses represent provided context (`0-1`, higher is better)
+- [`hallucination`](/reference/scorers/hallucination): Detects factual contradictions and unsupported claims (`0-1`, lower is better)
+- [`completeness`](/reference/scorers/completeness): Checks if responses include all necessary information (`0-1`, higher is better)
+- [`content-similarity`](/reference/scorers/content-similarity): Measures textual similarity using character-level matching (`0-1`, higher is better)
+- [`textual-difference`](/reference/scorers/textual-difference): Measures textual differences between strings (`0-1`, higher means more similar)
+
+### Output Quality
+
+These scorers evaluate adherence to format, style, and safety requirements:
+
+- [`tone-consistency`](/reference/scorers/tone-consistency): Measures consistency in formality, complexity, and style (`0-1`, higher is better)
+- [`toxicity`](/reference/scorers/toxicity): Detects harmful or inappropriate content (`0-1`, lower is better)
+- [`bias`](/reference/scorers/bias): Detects potential biases in the output (`0-1`, lower is better)
+- [`keyword-coverage`](/reference/scorers/keyword-coverage): Assesses technical terminology usage (`0-1`, higher is better)

package/.docs/raw/scorers/overview.mdx

@@ -0,0 +1,124 @@
+---
+title: "Overview"
+description: Overview of scorers in Mastra, detailing their capabilities for evaluating AI outputs and measuring performance.
+---
+
+# Scorers overview
+
+**Scorers** are evaluation tools that measure the quality, accuracy, or performance of AI-generated outputs. Scorers provide an automated way to assess whether your agents, workflows, or language models are producing the desired results by analyzing their responses against specific criteria.
+
+**Scores** are numerical values (typically between 0 and 1) that quantify how well an output meets your evaluation criteria. These scores enable you to objectively track performance, compare different approaches, and identify areas for improvement in your AI systems.
+
+## Evaluation pipeline
+
+Mastra scorers follow an optional three-step pipeline that allows for evaluation workflows:
+
+1. **Extract** (Optional): Identify and isolate relevant content for focused evaluation
+2. **Analyze** (Required): Perform the core evaluation and generate a score
+3. **Reason** (Optional): Provide explanations or justifications for the score
+
+This modular structure enables both simple single-step evaluations and complex multi-stage analysis workflows, allowing you to build evaluations that match your specific needs.
+
+### When to use each step
+
+**Extract step** - Use when your content is complex or needs preprocessing:
+- Separating facts from opinions in mixed responses
+- Focusing evaluation on specific sections of long outputs
+- Parsing multiple claims that need individual evaluation
+- Example: Bias detection that first identifies opinion statements
+
+**Analyze step** - Always required for core evaluation:
+- Straightforward scenarios: Direct scoring of input/output pairs
+- Complex scenarios: Evaluate preprocessed content and generate detailed results
+- Applies your scoring criteria and calculates the numerical score
+
+**Reason step** - Use when explanations are important:
+- Users need to understand why a score was assigned
+- Debugging and transparency are critical
+- Compliance or auditing requires explanations
+- Providing actionable feedback for improvement
+
+To learn how to create your own Scorers, see [Creating Custom Scorers](/docs/scorers/custom-scorers).
+
+## Live evaluations
+
+**Live evaluations** allow you to automatically score AI outputs in real-time as your agents and workflows operate. Instead of running evaluations manually or in batches, scorers run asynchronously alongside your AI systems, providing continuous quality monitoring.
+
+### Adding scorers to agents
+
+You can add built-in scorers to your agents to automatically evaluate their outputs. See the [full list of built-in scorers](/docs/scorers/off-the-shelf-scorers) for all available options.
+
+```typescript filename="src/mastra/agents/evaluated-agent.ts" showLineNumbers copy
+import { Agent } from "@mastra/core/agent";
+import { openai } from "@ai-sdk/openai";
+import { 
+  createAnswerRelevancyScorer,
+  createToxicityScorer 
+} from "@mastra/evals/scorers/llm";
+
+export const evaluatedAgent = new Agent({
+  // ...
+  scorers: {
+    relevancy: {
+      scorer: createAnswerRelevancyScorer({ model: openai("gpt-4o-mini") }),
+      sampling: { type: "ratio", rate: 0.5 }
+    },
+    safety: {
+      scorer: createToxicityScorer({ model: openai("gpt-4o-mini") }),
+      sampling: { type: "ratio", rate: 1 }
+    }
+  }
+});
+```
+
+### Adding scorers to workflow steps
+
+You can also add scorers to individual workflow steps to evaluate outputs at specific points in your process:
+
+```typescript filename="src/mastra/workflows/content-generation.ts" showLineNumbers copy
+import { createWorkflow, createStep } from "@mastra/core/workflows";
+import { z } from "zod";
+import { customStepScorer } from "../scorers/custom-step-scorer";
+
+const contentStep = createStep({
+  // ...
+  scorers: {
+    customStepScorer: {
+      scorer: customStepScorer(),
+      sampling: {
+        type: "ratio",
+        rate: 1, // Score every step execution
+      }
+    }
+  },
+});
+
+export const contentWorkflow = createWorkflow({ ... })
+  .then(contentStep)
+  .commit();
+```
+
+### How live evaluations work
+
+**Asynchronous execution**: Live evaluations run in the background without blocking your agent responses or workflow execution. This ensures your AI systems maintain their performance while still being monitored.
+
+**Sampling control**: The `sampling.rate` parameter (0-1) controls what percentage of outputs get scored:
+- `1.0`: Score every single response (100%)
+- `0.5`: Score half of all responses (50%) 
+- `0.1`: Score 10% of responses
+- `0.0`: Disable scoring
+
+**Automatic storage**: All scoring results are automatically stored in the `mastra_scorers` table in your configured database, allowing you to analyze performance trends over time.
+
+## Testing scorers locally
+
+Mastra provides a CLI command `mastra dev` to test your scorers. The playground includes a scorers section where you can run individual scorers against test inputs and view detailed results.
+
+For more details, see the [Local Dev Playground](/docs/server-db/local-dev-playground) docs.
+
+## Next steps
+
+- Learn how to create your own scorers in the [Creating Custom Scorers](/docs/scorers/custom-scorers) guide
+- Explore built-in scorers in the [Off-the-shelf Scorers](/docs/scorers/off-the-shelf-scorers) section
+- Test scorers with the [Local Dev Playground](/docs/server-db/local-dev-playground)
+- See example scorers in the [Examples Overview](/examples) section

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@mastra/mcp-docs-server",
-  "version": "0.13.7-alpha.1",
+  "version": "0.13.7-alpha.2",
   "description": "MCP server for accessing Mastra.ai documentation, changelogs, and news.",
   "type": "module",
   "main": "dist/index.js",
@@ -32,8 +32,8 @@
     "uuid": "^11.1.0",
     "zod": "^3.25.67",
     "zod-to-json-schema": "^3.24.5",
-    "@mastra/mcp": "^0.10.7",
-    "@mastra/core": "0.12.0-alpha.1"
+    "@mastra/core": "0.12.0-alpha.2",
+    "@mastra/mcp": "^0.10.7"
   },
   "devDependencies": {
     "@hono/node-server": "^1.17.1",
@@ -43,13 +43,13 @@
     "@wong2/mcp-cli": "^1.10.0",
     "cross-env": "^7.0.3",
     "eslint": "^9.30.1",
-    "hono": "^4.8.4",
+    "hono": "^4.8.9",
     "tsup": "^8.5.0",
     "tsx": "^4.19.4",
     "typescript": "^5.8.3",
     "vitest": "^3.2.4",
-    "@mastra/core": "0.12.0-alpha.1",
-    "@internal/lint": "0.0.23"
+    "@internal/lint": "0.0.23",
+    "@mastra/core": "0.12.0-alpha.2"
   },
   "scripts": {
     "prepare-docs": "cross-env PREPARE=true node dist/prepare-docs/prepare.js",