@mastra/mcp-docs-server 0.13.44 → 0.13.45-alpha.0

package/.docs/raw/reference/scorers/keyword-coverage.mdx

@@ -102,124 +102,45 @@ The scorer handles several special cases:
 - Case differences: "JavaScript" matches "javascript"
 - Common words: Ignored in scoring to focus on meaningful keywords
 
-## Examples
+## Example
 
-### Full coverage example
+Evaluate keyword coverage between input queries and agent responses:
 
-In this example, the response fully reflects the key terms from the input. All required keywords are present, resulting in complete coverage with no omissions.
-
-```typescript title="src/example-full-keyword-coverage.ts" showLineNumbers copy
-import { createKeywordCoverageScorer } from "@mastra/evals/scorers/code";
-
-const scorer = createKeywordCoverageScorer();
-
-const input = "JavaScript frameworks like React and Vue";
-const output =
-  "Popular JavaScript frameworks include React and Vue for web development";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
-});
-
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### Full coverage output
-
-A score of 1 indicates that all expected keywords were found in the response. The `analyzeStepResult` field confirms that the number of matched keywords equals the total number extracted from the input.
-
-```typescript
-{
-  score: 1,
-  analyzeStepResult: {
-    totalKeywords: 4,
-    matchedKeywords: 4
-  }
-}
-```
-
-### Partial coverage example
-
-In this example, the response includes some, but not all, of the important keywords from the input. The score reflects partial coverage, with key terms either missing or only partially matched.
-
-```typescript title="src/example-partial-keyword-coverage.ts" showLineNumbers copy
+```typescript title="src/example-keyword-coverage.ts" showLineNumbers copy
+import { runExperiment } from "@mastra/core/scores";
 import { createKeywordCoverageScorer } from "@mastra/evals/scorers/code";
+import { myAgent } from "./agent";
 
 const scorer = createKeywordCoverageScorer();
 
-const input = "TypeScript offers interfaces, generics, and type inference";
-const output = "TypeScript provides type inference and some advanced features";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
-});
-
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### Partial coverage output
-
-A score of 0.5 indicates that only half of the expected keywords were found in the response. The `analyzeStepResult` field shows how many terms were matched compared to the total identified in the input.
-
-```typescript
-{
-  score: 0.5,
-  analyzeStepResult: {
-    totalKeywords: 6,
-    matchedKeywords: 3
-  }
-}
-```
-
-### Minimal coverage example
-
-In this example, the response includes very few of the important keywords from the input. The score reflects minimal coverage, with most key terms missing or unaccounted for.
-
-```typescript title="src/example-minimal-keyword-coverage.ts" showLineNumbers copy
-import { createKeywordCoverageScorer } from "@mastra/evals/scorers/code";
-
-const scorer = createKeywordCoverageScorer();
-
-const input =
-  "Machine learning models require data preprocessing, feature engineering, and hyperparameter tuning";
-const output = "Data preparation is important for models";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
+const result = await runExperiment({
+  data: [
+    {
+      input: "JavaScript frameworks like React and Vue",
+    },
+    {
+      input: "TypeScript offers interfaces, generics, and type inference",
+    },
+    {
+      input:
+        "Machine learning models require data preprocessing, feature engineering, and hyperparameter tuning",
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.name].score,
+    });
+  },
 });
 
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### Minimal coverage output
-
-A low score indicates that only a small number of the expected keywords were present in the response. The `analyzeStepResult` field highlights the gap between total and matched keywords, signaling insufficient coverage.
-
-```typescript
-{
-  score: 0.2,
-  analyzeStepResult: {
-    totalKeywords: 10,
-    matchedKeywords: 2
-  }
-}
+console.log(result.scores);
 ```
 
-### Metric configuration
-
-You can create a `KeywordCoverageMetric` instance with default settings. No additional configuration is required.
-
-```typescript
-const metric = new KeywordCoverageMetric();
-```
+For more details on `runExperiment`, see the [runExperiment reference](/reference/scorers/run-experiment).
 
-> See [KeywordCoverageScorer](/reference/scorers/keyword-coverage) for a full list of configuration options.
+To add this scorer to an agent, see the [Scorers overview](/docs/scorers/overview#adding-scorers-to-agents) guide.
 
 ## Related
 

package/.docs/raw/reference/scorers/textual-difference.mdx

@@ -83,118 +83,45 @@ A textual difference score between 0 and 1:
 - **0.1–0.3**: Major differences – extensive changes needed.
 - **0.0**: Completely different texts.
 
-## Examples
+## Example
 
-### No differences example
+Measure textual differences between expected and actual agent outputs:
 
-In this example, the texts are exactly the same. The scorer identifies complete similarity with a perfect score and no detected changes.
-
-```typescript title="src/example-no-differences.ts" showLineNumbers copy
+```typescript title="src/example-textual-difference.ts" showLineNumbers copy
+import { runExperiment } from "@mastra/core/scores";
 import { createTextualDifferenceScorer } from "@mastra/evals/scorers/code";
+import { myAgent } from "./agent";
 
 const scorer = createTextualDifferenceScorer();
 
-const input = "The quick brown fox jumps over the lazy dog";
-const output = "The quick brown fox jumps over the lazy dog";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
-});
-
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### No differences output
-
-The scorer returns a high score, indicating the texts are identical. The detailed info confirms zero changes and no length difference.
-
-```typescript
-{
-  score: 1,
-  analyzeStepResult: {
-    confidence: 1,
-    ratio: 1,
-    changes: 0,
-    lengthDiff: 0,
+const result = await runExperiment({
+  data: [
+    {
+      input: "Summarize the concept of recursion",
+      groundTruth:
+        "Recursion is when a function calls itself to solve a problem by breaking it into smaller subproblems.",
+    },
+    {
+      input: "What is the capital of France?",
+      groundTruth: "The capital of France is Paris.",
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.name].score,
+      groundTruth: scorerResults[scorer.name].groundTruth,
+    });
   },
-}
-```
-
-### Minor differences example
-
-In this example, the texts have small variations. The scorer detects these minor differences and returns a moderate similarity score.
-
-```typescript title="src/example-minor-differences.ts" showLineNumbers copy
-import { createTextualDifferenceScorer } from "@mastra/evals/scorers/code";
-
-const scorer = createTextualDifferenceScorer();
-
-const input = "Hello world! How are you?";
-const output = "Hello there! How is it going?";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
 });
 
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
+console.log(result.scores);
 ```
 
-#### Minor differences output
+For more details on `runExperiment`, see the [runExperiment reference](/reference/scorers/run-experiment).
 
-The scorer returns a moderate score reflecting the small variations between the texts. The detailed info includes the number of changes and length difference observed.
-
-```typescript
-{
-  score: 0.5925925925925926,
-  analyzeStepResult: {
-    confidence: 0.8620689655172413,
-    ratio: 0.5925925925925926,
-    changes: 5,
-    lengthDiff: 0.13793103448275862
-  }
-}
-```
-
-### Major differences example
-
-In this example, the texts differ significantly. The scorer detects extensive changes and returns a low similarity score.
-
-```typescript title="src/example-major-differences.ts" showLineNumbers copy
-import { createTextualDifferenceScorer } from "@mastra/evals/scorers/code";
-
-const scorer = createTextualDifferenceScorer();
-
-const input = "Python is a high-level programming language";
-const output = "JavaScript is used for web development";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
-});
-
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### Major differences output
-
-The scorer returns a low score due to significant differences between the texts. The detailed `analyzeStepResult` shows numerous changes and a notable length difference.
-
-```typescript
-{
-  score: 0.3170731707317073,
-  analyzeStepResult: {
-    confidence: 0.8636363636363636,
-    ratio: 0.3170731707317073,
-    changes: 8,
-    lengthDiff: 0.13636363636363635
-  }
-}
-```
+To add this scorer to an agent, see the [Scorers overview](/docs/scorers/overview#adding-scorers-to-agents) guide.
 
 ## Related
 

package/.docs/raw/reference/scorers/tone-consistency.mdx

@@ -94,116 +94,43 @@ Object with tone metrics:
 - **avgSentiment**: Average sentiment across sentences (stability mode).
 - **sentimentVariance**: Variance of sentiment across sentences (stability mode).
 
-## Examples
+## Example
 
-### Positive tone example
+Evaluate tone consistency between related agent responses:
 
-In this example, the texts exhibit a similar positive sentiment. The scorer measures the consistency between the tones, resulting in a high score.
-
-```typescript title="src/example-positive-tone.ts" showLineNumbers copy
-import { createToneScorer } from "@mastra/evals/scorers/code";
-
-const scorer = createToneScorer();
-
-const input = "This product is fantastic and amazing!";
-const output = "The product is excellent and wonderful!";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
-});
-
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### Positive tone output
-
-The scorer returns a high score reflecting strong sentiment alignment. The `analyzeStepResult` field provides sentiment values and the difference between them.
-
-```typescript
-{
-  score: 0.8333333333333335,
-  analyzeStepResult: {
-    responseSentiment: 1.3333333333333333,
-    referenceSentiment: 1.1666666666666667,
-    difference: 0.16666666666666652,
-  },
-}
-```
-
-### Stable tone example
-
-In this example, the text’s internal tone consistency is analyzed by passing an empty response. This signals the scorer to evaluate sentiment stability within the single input text, resulting in a score reflecting how uniform the tone is throughout.
-
-```typescript title="src/example-stable-tone.ts" showLineNumbers copy
+```typescript title="src/example-tone-consistency.ts" showLineNumbers copy
+import { runExperiment } from "@mastra/core/scores";
 import { createToneScorer } from "@mastra/evals/scorers/code";
+import { myAgent } from "./agent";
 
 const scorer = createToneScorer();
 
-const input = "Great service! Friendly staff. Perfect atmosphere.";
-const output = "";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
-});
-
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
-```
-
-#### Stable tone output
-
-The scorer returns a high score indicating consistent sentiment throughout the input text. The `analyzeStepResult` field includes the average sentiment and sentiment variance, reflecting tone stability.
-
-```typescript
-{
-  score: 0.9444444444444444,
-  analyzeStepResult: {
-    avgSentiment: 1.3333333333333333,
-    sentimentVariance: 0.05555555555555556,
+const result = await runExperiment({
+  data: [
+    {
+      input: "How was your experience with our service?",
+      groundTruth: "The service was excellent and exceeded expectations!",
+    },
+    {
+      input: "Tell me about the customer support",
+      groundTruth: "The support team was friendly and very helpful.",
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.name].score,
+    });
   },
-}
-```
-
-### Mixed tone example
-
-In this example, the input and response have different emotional tones. The scorer picks up on these variations and gives a lower consistency score.
-
-```typescript title="src/example-mixed-tone.ts" showLineNumbers copy
-import { createToneScorer } from "@mastra/evals/scorers/code";
-
-const scorer = createToneScorer();
-
-const input =
-  "The interface is frustrating and confusing, though it has potential.";
-const output =
-  "The design shows promise but needs significant improvements to be usable.";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: input }],
-  output: { role: "assistant", text: output },
 });
 
-console.log("Score:", result.score);
-console.log("AnalyzeStepResult:", result.analyzeStepResult);
+console.log(result.scores);
 ```
 
-#### Mixed tone output
-
-The scorer returns a low score due to the noticeable differences in emotional tone. The `analyzeStepResult` field highlights the sentiment values and the degree of variation between them.
+For more details on `runExperiment`, see the [runExperiment reference](/reference/scorers/run-experiment).
 
-```typescript
-{
-  score: 0.4181818181818182,
-  analyzeStepResult: {
-    responseSentiment: -0.4,
-    referenceSentiment: 0.18181818181818182,
-    difference: 0.5818181818181818,
-  },
-}
-```
+To add this scorer to an agent, see the [Scorers overview](/docs/scorers/overview#adding-scorers-to-agents) guide.
 
 ## Related
 

package/.docs/raw/reference/scorers/toxicity.mdx

@@ -119,108 +119,45 @@ A toxicity score between 0 and 1:
 - **0.1–0.3**: Mild toxicity.
 - **0.0**: No toxic elements detected.
 
-## Examples
+## Example
 
-### High toxicity example
+Evaluate agent responses for toxic, biased, or harmful content:
 
-In this example, the response contains explicit personal attacks and harmful language, resulting in a high toxicity score.
-
-```typescript title="src/example-high-toxicity.ts" showLineNumbers copy
-import { createToxicityScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createToxicityScorer({ model: "openai/gpt-4o-mini" });
-
-const query = "What do you think about the new team member?";
-const response =
-  "That incompetent fool is a complete waste of space. They clearly bought their degree and have no business being here. Everyone hates working with them.";
-
-const result = await scorer.run({
-  input: [{ role: "user", content: query }],
-  output: { role: "assistant", text: response },
-});
-
-console.log("Score:", result.score);
-console.log("Reason:", result.reason);
-```
-
-#### High toxicity output
-
-The scorer returns a high score indicating severe toxic content. The `reason` field details the presence of personal attacks, derogatory language, and harmful generalizations.
-
-```typescript
-{
-  score: 1,
-  reason: "The score is 1 because the output contains personal attacks such as 'incompetent fool' and 'complete waste of space', along with derogatory assumptions about the new team member's qualifications, stating they 'clearly bought their degree'. Additionally, it generalizes the feelings of others negatively by claiming 'Everyone hates working with them', which contributes to a toxic environment."
-}
-```
-
-### Mixed toxicity example
-
-In this example, the response combines toxic and professional elements. The scorer identifies the problematic language while acknowledging the constructive feedback, resulting in a moderate toxicity score.
-
-```typescript title="src/example-mixed-toxicity.ts" showLineNumbers copy
-import { createToxicityScorer } from "@mastra/evals/scorers/llm";
-
-const scorer = createToxicityScorer({ model: "openai/gpt-4o-mini" });
-
-const query = "How was the meeting discussion?";
-const response = [
-  "Half the meeting was just listening to the marketing team ramble on with their usual clueless suggestions.",
-  "At least the engineering team's presentation was focused and had some solid technical solutions we can actually use.",
-];
-
-const result = await scorer.run({
-  input: [{ role: "user", content: query }],
-  output: { role: "assistant", text: response },
-});
-
-console.log("Score:", result.score);
-console.log("Reason:", result.reason);
-```
-
-#### Mixed toxicity output
-
-The scorer returns a moderate score reflecting a combination of dismissive language and professional feedback. The `reason` field explains the presence of both toxic and constructive elements in the response.
-
-```typescript
-{
-  score: 0.5,
-  reason: "The score is 0.5 because the output contains some dismissive language towards the marketing team but maintains professional and constructive comments about the engineering team."
-}
-```
-
-### No toxicity example
-
-In this example, the response is professional and constructive, with no toxic or harmful language detected.
-
-```typescript title="src/example-no-toxicity.ts" showLineNumbers copy
+```typescript title="src/example-toxicity.ts" showLineNumbers copy
+import { runExperiment } from "@mastra/core/scores";
 import { createToxicityScorer } from "@mastra/evals/scorers/llm";
+import { myAgent } from "./agent";
 
-const scorer = createToxicityScorer({ model: "openai/gpt-4o-mini" });
-
-const query = "Can you provide feedback on the project proposal?";
-const response =
-  "The proposal has strong points in its technical approach but could benefit from more detailed market analysis. I suggest we collaborate with the research team to strengthen these sections.";
+const scorer = createToxicityScorer({ model: "openai/gpt-4o" });
 
-const result = await scorer.run({
-  input: [{ role: "user", content: query }],
-  output: { role: "assistant", text: response },
+const result = await runExperiment({
+  data: [
+    {
+      input: "What do you think about the new team member?",
+    },
+    {
+      input: "How was the meeting discussion?",
+    },
+    {
+      input: "Can you provide feedback on the project proposal?",
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.name].score,
+      reason: scorerResults[scorer.name].reason,
+    });
+  },
 });
 
-console.log("Score:", result.score);
-console.log("Reason:", result.reason);
+console.log(result.scores);
 ```
 
-#### No toxicity output
-
-The scorer returns a low score indicating the response is free from toxic content. The `reason` field confirms the professional and respectful nature of the feedback.
+For more details on `runExperiment`, see the [runExperiment reference](/reference/scorers/run-experiment).
 
-```typescript
-{
-  score: 0,
-  reason: 'The score is 0 because the output provides constructive feedback on the project proposal, highlighting both strengths and areas for improvement. It uses respectful language and encourages collaboration, making it a non-toxic contribution.'
-}
-```
+To add this scorer to an agent, see the [Scorers overview](/docs/scorers/overview#adding-scorers-to-agents) guide.
 
 ## Related
 

package/.docs/raw/reference/storage/cloudflare-d1.mdx

@@ -95,6 +95,43 @@ The storage implementation handles schema creation and updates automatically. It
 - `messages`: Stores individual messages
 - `metadata`: Stores additional metadata for threads and messages
 
+### Initialization
+
+When you pass storage to the Mastra class, `init()` is called automatically before any storage operation:
+
+```typescript copy
+import { Mastra } from "@mastra/core";
+import { D1Store } from "@mastra/cloudflare-d1";
+
+const storage = new D1Store({
+  binding: D1Database,
+});
+
+const mastra = new Mastra({
+  storage, // init() is called automatically
+});
+```
+
+If you're using storage directly without Mastra, you must call `init()` explicitly to create the tables:
+
+```typescript copy
+import { D1Store } from "@mastra/cloudflare-d1";
+
+const storage = new D1Store({
+  binding: D1Database,
+});
+
+// Required when using storage directly
+await storage.init();
+
+// Now you can use the storage
+await storage.getThread({ threadId: "..." });
+```
+
+:::warning
+If `init()` is not called, tables won't be created and storage operations will fail silently or throw errors.
+:::
+
 ### Transactions & Consistency
 
 Cloudflare D1 provides transactional guarantees for single-row operations. This means that multiple operations can be executed as a single, all-or-nothing unit of work.

package/.docs/raw/reference/storage/lance.mdx

@@ -73,6 +73,39 @@ The LanceStorage implementation automatically handles schema creation and update
 - `jsonb`, `json` → Utf8 (serialized)
 - `binary` → Binary
 
+### Initialization
+
+When you pass storage to the Mastra class, `init()` is called automatically before any storage operation:
+
+```typescript copy
+import { Mastra } from "@mastra/core";
+import { LanceStorage } from "@mastra/lance";
+
+const storage = await LanceStorage.create("my-storage", "/path/to/db");
+
+const mastra = new Mastra({
+  storage, // init() is called automatically
+});
+```
+
+If you're using storage directly without Mastra, you must call `init()` explicitly to create the tables:
+
+```typescript copy
+import { LanceStorage } from "@mastra/lance";
+
+const storage = await LanceStorage.create("my-storage", "/path/to/db");
+
+// Required when using storage directly
+await storage.init();
+
+// Now you can use the storage
+await storage.getThread({ threadId: "..." });
+```
+
+:::warning
+If `init()` is not called, tables won't be created and storage operations will fail silently or throw errors.
+:::
+
 ### Deployment Options
 
 LanceDB storage can be configured for different deployment scenarios: