@mastra/mcp-docs-server 0.13.39 → 1.0.0-beta.0

package/.docs/raw/reference/evals/content-similarity.mdx

@@ -1,126 +1,75 @@
 ---
-title: "Reference: ContentSimilarityMetric | Evals | Mastra Docs"
-description: Documentation for the Content Similarity Metric in Mastra, which measures textual similarity between strings and provides a matching score.
+title: "Reference: Content Similarity Scorer | Evals | Mastra Docs"
+description: Documentation for the Content Similarity Scorer in Mastra, which measures textual similarity between strings and provides a matching score.
 ---
 
-# ContentSimilarityMetric
+# Content Similarity Scorer
 
-:::info Scorers
-This documentation refers to the legacy evals API. For the latest scorer features, see [Scorers](/docs/scorers/overview).
-:::
+The `createContentSimilarityScorer()` function measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.
 
-The `ContentSimilarityMetric` class measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.
+## Parameters
 
-## Basic Usage
-
-```typescript
-import { ContentSimilarityMetric } from "@mastra/evals/nlp";
-
-const metric = new ContentSimilarityMetric({
-  ignoreCase: true,
-  ignoreWhitespace: true,
-});
-
-const result = await metric.measure("Hello, world!", "hello world");
-
-console.log(result.score); // Similarity score from 0-1
-console.log(result.info); // Detailed similarity metrics
-```
-
-## Constructor Parameters
-
-<PropertiesTable
-  content={[
-    {
-      name: "options",
-      type: "ContentSimilarityOptions",
-      description: "Configuration options for similarity comparison",
-      isOptional: true,
-      defaultValue: "{ ignoreCase: true, ignoreWhitespace: true }",
-    },
-  ]}
-/>
-
-### ContentSimilarityOptions
+The `createContentSimilarityScorer()` function accepts a single options object with the following properties:
 
 <PropertiesTable
   content={[
     {
       name: "ignoreCase",
       type: "boolean",
-      description: "Whether to ignore case differences when comparing strings",
-      isOptional: true,
+      required: false,
       defaultValue: "true",
+      description: "Whether to ignore case differences when comparing strings.",
     },
     {
       name: "ignoreWhitespace",
       type: "boolean",
-      description: "Whether to normalize whitespace when comparing strings",
-      isOptional: true,
+      required: false,
       defaultValue: "true",
+      description: "Whether to normalize whitespace when comparing strings.",
     },
   ]}
 />
 
-## measure() Parameters
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](./mastra-scorer) for details on the `.run()` method and its input/output.
+
+## .run() Returns
 
 <PropertiesTable
   content={[
     {
-      name: "input",
+      name: "runId",
       type: "string",
-      description: "The reference text to compare against",
-      isOptional: false,
+      description: "The id of the run (optional).",
     },
     {
-      name: "output",
-      type: "string",
-      description: "The text to evaluate for similarity",
-      isOptional: false,
+      name: "preprocessStepResult",
+      type: "object",
+      description:
+        "Object with processed input and output: { processedInput: string, processedOutput: string }",
+    },
+    {
+      name: "analyzeStepResult",
+      type: "object",
+      description: "Object with similarity: { similarity: number }",
     },
-  ]}
-/>
-
-## Returns
-
-<PropertiesTable
-  content={[
     {
       name: "score",
       type: "number",
       description:
-        "Similarity score (0-1) where 1 indicates perfect similarity",
-    },
-    {
-      name: "info",
-      type: "object",
-      description: "Detailed similarity metrics",
-      properties: [
-        {
-          type: "number",
-          parameters: [
-            {
-              name: "similarity",
-              type: "number",
-              description: "Raw similarity score between the two texts",
-            },
-          ],
-        },
-      ],
+        "Similarity score (0-1) where 1 indicates perfect similarity.",
     },
   ]}
 />
 
 ## Scoring Details
 
-The metric evaluates textual similarity through character-level matching and configurable text normalization.
+The scorer evaluates textual similarity through character-level matching and configurable text normalization.
 
 ### Scoring Process
 
 1. Normalizes text:
    - Case normalization (if ignoreCase: true)
    - Whitespace normalization (if ignoreWhitespace: true)
-
 2. Compares processed strings using string-similarity algorithm:
    - Analyzes character sequences
    - Aligns word boundaries
@@ -129,56 +78,120 @@ The metric evaluates textual similarity through character-level matching and con
 
 Final score: `similarity_value * scale`
 
-### Score interpretation
+## Examples
+
+### High similarity example
+
+In this example, the response closely resembles the query in both structure and meaning. Minor differences in tense and phrasing do not significantly affect the overall similarity.
+
+```typescript title="src/example-high-similarity.ts" showLineNumbers copy
+import { createContentSimilarityScorer } from "@mastra/evals/scorers/prebuilt";
+
+const scorer = createContentSimilarityScorer();
 
-(0 to scale, default 0-1)
+const query = "The quick brown fox jumps over the lazy dog.";
+const response = "A quick brown fox jumped over a lazy dog.";
+
+const result = await scorer.run({
+  input: [{ role: "user", content: query }],
+  output: { text: response },
+});
+
+console.log(result);
+```
 
-- 1.0: Perfect match - identical texts
-- 0.7-0.9: High similarity - mostly matching content
-- 0.4-0.6: Moderate similarity - partial matches
-- 0.1-0.3: Low similarity - few matching patterns
-- 0.0: No similarity - completely different texts
+#### High similarity output
 
-## Example with Different Options
+The output receives a high score because the response preserves the intent and content of the query with only subtle wording changes.
 
 ```typescript
-import { ContentSimilarityMetric } from "@mastra/evals/nlp";
+{
+  score: 0.7761194029850746,
+  analyzeStepResult: {
+    similarity: 0.7761194029850746
+  },
+}
+```
+
+### Moderate similarity example
+
+In this example, the response shares some conceptual overlap with the query but diverges in structure and wording. Key elements remain present, but the phrasing introduces moderate variation.
+
+```typescript title="src/example-moderate-similarity.ts" showLineNumbers copy
+import { createContentSimilarityScorer } from "@mastra/evals/scorers/prebuilt";
 
-// Case-sensitive comparison
-const caseSensitiveMetric = new ContentSimilarityMetric({
-  ignoreCase: false,
-  ignoreWhitespace: true,
+const scorer = createContentSimilarityScorer();
+
+const query = "A brown fox quickly leaps across a sleeping dog.";
+const response = "The quick brown fox jumps over the lazy dog.";
+
+const result = await scorer.run({
+  input: [{ role: "user", content: query }],
+  output: { text: response },
 });
 
-const result1 = await caseSensitiveMetric.measure("Hello World", "hello world"); // Lower score due to case difference
+console.log(result);
+```
+
+#### Moderate similarity output
 
-// Example output:
-// {
-//   score: 0.75,
-//   info: { similarity: 0.75 }
-// }
+The output receives a mid-range score because the response captures the general idea of the query, though it differs enough in wording to reduce overall similarity.
 
-// Strict whitespace comparison
-const strictWhitespaceMetric = new ContentSimilarityMetric({
-  ignoreCase: true,
-  ignoreWhitespace: false,
+```typescript
+{
+  score: 0.40540540540540543,
+  analyzeStepResult: {
+    similarity: 0.40540540540540543
+  }
+}
+```
+
+### Low similarity example
+
+In this example, the response and query are unrelated in meaning, despite having a similar grammatical structure. There is little to no shared content overlap.
+
+```typescript title="src/example-low-similarity.ts" showLineNumbers copy
+import { createContentSimilarityScorer } from "@mastra/evals/scorers/prebuilt";
+
+const scorer = createContentSimilarityScorer();
+
+const query = "The cat sleeps on the windowsill.";
+const response = "The quick brown fox jumps over the lazy dog.";
+
+const result = await scorer.run({
+  input: [{ role: "user", content: query }],
+  output: { text: response },
 });
 
-const result2 = await strictWhitespaceMetric.measure(
-  "Hello   World",
-  "Hello World",
-); // Lower score due to whitespace difference
+console.log(result);
+```
+
+#### Low similarity output
 
-// Example output:
-// {
-//   score: 0.85,
-//   info: { similarity: 0.85 }
-// }
+The output receives a low score because the response does not align with the content or intent of the query.
+
+```typescript
+{
+  score: 0.25806451612903225,
+  analyzeStepResult: {
+    similarity: 0.25806451612903225
+  },
+}
 ```
 
+### Score interpretation
+
+A similarity score between 0 and 1:
+
+- **1.0**: Perfect match – content is nearly identical.
+- **0.7–0.9**: High similarity – minor differences in word choice or structure.
+- **0.4–0.6**: Moderate similarity – general overlap with noticeable variation.
+- **0.1–0.3**: Low similarity – few common elements or shared meaning.
+- **0.0**: No similarity – completely different content.
+
 ## Related
 
-- [Completeness Metric](./completeness)
-- [Textual Difference Metric](./textual-difference)
-- [Answer Relevancy Metric](./answer-relevancy)
-- [Keyword Coverage Metric](./keyword-coverage)
+- [Completeness Scorer](./completeness)
+- [Textual Difference Scorer](./textual-difference)
+- [Answer Relevancy Scorer](./answer-relevancy)
+- [Keyword Coverage Scorer](./keyword-coverage)