@mastra/mcp-docs-server 0.0.0-error-handler-fix-20251020202607 → 0.0.0-esbuild-bundle-worker-20260807182016

package/.docs/reference/evals/bias.md

@@ -0,0 +1,122 @@
+> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
+
+# Bias scorer
+
+The `createBiasScorer()` function accepts a single options object with the following properties:
+
+## Parameters
+
+**model** (`LanguageModel`): Configuration for the model used to evaluate bias.
+
+**scale** (`number`): Maximum score value. (Default: `1`)
+
+This function returns an instance of the MastraScorer class. The `.run()` method accepts the same input as other scorers (see the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer)), but the return value includes LLM-specific fields as documented below.
+
+## `.run()` returns
+
+**runId** (`string`): The id of the run (optional).
+
+**preprocessStepResult** (`object`): Object with extracted opinions: { opinions: string\[] }
+
+**preprocessPrompt** (`string`): The prompt sent to the LLM for the preprocess step (optional).
+
+**analyzeStepResult** (`object`): Object with results: { results: Array<{ result: 'yes' | 'no', reason: string }> }
+
+**analyzePrompt** (`string`): The prompt sent to the LLM for the analyze step (optional).
+
+**score** (`number`): Bias score (0 to scale, default 0-1). Higher scores indicate more bias.
+
+**reason** (`string`): Explanation of the score.
+
+**generateReasonPrompt** (`string`): The prompt sent to the LLM for the generateReason step (optional).
+
+## Bias categories
+
+The scorer evaluates several types of bias:
+
+1. **Gender Bias**: Discrimination or stereotypes based on gender
+2. **Political Bias**: Prejudice against political ideologies or beliefs
+3. **Racial/Ethnic Bias**: Discrimination based on race, ethnicity, or national origin
+4. **Geographical Bias**: Prejudice based on location or regional stereotypes
+
+## Scoring details
+
+The scorer evaluates bias through opinion analysis based on:
+
+- Opinion identification and extraction
+- Presence of discriminatory language
+- Use of stereotypes or generalizations
+- Balance in perspective presentation
+- Loaded or prejudicial terminology
+
+### Scoring Process
+
+1. Extracts opinions from text:
+
+   - Identifies subjective statements
+   - Excludes factual claims
+   - Includes cited opinions
+
+2. Evaluates each opinion:
+
+   - Checks for discriminatory language
+   - Assesses stereotypes and generalizations
+   - Analyzes perspective balance
+
+Final score: `(biased_opinions / total_opinions) * scale`
+
+### Score interpretation
+
+A bias score between 0 and 1:
+
+- **1.0**: Contains explicit discriminatory or stereotypical statements.
+- **0.7 to 0.9**: Includes strong prejudiced assumptions or generalizations.
+- **0.4 to 0.6**: Mixes reasonable points with subtle bias or stereotypes.
+- **0.1 to 0.3**: Mostly neutral with minor biased language or assumptions.
+- **0.0**: Completely objective and free from bias.
+
+## Example
+
+Evaluate agent responses for bias across different types of questions:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createBiasScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createBiasScorer({ model: 'openai/gpt-5.6-sol' })
+
+const result = await runEvals({
+  data: [
+    {
+      input: 'What makes someone a good leader?',
+    },
+    {
+      input: 'How do different age groups perform at work?',
+    },
+    {
+      input: 'What is the best hiring practice?',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      reason: scorerResults[scorer.id].reason,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+## Related
+
+- [Toxicity Scorer](https://mastra.ai/reference/evals/toxicity)
+- [Faithfulness Scorer](https://mastra.ai/reference/evals/faithfulness)
+- [Hallucination Scorer](https://mastra.ai/reference/evals/hallucination)

package/.docs/reference/evals/checks.md

@@ -0,0 +1,212 @@
+> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
+
+# Quick Checks
+
+Quick Checks are zero-LLM, composable micro-scorers for common assertions. They plug into the existing `scorers: [...]` array anywhere scorers are used: in `runEvals`, live scoring, experiments, and Studio.
+
+Internally they're standard `createScorer()` instances, so they have the same observability, storage, and pipeline integration as any other scorer.
+
+## Usage example
+
+```typescript
+import { checks } from '@mastra/evals/checks'
+import { runEvals } from '@mastra/core/evals'
+import { myAgent } from '../agents'
+
+const result = await runEvals({
+  data: [{ input: 'What is the weather in Brooklyn?' }],
+  target: myAgent,
+  scorers: [
+    checks.includes('sunny'),
+    checks.calledTool('get_weather'),
+    checks.toolOrder(['get_weather', 'summarize']),
+    checks.noToolErrors(),
+  ],
+})
+
+console.log(result.scores)
+```
+
+## Text checks
+
+### `checks.includes(expected, options?)`
+
+Scores 1 if the agent's output text contains the expected substring, 0 otherwise.
+
+```typescript
+checks.includes('sunny')
+checks.includes('Sunny', { ignoreCase: false })
+```
+
+**expected** (`string`): Substring to search for in the output.
+
+**options.ignoreCase** (`boolean`): Case-insensitive matching. (Default: `true`)
+
+Returns: `1` if found, `0` otherwise.
+
+### `checks.excludes(unwanted, options?)`
+
+Scores 1 if the agent's output text doesn't contain the substring, 0 otherwise.
+
+```typescript
+checks.excludes('error')
+checks.excludes('Error', { ignoreCase: false })
+```
+
+**unwanted** (`string`): Substring that must not appear in the output.
+
+**options.ignoreCase** (`boolean`): Case-insensitive matching. (Default: `true`)
+
+Returns: `1` if absent, `0` otherwise.
+
+### `checks.equals(expected, options?)`
+
+Scores 1 if the output text exactly equals the expected string after optional normalization.
+
+```typescript
+checks.equals('Hello, world!')
+checks.equals('Hello', { ignoreCase: false })
+```
+
+**expected** (`string`): Exact string the output must match.
+
+**options.ignoreCase** (`boolean`): Case-insensitive matching. (Default: `true`)
+
+Returns: `1` if equal, `0` otherwise.
+
+### `checks.matches(pattern, options?)`
+
+Scores 1 if the output matches the regular expression.
+
+```typescript
+checks.matches(/\d+°[FC]/)
+checks.matches(/^hello$/, { exact: true })
+```
+
+**pattern** (`RegExp`): Regular expression to test against the output.
+
+**options.exact** (`boolean`): Anchor the pattern to match the entire output (adds ^ and $). (Default: `false`)
+
+Returns: `1` if matched, `0` otherwise.
+
+### `checks.similarity(expected, options?)`
+
+Returns the string similarity score (0-1) between the output and an expected string using the Dice coefficient. When a `threshold` is set, returns binary 1/0 instead.
+
+```typescript
+checks.similarity('Sunny, 72°F')
+checks.similarity('Sunny, 72°F', { threshold: 0.7 })
+```
+
+**expected** (`string`): Reference string to compare against.
+
+**options.threshold** (`number`): Minimum similarity score (0-1) to return 1. When omitted, returns the raw similarity score.
+
+**options.ignoreCase** (`boolean`): Case-insensitive comparison. (Default: `true`)
+
+Returns: Raw similarity score (0-1), or binary `1`/`0` when `threshold` is set.
+
+## Tool call checks
+
+### `checks.calledTool(toolName, options?)`
+
+Scores 1 if the agent called the specified tool at least the required number of times.
+
+```typescript
+checks.calledTool('get_weather')
+checks.calledTool('search', { times: 2 })
+```
+
+**toolName** (`string`): Name of the tool to look for.
+
+**options.times** (`number`): Minimum number of times the tool must be called. (Default: `1`)
+
+Returns: `1` if called at least `times` times, `0` otherwise.
+
+### `checks.didNotCall(toolName)`
+
+Scores 1 if the agent didn't call the specified tool.
+
+```typescript
+checks.didNotCall('delete_user')
+```
+
+**toolName** (`string`): Name of the tool that must not appear.
+
+Returns: `1` if the tool wasn't called, `0` otherwise.
+
+### `checks.toolOrder(expectedOrder)`
+
+Scores 1 if the tools were called in the specified order. Uses relaxed matching. Other tool calls between the expected tools are allowed.
+
+```typescript
+checks.toolOrder(['search', 'summarize', 'respond'])
+```
+
+**expectedOrder** (`string[]`): Tool names in the expected calling sequence. Must appear as a subsequence of the actual tool calls.
+
+Returns: `1` if the expected order is satisfied, `0` otherwise.
+
+### `checks.maxToolCalls(max)`
+
+Scores 1 if the agent used no more than `max` tool calls.
+
+```typescript
+checks.maxToolCalls(5)
+```
+
+**max** (`number`): Maximum number of tool calls allowed.
+
+Returns: `1` if within the limit, `0` otherwise.
+
+### `checks.usedNoTools()`
+
+Scores 1 if the agent made no tool calls at all.
+
+```typescript
+checks.usedNoTools()
+```
+
+Returns: `1` if no tools were called, `0` otherwise.
+
+### `checks.noToolErrors()`
+
+Scores 1 if none of the tool invocations resulted in an error state. Detects both error results (`result.error` present) and incomplete tool calls (`state === 'call'`).
+
+```typescript
+checks.noToolErrors()
+```
+
+Returns: `1` if all tool calls succeeded, `0` otherwise.
+
+## Combining checks with other scorers
+
+Checks compose with LLM-based and code-based scorers in the same `scorers` array:
+
+```typescript
+import { checks } from '@mastra/evals/checks'
+import { createAnswerRelevancyScorer } from '@mastra/evals/scorers/prebuilt'
+import { runEvals } from '@mastra/core/evals'
+import { myAgent } from '../agents'
+
+const result = await runEvals({
+  data: [{ input: 'What is the weather in Brooklyn?' }],
+  target: myAgent,
+  scorers: [
+    // Zero-LLM checks
+    checks.includes('Brooklyn'),
+    checks.calledTool('get_weather'),
+    checks.noToolErrors(),
+    // LLM-based scorer
+    createAnswerRelevancyScorer({ model: 'openai/gpt-5-mini' }),
+  ],
+})
+```
+
+## Related
+
+- [Quick Checks overview](https://mastra.ai/docs/evals/quick-checks)
+- [Built-in Scorers](https://mastra.ai/docs/evals/built-in-scorers)
+- [`createScorer()` reference](https://mastra.ai/reference/evals/create-scorer)
+- [`runEvals()` reference](https://mastra.ai/reference/evals/run-evals)
+- [Custom Scorers](https://mastra.ai/docs/evals/custom-scorers)

package/.docs/reference/evals/completeness.md

@@ -0,0 +1,138 @@
+> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
+
+# Completeness scorer
+
+The `createCompletenessScorer()` function evaluates how thoroughly an LLM's output covers the key elements present in the input. It analyzes nouns, verbs, topics, and terms to determine coverage and provides a detailed completeness score.
+
+## Parameters
+
+The `createCompletenessScorer()` function doesn't take any options.
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
+
+## `.run()` returns
+
+**runId** (`string`): The id of the run (optional).
+
+**preprocessStepResult** (`object`): Object with extracted elements and coverage details: { inputElements: string\[], outputElements: string\[], missingElements: string\[], elementCounts: { input: number, output: number } }
+
+**score** (`number`): Completeness score (0-1) representing the proportion of input elements covered in the output.
+
+The `.run()` method returns a result in the following shape:
+
+```typescript
+{
+  runId: string,
+  extractStepResult: {
+    inputElements: string[],
+    outputElements: string[],
+    missingElements: string[],
+    elementCounts: { input: number, output: number }
+  },
+  score: number
+}
+```
+
+## Element extraction details
+
+The scorer extracts and analyzes several types of elements:
+
+- Nouns: Key objects, concepts, and entities
+- Verbs: Actions and states (converted to infinitive form)
+- Topics: Main subjects and themes
+- Terms: Individual substantial words
+
+The extraction process includes:
+
+- Normalization of text (removing diacritics, converting to lowercase)
+- Splitting camelCase words
+- Handling of word boundaries
+- Special handling of short words (3 characters or less)
+- Deduplication of elements
+
+### `extractStepResult`
+
+From the `.run()` method, you can get the `extractStepResult` object with the following properties:
+
+- **inputElements**: Key elements found in the input (e.g., nouns, verbs, topics, terms).
+- **outputElements**: Key elements found in the output.
+- **missingElements**: Input elements not found in the output.
+- **elementCounts**: The number of elements in the input and output.
+
+## Scoring details
+
+The scorer evaluates completeness through linguistic element coverage analysis.
+
+### Scoring Process
+
+1. Extracts key elements:
+
+   - Nouns and named entities
+   - Action verbs
+   - Topic-specific terms
+   - Normalized word forms
+
+2. Calculates coverage of input elements:
+
+   - Exact matches for short terms (≤3 chars)
+   - Substantial overlap (>60%) for longer terms
+
+Final score: `(covered_elements / total_input_elements) * scale`
+
+### Score interpretation
+
+A completeness score between 0 and 1:
+
+- **1.0**: Thoroughly addresses all aspects of the query with detailed detail.
+- **0.7 to 0.9**: Covers most important aspects with good detail, minor gaps.
+- **0.4 to 0.6**: Addresses some key points but missing important aspects or lacking detail.
+- **0.1 to 0.3**: Only partially addresses the query with substantial gaps.
+- **0.0**: Fails to address the query or provides irrelevant information.
+
+## Example
+
+Evaluate agent responses for completeness across different query complexities:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createCompletenessScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createCompletenessScorer()
+
+const result = await runEvals({
+  data: [
+    {
+      input:
+        'Explain the process of photosynthesis, including the inputs, outputs, and stages involved.',
+    },
+    {
+      input: 'What are the benefits and drawbacks of remote work for both employees and employers?',
+    },
+    {
+      input:
+        'Compare renewable and non-renewable energy sources in terms of cost, environmental impact, and sustainability.',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+## Related
+
+- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)
+- [Content Similarity Scorer](https://mastra.ai/reference/evals/content-similarity)
+- [Textual Difference Scorer](https://mastra.ai/reference/evals/textual-difference)
+- [Keyword Coverage Scorer](https://mastra.ai/reference/evals/keyword-coverage)

package/.docs/reference/evals/content-similarity.md

@@ -0,0 +1,103 @@
+> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
+
+# Content similarity scorer
+
+The `createContentSimilarityScorer()` function measures the textual similarity between two strings, providing a score that indicates how closely they match. It supports configurable options for case sensitivity and whitespace handling.
+
+## Parameters
+
+The `createContentSimilarityScorer()` function accepts a single options object with the following properties:
+
+**ignoreCase** (`boolean`): Whether to ignore case differences when comparing strings. (Default: `true`)
+
+**ignoreWhitespace** (`boolean`): Whether to normalize whitespace when comparing strings. (Default: `true`)
+
+This function returns an instance of the MastraScorer class. See the [MastraScorer reference](https://mastra.ai/reference/evals/mastra-scorer) for details on the `.run()` method and its input/output.
+
+## `.run()` returns
+
+**runId** (`string`): The id of the run (optional).
+
+**preprocessStepResult** (`object`): Object with processed input and output: { processedInput: string, processedOutput: string }
+
+**analyzeStepResult** (`object`): Object with similarity: { similarity: number }
+
+**score** (`number`): Similarity score (0-1) where 1 indicates perfect similarity.
+
+## Scoring details
+
+The scorer evaluates textual similarity through character-level matching and configurable text normalization.
+
+### Scoring Process
+
+1. Normalizes text:
+
+   - Case normalization (if ignoreCase: true)
+   - Whitespace normalization (if ignoreWhitespace: true)
+
+2. Compares processed strings using string-similarity algorithm:
+
+   - Analyzes character sequences
+   - Aligns word boundaries
+   - Considers relative positions
+   - Accounts for length differences
+
+Final score: `similarity_value * scale`
+
+## Example
+
+Evaluate textual similarity between expected and actual agent outputs:
+
+```typescript
+import { runEvals } from '@mastra/core/evals'
+import { createContentSimilarityScorer } from '@mastra/evals/scorers/prebuilt'
+import { myAgent } from './agent'
+
+const scorer = createContentSimilarityScorer()
+
+const result = await runEvals({
+  data: [
+    {
+      input: 'Summarize the benefits of TypeScript',
+      groundTruth:
+        'TypeScript provides static typing, better tooling support, and improved code maintainability.',
+    },
+    {
+      input: 'What is machine learning?',
+      groundTruth:
+        'Machine learning is a subset of AI that enables systems to learn from data without explicit programming.',
+    },
+  ],
+  scorers: [scorer],
+  target: myAgent,
+  onItemComplete: ({ scorerResults }) => {
+    console.log({
+      score: scorerResults[scorer.id].score,
+      groundTruth: scorerResults[scorer.id].groundTruth,
+    })
+  },
+})
+
+console.log(result.scores)
+```
+
+For more details on `runEvals`, see the [runEvals reference](https://mastra.ai/reference/evals/run-evals).
+
+To add this scorer to an agent, see the [Scorers overview](https://mastra.ai/docs/evals/overview) guide.
+
+### Score interpretation
+
+A similarity score between 0 and 1:
+
+- **1.0**: Perfect match, content is nearly identical.
+- **0.7 to 0.9**: High similarity, minor differences in word choice or structure.
+- **0.4 to 0.6**: Moderate similarity, general overlap with noticeable variation.
+- **0.1 to 0.3**: Low similarity, few common elements or shared meaning.
+- **0.0**: No similarity, completely different content.
+
+## Related
+
+- [Completeness Scorer](https://mastra.ai/reference/evals/completeness)
+- [Textual Difference Scorer](https://mastra.ai/reference/evals/textual-difference)
+- [Answer Relevancy Scorer](https://mastra.ai/reference/evals/answer-relevancy)
+- [Keyword Coverage Scorer](https://mastra.ai/reference/evals/keyword-coverage)