@mastra/mcp-docs-server 1.2.18-alpha.3 → 1.2.18-alpha.5

package/.docs/docs/sandbox/search.md

@@ -2,26 +2,16 @@
 
 # Search and indexing
 
-Search lets agents find relevant content in indexed workspace files. When an agent needs to answer a question or find information, it can search the indexed content instead of reading every file.
+Search gives agents a fast way to find relevant content without reading every file. It works with [direct filesystem access](https://mastra.ai/docs/sandbox/filesystem) and [`mounts`](https://mastra.ai/docs/sandbox/filesystem), but stores searchable content in a separate index.
 
-Search works with both [`mounts`](https://mastra.ai/docs/sandbox/filesystem) and a [workspace-only filesystem](https://mastra.ai/docs/sandbox/filesystem). With `mounts`, paths from every mounted filesystem are available through one composite filesystem. With `filesystem`, search uses that provider directly. In both cases, queries search the workspace index rather than reading live files on demand.
-
-## When to use search
-
-Use workspace search when your agent needs to:
-
-- Find exact terms, filenames, or error messages with BM25 keyword search
-- Find conceptually related content with vector search
-- Combine keyword and semantic results with hybrid search
-- Search a large set of files without reading each file in full
-- Index content from files, databases, or APIs
+Queries read that index, not the live filesystem. This separation also lets you index content from databases, APIs, or other application sources.
 
 ## Quickstart
 
-Enable BM25 search, index content, and query it through the workspace:
+Enable BM25 keyword search, add a document to the index, and search it:
 
 ```typescript
-import { Workspace, LocalFilesystem } from '@mastra/core/workspace'
+import { LocalFilesystem, Workspace } from '@mastra/core/workspace'
 
 const workspace = new Workspace({
   filesystem: new LocalFilesystem({ basePath: './workspace' }),
@@ -31,53 +21,34 @@ const workspace = new Workspace({
 await workspace.index('/docs/guide.md', 'Reset passwords from the account settings page.')
 
 const results = await workspace.search('password reset')
-console.log(results)
-```
-
-This configuration also gives agents tools for searching and indexing workspace content.
-
-## How it works
-
-Workspace search has two phases: indexing and querying.
 
-### Indexing
-
-Content must be indexed before it can be searched. When you index a document:
-
-- The content is tokenized (split into searchable terms)
-- For BM25: term frequencies and document statistics are computed
-- For vector: the content is embedded using your embedder function and stored in the vector store
-
-Each indexed document has:
-
-- **id** - A unique identifier (typically the file path)
-- **content** - The text content
-- **metadata** - Optional key-value data stored with the document
+for (const result of results) {
+  console.log(`${result.id}: ${result.score}`)
+  console.log(result.content)
+}
+```
 
-### Querying
+The first argument to `workspace.index()` is the document ID returned in search results. It looks like a file path in this example, but `index()` doesn't read or create that file.
 
-When you search:
+Configuring search also gives agents search and indexing tools. See [Agent tools](#agent-tools) to control them.
 
-1. The query is processed using the same tokenization/embedding as indexing
-2. Documents are scored based on relevance to the query
-3. Results are ranked by score and returned with the matching content
+## Choose a search mode
 
-Workspaces support three search modes: BM25 keyword search and vector semantic search, plus hybrid search that combines both.
+Mastra supports three search modes:
 
-## BM25 keyword search
+| Mode     | Best for                                 | Example queries                             |
+| -------- | ---------------------------------------- | ------------------------------------------- |
+| `bm25`   | Exact terms, technical queries, and code | "useState hook", "404 error", "config.yaml" |
+| `vector` | Concepts and natural-language questions  | "how to handle user authentication"         |
+| `hybrid` | A mix of exact and conceptual queries    | Most agent search use cases                 |
 
-BM25 scores documents based on term frequency and document length. It works well for exact matches and specific terminology.
+If you don't pass a mode to `workspace.search()`, Mastra uses hybrid search when both BM25 and vector search are configured. Otherwise, it uses the configured mode.
 
-```typescript
-import { Workspace, LocalFilesystem } from '@mastra/core/workspace'
+### BM25 keyword search
 
-const workspace = new Workspace({
-  filesystem: new LocalFilesystem({ basePath: './workspace' }),
-  bm25: true,
-})
-```
+BM25 scores documents by term frequency and document length. It needs no external service or embedding model. Pass `bm25: true` to use the defaults shown in the quickstart.
 
-For custom BM25 parameters (`k1` is term frequency saturation, `b` is document length normalization):
+To tune term-frequency saturation and document-length normalization, pass `k1` and `b`:
 
 ```typescript
 const workspace = new Workspace({
@@ -89,21 +60,21 @@ const workspace = new Workspace({
 })
 ```
 
-## Vector search
+### Vector search
 
-Vector search uses embeddings to find semantically similar content. It requires a vector store and embedder function.
+Vector search uses embeddings to find semantically similar content. Configure a vector store and a function that embeds one string at a time:
 
 ```typescript
+import { openai } from '@ai-sdk/openai'
 import { Workspace, LocalFilesystem } from '@mastra/core/workspace'
 import { PineconeVector } from '@mastra/pinecone'
 import { embed } from 'ai'
-import { openai } from '@ai-sdk/openai'
 
 const workspace = new Workspace({
   filesystem: new LocalFilesystem({ basePath: './workspace' }),
   vectorStore: new PineconeVector({
-    apiKey: process.env.PINECONE_API_KEY,
-    index: 'workspace-index',
+    id: 'workspace-search',
+    apiKey: process.env.PINECONE_API_KEY!,
   }),
   embedder: async (text: string) => {
     const { embedding } = await embed({
@@ -115,30 +86,42 @@ const workspace = new Workspace({
 })
 ```
 
-### Batch embedding
+#### Batch embedding
+
+A single-text embedder makes a separate provider call for each document. For large indexes, use a provider's batch API to embed several documents in one call.
+
+A batch embedder must:
+
+- Accept an array of strings.
+- Return one embedding per string, in the same order.
+- Have a `batch: true` property so Mastra can detect it at runtime.
+- Optionally set `maxBatchSize` to the largest array the provider accepts.
 
-The embedder above takes one text at a time. Indexing a workspace with hundreds of files calls the provider hundreds of times, which is slow and expensive.
+Mastra splits larger indexing sets according to `maxBatchSize` and can process the resulting groups concurrently. Set this value to the provider's documented limit:
 
-When the provider supports batching (for example, OpenAI's `embedMany`), pass an embedder that takes an array of texts and accepts many embeddings back in one call. To opt in, set a `batch: true` property on the function. Mastra checks for that property at runtime and switches to the batched path.
+| Provider | Maximum inputs |
+| -------- | -------------- |
+| OpenAI   | 2048           |
+| Cohere   | 96             |
+| Voyage   | 128            |
 
-The following example replaces the single-text embedder with a batched one. The embedder function takes an array and returns an array of embeddings in the same order, plus carries two extra properties:
+If you omit `maxBatchSize`, Mastra uses its internal batch size.
 
-- `batch: true`: marks the function as batch-capable. Without this property, Mastra calls it one text at a time.
-- `maxBatchSize`: the largest array the provider accepts in one call. Mastra splits larger requests into chunks of this size and sends them in parallel. Set this to your provider's documented limit. For example, OpenAI accepts 2048 and Cohere accepts 96. Voyage accepts 128. Omit it to send every pending text in one request.
+Replace the single-text embedder with a batch embedder:
 
 ```typescript
-import { Workspace, LocalFilesystem } from '@mastra/core/workspace'
+import { openai } from '@ai-sdk/openai'
+import { LocalFilesystem, Workspace } from '@mastra/core/workspace'
 import { PineconeVector } from '@mastra/pinecone'
 import { embedMany } from 'ai'
-import { openai } from '@ai-sdk/openai'
 
 const model = openai.embedding('text-embedding-3-small')
 
 const workspace = new Workspace({
   filesystem: new LocalFilesystem({ basePath: './workspace' }),
   vectorStore: new PineconeVector({
-    apiKey: process.env.PINECONE_API_KEY,
-    index: 'workspace-index',
+    id: 'workspace-search',
+    apiKey: process.env.PINECONE_API_KEY!,
   }),
   embedder: Object.assign(
     async (texts: string[]) => {
@@ -150,13 +133,11 @@ const workspace = new Workspace({
 })
 ```
 
-`Object.assign` adds the `batch` and `maxBatchSize` properties to the embedder function. Mastra reads them as metadata and never passes them to the provider.
-
-Single-text embedders still work. The function signature `(text: string) => Promise<number[]>` is unchanged, so existing code keeps running without modification.
+`Object.assign()` adds `batch` and `maxBatchSize` as properties on the function. Mastra reads them as metadata and doesn't pass them to the provider. Single-text embedders with the `(text: string) => Promise<number[]>` signature remain supported.
 
-## Hybrid search
+### Hybrid search
 
-Configure both BM25 and vector search to enable hybrid mode, which combines keyword matching with semantic understanding.
+Configure both BM25 and vector search to combine keyword and semantic results:
 
 ```typescript
 const workspace = new Workspace({
@@ -167,31 +148,30 @@ const workspace = new Workspace({
 })
 ```
 
-## Custom index name
+### Custom index name
 
-By default, the search index name is derived from the workspace ID. To set a custom name, use `searchIndexName`:
+The vector index name defaults to a sanitized version of the workspace ID followed by `_search`. Set `searchIndexName` when you need a stable or shared name:
 
 ```typescript
 const workspace = new Workspace({
   filesystem: new LocalFilesystem({ basePath: './workspace' }),
-  bm25: true,
+  vectorStore: pineconeVector,
+  embedder: embedderFn,
   searchIndexName: 'my_workspace_vectors',
 })
 ```
 
-The index name must be a valid SQL identifier: start with a letter or underscore, contain only letters, numbers, or shows, and be at most 63 characters long.
+The name must start with a letter or `_`, contain only letters, numbers, or `_`, and contain at most 63 characters.
 
-## Indexing content
+## Index content
 
-### Manual indexing
+Use `workspace.index()` to add supplied text to the search index. Mastra tokenizes the content for BM25 search and generates an embedding when vector search is configured.
 
-Use `workspace.index()` to add content to the search index programmatically. The file paths become document IDs. You can also pass metadata for each document.
+The document ID doesn't need to exist in the filesystem. Add metadata when results need filtering or more context:
 
 ```typescript
-// Basic indexing
 await workspace.index('/docs/guide.md', 'Content of the guide...')
 
-// Index with metadata for filtering or context
 await workspace.index('/docs/api.md', apiDocContent, {
   metadata: {
     category: 'api',
@@ -200,114 +180,81 @@ await workspace.index('/docs/api.md', apiDocContent, {
 })
 ```
 
-Manual indexing is useful when:
+Manual indexing works well for database records and API responses, or when you need to preprocess, chunk, or annotate content yourself.
 
-- You're indexing content that doesn't come from files (e.g., database records, API responses)
-- You want to pre-process or chunk content before indexing
-- You need to add custom metadata to documents
+### Keep files and the index in sync
 
-### Auto-indexing
+Filesystem and index mutations are independent. Writing, editing, or deleting a file doesn't update its indexed content, and `workspace.index()` doesn't read or change a file.
 
-Configure `autoIndexPaths` to automatically index files when the workspace initializes. Each entry can be a directory path (indexed recursively) or a glob pattern for selective indexing.
-
-`autoIndexPaths` works with a static `filesystem` or with `mounts`. For mounts, include the mount prefix in each path, such as `/docs/**/*.md`. Resolver-backed filesystems aren't auto-indexed during workspace initialization because the provider is only selected for a request; index their content manually instead.
+To write a file and make it searchable, perform both operations:
 
 ```typescript
-const workspace = new Workspace({
-  filesystem: new LocalFilesystem({ basePath: './workspace' }),
-  bm25: true,
-  autoIndexPaths: ['docs', 'support/faq'],
-})
-
-await workspace.init()
-```
+const path = 'notes/launch.md'
+const content = '# Launch notes\n\nShip the new dashboard on Friday.'
+const filesystem = workspace.filesystem
 
-When `init()` is called, all matching files are read and indexed for search. The file path becomes the document ID.
-
-Glob patterns let you index specific file types:
+if (!filesystem) {
+  throw new Error('This operation requires a static filesystem')
+}
 
-```typescript
-const workspace = new Workspace({
-  filesystem: new LocalFilesystem({ basePath: './workspace' }),
-  bm25: true,
-  autoIndexPaths: ['docs/**/*.md', 'support/**/*.txt'],
-})
+await filesystem.writeFile(path, content)
+await workspace.index(path, content)
 ```
 
-## Searching
-
-Use `workspace.search()` to find relevant content. Results are ranked by relevance score.
-
-```typescript
-const results = await workspace.search('password reset')
-
-for (const result of results) {
-  console.log(`${result.id}: ${result.score}`)
-  console.log(result.content)
-}
-```
+After editing a file, index its complete updated content again. Deleting a file leaves its indexed document in place, and Mastra doesn't currently expose a public method for removing one indexed document. Use the `grep` file tool instead when an agent must search the current filesystem state without maintaining a separate index.
 
-### Search options
+## Search content
 
-You can customize the search behavior with options:
+Use `workspace.search()` to return documents ranked by relevance:
 
 ```typescript
 const results = await workspace.search('authentication flow', {
   topK: 10,
   mode: 'hybrid',
-  minScore: 0.5,
   vectorWeight: 0.5,
 })
 ```
 
-| Option         | Description                                                                                                   |
-| -------------- | ------------------------------------------------------------------------------------------------------------- |
-| `topK`         | Maximum number of results to return. Default: 5                                                               |
-| `mode`         | Search mode: `'bm25'`, `'vector'`, or `'hybrid'`. Defaults to the best available mode based on configuration. |
-| `minScore`     | Filter out results below this score threshold (0-1).                                                          |
-| `vectorWeight` | In hybrid mode, how much to weight vector scores vs BM25. 0 = all BM25, 1 = all vector, 0.5 = equal.          |
+| Option         | Description                                                                                                                                                    |
+| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `topK`         | Maximum number of results. Direct API calls default to 10. The agent search tool defaults to 5.                                                                |
+| `mode`         | Search mode: `'bm25'`, `'vector'`, or `'hybrid'`. Defaults to the best available mode for the workspace configuration.                                         |
+| `minScore`     | Removes results below this score. The scale depends on the mode, vector provider, and indexed content.                                                         |
+| `vectorWeight` | In hybrid mode, the weight given to vector scores. `0` gives vector scores zero weight, `1` gives BM25 scores zero weight, and `0.5` weights both equally.     |
+| `filter`       | Vector-store metadata filter used by vector retrieval. In hybrid mode, it doesn't filter BM25-only results. The syntax depends on the configured vector store. |
 
-### Search results
-
-Each result contains:
+Each result contains the matching content and its score:
 
 ```typescript
 interface SearchResult {
-  id: string // Document ID (typically file path)
-  content: string // The matching content
-  score: number // Relevance score (0-1)
+  id: string
+  content: string
+  score: number
   lineRange?: {
-    // Lines where the match was found
     start: number
     end: number
   }
-  metadata?: Record<string, unknown> // Metadata stored with the document
+  metadata?: Record<string, unknown>
   scoreDetails?: {
-    // Score breakdown (hybrid mode only)
     vector?: number
     bm25?: number
   }
 }
 ```
 
-**Understanding scores:**
-
-- Scores range from 0 to 1, where 1 is a perfect match
-- BM25 scores are normalized based on the best match in the result set
-- Vector scores represent cosine similarity between query and document embeddings
-- In hybrid mode, scores are combined using the `vectorWeight` parameter
+Score scales aren't interchangeable:
 
-### When to use each mode
-
-| Mode     | Best for                             | Example queries                                                          |
-| -------- | ------------------------------------ | ------------------------------------------------------------------------ |
-| `bm25`   | Exact terms, technical queries, code | "useState hook", "404 error", "config.yaml"                              |
-| `vector` | Conceptual queries, natural language | "how to handle user authentication", "best practices for error handling" |
-| `hybrid` | General search, unknown query types  | Most agent use cases                                                     |
+- Standalone BM25 search returns raw BM25 scores, which aren't limited to 0 through 1.
+- Vector score meaning depends on the configured vector store.
+- Hybrid search min-max normalizes BM25 for its weighted calculation but uses the vector provider's score unchanged. The combined score isn't clamped to 0 through 1.
+- `scoreDetails` keeps the raw component scores, including the unnormalized BM25 score.
+- Tune `minScore` for the active mode, vector provider, and indexed content instead of reusing one threshold across indexes.
 
 ## Agent tools
 
-When you configure search on a workspace, agents receive `mastra_workspace_search` and `mastra_workspace_index` tools. Use `WORKSPACE_TOOLS.SEARCH` constants to configure them independently. For example, remove indexing when the agent should search existing content without changing the index:
+When search is configured, agents receive `mastra_workspace_search` and `mastra_workspace_index`. Configure them independently with `WORKSPACE_TOOLS.SEARCH`.
+
+Disable indexing when an agent should search existing content without changing the index:
 
 ```typescript
 import { LocalFilesystem, Workspace, WORKSPACE_TOOLS } from '@mastra/core/workspace'
@@ -323,10 +270,82 @@ const workspace = new Workspace({
 })
 ```
 
-See [workspace class reference](https://mastra.ai/reference/workspace/workspace-class) for the complete tool list and [`WorkspaceToolsConfig`](https://mastra.ai/reference/workspace/workspace-class) for shared settings.
+The index tool accepts the same supplied `path`, `content`, and optional metadata as `workspace.index()`. Because it can trigger embedding calls and vector-store writes, require approval when agents shouldn't mutate the index without review:
+
+```typescript
+const workspace = new Workspace({
+  filesystem: new LocalFilesystem({ basePath: './workspace' }),
+  bm25: true,
+  tools: {
+    [WORKSPACE_TOOLS.SEARCH.INDEX]: {
+      requireApproval: true,
+    },
+  },
+})
+```
+
+`requireReadBeforeWrite` applies to filesystem write tools, not index mutations. Mastra also excludes the index tool when a static filesystem is read-only, even though indexing doesn't write to that filesystem.
+
+See the [search tool reference](https://mastra.ai/reference/workspace/workspace-class) for the complete tool list and the [tool configuration reference](https://mastra.ai/reference/workspace/workspace-class) for shared settings.
+
+## Auto-indexing
+
+Use `autoIndexPaths` to build an index from static filesystem content during application startup.
+
+> **Warning:** `autoIndexPaths` runs as part of `workspace.init()`. Passing the workspace to `new Mastra({ workspace })` registers it but doesn't initialize it or build the index. Call and await `workspace.init()` unless another runtime, such as `AgentController`, owns workspace initialization.
+>
+> ```typescript
+> await workspace.init()
+> ```
+>
+> When your application owns initialization, call it once during startup before serving requests.
+
+In a normal Mastra application, construct `Mastra` first so its logger is available to the workspace. Then initialize the workspace at module startup:
+
+```typescript
+import { Mastra } from '@mastra/core'
+import { LocalFilesystem, Workspace } from '@mastra/core/workspace'
+
+const workspace = new Workspace({
+  filesystem: new LocalFilesystem({ basePath: './workspace' }),
+  bm25: true,
+  autoIndexPaths: ['docs', 'support/faq'],
+})
+
+export const mastra = new Mastra({ workspace })
+
+await workspace.init()
+```
+
+Top-level `await` keeps module startup pending until initialization finishes. When `init()` resolves, Mastra has attempted to read and index every matching file.
+
+Each `autoIndexPaths` entry can be a file, directory, or glob pattern. Recursive directory and glob traversal is limited to ten levels:
+
+```typescript
+const workspace = new Workspace({
+  filesystem: new LocalFilesystem({ basePath: './workspace' }),
+  bm25: true,
+  autoIndexPaths: ['docs/**/*.md', 'support/**/*.txt'],
+})
+
+export const mastra = new Mastra({ workspace })
+
+await workspace.init()
+```
+
+Auto-indexing supports a static `filesystem` or `mounts`. Include mount prefixes in paths, such as `/docs/**/*.md`. It doesn't support a filesystem resolver because no provider has been selected when the workspace initializes. Index resolver-backed content manually instead.
+
+Initialization performs one scan and doesn't watch for later changes. It splits large files into chunks before indexing them and generates embeddings in vector mode.
+
+Keep the configured paths narrow and avoid dependency or output directories. Don't include binary file trees. Use a batch embedder for large vector indexes.
+
+Calling `workspace.init()` again scans the paths again and clears the process-local BM25 index before rebuilding it. Don't call it from an agent call, tool, workflow step, or request handler.
+
+A persistent vector index behaves differently. Auto-indexing replaces the files it reads but can retain documents for files removed before a later scan. Use a stable workspace `id` or `searchIndexName` when vectors must survive process restarts, and plan cleanup around the vector store's persistence behavior.
 
 ## Related
 
-- [Sandbox](https://mastra.ai/docs/sandbox/overview)
-- [RAG overview](https://mastra.ai/reference/rag/overview)
-- [Workspace class reference](https://mastra.ai/reference/workspace/workspace-class)
+- [Sandboxes](https://mastra.ai/docs/sandbox/overview)
+- [Filesystem](https://mastra.ai/docs/sandbox/filesystem)
+- [Retrieval-Augmented Generation overview](https://mastra.ai/reference/rag/overview)
+- [Workspace configuration reference](https://mastra.ai/reference/workspace/workspace-class)