@mastra/pg 1.2.0-alpha.0 → 1.3.0-alpha.0

package/dist/docs/references/reference-processors-message-history-processor.md

@@ -0,0 +1,85 @@
+# MessageHistory
+
+The `MessageHistory` is a **hybrid processor** that handles both retrieval and persistence of message history. On input, it fetches historical messages from storage and prepends them to the conversation. On output, it persists new messages to storage.
+
+## Usage example
+
+```typescript
+import { MessageHistory } from "@mastra/core/processors";
+
+const processor = new MessageHistory({
+  storage: memoryStorage,
+  lastMessages: 50,
+});
+```
+
+## Constructor parameters
+
+**options:** (`MessageHistoryOptions`): Configuration options for the message history processor
+
+### Options
+
+**storage:** (`MemoryStorage`): Storage instance for retrieving and persisting messages
+
+**lastMessages?:** (`number`): Maximum number of historical messages to retrieve. If not specified, retrieves all messages
+
+## Returns
+
+**id:** (`string`): Processor identifier set to 'message-history'
+
+**name:** (`string`): Processor display name set to 'MessageHistory'
+
+**processInput:** (`(args: { messages: MastraDBMessage[]; messageList: MessageList; abort: (reason?: string) => never; tracingContext?: TracingContext; requestContext?: RequestContext }) => Promise<MessageList | MastraDBMessage[]>`): Fetches historical messages from storage and adds them to the message list
+
+**processOutputResult:** (`(args: { messages: MastraDBMessage[]; messageList: MessageList; abort: (reason?: string) => never; tracingContext?: TracingContext; requestContext?: RequestContext }) => Promise<MessageList>`): Persists new messages (user input and assistant response) to storage, excluding system messages
+
+## Extended usage example
+
+```typescript
+import { Agent } from "@mastra/core/agent";
+import { MessageHistory } from "@mastra/core/processors";
+import { PostgresStorage } from "@mastra/pg";
+
+const storage = new PostgresStorage({
+  connectionString: process.env.DATABASE_URL,
+});
+
+export const agent = new Agent({
+  name: "memory-agent",
+  instructions: "You are a helpful assistant with conversation memory",
+  model: "openai:gpt-4o",
+  inputProcessors: [
+    new MessageHistory({
+      storage,
+      lastMessages: 100,
+    }),
+  ],
+  outputProcessors: [
+    new MessageHistory({
+      storage,
+    }),
+  ],
+});
+```
+
+## Behavior
+
+### Input processing
+
+1. Retrieves `threadId` from the request context
+2. Fetches historical messages from storage (ordered by creation date, descending)
+3. Filters out system messages (they should not be stored in the database)
+4. Merges historical messages with incoming messages, avoiding duplicates by ID
+5. Adds historical messages with `source: 'memory'` tag
+
+### Output processing
+
+1. Retrieves `threadId` from the request context
+2. Skips persistence if `readOnly` is set in memory config
+3. Filters out incomplete tool calls from messages
+4. Persists new user input and assistant response messages to storage
+5. Updates the thread's `updatedAt` timestamp
+
+## Related
+
+- [Guardrails](https://mastra.ai/docs/agents/guardrails)

package/dist/docs/references/reference-processors-semantic-recall-processor.md

@@ -0,0 +1,123 @@
+# SemanticRecall
+
+The `SemanticRecall` is a **hybrid processor** that enables semantic search over conversation history using vector embeddings. On input, it performs semantic search to find relevant historical messages. On output, it creates embeddings for new messages to enable future semantic retrieval.
+
+## Usage example
+
+```typescript
+import { SemanticRecall } from "@mastra/core/processors";
+import { openai } from "@ai-sdk/openai";
+
+const processor = new SemanticRecall({
+  storage: memoryStorage,
+  vector: vectorStore,
+  embedder: openai.embedding("text-embedding-3-small"),
+  topK: 5,
+  messageRange: 2,
+  scope: "resource",
+});
+```
+
+## Constructor parameters
+
+**options:** (`SemanticRecallOptions`): Configuration options for the semantic recall processor
+
+### Options
+
+**storage:** (`MemoryStorage`): Storage instance for retrieving messages
+
+**vector:** (`MastraVector`): Vector store for semantic search
+
+**embedder:** (`MastraEmbeddingModel<string>`): Embedder for generating query embeddings
+
+**topK?:** (`number`): Number of most similar messages to retrieve
+
+**messageRange?:** (`number | { before: number; after: number }`): Number of context messages to include before/after each match. Can be a single number (same for both) or an object with separate values
+
+**scope?:** (`'thread' | 'resource'`): Scope of semantic search. 'thread' searches within the current thread only. 'resource' searches across all threads for the resource
+
+**threshold?:** (`number`): Minimum similarity score threshold (0-1). Messages below this threshold are filtered out
+
+**indexName?:** (`string`): Index name for the vector store. If not provided, auto-generated based on embedder model
+
+**logger?:** (`IMastraLogger`): Optional logger instance for structured logging
+
+## Returns
+
+**id:** (`string`): Processor identifier set to 'semantic-recall'
+
+**name:** (`string`): Processor display name set to 'SemanticRecall'
+
+**processInput:** (`(args: { messages: MastraDBMessage[]; messageList: MessageList; abort: (reason?: string) => never; tracingContext?: TracingContext; requestContext?: RequestContext }) => Promise<MessageList | MastraDBMessage[]>`): Performs semantic search on historical messages and adds relevant context to the message list
+
+**processOutputResult:** (`(args: { messages: MastraDBMessage[]; messageList?: MessageList; abort: (reason?: string) => never; tracingContext?: TracingContext; requestContext?: RequestContext }) => Promise<MessageList | MastraDBMessage[]>`): Creates embeddings for new messages to enable future semantic search
+
+## Extended usage example
+
+```typescript
+import { Agent } from "@mastra/core/agent";
+import { SemanticRecall, MessageHistory } from "@mastra/core/processors";
+import { PostgresStorage } from "@mastra/pg";
+import { PgVector } from "@mastra/pg";
+import { openai } from "@ai-sdk/openai";
+
+const storage = new PostgresStorage({
+  id: 'pg-storage',
+  connectionString: process.env.DATABASE_URL,
+});
+
+const vector = new PgVector({
+  id: 'pg-vector',
+  connectionString: process.env.DATABASE_URL,
+});
+
+const semanticRecall = new SemanticRecall({
+  storage,
+  vector,
+  embedder: openai.embedding("text-embedding-3-small"),
+  topK: 5,
+  messageRange: { before: 2, after: 1 },
+  scope: "resource",
+  threshold: 0.7,
+});
+
+export const agent = new Agent({
+  name: "semantic-memory-agent",
+  instructions: "You are a helpful assistant with semantic memory recall",
+  model: "openai:gpt-4o",
+  inputProcessors: [
+    semanticRecall,
+    new MessageHistory({ storage, lastMessages: 50 }),
+  ],
+  outputProcessors: [
+    semanticRecall,
+    new MessageHistory({ storage }),
+  ],
+});
+```
+
+## Behavior
+
+### Input processing
+
+1. Extracts the user query from the last user message
+2. Generates embeddings for the query
+3. Performs vector search to find semantically similar messages
+4. Retrieves matched messages along with surrounding context (based on `messageRange`)
+5. For `scope: 'resource'`, formats cross-thread messages as a system message with timestamps
+6. Adds recalled messages with `source: 'memory'` tag
+
+### Output processing
+
+1. Extracts text content from new user and assistant messages
+2. Generates embeddings for each message
+3. Stores embeddings in the vector store with metadata (message ID, thread ID, resource ID, role, content, timestamp)
+4. Uses LRU caching for embeddings to avoid redundant API calls
+
+### Cross-thread recall
+
+When `scope` is set to `'resource'`, the processor can recall messages from other threads. These cross-thread messages are formatted as a system message with timestamps and conversation labels to provide context about when and where the conversation occurred.
+
+## Related
+
+- [Guardrails](https://mastra.ai/docs/agents/guardrails)

package/dist/docs/references/reference-processors-working-memory-processor.md

@@ -0,0 +1,154 @@
+# WorkingMemory
+
+The `WorkingMemory` is an **input processor** that injects working memory data as a system message. It retrieves persistent information from storage and formats it as instructions for the LLM, enabling the agent to maintain context about users across conversations.
+
+## Usage example
+
+```typescript
+import { WorkingMemory } from "@mastra/core/processors";
+
+const processor = new WorkingMemory({
+  storage: memoryStorage,
+  scope: "resource",
+  template: {
+    format: "markdown",
+    content: `# User Profile
+- **Name**:
+- **Preferences**:
+- **Goals**:
+`,
+  },
+});
+```
+
+## Constructor parameters
+
+**options:** (`Options`): Configuration options for the working memory processor
+
+### Options
+
+**storage:** (`MemoryStorage`): Storage instance for retrieving working memory data
+
+**template?:** (`WorkingMemoryTemplate`): Template defining the format and structure of working memory
+
+**scope?:** (`'thread' | 'resource'`): Scope of working memory. 'thread' scopes to current thread, 'resource' shares across all threads for the resource
+
+**useVNext?:** (`boolean`): Use the next-generation instruction format with improved guidelines
+
+**readOnly?:** (`boolean`): When true, working memory is provided as read-only context. The data is injected into the conversation but without the updateWorkingMemory tool or update instructions. Useful for agents that should reference working memory without modifying it.
+
+**templateProvider?:** (`{ getWorkingMemoryTemplate(args: { memoryConfig?: MemoryConfig }): Promise<WorkingMemoryTemplate | null> }`): Dynamic template provider for runtime template resolution
+
+**logger?:** (`IMastraLogger`): Optional logger instance for structured logging
+
+### WorkingMemoryTemplate
+
+**format:** (`'markdown' | 'json'`): Format of the working memory content
+
+**content:** (`string`): Template content defining the structure of working memory data
+
+## Returns
+
+**id:** (`string`): Processor identifier set to 'working-memory'
+
+**name:** (`string`): Processor display name set to 'WorkingMemory'
+
+**defaultWorkingMemoryTemplate:** (`string`): The default markdown template used when no custom template is provided
+
+**processInput:** (`(args: { messages: MastraDBMessage[]; messageList: MessageList; abort: (reason?: string) => never; requestContext?: RequestContext }) => Promise<MessageList | MastraDBMessage[]>`): Retrieves working memory and adds it as a system message to the message list
+
+## Extended usage example
+
+```typescript
+import { Agent } from "@mastra/core/agent";
+import { WorkingMemory, MessageHistory } from "@mastra/core/processors";
+import { PostgresStorage } from "@mastra/pg";
+
+const storage = new PostgresStorage({
+  connectionString: process.env.DATABASE_URL,
+});
+
+export const agent = new Agent({
+  name: "personalized-agent",
+  instructions: "You are a helpful assistant that remembers user preferences",
+  model: "openai:gpt-4o",
+  inputProcessors: [
+    new WorkingMemory({
+      storage,
+      scope: "resource",
+      template: {
+        format: "markdown",
+        content: `# User Information
+- **Name**:
+- **Location**:
+- **Preferences**:
+- **Communication Style**:
+- **Current Projects**:
+`,
+      },
+    }),
+    new MessageHistory({ storage, lastMessages: 50 }),
+  ],
+  outputProcessors: [
+    new MessageHistory({ storage }),
+  ],
+});
+```
+
+## JSON format example
+
+```typescript
+import { WorkingMemory } from "@mastra/core/processors";
+
+const processor = new WorkingMemory({
+  storage: memoryStorage,
+  scope: "resource",
+  template: {
+    format: "json",
+    content: JSON.stringify({
+      user: {
+        name: { type: "string" },
+        preferences: { type: "object" },
+        goals: { type: "array" },
+      },
+    }),
+  },
+});
+```
+
+## Behavior
+
+### Input processing
+
+1. Retrieves `threadId` and `resourceId` from the request context
+
+2. Based on scope, fetches working memory from either:
+
+   - Thread metadata (`scope: 'thread'`)
+   - Resource record (`scope: 'resource'`)
+
+3. Resolves the template (from provider, options, or default)
+
+4. Generates system instructions based on mode:
+
+   - **Normal mode**: Includes guidelines for storing/updating information, template structure, and current data
+   - **Read-only mode** (`readOnly: true`): Includes only the current data as context without update instructions
+
+5. Adds the instruction as a system message with `source: 'memory'` tag
+
+### Working memory updates
+
+Working memory updates happen through the `updateWorkingMemory` tool provided by the Memory class, not through this processor. The processor only handles injecting the current working memory state into conversations.
+
+### Default template
+
+If no template is provided, the processor uses a default markdown template with fields for:
+
+- First Name, Last Name
+- Location, Occupation
+- Interests, Goals
+- Events, Facts, Projects
+
+## Related
+
+- [Guardrails](https://mastra.ai/docs/agents/guardrails)

package/dist/docs/{rag/04-reference.md → references/reference-rag-metadata-filters.md}

@@ -1,13 +1,4 @@
-# Rag API Reference
-
-> API reference for rag - 1 entries
-
-
----
-
-## Reference: Metadata Filters
-
-> Documentation for metadata filtering capabilities in Mastra, which allow for precise querying of vector search results across different vector stores.
+# Metadata Filters
 
 Mastra provides a unified metadata filtering syntax across all vector stores, based on MongoDB/Sift query syntax. Each vector store translates these filters into their native format.
 
@@ -35,188 +26,42 @@ const results = await store.query({
 
 ## Supported Operators
 
-<OperatorsTable
-  title="Basic Comparison"
-  operators={[
-    {
-      name: "$eq",
-      description: "Matches values equal to specified value",
-      example: "{ age: { $eq: 25 } }",
-      supportedBy: ["All except Couchbase"],
-    },
-    {
-      name: "$ne",
-      description: "Matches values not equal",
-      example: "{ status: { $ne: 'inactive' } }",
-      supportedBy: ["All except Couchbase"],
-    },
-    {
-      name: "$gt",
-      description: "Greater than",
-      example: "{ price: { $gt: 100 } }",
-      supportedBy: ["All except Couchbase"],
-    },
-    {
-      name: "$gte",
-      description: "Greater than or equal",
-      example: "{ rating: { $gte: 4.5 } }",
-      supportedBy: ["All except Couchbase"],
-    },
-    {
-      name: "$lt",
-      description: "Less than",
-      example: "{ stock: { $lt: 20 } }",
-      supportedBy: ["All except Couchbase"],
-    },
-    {
-      name: "$lte",
-      description: "Less than or equal",
-      example: "{ priority: { $lte: 3 } }",
-      supportedBy: ["All except Couchbase"],
-    },
-  ]}
-/>
-
-<OperatorsTable
-  title="Array Operators"
-  operators={[
-    {
-      name: "$in",
-      description: "Matches any value in array",
-      example: '{ category: { $in: ["A", "B"] } }',
-      supportedBy: ["All except Couchbase"],
-    },
-    {
-      name: "$nin",
-      description: "Matches none of the values",
-      example: '{ status: { $nin: ["deleted", "archived"] } }',
-      supportedBy: ["All except Couchbase"],
-    },
-    {
-      name: "$all",
-      description: "Matches arrays containing all elements",
-      example: '{ tags: { $all: ["urgent", "high"] } }',
-      supportedBy: ["Astra", "Pinecone", "Upstash", "MongoDB"],
-    },
-    {
-      name: "$elemMatch",
-      description: "Matches array elements meeting criteria",
-      example: "{ scores: { $elemMatch: { $gt: 80 } } }",
-      supportedBy: ["libSQL", "PgVector", "MongoDB"],
-    },
-  ]}
-/>
-
-<OperatorsTable
-  title="Logical Operators"
-  operators={[
-    {
-      name: "$and",
-      description: "Logical AND",
-      example: "{ $and: [{ price: { $gt: 100 } }, { stock: { $gt: 0 } }] }",
-      supportedBy: ["All except Vectorize, Couchbase"],
-    },
-    {
-      name: "$or",
-      description: "Logical OR",
-      example: '{ $or: [{ status: "active" }, { priority: "high" }] }',
-      supportedBy: ["All except Vectorize, Couchbase"],
-    },
-    {
-      name: "$not",
-      description: "Logical NOT",
-      example: "{ price: { $not: { $lt: 100 } } }",
-      supportedBy: [
-        "Astra",
-        "Qdrant",
-        "Upstash",
-        "PgVector",
-        "libSQL",
-        "MongoDB",
-      ],
-    },
-    {
-      name: "$nor",
-      description: "Logical NOR",
-      example: '{ $nor: [{ status: "deleted" }, { archived: true }] }',
-      supportedBy: ["Qdrant", "Upstash", "PgVector", "libSQL", "MongoDB"],
-    },
-  ]}
-/>
-
-<OperatorsTable
-  title="Element Operators"
-  operators={[
-    {
-      name: "$exists",
-      description: "Matches documents with field",
-      example: "{ rating: { $exists: true } }",
-      supportedBy: ["All except Vectorize, Chroma, Couchbase"],
-    },
-  ]}
-/>
-
-<OperatorsTable
-  title="Custom Operators"
-  operators={[
-    {
-      name: "$contains",
-      description: "Text contains substring",
-      example: '{ description: { $contains: "sale" } }',
-      supportedBy: ["Upstash", "libSQL", "PgVector"],
-    },
-    {
-      name: "$regex",
-      description: "Regular expression match",
-      example: '{ name: { $regex: "^test" } }',
-      supportedBy: ["Qdrant", "PgVector", "Upstash", "MongoDB"],
-    },
-    {
-      name: "$size",
-      description: "Array length check",
-      example: "{ tags: { $size: { $gt: 2 } } }",
-      supportedBy: ["Astra", "libSQL", "PgVector", "MongoDB"],
-    },
-    {
-      name: "$geo",
-      description: "Geospatial query",
-      example: '{ location: { $geo: { type: "radius", ... } } }',
-      supportedBy: ["Qdrant"],
-    },
-    {
-      name: "$datetime",
-      description: "Datetime range query",
-      example: '{ created: { $datetime: { range: { gt: "2024-01-01" } } } }',
-      supportedBy: ["Qdrant"],
-    },
-    {
-      name: "$hasId",
-      description: "Vector ID existence check",
-      example: '{ $hasId: ["id1", "id2"] }',
-      supportedBy: ["Qdrant"],
-    },
-    {
-      name: "$hasVector",
-      description: "Vector existence check",
-      example: "{ $hasVector: true }",
-      supportedBy: ["Qdrant"],
-    },
-  ]}
-/>
+### Basic Comparison
+
+`$eq`Matches values equal to specified value{ age: { $eq: 25 } }Supported by: All except Couchbase`$ne`Matches values not equal{ status: { $ne: 'inactive' } }Supported by: All except Couchbase`$gt`Greater than{ price: { $gt: 100 } }Supported by: All except Couchbase`$gte`Greater than or equal{ rating: { $gte: 4.5 } }Supported by: All except Couchbase`$lt`Less than{ stock: { $lt: 20 } }Supported by: All except Couchbase`$lte`Less than or equal{ priority: { $lte: 3 } }Supported by: All except Couchbase
+
+### Array Operators
+
+`$in`Matches any value in array{ category: { $in: \["A", "B"] } }Supported by: All except Couchbase`$nin`Matches none of the values{ status: { $nin: \["deleted", "archived"] } }Supported by: All except Couchbase`$all`Matches arrays containing all elements{ tags: { $all: \["urgent", "high"] } }Supported by: Astra, Pinecone, Upstash, MongoDB`$elemMatch`Matches array elements meeting criteria{ scores: { $elemMatch: { $gt: 80 } } }Supported by: libSQL, PgVector, MongoDB
+
+### Logical Operators
+
+`$and`Logical AND{ $and: \[{ price: { $gt: 100 } }, { stock: { $gt: 0 } }] }Supported by: All except Vectorize, Couchbase`$or`Logical OR{ $or: \[{ status: "active" }, { priority: "high" }] }Supported by: All except Vectorize, Couchbase`$not`Logical NOT{ price: { $not: { $lt: 100 } } }Supported by: Astra, Qdrant, Upstash, PgVector, libSQL, MongoDB`$nor`Logical NOR{ $nor: \[{ status: "deleted" }, { archived: true }] }Supported by: Qdrant, Upstash, PgVector, libSQL, MongoDB
+
+### Element Operators
+
+`$exists`Matches documents with field{ rating: { $exists: true } }Supported by: All except Vectorize, Chroma, Couchbase
+
+### Custom Operators
+
+`$contains`Text contains substring{ description: { $contains: "sale" } }Supported by: Upstash, libSQL, PgVector`$regex`Regular expression match{ name: { $regex: "^test" } }Supported by: Qdrant, PgVector, Upstash, MongoDB`$size`Array length check{ tags: { $size: { $gt: 2 } } }Supported by: Astra, libSQL, PgVector, MongoDB`$geo`Geospatial query{ location: { $geo: { type: "radius", ... } } }Supported by: Qdrant`$datetime`Datetime range query{ created: { $datetime: { range: { gt: "2024-01-01" } } } }Supported by: Qdrant`$hasId`Vector ID existence check{ $hasId: \["id1", "id2"] }Supported by: Qdrant`$hasVector`Vector existence check{ $hasVector: true }Supported by: Qdrant
 
 ## Common Rules and Restrictions
 
 1. Field names cannot:
+
    - Contain dots (.) unless referring to nested fields
    - Start with $ or contain null characters
    - Be empty strings
 
 2. Values must be:
+
    - Valid JSON types (string, number, boolean, object, array)
    - Not undefined
    - Properly typed for the operator (e.g., numbers for numeric comparisons)
 
 3. Logical operators:
+
    - Must contain valid conditions
    - Cannot be empty
    - Must be properly nested
@@ -229,6 +74,7 @@ const results = await store.query({
    - Invalid: `{ "field": { "$gt": { "$and": [{...}] } } }`
 
 4. $not operator:
+
    - Must be an object
    - Cannot be empty
    - Can be used at field level or top level
@@ -236,6 +82,7 @@ const results = await store.query({
    - Valid: `{ "field": { "$not": { "$eq": "value" } } }`
 
 5. Operator nesting:
+
    - Logical operators must contain field conditions, not direct operators
    - Valid: `{ "$and": [{ "field": { "$gt": 100 } }] }`
    - Invalid: `{ "$and": [{ "$gt": 100 }] }`
@@ -265,7 +112,7 @@ const results = await store.query({
 - Field names cannot contain dots (.) or start with $
 - Field names limited to 512 characters
 - Vectors must be re-upserted after creating new metadata indexes to be included in filtered results
-- Range queries may have reduced accuracy with very large datasets (~10M+ vectors)
+- Range queries may have reduced accuracy with very large datasets (\~10M+ vectors)
 
 ### libSQL