@mastra/pg 1.6.1 → 1.7.0

package/dist/docs/references/reference-rag-metadata-filters.md

@@ -0,0 +1,216 @@
+# Metadata Filters
+
+Mastra provides a unified metadata filtering syntax across all vector stores, based on MongoDB/Sift query syntax. Each vector store translates these filters into their native format.
+
+## Basic Example
+
+```typescript
+import { PgVector } from '@mastra/pg'
+
+const store = new PgVector({
+  id: 'pg-vector',
+  connectionString,
+})
+
+const results = await store.query({
+  indexName: 'my_index',
+  queryVector: queryVector,
+  topK: 10,
+  filter: {
+    category: 'electronics', // Simple equality
+    price: { $gt: 100 }, // Numeric comparison
+    tags: { $in: ['sale', 'new'] }, // Array membership
+  },
+})
+```
+
+## Supported Operators
+
+### Basic Comparison
+
+`$eq`Matches values equal to specified value{ age: { $eq: 25 } }Supported by: All except Couchbase`$ne`Matches values not equal{ status: { $ne: 'inactive' } }Supported by: All except Couchbase`$gt`Greater than{ price: { $gt: 100 } }Supported by: All except Couchbase`$gte`Greater than or equal{ rating: { $gte: 4.5 } }Supported by: All except Couchbase`$lt`Less than{ stock: { $lt: 20 } }Supported by: All except Couchbase`$lte`Less than or equal{ priority: { $lte: 3 } }Supported by: All except Couchbase
+
+### Array Operators
+
+`$in`Matches any value in array{ category: { $in: \["A", "B"] } }Supported by: All except Couchbase`$nin`Matches none of the values{ status: { $nin: \["deleted", "archived"] } }Supported by: All except Couchbase`$all`Matches arrays containing all elements{ tags: { $all: \["urgent", "high"] } }Supported by: Astra, Pinecone, Upstash, MongoDB`$elemMatch`Matches array elements meeting criteria{ scores: { $elemMatch: { $gt: 80 } } }Supported by: libSQL, PgVector, MongoDB
+
+### Logical Operators
+
+`$and`Logical AND{ $and: \[{ price: { $gt: 100 } }, { stock: { $gt: 0 } }] }Supported by: All except Vectorize, Couchbase`$or`Logical OR{ $or: \[{ status: "active" }, { priority: "high" }] }Supported by: All except Vectorize, Couchbase`$not`Logical NOT{ price: { $not: { $lt: 100 } } }Supported by: Astra, Qdrant, Upstash, PgVector, libSQL, MongoDB`$nor`Logical NOR{ $nor: \[{ status: "deleted" }, { archived: true }] }Supported by: Qdrant, Upstash, PgVector, libSQL, MongoDB
+
+### Element Operators
+
+`$exists`Matches documents with field{ rating: { $exists: true } }Supported by: All except Vectorize, Chroma, Couchbase
+
+### Custom Operators
+
+`$contains`Text contains substring{ description: { $contains: "sale" } }Supported by: Upstash, libSQL, PgVector`$regex`Regular expression match{ name: { $regex: "^test" } }Supported by: Qdrant, PgVector, Upstash, MongoDB`$size`Array length check{ tags: { $size: { $gt: 2 } } }Supported by: Astra, libSQL, PgVector, MongoDB`$geo`Geospatial query{ location: { $geo: { type: "radius", ... } } }Supported by: Qdrant`$datetime`Datetime range query{ created: { $datetime: { range: { gt: "2024-01-01" } } } }Supported by: Qdrant`$hasId`Vector ID existence check{ $hasId: \["id1", "id2"] }Supported by: Qdrant`$hasVector`Vector existence check{ $hasVector: true }Supported by: Qdrant
+
+## Common Rules and Restrictions
+
+1. Field names cannot:
+
+   - Contain dots (.) unless referring to nested fields
+   - Start with $ or contain null characters
+   - Be empty strings
+
+2. Values must be:
+
+   - Valid JSON types (string, number, boolean, object, array)
+   - Not undefined
+   - Properly typed for the operator (e.g., numbers for numeric comparisons)
+
+3. Logical operators:
+
+   - Must contain valid conditions
+   - Cannot be empty
+   - Must be properly nested
+   - Can only be used at top level or nested within other logical operators
+   - Cannot be used at field level or nested inside a field
+   - Cannot be used inside an operator
+   - Valid: `{ "$and": [{ "field": { "$gt": 100 } }] }`
+   - Valid: `{ "$or": [{ "$and": [{ "field": { "$gt": 100 } }] }] }`
+   - Invalid: `{ "field": { "$and": [{ "$gt": 100 }] } }`
+   - Invalid: `{ "field": { "$gt": { "$and": [{...}] } } }`
+
+4. $not operator:
+
+   - Must be an object
+   - Cannot be empty
+   - Can be used at field level or top level
+   - Valid: `{ "$not": { "field": "value" } }`
+   - Valid: `{ "field": { "$not": { "$eq": "value" } } }`
+
+5. Operator nesting:
+
+   - Logical operators must contain field conditions, not direct operators
+   - Valid: `{ "$and": [{ "field": { "$gt": 100 } }] }`
+   - Invalid: `{ "$and": [{ "$gt": 100 }] }`
+
+## Store-Specific Notes
+
+### Astra
+
+- Nested field queries are supported using dot notation
+- Array fields must be explicitly defined as arrays in the metadata
+- Metadata values are case-sensitive
+
+### ChromaDB
+
+- Where filters only return results where the filtered field exists in metadata
+- Empty metadata fields are not included in filter results
+- Metadata fields must be present for negative matches (e.g., $ne won't match documents missing the field)
+
+### Cloudflare Vectorize
+
+- Requires explicit metadata indexing before filtering can be used
+- Use `createMetadataIndex()` to index fields you want to filter on
+- Up to 10 metadata indexes per Vectorize index
+- String values are indexed up to first 64 bytes (truncated on UTF-8 boundaries)
+- Number values use float64 precision
+- Filter JSON must be under 2048 bytes
+- Field names cannot contain dots (.) or start with $
+- Field names limited to 512 characters
+- Vectors must be re-upserted after creating new metadata indexes to be included in filtered results
+- Range queries may have reduced accuracy with very large datasets (\~10M+ vectors)
+
+### libSQL
+
+- Supports nested object queries with dot notation
+- Array fields are validated to ensure they contain valid JSON arrays
+- Numeric comparisons maintain proper type handling
+- Empty arrays in conditions are handled gracefully
+- Metadata is stored in a JSONB column for efficient querying
+
+### PgVector
+
+- Full support for PostgreSQL's native JSON querying capabilities
+- Efficient handling of array operations using native array functions
+- Proper type handling for numbers, strings, and booleans
+- Nested field queries use PostgreSQL's JSON path syntax internally
+- Metadata is stored in a JSONB column for efficient indexing
+
+### Pinecone
+
+- Metadata field names are limited to 512 characters
+- Numeric values must be within the range of ±1e38
+- Arrays in metadata are limited to 64KB total size
+- Nested objects are flattened with dot notation
+- Metadata updates replace the entire metadata object
+
+### Qdrant
+
+- Supports advanced filtering with nested conditions
+- Payload (metadata) fields must be explicitly indexed for filtering
+- Use `createPayloadIndex()` to index fields you want to filter on:
+
+```typescript
+// Index a field before filtering on it
+await store.createPayloadIndex({
+  indexName: 'my_index',
+  fieldName: 'source',
+  fieldSchema: 'keyword', // 'keyword' | 'integer' | 'float' | 'geo' | 'text' | 'bool' | 'datetime' | 'uuid'
+})
+
+// Now filtering works
+const results = await store.query({
+  indexName: 'my_index',
+  queryVector: queryVector,
+  filter: { source: 'document-a' },
+})
+```
+
+- Efficient handling of geo-spatial queries
+- Special handling for null and empty values
+- Vector-specific filtering capabilities
+- Datetime values must be in RFC 3339 format
+
+### Upstash
+
+- 512-character limit for metadata field keys
+- Query size is limited (avoid large IN clauses)
+- No support for null/undefined values in filters
+- Translates to SQL-like syntax internally
+- Case-sensitive string comparisons
+- Metadata updates are atomic
+
+### MongoDB
+
+- Full support for MongoDB/Sift query syntax for metadata filters
+- Supports all standard comparison, array, logical, and element operators
+- Supports nested fields and arrays in metadata
+- Filtering can be applied to both `metadata` and the original document content using the `filter` and `documentFilter` options, respectively
+- `filter` applies to the metadata object; `documentFilter` applies to the original document fields
+- No artificial limits on filter size or complexity (subject to MongoDB query limits)
+- Indexing metadata fields is recommended for optimal performance
+
+### Couchbase
+
+- Currently does not have support for metadata filters. Filtering must be done client-side after retrieving results or by using the Couchbase SDK's Search capabilities directly for more complex queries.
+
+### Amazon S3 Vectors
+
+- Equality values must be primitives (string/number/boolean). `null`/`undefined`, arrays, objects, and Date are not allowed for equality. Range operators accept numbers or Date (Dates are normalized to epoch ms).
+- `$in`/`$nin` require **non-empty arrays of primitives**; Date elements are allowed and normalized to epoch ms. **Array equality** is not supported.
+- Implicit AND is canonicalized (`{a:1,b:2}` → `{$and:[{a:1},{b:2}]}`). Logical operators must contain field conditions, use non-empty arrays, and appear only at the root or within other logical operators (not inside field values).
+- Keys listed in `nonFilterableMetadataKeys` at index creation are stored but not filterable; this setting is immutable.
+- $exists requires a boolean value.
+- undefined/null/empty filters are treated as no filter.
+- Each metadata key name limited to 63 characters.
+- Total metadata per vector: Up to 40 KB (filterable + non-filterable)
+- Total metadata keys per vector: Up to 10
+- Filterable metadata per vector: Up to 2 KB
+- Non-filterable metadata keys per vector index: Up to 10
+
+## Related
+
+- [Astra](https://mastra.ai/reference/vectors/astra)
+- [Chroma](https://mastra.ai/reference/vectors/chroma)
+- [Cloudflare Vectorize](https://mastra.ai/reference/vectors/vectorize)
+- [libSQL](https://mastra.ai/reference/vectors/libsql)
+- [MongoDB](https://mastra.ai/reference/vectors/mongodb)
+- [PgStore](https://mastra.ai/reference/vectors/pg)
+- [Pinecone](https://mastra.ai/reference/vectors/pinecone)
+- [Qdrant](https://mastra.ai/reference/vectors/qdrant)
+- [Upstash](https://mastra.ai/reference/vectors/upstash)
+- [Amazon S3 Vectors](https://mastra.ai/reference/vectors/s3vectors)

package/dist/docs/references/reference-storage-composite.md

@@ -0,0 +1,235 @@
+# Composite Storage
+
+`MastraCompositeStore` can compose storage domains from different providers. Use it when you need different databases for different purposes. For example, use LibSQL for memory and PostgreSQL for workflows.
+
+## Installation
+
+`MastraCompositeStore` is included in `@mastra/core`:
+
+**npm**:
+
+```bash
+npm install @mastra/core@latest
+```
+
+**pnpm**:
+
+```bash
+pnpm add @mastra/core@latest
+```
+
+**Yarn**:
+
+```bash
+yarn add @mastra/core@latest
+```
+
+**Bun**:
+
+```bash
+bun add @mastra/core@latest
+```
+
+You'll also need to install the storage providers you want to compose:
+
+**npm**:
+
+```bash
+npm install @mastra/pg@latest @mastra/libsql@latest
+```
+
+**pnpm**:
+
+```bash
+pnpm add @mastra/pg@latest @mastra/libsql@latest
+```
+
+**Yarn**:
+
+```bash
+yarn add @mastra/pg@latest @mastra/libsql@latest
+```
+
+**Bun**:
+
+```bash
+bun add @mastra/pg@latest @mastra/libsql@latest
+```
+
+## Storage domains
+
+Mastra organizes storage into five specialized domains, each handling a specific type of data. Each domain can be backed by a different storage adapter, and domain classes are exported from each storage package.
+
+| Domain          | Description                                                                                                                                                                           |
+| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `memory`        | Conversation persistence for agents. Stores threads (conversation sessions), messages, resources (user identities), and working memory (persistent context across conversations).     |
+| `workflows`     | Workflow execution state. When workflows suspend for human input, external events, or scheduled resumption, their state is persisted here to enable resumption after server restarts. |
+| `scores`        | Evaluation results from Mastra's evals system. Scores and metrics are persisted here for analysis and comparison over time.                                                           |
+| `observability` | Telemetry data including traces and spans. Agent interactions, tool calls, and LLM requests generate spans collected into traces for debugging and performance analysis.              |
+| `agents`        | Agent configurations for stored agents. Enables agents to be defined and updated at runtime without code deployments.                                                                 |
+
+## Usage
+
+### Basic composition
+
+Import domain classes directly from each store package and compose them:
+
+```typescript
+import { MastraCompositeStore } from '@mastra/core/storage'
+import { WorkflowsPG, ScoresPG } from '@mastra/pg'
+import { MemoryLibSQL } from '@mastra/libsql'
+import { Mastra } from '@mastra/core'
+
+export const mastra = new Mastra({
+  storage: new MastraCompositeStore({
+    id: 'composite',
+    domains: {
+      memory: new MemoryLibSQL({ url: 'file:./local.db' }),
+      workflows: new WorkflowsPG({ connectionString: process.env.DATABASE_URL }),
+      scores: new ScoresPG({ connectionString: process.env.DATABASE_URL }),
+    },
+  }),
+})
+```
+
+### With a default storage
+
+Use `default` to specify a fallback storage, then override specific domains:
+
+```typescript
+import { MastraCompositeStore } from '@mastra/core/storage'
+import { PostgresStore } from '@mastra/pg'
+import { MemoryLibSQL } from '@mastra/libsql'
+import { Mastra } from '@mastra/core'
+
+const pgStore = new PostgresStore({
+  id: 'pg',
+  connectionString: process.env.DATABASE_URL,
+})
+
+export const mastra = new Mastra({
+  storage: new MastraCompositeStore({
+    id: 'composite',
+    default: pgStore,
+    domains: {
+      memory: new MemoryLibSQL({ url: 'file:./local.db' }),
+    },
+  }),
+})
+```
+
+## Options
+
+**id:** (`string`): Unique identifier for this storage instance.
+
+**default?:** (`MastraCompositeStore`): Default storage adapter. Domains not explicitly specified in \`domains\` will use this storage's domains as fallbacks.
+
+**domains?:** (`object`): Individual domain overrides. Each domain can come from a different storage adapter. These take precedence over the default storage.
+
+**domains.memory?:** (`MemoryStorage`): Storage for threads, messages, and resources.
+
+**domains.workflows?:** (`WorkflowsStorage`): Storage for workflow snapshots.
+
+**domains.scores?:** (`ScoresStorage`): Storage for evaluation scores.
+
+**domains.observability?:** (`ObservabilityStorage`): Storage for traces and spans.
+
+**domains.agents?:** (`AgentsStorage`): Storage for stored agent configurations.
+
+**disableInit?:** (`boolean`): When true, automatic initialization is disabled. You must call init() explicitly.
+
+## Initialization
+
+`MastraCompositeStore` initializes each configured domain independently. When passed to the Mastra class, `init()` is called automatically:
+
+```typescript
+import { MastraCompositeStore } from '@mastra/core/storage'
+import { MemoryPG, WorkflowsPG, ScoresPG } from '@mastra/pg'
+import { Mastra } from '@mastra/core'
+
+const storage = new MastraCompositeStore({
+  id: 'composite',
+  domains: {
+    memory: new MemoryPG({ connectionString: process.env.DATABASE_URL }),
+    workflows: new WorkflowsPG({ connectionString: process.env.DATABASE_URL }),
+    scores: new ScoresPG({ connectionString: process.env.DATABASE_URL }),
+  },
+})
+
+export const mastra = new Mastra({
+  storage, // init() called automatically
+})
+```
+
+If using storage directly, call `init()` explicitly:
+
+```typescript
+import { MastraCompositeStore } from '@mastra/core/storage'
+import { MemoryPG } from '@mastra/pg'
+
+const storage = new MastraCompositeStore({
+  id: 'composite',
+  domains: {
+    memory: new MemoryPG({ connectionString: process.env.DATABASE_URL }),
+  },
+})
+
+await storage.init()
+
+// Access domain-specific stores via getStore()
+const memoryStore = await storage.getStore('memory')
+const thread = await memoryStore?.getThreadById({ threadId: '...' })
+```
+
+## Use cases
+
+### Separate databases for different workloads
+
+Use a local database for development while keeping production data in a managed service:
+
+```typescript
+import { MastraCompositeStore } from '@mastra/core/storage'
+import { MemoryPG, WorkflowsPG, ScoresPG } from '@mastra/pg'
+import { MemoryLibSQL } from '@mastra/libsql'
+
+const storage = new MastraCompositeStore({
+  id: 'composite',
+  domains: {
+    // Use local SQLite for development, PostgreSQL for production
+    memory:
+      process.env.NODE_ENV === 'development'
+        ? new MemoryLibSQL({ url: 'file:./dev.db' })
+        : new MemoryPG({ connectionString: process.env.DATABASE_URL }),
+    workflows: new WorkflowsPG({ connectionString: process.env.DATABASE_URL }),
+    scores: new ScoresPG({ connectionString: process.env.DATABASE_URL }),
+  },
+})
+```
+
+### Specialized storage for observability
+
+Observability data can quickly overwhelm general-purpose databases in production. A single agent interaction can generate hundreds of spans, and high-traffic applications can produce thousands of traces per day.
+
+**ClickHouse** is recommended for production observability because it's optimized for high-volume, write-heavy analytics workloads. Use composite storage to route observability to ClickHouse while keeping other data in your primary database:
+
+```typescript
+import { MastraCompositeStore } from '@mastra/core/storage'
+import { MemoryPG, WorkflowsPG, ScoresPG } from '@mastra/pg'
+import { ObservabilityStorageClickhouse } from '@mastra/clickhouse'
+
+const storage = new MastraCompositeStore({
+  id: 'composite',
+  domains: {
+    memory: new MemoryPG({ connectionString: process.env.DATABASE_URL }),
+    workflows: new WorkflowsPG({ connectionString: process.env.DATABASE_URL }),
+    scores: new ScoresPG({ connectionString: process.env.DATABASE_URL }),
+    observability: new ObservabilityStorageClickhouse({
+      url: process.env.CLICKHOUSE_URL,
+      username: process.env.CLICKHOUSE_USERNAME,
+      password: process.env.CLICKHOUSE_PASSWORD,
+    }),
+  },
+})
+```
+
+> **Info:** This approach is also required when using storage providers that don't support observability (like Convex, DynamoDB, or Cloudflare). See the [DefaultExporter documentation](https://mastra.ai/docs/observability/tracing/exporters/default) for the full list of supported providers.