@mastra/pg 1.0.0-beta.11 → 1.0.0-beta.12

package/dist/docs/rag/04-reference.md

@@ -0,0 +1,351 @@
+# Rag API Reference
+
+> API reference for rag - 1 entries
+
+
+---
+
+## Reference: Metadata Filters
+
+> Documentation for metadata filtering capabilities in Mastra, which allow for precise querying of vector search results across different vector stores.
+
+Mastra provides a unified metadata filtering syntax across all vector stores, based on MongoDB/Sift query syntax. Each vector store translates these filters into their native format.
+
+## Basic Example
+
+```typescript
+import { PgVector } from "@mastra/pg";
+
+const store = new PgVector({
+  id: 'pg-vector',
+  connectionString
+});
+
+const results = await store.query({
+  indexName: "my_index",
+  queryVector: queryVector,
+  topK: 10,
+  filter: {
+    category: "electronics", // Simple equality
+    price: { $gt: 100 }, // Numeric comparison
+    tags: { $in: ["sale", "new"] }, // Array membership
+  },
+});
+```
+
+## Supported Operators
+
+<OperatorsTable
+  title="Basic Comparison"
+  operators={[
+    {
+      name: "$eq",
+      description: "Matches values equal to specified value",
+      example: "{ age: { $eq: 25 } }",
+      supportedBy: ["All except Couchbase"],
+    },
+    {
+      name: "$ne",
+      description: "Matches values not equal",
+      example: "{ status: { $ne: 'inactive' } }",
+      supportedBy: ["All except Couchbase"],
+    },
+    {
+      name: "$gt",
+      description: "Greater than",
+      example: "{ price: { $gt: 100 } }",
+      supportedBy: ["All except Couchbase"],
+    },
+    {
+      name: "$gte",
+      description: "Greater than or equal",
+      example: "{ rating: { $gte: 4.5 } }",
+      supportedBy: ["All except Couchbase"],
+    },
+    {
+      name: "$lt",
+      description: "Less than",
+      example: "{ stock: { $lt: 20 } }",
+      supportedBy: ["All except Couchbase"],
+    },
+    {
+      name: "$lte",
+      description: "Less than or equal",
+      example: "{ priority: { $lte: 3 } }",
+      supportedBy: ["All except Couchbase"],
+    },
+  ]}
+/>
+
+<OperatorsTable
+  title="Array Operators"
+  operators={[
+    {
+      name: "$in",
+      description: "Matches any value in array",
+      example: '{ category: { $in: ["A", "B"] } }',
+      supportedBy: ["All except Couchbase"],
+    },
+    {
+      name: "$nin",
+      description: "Matches none of the values",
+      example: '{ status: { $nin: ["deleted", "archived"] } }',
+      supportedBy: ["All except Couchbase"],
+    },
+    {
+      name: "$all",
+      description: "Matches arrays containing all elements",
+      example: '{ tags: { $all: ["urgent", "high"] } }',
+      supportedBy: ["Astra", "Pinecone", "Upstash", "MongoDB"],
+    },
+    {
+      name: "$elemMatch",
+      description: "Matches array elements meeting criteria",
+      example: "{ scores: { $elemMatch: { $gt: 80 } } }",
+      supportedBy: ["libSQL", "PgVector", "MongoDB"],
+    },
+  ]}
+/>
+
+<OperatorsTable
+  title="Logical Operators"
+  operators={[
+    {
+      name: "$and",
+      description: "Logical AND",
+      example: "{ $and: [{ price: { $gt: 100 } }, { stock: { $gt: 0 } }] }",
+      supportedBy: ["All except Vectorize, Couchbase"],
+    },
+    {
+      name: "$or",
+      description: "Logical OR",
+      example: '{ $or: [{ status: "active" }, { priority: "high" }] }',
+      supportedBy: ["All except Vectorize, Couchbase"],
+    },
+    {
+      name: "$not",
+      description: "Logical NOT",
+      example: "{ price: { $not: { $lt: 100 } } }",
+      supportedBy: [
+        "Astra",
+        "Qdrant",
+        "Upstash",
+        "PgVector",
+        "libSQL",
+        "MongoDB",
+      ],
+    },
+    {
+      name: "$nor",
+      description: "Logical NOR",
+      example: '{ $nor: [{ status: "deleted" }, { archived: true }] }',
+      supportedBy: ["Qdrant", "Upstash", "PgVector", "libSQL", "MongoDB"],
+    },
+  ]}
+/>
+
+<OperatorsTable
+  title="Element Operators"
+  operators={[
+    {
+      name: "$exists",
+      description: "Matches documents with field",
+      example: "{ rating: { $exists: true } }",
+      supportedBy: ["All except Vectorize, Chroma, Couchbase"],
+    },
+  ]}
+/>
+
+<OperatorsTable
+  title="Custom Operators"
+  operators={[
+    {
+      name: "$contains",
+      description: "Text contains substring",
+      example: '{ description: { $contains: "sale" } }',
+      supportedBy: ["Upstash", "libSQL", "PgVector"],
+    },
+    {
+      name: "$regex",
+      description: "Regular expression match",
+      example: '{ name: { $regex: "^test" } }',
+      supportedBy: ["Qdrant", "PgVector", "Upstash", "MongoDB"],
+    },
+    {
+      name: "$size",
+      description: "Array length check",
+      example: "{ tags: { $size: { $gt: 2 } } }",
+      supportedBy: ["Astra", "libSQL", "PgVector", "MongoDB"],
+    },
+    {
+      name: "$geo",
+      description: "Geospatial query",
+      example: '{ location: { $geo: { type: "radius", ... } } }',
+      supportedBy: ["Qdrant"],
+    },
+    {
+      name: "$datetime",
+      description: "Datetime range query",
+      example: '{ created: { $datetime: { range: { gt: "2024-01-01" } } } }',
+      supportedBy: ["Qdrant"],
+    },
+    {
+      name: "$hasId",
+      description: "Vector ID existence check",
+      example: '{ $hasId: ["id1", "id2"] }',
+      supportedBy: ["Qdrant"],
+    },
+    {
+      name: "$hasVector",
+      description: "Vector existence check",
+      example: "{ $hasVector: true }",
+      supportedBy: ["Qdrant"],
+    },
+  ]}
+/>
+
+## Common Rules and Restrictions
+
+1. Field names cannot:
+   - Contain dots (.) unless referring to nested fields
+   - Start with $ or contain null characters
+   - Be empty strings
+
+2. Values must be:
+   - Valid JSON types (string, number, boolean, object, array)
+   - Not undefined
+   - Properly typed for the operator (e.g., numbers for numeric comparisons)
+
+3. Logical operators:
+   - Must contain valid conditions
+   - Cannot be empty
+   - Must be properly nested
+   - Can only be used at top level or nested within other logical operators
+   - Cannot be used at field level or nested inside a field
+   - Cannot be used inside an operator
+   - Valid: `{ "$and": [{ "field": { "$gt": 100 } }] }`
+   - Valid: `{ "$or": [{ "$and": [{ "field": { "$gt": 100 } }] }] }`
+   - Invalid: `{ "field": { "$and": [{ "$gt": 100 }] } }`
+   - Invalid: `{ "field": { "$gt": { "$and": [{...}] } } }`
+
+4. $not operator:
+   - Must be an object
+   - Cannot be empty
+   - Can be used at field level or top level
+   - Valid: `{ "$not": { "field": "value" } }`
+   - Valid: `{ "field": { "$not": { "$eq": "value" } } }`
+
+5. Operator nesting:
+   - Logical operators must contain field conditions, not direct operators
+   - Valid: `{ "$and": [{ "field": { "$gt": 100 } }] }`
+   - Invalid: `{ "$and": [{ "$gt": 100 }] }`
+
+## Store-Specific Notes
+
+### Astra
+
+- Nested field queries are supported using dot notation
+- Array fields must be explicitly defined as arrays in the metadata
+- Metadata values are case-sensitive
+
+### ChromaDB
+
+- Where filters only return results where the filtered field exists in metadata
+- Empty metadata fields are not included in filter results
+- Metadata fields must be present for negative matches (e.g., $ne won't match documents missing the field)
+
+### Cloudflare Vectorize
+
+- Requires explicit metadata indexing before filtering can be used
+- Use `createMetadataIndex()` to index fields you want to filter on
+- Up to 10 metadata indexes per Vectorize index
+- String values are indexed up to first 64 bytes (truncated on UTF-8 boundaries)
+- Number values use float64 precision
+- Filter JSON must be under 2048 bytes
+- Field names cannot contain dots (.) or start with $
+- Field names limited to 512 characters
+- Vectors must be re-upserted after creating new metadata indexes to be included in filtered results
+- Range queries may have reduced accuracy with very large datasets (~10M+ vectors)
+
+### libSQL
+
+- Supports nested object queries with dot notation
+- Array fields are validated to ensure they contain valid JSON arrays
+- Numeric comparisons maintain proper type handling
+- Empty arrays in conditions are handled gracefully
+- Metadata is stored in a JSONB column for efficient querying
+
+### PgVector
+
+- Full support for PostgreSQL's native JSON querying capabilities
+- Efficient handling of array operations using native array functions
+- Proper type handling for numbers, strings, and booleans
+- Nested field queries use PostgreSQL's JSON path syntax internally
+- Metadata is stored in a JSONB column for efficient indexing
+
+### Pinecone
+
+- Metadata field names are limited to 512 characters
+- Numeric values must be within the range of ±1e38
+- Arrays in metadata are limited to 64KB total size
+- Nested objects are flattened with dot notation
+- Metadata updates replace the entire metadata object
+
+### Qdrant
+
+- Supports advanced filtering with nested conditions
+- Payload (metadata) fields must be explicitly indexed for filtering
+- Efficient handling of geo-spatial queries
+- Special handling for null and empty values
+- Vector-specific filtering capabilities
+- Datetime values must be in RFC 3339 format
+
+### Upstash
+
+- 512-character limit for metadata field keys
+- Query size is limited (avoid large IN clauses)
+- No support for null/undefined values in filters
+- Translates to SQL-like syntax internally
+- Case-sensitive string comparisons
+- Metadata updates are atomic
+
+### MongoDB
+
+- Full support for MongoDB/Sift query syntax for metadata filters
+- Supports all standard comparison, array, logical, and element operators
+- Supports nested fields and arrays in metadata
+- Filtering can be applied to both `metadata` and the original document content using the `filter` and `documentFilter` options, respectively
+- `filter` applies to the metadata object; `documentFilter` applies to the original document fields
+- No artificial limits on filter size or complexity (subject to MongoDB query limits)
+- Indexing metadata fields is recommended for optimal performance
+
+### Couchbase
+
+- Currently does not have support for metadata filters. Filtering must be done client-side after retrieving results or by using the Couchbase SDK's Search capabilities directly for more complex queries.
+
+### Amazon S3 Vectors
+
+- Equality values must be primitives (string/number/boolean). `null`/`undefined`, arrays, objects, and Date are not allowed for equality. Range operators accept numbers or Date (Dates are normalized to epoch ms).
+- `$in`/`$nin` require **non-empty arrays of primitives**; Date elements are allowed and normalized to epoch ms. **Array equality** is not supported.
+- Implicit AND is canonicalized (`{a:1,b:2}` → `{$and:[{a:1},{b:2}]}`). Logical operators must contain field conditions, use non-empty arrays, and appear only at the root or within other logical operators (not inside field values).
+- Keys listed in `nonFilterableMetadataKeys` at index creation are stored but not filterable; this setting is immutable.
+- $exists requires a boolean value.
+- undefined/null/empty filters are treated as no filter.
+- Each metadata key name limited to 63 characters.
+- Total metadata per vector: Up to 40 KB (filterable + non-filterable)
+- Total metadata keys per vector: Up to 10
+- Filterable metadata per vector: Up to 2 KB
+- Non-filterable metadata keys per vector index: Up to 10
+
+## Related
+
+- [Astra](https://mastra.ai/reference/v1/vectors/astra)
+- [Chroma](https://mastra.ai/reference/v1/vectors/chroma)
+- [Cloudflare Vectorize](https://mastra.ai/reference/v1/vectors/vectorize)
+- [libSQL](https://mastra.ai/reference/v1/vectors/libsql)
+- [MongoDB](https://mastra.ai/reference/v1/vectors/mongodb)
+- [PgStore](https://mastra.ai/reference/v1/vectors/pg)
+- [Pinecone](https://mastra.ai/reference/v1/vectors/pinecone)
+- [Qdrant](https://mastra.ai/reference/v1/vectors/qdrant)
+- [Upstash](https://mastra.ai/reference/v1/vectors/upstash)
+- [Amazon S3 Vectors](https://mastra.ai/reference/v1/vectors/s3vectors)