@mastra/mongodb 1.13.0-alpha.0 → 1.14.0-alpha.0

package/CHANGELOG.md

@@ -1,5 +1,55 @@
 # @mastra/mongodb
 
+## 1.14.0-alpha.0
+
+### Minor Changes
+
+- Added a `filterFields` option to `MongoDBVector.createIndex()`. Declaring the metadata fields you filter on lets Mastra register them as native filter fields in the Atlas vectorSearch index, so filtered queries are pushed straight into `$vectorSearch` instead of first materialising matching document `_id`s. This removes the 16 MB BSON ceiling that previously capped metadata-filtered queries at roughly 342,000 matching documents. ([#19006](https://github.com/mastra-ai/mastra/pull/19006))
+
+  ```ts
+  await vectorStore.createIndex({
+    indexName: 'my-index',
+    dimension: 1536,
+    metric: 'cosine',
+    filterFields: ['category', 'tenant_id'],
+  });
+  ```
+
+  Queries that filter only on declared fields (using operators Atlas Vector Search supports) take the fast path automatically. Filters that reference an undeclared field, or use an unsupported operator, keep working through the existing pre-filter.
+
+### Patch Changes
+
+- Fixed `MongoDBVector.createIndex()` leaving index setup incomplete. Previously, when the vector search index already existed, the call silently skipped creating the companion full-text search index. Repeated or interrupted `createIndex()` calls now finish creating the remaining search index instead of leaving setup incomplete. ([#19006](https://github.com/mastra-ai/mastra/pull/19006))
+
+- Updated dependencies [[`ec857fc`](https://github.com/mastra-ai/mastra/commit/ec857fc79c264b53b38e16478c789b7177f2ad59), [`e1f2fae`](https://github.com/mastra-ai/mastra/commit/e1f2faebaf048c3d4c2e2c01d293767c195d5794), [`63aa799`](https://github.com/mastra-ai/mastra/commit/63aa799c6b44eacc7806cda6846b7c5bbee06b37), [`73db8db`](https://github.com/mastra-ai/mastra/commit/73db8db90d69ab6153c7942749f624db0d96952d), [`73db8db`](https://github.com/mastra-ai/mastra/commit/73db8db90d69ab6153c7942749f624db0d96952d), [`76b7181`](https://github.com/mastra-ai/mastra/commit/76b71810366e6d90b9d3973149d1c7ba3659ffb9), [`0c0e8d7`](https://github.com/mastra-ai/mastra/commit/0c0e8d7becd4d1445c656b78d5d845f606c1ff9d), [`9f7c67a`](https://github.com/mastra-ai/mastra/commit/9f7c67abeeb52c41c51a9b5edee60b62afe7cd8d), [`3b65e68`](https://github.com/mastra-ai/mastra/commit/3b65e68d7f1c771c7a70eea42d83fefdd28cad88), [`e3868e2`](https://github.com/mastra-ai/mastra/commit/e3868e22babfffd0133771669ca724501c2dd58e)]:
+  - @mastra/core@1.52.0-alpha.5
+
+## 1.13.0
+
+### Minor Changes
+
+- Added atomic caller-defined dataset IDs for idempotent dataset creation across built-in storage adapters. Supplying `id` to `mastra.datasets.create()` now creates the dataset once and resolves compatible retries to the persisted record; incompatible immutable identity fields throw `DATASET_ID_CONFLICT`. ([#19370](https://github.com/mastra-ai/mastra/pull/19370))
+
+- Added caller-defined dataset item identities for safe retries across all dataset storage adapters. ([#19384](https://github.com/mastra-ai/mastra/pull/19384))
+
+  Dataset items can now include an `externalId` when calling `addItem` or `addItems`:
+
+  ```ts
+  await dataset.addItem({
+    externalId: 'source-item-123',
+    input: { prompt: 'Hello' },
+  });
+  ```
+
+  Retrying with the same identity and payload returns the existing item. Reusing an identity with different content returns a typed conflict, including during concurrent writes. Updates and deletes preserve the identity, Spanner retries transactions without changing the outcome, and MySQL batch writes now preserve every supported dataset item field.
+
+### Patch Changes
+
+- Raised the `@mastra/core` peer dependency floor to `>=1.51.0-0` so the dataset item identity helpers used by the storage adapters are available at runtime. ([#19384](https://github.com/mastra-ai/mastra/pull/19384))
+
+- Updated dependencies [[`bd6d240`](https://github.com/mastra-ai/mastra/commit/bd6d2402db93dddaef0721667e7e8a030e7c6e16), [`0111486`](https://github.com/mastra-ai/mastra/commit/01114867612593eef5cfa2fda6a1194dfedda841), [`96a3749`](https://github.com/mastra-ai/mastra/commit/96a37492235f5b8076b3e3177d83ed5a5e44a640), [`fe1bda0`](https://github.com/mastra-ai/mastra/commit/fe1bda06f6af92a694a51712db747cda1e7185f0), [`25e7c12`](https://github.com/mastra-ai/mastra/commit/25e7c126a770069ae7fb7ecf1d2adb40e017b009), [`1ce5121`](https://github.com/mastra-ai/mastra/commit/1ce512155d122bb21f47d98383e82ffbf84b39e8), [`fb8aea3`](https://github.com/mastra-ai/mastra/commit/fb8aea384291e77311be3a64ee1717320d5c3c73), [`4adc391`](https://github.com/mastra-ai/mastra/commit/4adc3911075249c352bb4832d2471922826344de), [`a5c6337`](https://github.com/mastra-ai/mastra/commit/a5c6337d23c7686c81a32ce62f550f610543a240), [`3cfc47a`](https://github.com/mastra-ai/mastra/commit/3cfc47a6b89940aadd0f46fb01ae9624a73a865d), [`2bb7817`](https://github.com/mastra-ai/mastra/commit/2bb78176112fde628483de2830528f7eee911e56), [`51d9870`](https://github.com/mastra-ai/mastra/commit/51d987032c689c2855374d0f244f5d654da809d1), [`5cab274`](https://github.com/mastra-ai/mastra/commit/5cab2744250e22d12fefa7b32637dce224233cee), [`7fa27d3`](https://github.com/mastra-ai/mastra/commit/7fa27d3b6f5ed68cd34e454a4d3ad9c482a0cfbc), [`8b97958`](https://github.com/mastra-ai/mastra/commit/8b979589f9aa59ba67cac565949475f2ffeb4ac3), [`8410541`](https://github.com/mastra-ai/mastra/commit/84105412c60ecd3bb33a9838146f59c4b588228f), [`a58dcbb`](https://github.com/mastra-ai/mastra/commit/a58dcbb546d7e1d65ebdc1f39e55f0908fcd9391), [`aa38805`](https://github.com/mastra-ai/mastra/commit/aa38805b878b827403be785eb90688d7172f5a40), [`153bd3b`](https://github.com/mastra-ai/mastra/commit/153bd3b396bdfed6b74cf43de12db8fd2d83c04a), [`45a8e65`](https://github.com/mastra-ai/mastra/commit/45a8e65e1556d1362cb3f25187023c36de26661d), [`e955965`](https://github.com/mastra-ai/mastra/commit/e955965dce575a903e37cf054d28ea99aa48785e), [`2d22570`](https://github.com/mastra-ai/mastra/commit/2d22570c7dfdd02123d0ecc529efb05ccba2d9fc), [`07bb863`](https://github.com/mastra-ai/mastra/commit/07bb8631919c6f7cf377dccd45b096e0f17fbed0), [`c8ed116`](https://github.com/mastra-ai/mastra/commit/c8ed11699f62bcac70102ab4ec84d80d20541da6), [`01b338c`](https://github.com/mastra-ai/mastra/commit/01b338c56271f0219606710e3e8b26dee27ac6c2), [`a99eae8`](https://github.com/mastra-ai/mastra/commit/a99eae8908e500c1b2d12f9d277be616b98617a5), [`860ef7e`](https://github.com/mastra-ai/mastra/commit/860ef7e77d92b63469cbe5857aa1e626197e43e9), [`17e818c`](https://github.com/mastra-ai/mastra/commit/17e818c51a958ba90641b1a959dc38faf8c034e9), [`edce8d2`](https://github.com/mastra-ai/mastra/commit/edce8d2769f19e27a05737c627af2d765472a4f8), [`8a586ec`](https://github.com/mastra-ai/mastra/commit/8a586eca9a4914f31dff6140d0d45ac375b00669), [`4451dfe`](https://github.com/mastra-ai/mastra/commit/4451dfe857428e7abcc0261a507a2e186dae6d47), [`8b7361d`](https://github.com/mastra-ai/mastra/commit/8b7361d35de68b80d05d30a74e0c69e7218fd612), [`1d39058`](https://github.com/mastra-ai/mastra/commit/1d39058e548efd691799985d5c8af2737f1c3bd2), [`3927473`](https://github.com/mastra-ai/mastra/commit/392747323ddb10c643d12be7b9ae913159dfaeed), [`dce50dc`](https://github.com/mastra-ai/mastra/commit/dce50dc9a1c1fcd0f427bb5f6250ec74910cb04b), [`fd13f8e`](https://github.com/mastra-ai/mastra/commit/fd13f8e21990f9904c3eedba3a626bb4a929cdb8), [`634caff`](https://github.com/mastra-ai/mastra/commit/634caff29a9200ad058b67d53f96d9e5832fb8a2), [`f703f87`](https://github.com/mastra-ai/mastra/commit/f703f878de072d51fda557f9c50867d8252bef05), [`3e26c87`](https://github.com/mastra-ai/mastra/commit/3e26c87de0c5bc2583b795ce6ca5889b6b161acb), [`33f2b88`](https://github.com/mastra-ai/mastra/commit/33f2b88842c09a567f906fac4cb61cd5277ced59), [`177010f`](https://github.com/mastra-ai/mastra/commit/177010ff096d2e4b28d89803be5b1a4cad2a0d6b), [`0ad646f`](https://github.com/mastra-ai/mastra/commit/0ad646f71a530f2454664299e5e01bfd13fa12e5), [`b486abf`](https://github.com/mastra-ai/mastra/commit/b486abfa2a7528c6f527e4015c819ea9fa54aaad), [`54a51e0`](https://github.com/mastra-ai/mastra/commit/54a51e0a484fe1ebad3fb1f7ef5282a075709eb7), [`c43f3a9`](https://github.com/mastra-ai/mastra/commit/c43f3a9d1efde99b38789364ba4d0ba670f430e3), [`a5008f2`](https://github.com/mastra-ai/mastra/commit/a5008f22ae710ad9402ea9f2547d8c02f74d384b), [`e2d5f37`](https://github.com/mastra-ai/mastra/commit/e2d5f373bd289be534d5f8694d34465010533df6), [`4ce0163`](https://github.com/mastra-ai/mastra/commit/4ce0163dc86e675a86809685c8ce6c49f1aeb87e), [`4378341`](https://github.com/mastra-ai/mastra/commit/43783412df5ea3dd35f5b1f6e4851e79c346fc89)]:
+  - @mastra/core@1.51.0
+
 ## 1.13.0-alpha.0
 
 ### Minor Changes
@@ -660,9 +710,7 @@
   **Example**
 
   ```ts
-  const storage = new LibSQLStore({
-    /* config */
-  });
+  const storage = new LibSQLStore({/* config */});
   const favorites = await storage.getStore('favorites');
 
   await favorites?.favorite({
@@ -688,9 +736,7 @@
   **Example**
 
   ```ts
-  const storage = new LibSQLStore({
-    /* config */
-  });
+  const storage = new LibSQLStore({/* config */});
   const favorites = await storage.getStore('favorites');
 
   await favorites?.favorite({

package/dist/docs/SKILL.md

@@ -3,7 +3,7 @@ name: mastra-mongodb
 description: Documentation for @mastra/mongodb. Use when working with @mastra/mongodb APIs, configuration, or implementation.
 metadata:
   package: "@mastra/mongodb"
-  version: "1.13.0-alpha.0"
+  version: "1.14.0-alpha.0"
 ---
 
 ## When to use

package/dist/docs/assets/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.13.0-alpha.0",
+  "version": "1.14.0-alpha.0",
   "package": "@mastra/mongodb",
   "exports": {},
   "modules": {}

package/dist/docs/references/docs-memory-working-memory.md

@@ -403,7 +403,7 @@ By default, working memory reaches the model as part of the system message. You
 
 ```typescript
 const memory = new Memory({
-  storage: new LibSQLStore({ url: 'file:./mastra.db' }),
+  storage: new LibSQLStore({ id: 'mastra-storage', url: 'file:./mastra.db' }),
   options: {
     workingMemory: {
       enabled: true,

package/dist/docs/references/reference-vectors-mongodb.md

@@ -89,6 +89,8 @@ Creates a new vector index (collection) in MongoDB.
 
 **metric** (`'cosine' | 'euclidean' | 'dotproduct'`): Distance metric for similarity search (Default: `cosine`)
 
+**filterFields** (`string[]`): Metadata field names to declare as filter fields in the Atlas vectorSearch index (registered as metadata.\<field>). Queries that filter only on declared fields are pushed directly into $vectorSearch instead of pre-filtering candidate \_ids, avoiding the 16 MB BSON limit on large result sets. Filters that reference an undeclared field, or use an operator $vectorSearch does not support, fall back to the pre-filter automatically.
+
 ### `waitForIndexReady()`
 
 Waits for an index to become ready after creation. Useful when you need to ensure an index is ready before performing operations.

package/dist/index.cjs

@@ -15,7 +15,7 @@ var skills = require('@mastra/core/storage/domains/skills');
 
 // package.json
 var package_default = {
-  version: "1.13.0-alpha.0"};
+  version: "1.14.0-alpha.0"};
 var MongoDBFilterTranslator = class extends filter.BaseFilterTranslator {
   getSupportedOperators() {
     return {
@@ -105,13 +105,38 @@ var MongoDBFilterTranslator = class extends filter.BaseFilterTranslator {
 };
 
 // src/vector/index.ts
-var MongoDBVector = class extends vector.MastraVector {
+var MongoDBVector = class _MongoDBVector extends vector.MastraVector {
   client;
   db;
   collections;
   embeddingFieldName;
   metadataFieldName = "metadata";
   documentFieldName = "document";
+  /**
+   * Per-index cache of the filter paths declared in the Atlas vectorSearch index
+   * (e.g. `document`, `metadata.category`). Populated when an index is created in
+   * this process and lazily hydrated from the live index definition otherwise.
+   * `_id` is excluded because it is materialised separately by the fallback path.
+   */
+  declaredFilterPaths = /* @__PURE__ */ new Map();
+  /**
+   * MongoDB query operators supported inside `$vectorSearch.filter`. Intentionally
+   * conservative: filters using any operator outside this set fall back to the
+   * `$match` pre-filter, which supports the full query language. Widening this set
+   * is safe only for operators Atlas Vector Search actually accepts in `filter`.
+   */
+  static PUSHDOWN_OPERATORS = /* @__PURE__ */ new Set([
+    "$and",
+    "$or",
+    "$eq",
+    "$ne",
+    "$gt",
+    "$gte",
+    "$lt",
+    "$lte",
+    "$in",
+    "$nin"
+  ]);
   mongoMetricMap = {
     cosine: "cosine",
     euclidean: "euclidean",
@@ -174,7 +199,12 @@ var MongoDBVector = class extends vector.MastraVector {
    * queryable. Skipping that step on a real Atlas cluster may cause
    * "index not found" or "index not ready" errors on subsequent operations.
    */
-  async createIndex({ indexName, dimension, metric = "cosine" }) {
+  async createIndex({
+    indexName,
+    dimension,
+    metric = "cosine",
+    filterFields
+  }) {
     let mongoMetric;
     try {
       if (!Number.isInteger(dimension) || dimension <= 0) {
@@ -209,29 +239,33 @@ var MongoDBVector = class extends vector.MastraVector {
       const indexNameInternal = `${indexName}_vector_index`;
       const embeddingField = this.embeddingFieldName;
       const numDimensions = dimension;
-      await collection.createSearchIndex({
-        definition: {
-          fields: [
-            {
-              type: "vector",
-              path: embeddingField,
-              numDimensions,
-              similarity: mongoMetric
-            },
-            {
-              type: "filter",
-              path: "_id"
-            },
-            {
-              type: "filter",
-              path: "document"
-            }
-          ]
+      const declaredMetadataPaths = this.buildDeclaredMetadataPaths(filterFields);
+      const fields = [
+        {
+          type: "vector",
+          path: embeddingField,
+          numDimensions,
+          similarity: mongoMetric
+        },
+        {
+          type: "filter",
+          path: "_id"
         },
+        {
+          type: "filter",
+          path: this.documentFieldName
+        },
+        ...declaredMetadataPaths.map((path) => ({ type: "filter", path }))
+      ];
+      const vectorIndexCreated = await this.createSearchIndexIgnoringExisting(collection, {
+        definition: { fields },
         name: indexNameInternal,
         type: "vectorSearch"
       });
-      await collection.createSearchIndex({
+      if (vectorIndexCreated) {
+        this.declaredFilterPaths.set(indexName, /* @__PURE__ */ new Set([this.documentFieldName, ...declaredMetadataPaths]));
+      }
+      await this.createSearchIndexIgnoringExisting(collection, {
         definition: {
           mappings: {
             dynamic: true
@@ -241,16 +275,30 @@ var MongoDBVector = class extends vector.MastraVector {
         type: "search"
       });
     } catch (error$1) {
-      if (error$1.codeName !== "IndexAlreadyExists") {
-        throw new error.MastraError(
-          {
-            id: storage.createVectorErrorId("MONGODB", "CREATE_INDEX", "FAILED"),
-            domain: error.ErrorDomain.STORAGE,
-            category: error.ErrorCategory.THIRD_PARTY
-          },
-          error$1
-        );
+      throw new error.MastraError(
+        {
+          id: storage.createVectorErrorId("MONGODB", "CREATE_INDEX", "FAILED"),
+          domain: error.ErrorDomain.STORAGE,
+          category: error.ErrorCategory.THIRD_PARTY
+        },
+        error$1
+      );
+    }
+  }
+  /**
+   * Creates an Atlas Search index, treating IndexAlreadyExists as a no-op so
+   * `createIndex` stays idempotent per index. Returns true when the index was
+   * actually created by this call.
+   */
+  async createSearchIndexIgnoringExisting(collection, description) {
+    try {
+      await collection.createSearchIndex(description);
+      return true;
+    } catch (error) {
+      if (error?.codeName === "IndexAlreadyExists") {
+        return false;
       }
+      throw error;
     }
   }
   /**
@@ -392,9 +440,19 @@ var MongoDBVector = class extends vector.MastraVector {
         limit: Math.min(1e4, topK)
       };
       if (hasMetadataFilter) {
-        const candidateIds = await collection.aggregate([{ $match: metadataFilter }, { $project: { _id: 1 } }]).map((doc) => doc._id).toArray();
-        if (candidateIds.length === 0) return [];
-        vectorSearch.filter = documentFilter ? { $and: [{ _id: { $in: candidateIds } }, { [this.documentFieldName]: documentFilter }] } : { _id: { $in: candidateIds } };
+        let declaredPaths;
+        try {
+          declaredPaths = await this.getDeclaredFilterPaths(indexName);
+        } catch {
+          declaredPaths = /* @__PURE__ */ new Set();
+        }
+        if (this.canPushDownFilter(metadataFilter, declaredPaths)) {
+          vectorSearch.filter = documentFilter ? { $and: [metadataFilter, { [this.documentFieldName]: documentFilter }] } : metadataFilter;
+        } else {
+          const candidateIds = await collection.aggregate([{ $match: metadataFilter }, { $project: { _id: 1 } }]).map((doc) => doc._id).toArray();
+          if (candidateIds.length === 0) return [];
+          vectorSearch.filter = documentFilter ? { $and: [{ _id: { $in: candidateIds } }, { [this.documentFieldName]: documentFilter }] } : { _id: { $in: candidateIds } };
+        }
       } else if (documentFilter) {
         vectorSearch.filter = { [this.documentFieldName]: documentFilter };
       }
@@ -511,6 +569,7 @@ var MongoDBVector = class extends vector.MastraVector {
       if (collection) {
         await collection.drop();
         this.collections.delete(indexName);
+        this.declaredFilterPaths.delete(indexName);
       } else {
         throw new Error(`Index (Collection) "${indexName}" does not exist`);
       }
@@ -777,6 +836,66 @@ var MongoDBVector = class extends vector.MastraVector {
       }
     }
   }
+  /**
+   * Maps user-facing `filterFields` (bare metadata field names) to the fully
+   * qualified filter paths stored in the index, e.g. `category` → `metadata.category`.
+   * Blank entries are dropped and duplicates collapsed. A name already carrying the
+   * `metadata.` prefix is left untouched so callers can pass either form.
+   */
+  buildDeclaredMetadataPaths(filterFields) {
+    if (!filterFields?.length) return [];
+    const paths = /* @__PURE__ */ new Set();
+    for (const field of filterFields) {
+      const trimmed = field?.trim();
+      if (!trimmed) continue;
+      paths.add(trimmed.startsWith(`${this.metadataFieldName}.`) ? trimmed : `${this.metadataFieldName}.${trimmed}`);
+    }
+    return [...paths];
+  }
+  /**
+   * Returns the set of filter paths declared in the vectorSearch index (e.g.
+   * `document`, `metadata.category`), excluding `_id`. Reads the live index
+   * definition once per index and caches the result. Used to decide whether a
+   * metadata filter can be pushed down into `$vectorSearch.filter`.
+   */
+  async getDeclaredFilterPaths(indexName) {
+    const cached = this.declaredFilterPaths.get(indexName);
+    if (cached) return cached;
+    const collection = await this.getCollection(indexName, true);
+    const indexNameInternal = `${indexName}_vector_index`;
+    const indexInfo = await collection.listSearchIndexes().toArray();
+    const indexData = indexInfo.find((idx) => idx.name === indexNameInternal);
+    const paths = /* @__PURE__ */ new Set();
+    if (indexData?.status !== "READY") {
+      return paths;
+    }
+    for (const field of indexData.latestDefinition?.fields ?? []) {
+      if (field?.type === "filter" && typeof field.path === "string" && field.path !== "_id") {
+        paths.add(field.path);
+      }
+    }
+    this.declaredFilterPaths.set(indexName, paths);
+    return paths;
+  }
+  /**
+   * True when `filter` can be passed directly to `$vectorSearch.filter`: every
+   * field path it references is a declared filter field and every operator is in
+   * {@link MongoDBVector.PUSHDOWN_OPERATORS}. Conservative by design — anything it
+   * can't prove safe returns false so the caller uses the `$match` pre-filter.
+   */
+  canPushDownFilter(filter, declaredPaths) {
+    if (filter === null || typeof filter !== "object") return true;
+    if (Array.isArray(filter)) return filter.every((item) => this.canPushDownFilter(item, declaredPaths));
+    for (const [key, value] of Object.entries(filter)) {
+      if (key.startsWith("$")) {
+        if (!_MongoDBVector.PUSHDOWN_OPERATORS.has(key)) return false;
+      } else if (!declaredPaths.has(key)) {
+        return false;
+      }
+      if (!this.canPushDownFilter(value, declaredPaths)) return false;
+    }
+    return true;
+  }
   transformFilter(filter) {
     const translator = new MongoDBFilterTranslator();
     if (!filter) return {};