@mastra/mongodb 1.0.0-beta.12 → 1.0.0-beta.14

package/CHANGELOG.md

@@ -1,5 +1,109 @@
 # @mastra/mongodb
 
+## 1.0.0-beta.14
+
+### Patch Changes
+
+- Fixed duplicate spans migration issue across all storage backends. When upgrading from older versions, existing duplicate (traceId, spanId) combinations in the spans table could prevent the unique constraint from being created. The migration deduplicates spans before adding the constraint. ([#12073](https://github.com/mastra-ai/mastra/pull/12073))
+
+  **Deduplication rules (in priority order):**
+  1. Keep completed spans (those with `endedAt` set) over incomplete spans
+  2. Among spans with the same completion status, keep the one with the newest `updatedAt`
+  3. Use `createdAt` as the final tiebreaker
+
+  **What changed:**
+  - Added `migrateSpans()` method to observability stores for manual migration
+  - Added `checkSpansMigrationStatus()` method to check if migration is needed
+  - All stores use optimized single-query deduplication to avoid memory issues on large tables
+
+  **Usage example:**
+
+  ```typescript
+  const observability = await storage.getStore('observability');
+  const status = await observability.checkSpansMigrationStatus();
+  if (status.needsMigration) {
+    const result = await observability.migrateSpans();
+    console.log(`Migration complete: ${result.duplicatesRemoved} duplicates removed`);
+  }
+  ```
+
+  Fixes #11840
+
+- Renamed MastraStorage to MastraCompositeStore for better clarity. The old MastraStorage name remains available as a deprecated alias for backward compatibility, but will be removed in a future version. ([#12093](https://github.com/mastra-ai/mastra/pull/12093))
+
+  **Migration:**
+
+  Update your imports and usage:
+
+  ```typescript
+  // Before
+  import { MastraStorage } from '@mastra/core/storage';
+
+  const storage = new MastraStorage({
+    id: 'composite',
+    domains: { ... }
+  });
+
+  // After
+  import { MastraCompositeStore } from '@mastra/core/storage';
+
+  const storage = new MastraCompositeStore({
+    id: 'composite',
+    domains: { ... }
+  });
+  ```
+
+  The new name better reflects that this is a composite storage implementation that routes different domains (workflows, traces, messages) to different underlying stores, avoiding confusion with the general "Mastra Storage" concept.
+
+- Updated dependencies [[`026b848`](https://github.com/mastra-ai/mastra/commit/026b8483fbf5b6d977be8f7e6aac8d15c75558ac), [`ffa553a`](https://github.com/mastra-ai/mastra/commit/ffa553a3edc1bd17d73669fba66d6b6f4ac10897)]:
+  - @mastra/core@1.0.0-beta.26
+
+## 1.0.0-beta.13
+
+### Patch Changes
+
+- Added new `listThreads` method for flexible thread filtering across all storage adapters. ([#11832](https://github.com/mastra-ai/mastra/pull/11832))
+
+  **New Features**
+  - Filter threads by `resourceId`, `metadata`, or both (with AND logic for metadata key-value pairs)
+  - All filter parameters are optional, allowing you to list all threads or filter as needed
+  - Full pagination and sorting support
+
+  **Example Usage**
+
+  ```typescript
+  // List all threads
+  const allThreads = await memory.listThreads({});
+
+  // Filter by resourceId only
+  const userThreads = await memory.listThreads({
+    filter: { resourceId: 'user-123' },
+  });
+
+  // Filter by metadata only
+  const supportThreads = await memory.listThreads({
+    filter: { metadata: { category: 'support' } },
+  });
+
+  // Filter by both with pagination
+  const filteredThreads = await memory.listThreads({
+    filter: {
+      resourceId: 'user-123',
+      metadata: { priority: 'high', status: 'open' },
+    },
+    orderBy: { field: 'updatedAt', direction: 'DESC' },
+    page: 0,
+    perPage: 20,
+  });
+  ```
+
+  **Security Improvements**
+  - Added validation to prevent SQL injection via malicious metadata keys
+  - Added pagination parameter validation to prevent integer overflow attacks
+
+- Updated dependencies [[`ed3e3dd`](https://github.com/mastra-ai/mastra/commit/ed3e3ddec69d564fe2b125e083437f76331f1283), [`6833c69`](https://github.com/mastra-ai/mastra/commit/6833c69607418d257750bbcdd84638993d343539), [`47b1c16`](https://github.com/mastra-ai/mastra/commit/47b1c16a01c7ffb6765fe1e499b49092f8b7eba3), [`3a76a80`](https://github.com/mastra-ai/mastra/commit/3a76a80284cb71a0faa975abb3d4b2a9631e60cd), [`8538a0d`](https://github.com/mastra-ai/mastra/commit/8538a0d232619bf55dad7ddc2a8b0ca77c679a87), [`9312dcd`](https://github.com/mastra-ai/mastra/commit/9312dcd1c6f5b321929e7d382e763d95fdc030f5)]:
+  - @mastra/core@1.0.0-beta.25
+
 ## 1.0.0-beta.12
 
 ### Minor Changes

package/dist/docs/README.md

@@ -31,4 +31,4 @@ docs/
 ## Version
 
 Package: @mastra/mongodb
-Version: 1.0.0-beta.12
+Version: 1.0.0-beta.14

package/dist/docs/SKILL.md

@@ -5,7 +5,7 @@ description: Documentation for @mastra/mongodb. Includes links to type definitio
 
 # @mastra/mongodb Documentation
 
-> **Version**: 1.0.0-beta.12
+> **Version**: 1.0.0-beta.14
 > **Package**: @mastra/mongodb
 
 ## Quick Navigation

package/dist/docs/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.0.0-beta.12",
+  "version": "1.0.0-beta.14",
   "package": "@mastra/mongodb",
   "exports": {},
   "modules": {}

package/dist/docs/rag/01-vector-databases.md

@@ -27,7 +27,7 @@ await store.upsert({
 });
 ```
 
-### Using MongoDB Atlas Vector search
+<h3>Using MongoDB Atlas Vector search</h3>
 
 For detailed setup instructions and best practices, see the [official MongoDB Atlas Vector Search documentation](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/?utm_campaign=devrel&utm_source=third-party-content&utm_medium=cta&utm_content=mastra-docs).
 
@@ -55,7 +55,7 @@ await store.upsert({
 });
 ```
 
-### Using PostgreSQL with pgvector
+<h3>Using PostgreSQL with pgvector</h3>
 
 PostgreSQL with the pgvector extension is a good solution for teams already using PostgreSQL who want to minimize infrastructure complexity.
 For detailed setup instructions and best practices, see the [official pgvector repository](https://github.com/pgvector/pgvector).
@@ -323,7 +323,7 @@ await store.upsert({
 });
 ```
 
-### Using LanceDB
+<h3>Using LanceDB</h3>
 
 LanceDB is an embedded vector database built on the Lance columnar format, suitable for local development or cloud deployment.
 For detailed setup instructions and best practices, see the [official LanceDB documentation](https://lancedb.github.io/lancedb/).

package/dist/index.cjs

@@ -13,7 +13,7 @@ var evals = require('@mastra/core/evals');
 
 // package.json
 var package_default = {
-  version: "1.0.0-beta.12"};
+  version: "1.0.0-beta.14"};
 var MongoDBFilterTranslator = class extends filter.BaseFilterTranslator {
   getSupportedOperators() {
     return {
@@ -280,6 +280,8 @@ var MongoDBVector = class extends vector.MastraVector {
     throw new Error(`Index "${indexNameInternal}" did not become ready within timeout`);
   }
   async upsert({ indexName, vectors, metadata, ids, documents }) {
+    vector.validateUpsertInput("MONGODB", vectors, metadata, ids);
+    vector.validateVectorValues("MONGODB", vectors);
     try {
       const collection = await this.getCollection(indexName);
       this.collectionForValidation = collection;
@@ -355,8 +357,8 @@ var MongoDBVector = class extends vector.MastraVector {
         index: indexNameInternal,
         queryVector,
         path: this.embeddingFieldName,
-        numCandidates: 100,
-        limit: topK
+        numCandidates: Math.min(1e4, Math.max(100, topK)),
+        limit: Math.min(1e4, topK)
       };
       if (Object.keys(combinedFilter).length > 0) {
         const filterWithExclusion = {
@@ -1687,25 +1689,48 @@ var MemoryStorageMongoDB = class _MemoryStorageMongoDB extends storage.MemorySto
       );
     }
   }
-  async listThreadsByResourceId(args) {
+  async listThreads(args) {
+    const { page = 0, perPage: perPageInput, orderBy, filter } = args;
+    try {
+      this.validatePaginationInput(page, perPageInput ?? 100);
+    } catch (error$1) {
+      throw new error.MastraError(
+        {
+          id: storage.createStorageErrorId("MONGODB", "LIST_THREADS", "INVALID_PAGE"),
+          domain: error.ErrorDomain.STORAGE,
+          category: error.ErrorCategory.USER,
+          details: { page, ...perPageInput !== void 0 && { perPage: perPageInput } }
+        },
+        error$1 instanceof Error ? error$1 : new Error("Invalid pagination parameters")
+      );
+    }
+    const perPage = storage.normalizePerPage(perPageInput, 100);
+    try {
+      this.validateMetadataKeys(filter?.metadata);
+    } catch (error$1) {
+      throw new error.MastraError(
+        {
+          id: storage.createStorageErrorId("MONGODB", "LIST_THREADS", "INVALID_METADATA_KEY"),
+          domain: error.ErrorDomain.STORAGE,
+          category: error.ErrorCategory.USER,
+          details: { metadataKeys: filter?.metadata ? Object.keys(filter.metadata).join(", ") : "" }
+        },
+        error$1 instanceof Error ? error$1 : new Error("Invalid metadata key")
+      );
+    }
     try {
-      const { resourceId, page = 0, perPage: perPageInput, orderBy } = args;
-      if (page < 0) {
-        throw new error.MastraError(
-          {
-            id: storage.createStorageErrorId("MONGODB", "LIST_THREADS_BY_RESOURCE_ID", "INVALID_PAGE"),
-            domain: error.ErrorDomain.STORAGE,
-            category: error.ErrorCategory.USER,
-            details: { page }
-          },
-          new Error("page must be >= 0")
-        );
-      }
-      const perPage = storage.normalizePerPage(perPageInput, 100);
       const { offset, perPage: perPageForResponse } = storage.calculatePagination(page, perPageInput, perPage);
       const { field, direction } = this.parseOrderBy(orderBy);
       const collection = await this.getCollection(storage.TABLE_THREADS);
-      const query = { resourceId };
+      const query = {};
+      if (filter?.resourceId) {
+        query.resourceId = filter.resourceId;
+      }
+      if (filter?.metadata && Object.keys(filter.metadata).length > 0) {
+        for (const [key, value] of Object.entries(filter.metadata)) {
+          query[`metadata.${key}`] = value;
+        }
+      }
       const total = await collection.countDocuments(query);
       if (perPage === 0) {
         return {
@@ -1739,10 +1764,13 @@ var MemoryStorageMongoDB = class _MemoryStorageMongoDB extends storage.MemorySto
     } catch (error$1) {
       throw new error.MastraError(
         {
-          id: storage.createStorageErrorId("MONGODB", "LIST_THREADS_BY_RESOURCE_ID", "FAILED"),
+          id: storage.createStorageErrorId("MONGODB", "LIST_THREADS", "FAILED"),
           domain: error.ErrorDomain.STORAGE,
           category: error.ErrorCategory.THIRD_PARTY,
-          details: { resourceId: args.resourceId }
+          details: {
+            ...filter?.resourceId && { resourceId: filter.resourceId },
+            hasMetadataFilter: !!filter?.metadata
+          }
         },
         error$1
       );
@@ -1924,9 +1952,221 @@ var ObservabilityMongoDB = class _ObservabilityMongoDB extends storage.Observabi
     }
   }
   async init() {
+    const uniqueIndexExists = await this.spansUniqueIndexExists();
+    if (!uniqueIndexExists) {
+      const duplicateInfo = await this.checkForDuplicateSpans();
+      if (duplicateInfo.hasDuplicates) {
+        const errorMessage = `
+===========================================================================
+MIGRATION REQUIRED: Duplicate spans detected in ${storage.TABLE_SPANS} collection
+===========================================================================
+
+Found ${duplicateInfo.duplicateCount} duplicate (traceId, spanId) combinations.
+
+The spans collection requires a unique index on (traceId, spanId), but your
+database contains duplicate entries that must be resolved first.
+
+To fix this, run the manual migration command:
+
+  npx mastra migrate
+
+This command will:
+  1. Remove duplicate spans (keeping the most complete/recent version)
+  2. Add the required unique index
+
+Note: This migration may take some time for large collections.
+===========================================================================
+`;
+        throw new error.MastraError({
+          id: storage.createStorageErrorId("MONGODB", "MIGRATION_REQUIRED", "DUPLICATE_SPANS"),
+          domain: error.ErrorDomain.STORAGE,
+          category: error.ErrorCategory.USER,
+          text: errorMessage
+        });
+      }
+    }
     await this.createDefaultIndexes();
     await this.createCustomIndexes();
   }
+  /**
+   * Checks if the unique index on (spanId, traceId) already exists on the spans collection.
+   * Used to skip deduplication when the index already exists (migration already complete).
+   */
+  async spansUniqueIndexExists() {
+    try {
+      const collection = await this.getCollection(storage.TABLE_SPANS);
+      const indexes = await collection.indexes();
+      return indexes.some((idx) => idx.unique === true && idx.key?.spanId === 1 && idx.key?.traceId === 1);
+    } catch {
+      return false;
+    }
+  }
+  /**
+   * Checks for duplicate (traceId, spanId) combinations in the spans collection.
+   * Returns information about duplicates for logging/CLI purposes.
+   */
+  async checkForDuplicateSpans() {
+    try {
+      const collection = await this.getCollection(storage.TABLE_SPANS);
+      const result = await collection.aggregate([
+        {
+          $group: {
+            _id: { traceId: "$traceId", spanId: "$spanId" },
+            count: { $sum: 1 }
+          }
+        },
+        { $match: { count: { $gt: 1 } } },
+        { $count: "duplicateCount" }
+      ]).toArray();
+      const duplicateCount = result[0]?.duplicateCount ?? 0;
+      return {
+        hasDuplicates: duplicateCount > 0,
+        duplicateCount
+      };
+    } catch (error) {
+      this.logger?.debug?.(`Could not check for duplicates: ${error}`);
+      return { hasDuplicates: false, duplicateCount: 0 };
+    }
+  }
+  /**
+   * Manually run the spans migration to deduplicate and add the unique index.
+   * This is intended to be called from the CLI when duplicates are detected.
+   *
+   * @returns Migration result with status and details
+   */
+  async migrateSpans() {
+    const indexExists = await this.spansUniqueIndexExists();
+    if (indexExists) {
+      return {
+        success: true,
+        alreadyMigrated: true,
+        duplicatesRemoved: 0,
+        message: `Migration already complete. Unique index exists on ${storage.TABLE_SPANS} collection.`
+      };
+    }
+    const duplicateInfo = await this.checkForDuplicateSpans();
+    if (duplicateInfo.hasDuplicates) {
+      this.logger?.info?.(
+        `Found ${duplicateInfo.duplicateCount} duplicate (traceId, spanId) combinations. Starting deduplication...`
+      );
+      await this.deduplicateSpans();
+    } else {
+      this.logger?.info?.(`No duplicate spans found.`);
+    }
+    await this.createDefaultIndexes();
+    await this.createCustomIndexes();
+    return {
+      success: true,
+      alreadyMigrated: false,
+      duplicatesRemoved: duplicateInfo.duplicateCount,
+      message: duplicateInfo.hasDuplicates ? `Migration complete. Removed duplicates and added unique index to ${storage.TABLE_SPANS} collection.` : `Migration complete. Added unique index to ${storage.TABLE_SPANS} collection.`
+    };
+  }
+  /**
+   * Check migration status for the spans collection.
+   * Returns information about whether migration is needed.
+   */
+  async checkSpansMigrationStatus() {
+    const indexExists = await this.spansUniqueIndexExists();
+    if (indexExists) {
+      return {
+        needsMigration: false,
+        hasDuplicates: false,
+        duplicateCount: 0,
+        constraintExists: true,
+        tableName: storage.TABLE_SPANS
+      };
+    }
+    const duplicateInfo = await this.checkForDuplicateSpans();
+    return {
+      needsMigration: true,
+      hasDuplicates: duplicateInfo.hasDuplicates,
+      duplicateCount: duplicateInfo.duplicateCount,
+      constraintExists: false,
+      tableName: storage.TABLE_SPANS
+    };
+  }
+  /**
+   * Deduplicates spans with the same (traceId, spanId) combination.
+   * This is needed for databases that existed before the unique constraint was added.
+   *
+   * Priority for keeping spans:
+   * 1. Completed spans (endedAt IS NOT NULL) over incomplete spans
+   * 2. Most recent updatedAt
+   * 3. Most recent createdAt (as tiebreaker)
+   *
+   * Note: This prioritizes migration completion over perfect data preservation.
+   * Old trace data may be lost, which is acceptable for this use case.
+   */
+  async deduplicateSpans() {
+    try {
+      const collection = await this.getCollection(storage.TABLE_SPANS);
+      const duplicateCheck = await collection.aggregate([
+        {
+          $group: {
+            _id: { traceId: "$traceId", spanId: "$spanId" },
+            count: { $sum: 1 }
+          }
+        },
+        { $match: { count: { $gt: 1 } } },
+        { $limit: 1 }
+      ]).toArray();
+      if (duplicateCheck.length === 0) {
+        this.logger?.debug?.("No duplicate spans found");
+        return;
+      }
+      this.logger?.info?.("Duplicate spans detected, starting deduplication...");
+      const idsToDelete = await collection.aggregate([
+        // Sort by priority (affects which document $first picks within each group)
+        {
+          $sort: {
+            // Completed spans first (endedAt exists and is not null)
+            endedAt: -1,
+            updatedAt: -1,
+            createdAt: -1
+          }
+        },
+        // Group by (traceId, spanId), keeping the first (best) _id and all _ids
+        {
+          $group: {
+            _id: { traceId: "$traceId", spanId: "$spanId" },
+            keepId: { $first: "$_id" },
+            // The best one to keep (after sort)
+            allIds: { $push: "$_id" },
+            // All ObjectIds (just 12 bytes each, not full docs)
+            count: { $sum: 1 }
+          }
+        },
+        // Only consider groups with duplicates
+        { $match: { count: { $gt: 1 } } },
+        // Get IDs to delete (allIds minus keepId)
+        {
+          $project: {
+            idsToDelete: {
+              $filter: {
+                input: "$allIds",
+                cond: { $ne: ["$$this", "$keepId"] }
+              }
+            }
+          }
+        },
+        // Unwind to get flat list of IDs
+        { $unwind: "$idsToDelete" },
+        // Just output the ID
+        { $project: { _id: "$idsToDelete" } }
+      ]).toArray();
+      if (idsToDelete.length === 0) {
+        this.logger?.debug?.("No duplicates to delete after aggregation");
+        return;
+      }
+      const deleteResult = await collection.deleteMany({
+        _id: { $in: idsToDelete.map((d) => d._id) }
+      });
+      this.logger?.info?.(`Deduplication complete: removed ${deleteResult.deletedCount} duplicate spans`);
+    } catch (error) {
+      this.logger?.warn?.("Failed to deduplicate spans:", error);
+    }
+  }
   async dangerouslyClearAll() {
     const collection = await this.getCollection(storage.TABLE_SPANS);
     await collection.deleteMany({});
@@ -2207,7 +2447,7 @@ var ObservabilityMongoDB = class _ObservabilityMongoDB extends storage.Observabi
             // No children with errors
           }
         ];
-        const countResult = await collection.aggregate([...pipeline, { $count: "total" }]).toArray();
+        const countResult = await collection.aggregate([...pipeline, { $count: "total" }], { allowDiskUse: true }).toArray();
         const count2 = countResult[0]?.total || 0;
         if (count2 === 0) {
           return {
@@ -2244,7 +2484,7 @@ var ObservabilityMongoDB = class _ObservabilityMongoDB extends storage.Observabi
             { $project: { _errorSpans: 0 } }
           ];
         }
-        const spans2 = await collection.aggregate(aggregationPipeline).toArray();
+        const spans2 = await collection.aggregate(aggregationPipeline, { allowDiskUse: true }).toArray();
         return {
           pagination: {
             total: count2,
@@ -2270,18 +2510,21 @@ var ObservabilityMongoDB = class _ObservabilityMongoDB extends storage.Observabi
       let spans;
       if (sortField === "endedAt") {
         const nullSortValue = sortDirection === -1 ? 0 : 1;
-        spans = await collection.aggregate([
-          { $match: mongoFilter },
-          {
-            $addFields: {
-              _endedAtNull: { $cond: [{ $eq: ["$endedAt", null] }, nullSortValue, sortDirection === -1 ? 1 : 0] }
-            }
-          },
-          { $sort: { _endedAtNull: 1, [sortField]: sortDirection } },
-          { $skip: page * perPage },
-          { $limit: perPage },
-          { $project: { _endedAtNull: 0 } }
-        ]).toArray();
+        spans = await collection.aggregate(
+          [
+            { $match: mongoFilter },
+            {
+              $addFields: {
+                _endedAtNull: { $cond: [{ $eq: ["$endedAt", null] }, nullSortValue, sortDirection === -1 ? 1 : 0] }
+              }
+            },
+            { $sort: { _endedAtNull: 1, [sortField]: sortDirection } },
+            { $skip: page * perPage },
+            { $limit: perPage },
+            { $project: { _endedAtNull: 0 } }
+          ],
+          { allowDiskUse: true }
+        ).toArray();
       } else {
         spans = await collection.find(mongoFilter).sort({ [sortField]: sortDirection }).skip(page * perPage).limit(perPage).toArray();
       }
@@ -3055,7 +3298,7 @@ var WorkflowsStorageMongoDB = class _WorkflowsStorageMongoDB extends storage.Wor
 };
 
 // src/storage/index.ts
-var MongoDBStore = class extends storage.MastraStorage {
+var MongoDBStore = class extends storage.MastraCompositeStore {
   #connector;
   stores;
   constructor(config) {