@mastra/libsql 1.0.0-beta.13 → 1.0.0-beta.14

package/CHANGELOG.md

@@ -1,5 +1,63 @@
 # @mastra/libsql
 
+## 1.0.0-beta.14
+
+### Patch Changes
+
+- Fixed duplicate spans migration issue across all storage backends. When upgrading from older versions, existing duplicate (traceId, spanId) combinations in the spans table could prevent the unique constraint from being created. The migration deduplicates spans before adding the constraint. ([#12073](https://github.com/mastra-ai/mastra/pull/12073))
+
+  **Deduplication rules (in priority order):**
+  1. Keep completed spans (those with `endedAt` set) over incomplete spans
+  2. Among spans with the same completion status, keep the one with the newest `updatedAt`
+  3. Use `createdAt` as the final tiebreaker
+
+  **What changed:**
+  - Added `migrateSpans()` method to observability stores for manual migration
+  - Added `checkSpansMigrationStatus()` method to check if migration is needed
+  - All stores use optimized single-query deduplication to avoid memory issues on large tables
+
+  **Usage example:**
+
+  ```typescript
+  const observability = await storage.getStore('observability');
+  const status = await observability.checkSpansMigrationStatus();
+  if (status.needsMigration) {
+    const result = await observability.migrateSpans();
+    console.log(`Migration complete: ${result.duplicatesRemoved} duplicates removed`);
+  }
+  ```
+
+  Fixes #11840
+
+- Renamed MastraStorage to MastraCompositeStore for better clarity. The old MastraStorage name remains available as a deprecated alias for backward compatibility, but will be removed in a future version. ([#12093](https://github.com/mastra-ai/mastra/pull/12093))
+
+  **Migration:**
+
+  Update your imports and usage:
+
+  ```typescript
+  // Before
+  import { MastraStorage } from '@mastra/core/storage';
+
+  const storage = new MastraStorage({
+    id: 'composite',
+    domains: { ... }
+  });
+
+  // After
+  import { MastraCompositeStore } from '@mastra/core/storage';
+
+  const storage = new MastraCompositeStore({
+    id: 'composite',
+    domains: { ... }
+  });
+  ```
+
+  The new name better reflects that this is a composite storage implementation that routes different domains (workflows, traces, messages) to different underlying stores, avoiding confusion with the general "Mastra Storage" concept.
+
+- Updated dependencies [[`026b848`](https://github.com/mastra-ai/mastra/commit/026b8483fbf5b6d977be8f7e6aac8d15c75558ac), [`ffa553a`](https://github.com/mastra-ai/mastra/commit/ffa553a3edc1bd17d73669fba66d6b6f4ac10897)]:
+  - @mastra/core@1.0.0-beta.26
+
 ## 1.0.0-beta.13
 
 ### Patch Changes

package/dist/docs/README.md

@@ -36,4 +36,4 @@ docs/
 ## Version
 
 Package: @mastra/libsql
-Version: 1.0.0-beta.13
+Version: 1.0.0-beta.14

package/dist/docs/SKILL.md

@@ -5,7 +5,7 @@ description: Documentation for @mastra/libsql. Includes links to type definition
 
 # @mastra/libsql Documentation
 
-> **Version**: 1.0.0-beta.13
+> **Version**: 1.0.0-beta.14
 > **Package**: @mastra/libsql
 
 ## Quick Navigation

package/dist/docs/SOURCE_MAP.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.0.0-beta.13",
+  "version": "1.0.0-beta.14",
   "package": "@mastra/libsql",
   "exports": {},
   "modules": {}

package/dist/docs/memory/02-storage.md

@@ -63,17 +63,17 @@ This is useful when all primitives share the same storage backend and have simil
 
 #### Composite storage
 
-Add storage to your Mastra instance using `MastraStorage` and configure individual storage domains to use different storage providers.
+Add storage to your Mastra instance using `MastraCompositeStore` and configure individual storage domains to use different storage providers.
 
 ```typescript title="src/mastra/index.ts"
 import { Mastra } from "@mastra/core";
-import { MastraStorage } from "@mastra/core/storage";
+import { MastraCompositeStore } from "@mastra/core/storage";
 import { MemoryLibSQL } from "@mastra/libsql";
 import { WorkflowsPG } from "@mastra/pg";
 import { ObservabilityStorageClickhouse } from "@mastra/clickhouse";
 
 export const mastra = new Mastra({
-  storage: new MastraStorage({
+  storage: new MastraCompositeStore({
     id: "composite",
     domains: {
       memory: new MemoryLibSQL({ url: "file:./memory.db" }),

package/dist/docs/storage/01-reference.md

@@ -9,11 +9,11 @@
 
 > Documentation for combining multiple storage backends in Mastra.
 
-`MastraStorage` can compose storage domains from different providers. Use it when you need different databases for different purposes. For example, use LibSQL for memory and PostgreSQL for workflows.
+`MastraCompositeStore` can compose storage domains from different providers. Use it when you need different databases for different purposes. For example, use LibSQL for memory and PostgreSQL for workflows.
 
 ## Installation
 
-`MastraStorage` is included in `@mastra/core`:
+`MastraCompositeStore` is included in `@mastra/core`:
 
 ```bash
 npm install @mastra/core@beta
@@ -44,13 +44,13 @@ Mastra organizes storage into five specialized domains, each handling a specific
 Import domain classes directly from each store package and compose them:
 
 ```typescript title="src/mastra/index.ts"
-import { MastraStorage } from "@mastra/core/storage";
+import { MastraCompositeStore } from "@mastra/core/storage";
 import { WorkflowsPG, ScoresPG } from "@mastra/pg";
 import { MemoryLibSQL } from "@mastra/libsql";
 import { Mastra } from "@mastra/core";
 
 export const mastra = new Mastra({
-  storage: new MastraStorage({
+  storage: new MastraCompositeStore({
     id: "composite",
     domains: {
       memory: new MemoryLibSQL({ url: "file:./local.db" }),
@@ -66,7 +66,7 @@ export const mastra = new Mastra({
 Use `default` to specify a fallback storage, then override specific domains:
 
 ```typescript title="src/mastra/index.ts"
-import { MastraStorage } from "@mastra/core/storage";
+import { MastraCompositeStore } from "@mastra/core/storage";
 import { PostgresStore } from "@mastra/pg";
 import { MemoryLibSQL } from "@mastra/libsql";
 import { Mastra } from "@mastra/core";
@@ -77,7 +77,7 @@ const pgStore = new PostgresStore({
 });
 
 export const mastra = new Mastra({
-  storage: new MastraStorage({
+  storage: new MastraCompositeStore({
     id: "composite",
     default: pgStore,
     domains: {
@@ -91,14 +91,14 @@ export const mastra = new Mastra({
 
 ## Initialization
 
-`MastraStorage` initializes each configured domain independently. When passed to the Mastra class, `init()` is called automatically:
+`MastraCompositeStore` initializes each configured domain independently. When passed to the Mastra class, `init()` is called automatically:
 
 ```typescript title="src/mastra/index.ts"
-import { MastraStorage } from "@mastra/core/storage";
+import { MastraCompositeStore } from "@mastra/core/storage";
 import { MemoryPG, WorkflowsPG, ScoresPG } from "@mastra/pg";
 import { Mastra } from "@mastra/core";
 
-const storage = new MastraStorage({
+const storage = new MastraCompositeStore({
   id: "composite",
   domains: {
     memory: new MemoryPG({ connectionString: process.env.DATABASE_URL }),
@@ -115,10 +115,10 @@ export const mastra = new Mastra({
 If using storage directly, call `init()` explicitly:
 
 ```typescript
-import { MastraStorage } from "@mastra/core/storage";
+import { MastraCompositeStore } from "@mastra/core/storage";
 import { MemoryPG } from "@mastra/pg";
 
-const storage = new MastraStorage({
+const storage = new MastraCompositeStore({
   id: "composite",
   domains: {
     memory: new MemoryPG({ connectionString: process.env.DATABASE_URL }),
@@ -139,11 +139,11 @@ const thread = await memoryStore?.getThreadById({ threadId: "..." });
 Use a local database for development while keeping production data in a managed service:
 
 ```typescript
-import { MastraStorage } from "@mastra/core/storage";
+import { MastraCompositeStore } from "@mastra/core/storage";
 import { MemoryPG, WorkflowsPG, ScoresPG } from "@mastra/pg";
 import { MemoryLibSQL } from "@mastra/libsql";
 
-const storage = new MastraStorage({
+const storage = new MastraCompositeStore({
   id: "composite",
   domains: {
     // Use local SQLite for development, PostgreSQL for production
@@ -162,11 +162,11 @@ const storage = new MastraStorage({
 Use a time-series database for traces while keeping other data in PostgreSQL:
 
 ```typescript
-import { MastraStorage } from "@mastra/core/storage";
+import { MastraCompositeStore } from "@mastra/core/storage";
 import { MemoryPG, WorkflowsPG, ScoresPG } from "@mastra/pg";
 import { ObservabilityStorageClickhouse } from "@mastra/clickhouse";
 
-const storage = new MastraStorage({
+const storage = new MastraCompositeStore({
   id: "composite",
   domains: {
     memory: new MemoryPG({ connectionString: process.env.DATABASE_URL }),

package/dist/index.cjs

@@ -244,10 +244,10 @@ var FILTER_OPERATORS = {
     };
   },
   // Element Operators
-  $exists: (key) => {
+  $exists: (key, value) => {
     const jsonPath = getJsonPath(key);
     return {
-      sql: `json_extract(metadata, ${jsonPath}) IS NOT NULL`,
+      sql: value === false ? `json_extract(metadata, ${jsonPath}) IS NULL` : `json_extract(metadata, ${jsonPath}) IS NOT NULL`,
       needsValue: false
     };
   },
@@ -540,7 +540,7 @@ var LibSQLVector = class extends vector.MastraVector {
       try {
         return await operation();
       } catch (error) {
-        if (error.code === "SQLITE_BUSY" || error.message && error.message.toLowerCase().includes("database is locked")) {
+        if (error.code === "SQLITE_BUSY" || error.code === "SQLITE_LOCKED" || error.code === "SQLITE_LOCKED_SHAREDCACHE" || error.message && error.message.toLowerCase().includes("database is locked") || error.message && error.message.toLowerCase().includes("database table is locked")) {
           attempts++;
           if (attempts >= this.maxRetries) {
             this.logger.error(
@@ -574,22 +574,14 @@ var LibSQLVector = class extends vector.MastraVector {
     minScore = -1
     // Default to -1 to include all results (cosine similarity ranges from -1 to 1)
   }) {
-    try {
-      if (!Number.isInteger(topK) || topK <= 0) {
-        throw new Error("topK must be a positive integer");
-      }
-      if (!Array.isArray(queryVector) || !queryVector.every((x) => typeof x === "number" && Number.isFinite(x))) {
-        throw new Error("queryVector must be an array of finite numbers");
-      }
-    } catch (error$1) {
-      throw new error.MastraError(
-        {
-          id: storage.createVectorErrorId("LIBSQL", "QUERY", "INVALID_ARGS"),
-          domain: error.ErrorDomain.STORAGE,
-          category: error.ErrorCategory.USER
-        },
-        error$1
-      );
+    vector.validateTopK("LIBSQL", topK);
+    if (!Array.isArray(queryVector) || !queryVector.every((x) => typeof x === "number" && Number.isFinite(x))) {
+      throw new error.MastraError({
+        id: storage.createVectorErrorId("LIBSQL", "QUERY", "INVALID_ARGS"),
+        domain: error.ErrorDomain.STORAGE,
+        category: error.ErrorCategory.USER,
+        details: { message: "queryVector must be an array of finite numbers" }
+      });
     }
     try {
       const parsedIndexName = utils.parseSqlIdentifier(indexName, "index name");
@@ -649,6 +641,7 @@ var LibSQLVector = class extends vector.MastraVector {
     }
   }
   async doUpsert({ indexName, vectors, metadata, ids }) {
+    vector.validateUpsertInput("LIBSQL", vectors, metadata, ids);
     const tx = await this.turso.transaction("write");
     try {
       const parsedIndexName = utils.parseSqlIdentifier(indexName, "index name");
@@ -1712,6 +1705,9 @@ var LibSQLDB = class extends base.MastraBase {
       if (tableName === storage.TABLE_WORKFLOW_SNAPSHOT) {
         tableConstraints.push("UNIQUE (workflow_name, run_id)");
       }
+      if (tableName === storage.TABLE_SPANS) {
+        tableConstraints.push("UNIQUE (spanId, traceId)");
+      }
       const allDefinitions = [...columnDefinitions, ...tableConstraints].join(",\n  ");
       const sql = `CREATE TABLE IF NOT EXISTS ${parsedTableName} (
   ${allDefinitions}
@@ -1722,6 +1718,9 @@ var LibSQLDB = class extends base.MastraBase {
         await this.migrateSpansTable();
       }
     } catch (error$1) {
+      if (error$1 instanceof error.MastraError) {
+        throw error$1;
+      }
       throw new error.MastraError(
         {
           id: storage.createStorageErrorId("LIBSQL", "CREATE_TABLE", "FAILED"),
@@ -1735,7 +1734,7 @@ var LibSQLDB = class extends base.MastraBase {
   }
   /**
    * Migrates the spans table schema from OLD_SPAN_SCHEMA to current SPAN_SCHEMA.
-   * This adds new columns that don't exist in old schema.
+   * This adds new columns that don't exist in old schema and ensures required indexes exist.
    */
   async migrateSpansTable() {
     const schema = storage.TABLE_SCHEMAS[storage.TABLE_SPANS];
@@ -1749,9 +1748,203 @@ var LibSQLDB = class extends base.MastraBase {
           this.logger.debug(`LibSQLDB: Added column '${columnName}' to ${storage.TABLE_SPANS}`);
         }
       }
+      const indexExists = await this.spansUniqueIndexExists();
+      if (!indexExists) {
+        const duplicateInfo = await this.checkForDuplicateSpans();
+        if (duplicateInfo.hasDuplicates) {
+          const errorMessage = `
+===========================================================================
+MIGRATION REQUIRED: Duplicate spans detected in ${storage.TABLE_SPANS}
+===========================================================================
+
+Found ${duplicateInfo.duplicateCount} duplicate (traceId, spanId) combinations.
+
+The spans table requires a unique constraint on (traceId, spanId), but your
+database contains duplicate entries that must be resolved first.
+
+To fix this, run the manual migration command:
+
+  npx mastra migrate
+
+This command will:
+  1. Remove duplicate spans (keeping the most complete/recent version)
+  2. Add the required unique constraint
+
+Note: This migration may take some time for large tables.
+===========================================================================
+`;
+          throw new error.MastraError({
+            id: storage.createStorageErrorId("LIBSQL", "MIGRATION_REQUIRED", "DUPLICATE_SPANS"),
+            domain: error.ErrorDomain.STORAGE,
+            category: error.ErrorCategory.USER,
+            text: errorMessage
+          });
+        } else {
+          await this.client.execute(
+            `CREATE UNIQUE INDEX IF NOT EXISTS "mastra_ai_spans_spanid_traceid_idx" ON "${storage.TABLE_SPANS}" ("spanId", "traceId")`
+          );
+          this.logger.debug(`LibSQLDB: Created unique index on (spanId, traceId) for ${storage.TABLE_SPANS}`);
+        }
+      }
       this.logger.info(`LibSQLDB: Migration completed for ${storage.TABLE_SPANS}`);
+    } catch (error$1) {
+      if (error$1 instanceof error.MastraError) {
+        throw error$1;
+      }
+      this.logger.warn(`LibSQLDB: Failed to migrate spans table ${storage.TABLE_SPANS}:`, error$1);
+    }
+  }
+  /**
+   * Checks if the unique index on (spanId, traceId) already exists on the spans table.
+   * Used to skip deduplication when the index already exists (migration already complete).
+   */
+  async spansUniqueIndexExists() {
+    try {
+      const result = await this.client.execute(
+        `SELECT 1 FROM sqlite_master WHERE type = 'index' AND name = 'mastra_ai_spans_spanid_traceid_idx'`
+      );
+      return (result.rows?.length ?? 0) > 0;
+    } catch {
+      return false;
+    }
+  }
+  /**
+   * Checks for duplicate (traceId, spanId) combinations in the spans table.
+   * Returns information about duplicates for logging/CLI purposes.
+   */
+  async checkForDuplicateSpans() {
+    try {
+      const result = await this.client.execute(`
+        SELECT COUNT(*) as duplicate_count FROM (
+          SELECT "spanId", "traceId"
+          FROM "${storage.TABLE_SPANS}"
+          GROUP BY "spanId", "traceId"
+          HAVING COUNT(*) > 1
+        )
+      `);
+      const duplicateCount = Number(result.rows?.[0]?.duplicate_count ?? 0);
+      return {
+        hasDuplicates: duplicateCount > 0,
+        duplicateCount
+      };
     } catch (error) {
-      this.logger.warn(`LibSQLDB: Failed to migrate spans table ${storage.TABLE_SPANS}:`, error);
+      this.logger.debug(`LibSQLDB: Could not check for duplicates: ${error}`);
+      return { hasDuplicates: false, duplicateCount: 0 };
+    }
+  }
+  /**
+   * Manually run the spans migration to deduplicate and add the unique constraint.
+   * This is intended to be called from the CLI when duplicates are detected.
+   *
+   * @returns Migration result with status and details
+   */
+  async migrateSpans() {
+    const indexExists = await this.spansUniqueIndexExists();
+    if (indexExists) {
+      return {
+        success: true,
+        alreadyMigrated: true,
+        duplicatesRemoved: 0,
+        message: `Migration already complete. Unique index exists on ${storage.TABLE_SPANS}.`
+      };
+    }
+    const duplicateInfo = await this.checkForDuplicateSpans();
+    if (duplicateInfo.hasDuplicates) {
+      this.logger.info(
+        `Found ${duplicateInfo.duplicateCount} duplicate (traceId, spanId) combinations. Starting deduplication...`
+      );
+      await this.deduplicateSpans();
+    } else {
+      this.logger.info(`No duplicate spans found.`);
+    }
+    await this.client.execute(
+      `CREATE UNIQUE INDEX IF NOT EXISTS "mastra_ai_spans_spanid_traceid_idx" ON "${storage.TABLE_SPANS}" ("spanId", "traceId")`
+    );
+    return {
+      success: true,
+      alreadyMigrated: false,
+      duplicatesRemoved: duplicateInfo.duplicateCount,
+      message: duplicateInfo.hasDuplicates ? `Migration complete. Removed duplicates and added unique index to ${storage.TABLE_SPANS}.` : `Migration complete. Added unique index to ${storage.TABLE_SPANS}.`
+    };
+  }
+  /**
+   * Check migration status for the spans table.
+   * Returns information about whether migration is needed.
+   */
+  async checkSpansMigrationStatus() {
+    const indexExists = await this.spansUniqueIndexExists();
+    if (indexExists) {
+      return {
+        needsMigration: false,
+        hasDuplicates: false,
+        duplicateCount: 0,
+        constraintExists: true,
+        tableName: storage.TABLE_SPANS
+      };
+    }
+    const duplicateInfo = await this.checkForDuplicateSpans();
+    return {
+      needsMigration: true,
+      hasDuplicates: duplicateInfo.hasDuplicates,
+      duplicateCount: duplicateInfo.duplicateCount,
+      constraintExists: false,
+      tableName: storage.TABLE_SPANS
+    };
+  }
+  /**
+   * Deduplicates spans table by removing duplicate (spanId, traceId) combinations.
+   * Keeps the "best" record for each duplicate group based on:
+   * 1. Completed spans (endedAt IS NOT NULL) over incomplete ones
+   * 2. Most recently updated (updatedAt DESC)
+   * 3. Most recently created (createdAt DESC) as tiebreaker
+   */
+  async deduplicateSpans() {
+    try {
+      const duplicateCheck = await this.client.execute(`
+        SELECT COUNT(*) as duplicate_count FROM (
+          SELECT "spanId", "traceId"
+          FROM "${storage.TABLE_SPANS}"
+          GROUP BY "spanId", "traceId"
+          HAVING COUNT(*) > 1
+        )
+      `);
+      const duplicateCount = Number(duplicateCheck.rows?.[0]?.duplicate_count ?? 0);
+      if (duplicateCount === 0) {
+        this.logger.debug(`LibSQLDB: No duplicate spans found, skipping deduplication`);
+        return;
+      }
+      this.logger.warn(`LibSQLDB: Found ${duplicateCount} duplicate (spanId, traceId) combinations, deduplicating...`);
+      const deleteResult = await this.client.execute(`
+        DELETE FROM "${storage.TABLE_SPANS}"
+        WHERE rowid NOT IN (
+          SELECT MIN(best_rowid) FROM (
+            SELECT
+              rowid as best_rowid,
+              "spanId",
+              "traceId",
+              ROW_NUMBER() OVER (
+                PARTITION BY "spanId", "traceId"
+                ORDER BY
+                  CASE WHEN "endedAt" IS NOT NULL THEN 0 ELSE 1 END,
+                  "updatedAt" DESC,
+                  "createdAt" DESC
+              ) as rn
+            FROM "${storage.TABLE_SPANS}"
+          ) ranked
+          WHERE rn = 1
+          GROUP BY "spanId", "traceId"
+        )
+        AND ("spanId", "traceId") IN (
+          SELECT "spanId", "traceId"
+          FROM "${storage.TABLE_SPANS}"
+          GROUP BY "spanId", "traceId"
+          HAVING COUNT(*) > 1
+        )
+      `);
+      const deletedCount = deleteResult.rowsAffected ?? 0;
+      this.logger.warn(`LibSQLDB: Deleted ${deletedCount} duplicate span records`);
+    } catch (error) {
+      this.logger.warn(`LibSQLDB: Failed to deduplicate spans:`, error);
     }
   }
   /**
@@ -3021,6 +3214,22 @@ var ObservabilityLibSQL = class extends storage.ObservabilityStorage {
   async dangerouslyClearAll() {
     await this.#db.deleteData({ tableName: storage.TABLE_SPANS });
   }
+  /**
+   * Manually run the spans migration to deduplicate and add the unique constraint.
+   * This is intended to be called from the CLI when duplicates are detected.
+   *
+   * @returns Migration result with status and details
+   */
+  async migrateSpans() {
+    return this.#db.migrateSpans();
+  }
+  /**
+   * Check migration status for the spans table.
+   * Returns information about whether migration is needed.
+   */
+  async checkSpansMigrationStatus() {
+    return this.#db.checkSpansMigrationStatus();
+  }
   get tracingStrategy() {
     return {
       preferred: "batch-with-updates",
@@ -4056,7 +4265,7 @@ var WorkflowsLibSQL = class extends storage.WorkflowsStorage {
 };
 
 // src/storage/index.ts
-var LibSQLStore = class extends storage.MastraStorage {
+var LibSQLStore = class extends storage.MastraCompositeStore {
   client;
   maxRetries;
   initialBackoffMs;