@mastra/pg 1.15.1 → 1.16.0-alpha.0

package/dist/docs/references/reference-storage-retention.md

@@ -6,7 +6,7 @@ Storage grows without bound by default. Retention is an opt-in, age-based cleanu
 
 `prune()` deletes rows. It caps growth and is safe to run against large tables (batched, bounded, resumable, cancellable). It never reclaims disk — on SQLite/libSQL the freed pages are reused by future writes so the file stops growing, but handing disk back to the OS (for example a `VACUUM`) is left to the underlying database and the operator to manage.
 
-Retention covers **growth tables** only — tables that accumulate rows unbounded as a side effect of normal operation (conversation history, telemetry, job and run records, schedule fire history, event feeds). User-authored artifacts and config (agents, skills, workspaces, prompt blocks, datasets, schedule definitions, channel installations, and so on) grow with user intent and are edited or deleted explicitly, so they are not valid retention keys.
+Retention covers **growth tables** only — tables that accumulate rows unbounded as a side effect of normal operation (conversation history, telemetry, job and run records, schedule fire history, event feeds). User-authored artifacts and config (agents, skills, workspaces, prompt blocks, datasets, schedule definitions, channel installations, and so on) grow with user intent and are edited or deleted explicitly, so they're not valid retention keys.
 
 The reference implementations are [libSQL](https://mastra.ai/reference/storage/libsql), [PostgreSQL](https://mastra.ai/reference/storage/postgresql), and [MongoDB](https://mastra.ai/reference/storage/mongodb). Other adapters keep rows forever until they implement retention.
 
@@ -89,8 +89,8 @@ Each domain declares which of its tables can be age-pruned and which timestamp c
 > **Note:**
 >
 > - The memory `observational_memory` table has no timestamp anchor, so it can't be age-pruned and isn't a valid retention key.
-> - Experiments prune as whole units: an aged experiment's result rows are deleted together with it (results cascade with their parent), so a run is never left partially deleted. There is no separate `results` key.
-> - For `schedules`, the growth table is the fire history (`schedule_triggers`, one row per fire) — schedule definitions are config and are not pruned.
+> - Experiments prune as whole units: an aged experiment's result rows are deleted together with it (results cascade with their parent), so a run is never left partially deleted. Retention doesn't have a separate `results` key.
+> - For `schedules`, the growth table is the fire history (`schedule_triggers`, one row per fire) — schedule definitions are config and aren't pruned.
 > - On PostgreSQL, timestamp anchors use the timezone-aware mirror columns (for example `createdAtZ`, `completedAtZ`).
 > - LibSQL supports all domains above; PostgreSQL and MongoDB support all except `threadState` and `harness`, which they don't implement.
 > - The v-next PostgreSQL observability domain stores signal events in day-partitioned tables (`spans`, `metrics`, `logs`, `scores`, `feedback`). For it, `prune()` drops whole day partitions (or TimescaleDB chunks) that are entirely older than the cutoff instead of deleting rows — effective granularity is one day, and a partition is only dropped once its entire day is past `maxAge`. `PruneResult.deleted` reports the number of rows in the dropped partitions.
@@ -105,6 +105,8 @@ Deletes rows older than their configured `maxAge` across every domain that has a
 
 `prune()` is designed to be safe on tables with millions of rows. It deletes in bounded, batched chunks — each batch is its own transaction — so it never takes a long lock or bloats the transaction log. It never runs a `VACUUM`.
 
+Pass `options.retention` to replace the configured policies for that call only — for example to skip a domain (keep chat history) or prune more aggressively than the standing config. The store's configured `retention` is unchanged.
+
 Anchor-column indexes are created lazily on the first `prune()` call for each table with a policy — never at `init()` — so deployments that don't configure retention pay no extra index write or disk overhead. The first prune of an existing large table pays a one-time index build; subsequent prunes reuse the index.
 
 ```typescript
@@ -116,6 +118,13 @@ const results = await storage.prune({
 for (const r of results) {
   console.log(`${r.domain}.${r.table}: deleted ${r.deleted}, done=${r.done}`)
 }
+
+// One-off pass with different policies (configured retention untouched):
+await storage.prune({
+  retention: {
+    observability: { spans: { maxAge: '1d' } },
+  },
+})
 ```
 
 Returns: `Promise<PruneResult[]>`
@@ -130,6 +139,8 @@ Returns: `Promise<PruneResult[]>`
 
 **signal** (`AbortSignal`): Cooperative cancellation. The batch loop checks it between batches and stops cleanly, returning partial results with done: false.
 
+**retention** (`RetentionConfig`): Replaces the store's configured retention policies for this call only — e.g. to skip a domain or prune more aggressively. The configured retention is unchanged.
+
 ##### PruneResult
 
 Each result describes one table's progress:
@@ -145,7 +156,7 @@ interface PruneResult {
 
 ## Running prune on a schedule
 
-`prune()` has no built-in scheduler — you decide when it runs. Because it is bounded, a single call may not delete everything. When any result has `done: false`, eligible rows remain and you call again on the next tick. This keeps each invocation short and lets a large backlog drain over several runs.
+`prune()` has no built-in scheduler — you decide when it runs. Because it's bounded, a single call may not delete everything. When any result has `done: false`, eligible rows remain and you call again on the next tick. This keeps each invocation short and lets a large backlog drain over several runs.
 
 ```typescript
 // Runs on your own cron (node-cron, a workflow schedule, an external job, etc.).
@@ -164,11 +175,68 @@ async function retentionTick() {
 
 You can also cancel a long-running prune with an `AbortSignal` — the loop stops between batches and returns partial results with `done: false`, so the next run resumes cleanly.
 
+## MongoDB TTL indexes (alternative to prune)
+
+MongoDB offers native [TTL (Time-To-Live) indexes](https://www.mongodb.com/docs/manual/core/index-ttl/) that automatically delete expired documents without requiring manual `prune()` calls. This is a database-level feature that runs as a background thread.
+
+> **When to use TTL vs prune():** **Use MongoDB TTL indexes when:**
+>
+> - You want automated, zero-maintenance deletion
+> - Your retention periods are fixed (e.g., "always 30 days")
+> - You prefer database-native solutions
+>
+> **Use `prune()` when:**
+>
+> - You need fine-grained control over deletion timing
+> - You want to cap deletion rate during business hours
+> - You need resumable, cancellable cleanup operations
+> - You're using composite storage with multiple databases
+>
+> Both approaches are valid. TTL is simpler; `prune()` gives more control.
+
+### Setting up TTL indexes on MongoDB
+
+TTL indexes work on date fields. MongoDB checks the index every 60 seconds and deletes documents where the date field + TTL duration < current time.
+
+```typescript
+import { MongoDBStore } from '@mastra/mongodb'
+
+const storage = new MongoDBStore({
+  id: 'mongodb-storage',
+  uri: process.env.MONGODB_URI!,
+  dbName: process.env.MONGODB_DB_NAME!,
+  indexes: [
+    // Messages expire after 30 days
+    {
+      collection: 'mastra_messages',
+      keys: { createdAt: 1 },
+      options: { expireAfterSeconds: 30 * 24 * 60 * 60 }, // 30 days
+    },
+    // Threads expire after 90 days
+    {
+      collection: 'mastra_threads',
+      keys: { createdAt: 1 },
+      options: { expireAfterSeconds: 90 * 24 * 60 * 60 }, // 90 days
+    },
+    // Spans expire after 7 days
+    {
+      collection: 'mastra_ai_spans',
+      keys: { startedAt: 1 },
+      options: { expireAfterSeconds: 7 * 24 * 60 * 60 }, // 7 days
+    },
+  ],
+})
+```
+
+> **Tip:** TTL indexes delete documents shortly after they expire (background thread runs every \~60 seconds), but the exact timing is not guaranteed. For precise, immediate cleanup, use `prune()` instead.
+
 ## Reclaiming disk
 
-`prune()` deletes rows but does not shrink the database file. On SQLite/libSQL the freed pages go on a freelist and are reused by future writes, so the file stops growing — for most users this alone solves the unbounded-growth problem.
+`prune()` deletes rows but doesn't shrink the database file. On SQLite/libSQL the freed pages go on a freelist and are reused by future writes, so the file stops growing — for most users this alone solves the unbounded-growth problem.
+
+Handing that free space back to the OS is a separate concern that Mastra doesn't manage. If you specifically need to shrink the file, run the underlying database's compaction (for example `VACUUM` on self-hosted libSQL) yourself, in a maintenance window — a full `VACUUM` locks the file and needs roughly twice the file size in free disk. On PostgreSQL, autovacuum reclaims dead tuples for reuse automatically; a manual `VACUUM FULL` is only needed if you must return disk to the OS.
 
-Handing that free space back to the OS is a separate concern that Mastra does not manage. If you specifically need to shrink the file, run the underlying database's compaction (for example `VACUUM` on self-hosted libSQL) yourself, in a maintenance window — a full `VACUUM` locks the file and needs roughly twice the file size in free disk. On PostgreSQL, autovacuum reclaims dead tuples for reuse automatically; a manual `VACUUM FULL` is only needed if you must return disk to the OS.
+For MongoDB, deleted documents are reused by future insertions. To reclaim disk space, run [`db.runCommand({ compact: "collection_name" })`](https://www.mongodb.com/docs/manual/reference/command/compact/) during a maintenance window.
 
 > **LibSQL and Turso:** [Turso Cloud](https://mastra.ai/reference/storage/libsql) manages storage compaction for you, so there's nothing to reclaim manually. This applies only to self-hosted libSQL files.
 

package/dist/index.cjs

@@ -5452,6 +5452,7 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
     await this.#addColumnIfNotExists(storage.TABLE_DATASET_ITEMS, "organizationId", "TEXT");
     await this.#addColumnIfNotExists(storage.TABLE_DATASET_ITEMS, "projectId", "TEXT");
     await this.#addColumnIfNotExists(storage.TABLE_DATASET_ITEMS, "toolMocks", "JSONB");
+    await this.#addColumnIfNotExists(storage.TABLE_DATASET_ITEMS, "externalId", "TEXT");
     await this.createDefaultIndexes();
     await this.createCustomIndexes();
   }
@@ -5470,6 +5471,11 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
         table: storage.TABLE_DATASET_ITEMS,
         columns: ["datasetId", "datasetVersion"]
       },
+      {
+        name: "idx_dataset_items_external_id_history",
+        table: storage.TABLE_DATASET_ITEMS,
+        columns: ["datasetId", "externalId", "datasetVersion"]
+      },
       {
         name: "idx_dataset_items_dataset_validto_deleted",
         table: storage.TABLE_DATASET_ITEMS,
@@ -5552,6 +5558,7 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
       id: row.id,
       datasetId: row.datasetId,
       datasetVersion: row.datasetVersion,
+      externalId: row.externalId ?? null,
       organizationId: row.organizationId ?? null,
       projectId: row.projectId ?? null,
       input: storage.safelyParseJSON(row.input),
@@ -5570,6 +5577,7 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
       id: row.id,
       datasetId: row.datasetId,
       datasetVersion: row.datasetVersion,
+      externalId: row.externalId ?? null,
       organizationId: row.organizationId ?? null,
       projectId: row.projectId ?? null,
       validTo: row.validTo,
@@ -5596,7 +5604,8 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
   // --- Dataset CRUD ---
   async createDataset(input) {
     try {
-      const id = crypto.randomUUID();
+      const id = input.id ?? crypto.randomUUID();
+      if (input.id !== void 0) this.validateCallerDefinedDatasetId(input.id);
       const now = /* @__PURE__ */ new Date();
       const nowIso = now.toISOString();
       await this.#db.insert({
@@ -5641,6 +5650,11 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
         updatedAt: now
       };
     } catch (error$1) {
+      if (input.id !== void 0 && storage.hasErrorCode(error$1, /* @__PURE__ */ new Set(["23505"]))) {
+        const existing = await this.getDatasetById({ id: input.id });
+        if (existing) return this.resolveExistingDataset(existing, { ...input, id: input.id });
+      }
+      if (error$1 instanceof error.MastraError) throw error$1;
       throw new error.MastraError(
         {
           id: storage.createStorageErrorId("PG", "CREATE_DATASET", "FAILED"),
@@ -5913,11 +5927,12 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
         parentOrganizationId = row.organizationId ?? null;
         parentProjectId = row.projectId ?? null;
         await t.none(
-          `INSERT INTO ${itemsTable} ("id","datasetId","datasetVersion","organizationId","projectId","validTo","isDeleted","input","groundTruth","expectedTrajectory","toolMocks","requestContext","metadata","source","createdAt","createdAtZ","updatedAt","updatedAtZ") VALUES ($1,$2,$3,$4,$5,NULL,false,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16)`,
+          `INSERT INTO ${itemsTable} ("id","datasetId","datasetVersion","externalId","organizationId","projectId","validTo","isDeleted","input","groundTruth","expectedTrajectory","toolMocks","requestContext","metadata","source","createdAt","createdAtZ","updatedAt","updatedAtZ") VALUES ($1,$2,$3,$4,$5,$6,NULL,false,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17)`,
           [
             id,
             args.datasetId,
             newVersion,
+            args.externalId ?? null,
             parentOrganizationId,
             parentProjectId,
             JSON.stringify(args.input),
@@ -6017,11 +6032,12 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
           [newVersion, args.id]
         );
         await t.none(
-          `INSERT INTO ${itemsTable} ("id","datasetId","datasetVersion","organizationId","projectId","validTo","isDeleted","input","groundTruth","expectedTrajectory","toolMocks","requestContext","metadata","source","createdAt","createdAtZ","updatedAt","updatedAtZ") VALUES ($1,$2,$3,$4,$5,NULL,false,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16)`,
+          `INSERT INTO ${itemsTable} ("id","datasetId","datasetVersion","externalId","organizationId","projectId","validTo","isDeleted","input","groundTruth","expectedTrajectory","toolMocks","requestContext","metadata","source","createdAt","createdAtZ","updatedAt","updatedAtZ") VALUES ($1,$2,$3,$4,$5,$6,NULL,false,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17)`,
           [
             args.id,
             args.datasetId,
             newVersion,
+            existing.externalId ?? null,
             parentOrganizationId,
             parentProjectId,
             JSON.stringify(mergedInput),
@@ -6101,11 +6117,12 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
           [newVersion, id]
         );
         await t.none(
-          `INSERT INTO ${itemsTable} ("id","datasetId","datasetVersion","organizationId","projectId","validTo","isDeleted","input","groundTruth","expectedTrajectory","toolMocks","requestContext","metadata","source","createdAt","createdAtZ","updatedAt","updatedAtZ") VALUES ($1,$2,$3,$4,$5,NULL,true,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16)`,
+          `INSERT INTO ${itemsTable} ("id","datasetId","datasetVersion","externalId","organizationId","projectId","validTo","isDeleted","input","groundTruth","expectedTrajectory","toolMocks","requestContext","metadata","source","createdAt","createdAtZ","updatedAt","updatedAtZ") VALUES ($1,$2,$3,$4,$5,$6,NULL,true,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17)`,
           [
             id,
             datasetId,
             newVersion,
+            existing.externalId ?? null,
             parentOrganizationId,
             parentProjectId,
             JSON.stringify(existing.input),
@@ -6140,78 +6157,93 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
   }
   async _doBatchInsertItems(input) {
     try {
-      const dataset = await this.getDatasetById({ id: input.datasetId });
-      if (!dataset) {
-        throw new error.MastraError({
-          id: storage.createStorageErrorId("PG", "BULK_ADD_ITEMS", "DATASET_NOT_FOUND"),
-          domain: error.ErrorDomain.STORAGE,
-          category: error.ErrorCategory.USER,
-          details: { datasetId: input.datasetId }
-        });
-      }
+      if (input.items.length === 0) return [];
       const datasetsTable = getTableName2({ indexName: storage.TABLE_DATASETS, schemaName: getSchemaName2(this.#schema) });
       const itemsTable = getTableName2({ indexName: storage.TABLE_DATASET_ITEMS, schemaName: getSchemaName2(this.#schema) });
       const versionsTable = getTableName2({
         indexName: storage.TABLE_DATASET_VERSIONS,
         schemaName: getSchemaName2(this.#schema)
       });
-      const now = /* @__PURE__ */ new Date();
-      const nowIso = now.toISOString();
-      const versionId = crypto.randomUUID();
-      const itemsWithIds = input.items.map((itemInput) => ({ id: crypto.randomUUID(), input: itemInput }));
-      const parentOrganizationId = dataset.organizationId ?? null;
-      const parentProjectId = dataset.projectId ?? null;
-      let newVersion;
+      let results = [];
       await this.#db.client.tx(async (t) => {
-        const row = await t.one(
-          `UPDATE ${datasetsTable} SET "version" = "version" + 1 WHERE "id" = $1 RETURNING "version"`,
+        const dataset = await t.oneOrNone(
+          `SELECT "version", "organizationId", "projectId" FROM ${datasetsTable} WHERE "id" = $1 FOR UPDATE`,
           [input.datasetId]
         );
-        newVersion = row.version;
-        for (const { id, input: itemInput } of itemsWithIds) {
-          await t.none(
-            `INSERT INTO ${itemsTable} ("id","datasetId","datasetVersion","organizationId","projectId","validTo","isDeleted","input","groundTruth","expectedTrajectory","toolMocks","requestContext","metadata","source","createdAt","createdAtZ","updatedAt","updatedAtZ") VALUES ($1,$2,$3,$4,$5,NULL,false,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16)`,
-            [
+        if (!dataset)
+          throw new error.MastraError({
+            id: storage.createStorageErrorId("PG", "BULK_ADD_ITEMS", "DATASET_NOT_FOUND"),
+            domain: error.ErrorDomain.STORAGE,
+            category: error.ErrorCategory.USER,
+            details: { datasetId: input.datasetId }
+          });
+        const externalIds = [...new Set(input.items.flatMap((item) => item.externalId ? [item.externalId] : []))];
+        const historyRows = externalIds.length ? await t.manyOrNone(
+          `SELECT * FROM ${itemsTable} WHERE "datasetId" = $1 AND "externalId" = ANY($2::text[]) ORDER BY "datasetVersion"`,
+          [input.datasetId, externalIds]
+        ) : [];
+        const plan = this.planDatasetItemBatch(
+          input.items,
+          historyRows.map((row) => this.transformItemRowFull(row)),
+          () => crypto.randomUUID()
+        );
+        const resolved = new Map(
+          [...plan.existingCurrentItems].map(([id, row]) => [id, this.datasetItemFromRow(row)])
+        );
+        if (plan.inserts.length > 0) {
+          const newVersion = Number(dataset.version) + 1;
+          const now = /* @__PURE__ */ new Date();
+          const nowIso = now.toISOString();
+          await t.none(`UPDATE ${datasetsTable} SET "version" = $2 WHERE "id" = $1`, [input.datasetId, newVersion]);
+          for (const { id, item } of plan.inserts) {
+            await t.none(
+              `INSERT INTO ${itemsTable} ("id","datasetId","datasetVersion","externalId","organizationId","projectId","validTo","isDeleted","input","groundTruth","expectedTrajectory","toolMocks","requestContext","metadata","source","createdAt","createdAtZ","updatedAt","updatedAtZ") VALUES ($1,$2,$3,$4,$5,$6,NULL,false,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17)`,
+              [
+                id,
+                input.datasetId,
+                newVersion,
+                item.externalId ?? null,
+                dataset.organizationId ?? null,
+                dataset.projectId ?? null,
+                JSON.stringify(item.input),
+                jsonbArg(item.groundTruth),
+                jsonbArg(item.expectedTrajectory),
+                jsonbArg(item.toolMocks),
+                jsonbArg(item.requestContext),
+                jsonbArg(item.metadata),
+                jsonbArg(item.source),
+                nowIso,
+                nowIso,
+                nowIso,
+                nowIso
+              ]
+            );
+            resolved.set(id, {
               id,
-              input.datasetId,
-              newVersion,
-              parentOrganizationId,
-              parentProjectId,
-              JSON.stringify(itemInput.input),
-              jsonbArg(itemInput.groundTruth),
-              jsonbArg(itemInput.expectedTrajectory),
-              jsonbArg(itemInput.toolMocks),
-              jsonbArg(itemInput.requestContext),
-              jsonbArg(itemInput.metadata),
-              jsonbArg(itemInput.source),
-              nowIso,
-              nowIso,
-              nowIso,
-              nowIso
-            ]
+              datasetId: input.datasetId,
+              datasetVersion: newVersion,
+              externalId: item.externalId ?? null,
+              organizationId: dataset.organizationId ?? null,
+              projectId: dataset.projectId ?? null,
+              input: item.input,
+              groundTruth: item.groundTruth,
+              expectedTrajectory: item.expectedTrajectory,
+              toolMocks: item.toolMocks,
+              requestContext: item.requestContext,
+              metadata: item.metadata,
+              source: item.source,
+              createdAt: now,
+              updatedAt: now
+            });
+          }
+          await t.none(
+            `INSERT INTO ${versionsTable} ("id","datasetId","version","createdAt","createdAtZ") VALUES ($1,$2,$3,$4,$5)`,
+            [crypto.randomUUID(), input.datasetId, newVersion, nowIso, nowIso]
           );
         }
-        await t.none(
-          `INSERT INTO ${versionsTable} ("id","datasetId","version","createdAt","createdAtZ") VALUES ($1,$2,$3,$4,$5)`,
-          [versionId, input.datasetId, newVersion, nowIso, nowIso]
-        );
+        results = plan.resolvedIds.map((id) => resolved.get(id));
       });
-      return itemsWithIds.map(({ id, input: itemInput }) => ({
-        id,
-        datasetId: input.datasetId,
-        datasetVersion: newVersion,
-        organizationId: parentOrganizationId,
-        projectId: parentProjectId,
-        input: itemInput.input,
-        groundTruth: itemInput.groundTruth,
-        expectedTrajectory: itemInput.expectedTrajectory,
-        toolMocks: itemInput.toolMocks,
-        requestContext: itemInput.requestContext,
-        metadata: itemInput.metadata,
-        source: itemInput.source,
-        createdAt: now,
-        updatedAt: now
-      }));
+      return results;
     } catch (error$1) {
       if (error$1 instanceof error.MastraError) throw error$1;
       throw new error.MastraError(
@@ -6265,11 +6297,12 @@ var DatasetsPG = class _DatasetsPG extends storage.DatasetsStorage {
             [newVersion, item.id]
           );
           await t.none(
-            `INSERT INTO ${itemsTable} ("id","datasetId","datasetVersion","organizationId","projectId","validTo","isDeleted","input","groundTruth","expectedTrajectory","toolMocks","requestContext","metadata","source","createdAt","createdAtZ","updatedAt","updatedAtZ") VALUES ($1,$2,$3,$4,$5,NULL,true,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16)`,
+            `INSERT INTO ${itemsTable} ("id","datasetId","datasetVersion","externalId","organizationId","projectId","validTo","isDeleted","input","groundTruth","expectedTrajectory","toolMocks","requestContext","metadata","source","createdAt","createdAtZ","updatedAt","updatedAtZ") VALUES ($1,$2,$3,$4,$5,$6,NULL,true,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17)`,
             [
               item.id,
               input.datasetId,
               newVersion,
+              item.externalId ?? null,
               parentOrganizationId,
               parentProjectId,
               JSON.stringify(item.input),