@mastra/pg 1.26.0-alpha.1 → 1.26.0-alpha.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/docs/SKILL.md +1 -1
- package/dist/docs/assets/SOURCE_MAP.json +1 -1
- package/dist/docs/references/reference-rag-vector-databases.md +34 -0
- package/dist/docs/references/reference-storage-retention.md +56 -4
- package/dist/index.cjs +194 -161
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +195 -162
- package/dist/index.js.map +1 -1
- package/dist/storage/domains/memory/index.d.ts.map +1 -1
- package/dist/storage/domains/observability/v-next/ddl.d.ts.map +1 -1
- package/dist/storage/domains/observability/v-next/index.d.ts.map +1 -1
- package/dist/storage/domains/observability/v-next/scores.d.ts +1 -0
- package/dist/storage/domains/observability/v-next/scores.d.ts.map +1 -1
- package/dist/storage/domains/observability/v-next/trace-query.d.ts +4 -2
- package/dist/storage/domains/observability/v-next/trace-query.d.ts.map +1 -1
- package/dist/storage/retention.d.ts +5 -58
- package/dist/storage/retention.d.ts.map +1 -1
- package/package.json +4 -4
package/dist/docs/SKILL.md
CHANGED
|
@@ -345,6 +345,29 @@ await store.upsert({
|
|
|
345
345
|
|
|
346
346
|
For detailed setup instructions and best practices, see the [official Elasticsearch documentation](https://www.elastic.co/docs/solutions/search/get-started).
|
|
347
347
|
|
|
348
|
+
**Azure AI Search**:
|
|
349
|
+
|
|
350
|
+
```ts
|
|
351
|
+
import { AzureAISearchVector } from '@mastra/azure-ai-search'
|
|
352
|
+
|
|
353
|
+
const store = new AzureAISearchVector({
|
|
354
|
+
id: 'azure-search-vectors',
|
|
355
|
+
endpoint: process.env.AZURE_AI_SEARCH_ENDPOINT!,
|
|
356
|
+
credential: process.env.AZURE_AI_SEARCH_CREDENTIAL!,
|
|
357
|
+
})
|
|
358
|
+
|
|
359
|
+
await store.createIndex({
|
|
360
|
+
indexName: 'my-collection',
|
|
361
|
+
dimension: 1536,
|
|
362
|
+
})
|
|
363
|
+
|
|
364
|
+
await store.upsert({
|
|
365
|
+
indexName: 'my-collection',
|
|
366
|
+
vectors: embeddings,
|
|
367
|
+
metadata: chunks.map(chunk => ({ text: chunk.text })),
|
|
368
|
+
})
|
|
369
|
+
```
|
|
370
|
+
|
|
348
371
|
**Couchbase**:
|
|
349
372
|
|
|
350
373
|
```ts
|
|
@@ -626,6 +649,17 @@ Index names must:
|
|
|
626
649
|
- Example: `myindex-` isn't valid (ends with hyphen)
|
|
627
650
|
- Example: `MyIndex` isn't valid (contains uppercase letters)
|
|
628
651
|
|
|
652
|
+
**Azure AI Search**:
|
|
653
|
+
|
|
654
|
+
Index names must:
|
|
655
|
+
|
|
656
|
+
- Use only lowercase letters, numbers, dashes (`-`), and underscore (`_`) characters
|
|
657
|
+
- Not start or end with a dash
|
|
658
|
+
- Be between 2 and 128 characters long
|
|
659
|
+
- Example: `my-index-123` and `my_index` are valid
|
|
660
|
+
- Example: `MyIndex` isn't valid (contains uppercase letters)
|
|
661
|
+
- Example: `my-index-` isn't valid (ends with a dash)
|
|
662
|
+
|
|
629
663
|
### Upserting Embeddings
|
|
630
664
|
|
|
631
665
|
After creating an index, you can store embeddings along with their basic metadata:
|
|
@@ -6,11 +6,37 @@
|
|
|
6
6
|
|
|
7
7
|
Because storage grows without bound by default, Mastra provides an opt-in, age-based retention system. Declare per-table `maxAge` policies in the `retention` config, then call `storage.prune()` to delete rows older than their configured age. Unconfigured data is kept forever, so behavior doesn't change until you opt in.
|
|
8
8
|
|
|
9
|
-
`prune()` deletes rows.
|
|
9
|
+
`prune()` deletes rows in bounded batches. Runs are resumable and cancellable, so you can limit how much work each maintenance window performs. Pruning doesn't reclaim disk space by itself. Use the database-specific maintenance guidance below when you need to return freed space to the operating system.
|
|
10
10
|
|
|
11
11
|
Retention covers **growth tables** only: tables that accumulate rows unbounded as a side effect of normal operation (conversation history, telemetry, job and run records, schedule fire history, event feeds). User-authored artifacts and config (agents, skills, workspaces, prompt blocks, datasets, schedule definitions, channel installations, and so on) grow with user intent and are edited or deleted explicitly, so they're not valid retention keys.
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
Storage adapters use the shared core retention contract for `prune()`, or a database-native mechanism when that better matches the backend.
|
|
14
|
+
|
|
15
|
+
| Adapter | Mechanism | Retention support |
|
|
16
|
+
| -------------------- | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
17
|
+
| libSQL | `prune()` | All supported growth domains |
|
|
18
|
+
| PostgreSQL | `prune()` | All supported growth domains. V-next observability drops expired partitions or chunks |
|
|
19
|
+
| MongoDB | `prune()` or native TTL | All supported growth domains. Native TTL indexes are also available |
|
|
20
|
+
| DuckDB | `prune()` | Observability spans, metrics, logs, scores, and feedback |
|
|
21
|
+
| MySQL | `prune()` | Observability spans |
|
|
22
|
+
| Microsoft SQL Server | `prune()` | Observability spans |
|
|
23
|
+
| Oracle Database | `prune()` | Observability spans and logs |
|
|
24
|
+
| Amazon Aurora DSQL | `prune()` | Observability spans |
|
|
25
|
+
| Google Cloud Spanner | `prune()` | Observability spans, plus metrics when metrics storage is enabled |
|
|
26
|
+
| ClickHouse | Native TTL | Observability spans, metrics, logs, scores, and feedback. When all five signals have finite retention, deletion-request records expire after the longest signal retention plus 30 days |
|
|
27
|
+
|
|
28
|
+
## Storage-specific maintenance
|
|
29
|
+
|
|
30
|
+
| Adapter | Maintenance guidance |
|
|
31
|
+
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
32
|
+
| SQLite and libSQL | Freed pages are reused by future writes, which stops the database file from growing. Reclaiming disk space requires database-level maintenance. |
|
|
33
|
+
| DuckDB | For file-backed stores, run `CHECKPOINT` after pruning to reclaim deleted rows in storage. DuckDB's `VACUUM` doesn't reclaim deleted rows. |
|
|
34
|
+
|
|
35
|
+
## Schedule pruning
|
|
36
|
+
|
|
37
|
+
Run `prune()` from a scheduler or maintenance worker, not from application startup or shutdown hooks. For deployments that share a database, prefer a single active scheduler or worker for pruning.
|
|
38
|
+
|
|
39
|
+
Prefer lower-traffic periods when pruning large tables. Use `maxBatches`, `maxRows`, and `pauseMs` to bound each run, and pass an `AbortSignal` when the maintenance process needs to stop promptly. These recommendations apply to adapters that expose `prune()`. ClickHouse applies its native time to live (TTL) policy within the database.
|
|
14
40
|
|
|
15
41
|
## Usage example
|
|
16
42
|
|
|
@@ -94,7 +120,8 @@ Each domain specifies its age-prunable tables and the timestamp column that anch
|
|
|
94
120
|
> - Experiments prune as whole units: an aged experiment's result rows are deleted together with it (results cascade with their parent), so a run is never left partially deleted. Retention doesn't have a separate `results` key.
|
|
95
121
|
> - For `schedules`, the growth table is the fire history (`schedule_triggers`, one row per fire): schedule definitions are config and aren't pruned.
|
|
96
122
|
> - On PostgreSQL, timestamp anchors use the timezone-aware mirror columns (for example `createdAtZ`, `completedAtZ`).
|
|
97
|
-
> -
|
|
123
|
+
> - DuckDB observability stores append-only events for all five signals. Its `spans` policy uses the event `timestamp` column rather than `startedAt`.
|
|
124
|
+
> - LibSQL and PostgreSQL support all domains above except `harness`, which PostgreSQL doesn't implement. MongoDB supports all except `threadState` and `harness`. DuckDB, MySQL, Microsoft SQL Server, Oracle Database, Amazon Aurora DSQL, and Google Cloud Spanner currently support retention only in their `observability` domains, with the signal coverage shown in the support matrix.
|
|
98
125
|
> - The v-next PostgreSQL observability domain stores signal events in day-partitioned tables (`spans`, `metrics`, `logs`, `scores`, `feedback`). For it, `prune()` drops whole day partitions (or TimescaleDB chunks) that are entirely older than the cutoff instead of deleting rows: effective level of detail is one day, and a partition is only dropped once its entire day is past `maxAge`. `PruneResult.deleted` reports the number of rows in the dropped partitions.
|
|
99
126
|
|
|
100
127
|
## Methods
|
|
@@ -109,7 +136,7 @@ Deletes rows older than their configured `maxAge` across every domain that has a
|
|
|
109
136
|
|
|
110
137
|
Pass `options.retention` to replace the configured policies for that call only: for example to skip a domain (keep chat history) or prune more aggressively than the standing config. The store's configured `retention` is unchanged.
|
|
111
138
|
|
|
112
|
-
|
|
139
|
+
Adapters that use anchor-column indexes create them lazily on the first `prune()` call for each table with a policy (never at `init()`) so deployments that don't configure retention pay no extra index write or disk overhead. The first prune of an existing large table pays a one-time index build. Subsequent prunes reuse the index. DuckDB uses its built-in zone maps instead of creating retention indexes.
|
|
113
140
|
|
|
114
141
|
```typescript
|
|
115
142
|
const results = await storage.prune({
|
|
@@ -177,6 +204,31 @@ async function retentionTick() {
|
|
|
177
204
|
|
|
178
205
|
You can also cancel a long-running prune with an `AbortSignal`: the loop stops between batches and returns partial results with `done: false`, so the next run resumes cleanly.
|
|
179
206
|
|
|
207
|
+
## ClickHouse native TTL
|
|
208
|
+
|
|
209
|
+
ClickHouse observability storage uses native table TTLs instead of `prune()`. Configure retention as days per signal. `init()` applies the TTLs to new and existing tables and skips `ALTER TABLE` statements when the configured TTL is already present.
|
|
210
|
+
|
|
211
|
+
For deployments that need to update TTL configuration without running the full initialization path, call `applyRetention()` on the v-next observability store:
|
|
212
|
+
|
|
213
|
+
```typescript
|
|
214
|
+
import { ObservabilityStorageClickhouseVNext } from '@mastra/clickhouse'
|
|
215
|
+
|
|
216
|
+
const observability = new ObservabilityStorageClickhouseVNext({
|
|
217
|
+
client,
|
|
218
|
+
retention: {
|
|
219
|
+
tracing: 30,
|
|
220
|
+
logs: 7,
|
|
221
|
+
metrics: 14,
|
|
222
|
+
scores: 90,
|
|
223
|
+
feedback: 60,
|
|
224
|
+
},
|
|
225
|
+
})
|
|
226
|
+
|
|
227
|
+
await observability.applyRetention()
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
Deletion requests are retained long enough to keep enforcing erasure after signal rows expire. Mastra applies a TTL to `mastra_deletion_requests` only when tracing, logs, metrics, scores, and feedback all have finite retention. The deletion-request TTL is the longest of those periods plus 30 days. For example, if score retention is the longest period at 90 days, deletion requests expire after 120 days. When any signal is unbounded, deletion requests remain unbounded because trace deletion requests cover rows across all five signals.
|
|
231
|
+
|
|
180
232
|
## MongoDB TTL indexes (alternative to prune)
|
|
181
233
|
|
|
182
234
|
MongoDB offers native [TTL (Time-To-Live) indexes](https://www.mongodb.com/docs/manual/core/index-ttl/) that automatically delete expired documents without requiring manual `prune()` calls. This is a database-level feature that runs as a background thread.
|