@mastra/pg 1.20.0-alpha.3 → 1.20.0-alpha.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +11 -0
- package/dist/docs/SKILL.md +9 -7
- package/dist/docs/assets/SOURCE_MAP.json +1 -1
- package/dist/docs/references/docs-deployment-workers.md +1 -1
- package/dist/docs/references/docs-storage-overview.md +14 -13
- package/dist/docs/references/{reference-storage-dynamodb.md → integrations-databases-dynamodb.md} +1 -1
- package/dist/docs/references/integrations-databases-neon.md +220 -0
- package/dist/docs/references/{reference-storage-postgresql.md → integrations-databases-postgresql.md} +1 -1
- package/dist/docs/references/reference-rag-chunking-and-embedding.md +182 -0
- package/dist/docs/references/{guides-rag-overview.md → reference-rag-overview.md} +2 -2
- package/dist/docs/references/{guides-rag-retrieval.md → reference-rag-retrieval.md} +1 -1
- package/dist/docs/references/{guides-rag-vector-databases.md → reference-rag-vector-databases.md} +1 -1
- package/dist/docs/references/reference-storage-composite.md +4 -4
- package/dist/docs/references/reference-storage-retention.md +4 -4
- package/dist/index.cjs +1 -0
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/storage/domains/memory/index.d.ts +1 -0
- package/dist/storage/domains/memory/index.d.ts.map +1 -1
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,16 @@
|
|
|
1
1
|
# @mastra/pg
|
|
2
2
|
|
|
3
|
+
## 1.20.0-alpha.4
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- Fixed a crash where updating a thread without a title (for example during observational memory buffering) could write a null title and violate the database's not-null constraint when running a newer @mastra/memory against an older storage package. Memory now checks whether the connected storage adapter supports partial thread updates and backfills the existing title for older adapters, so mixed-version deployments keep working. See #21041 for the original title-clobbering fix this makes backward compatible. ([#21257](https://github.com/mastra-ai/mastra/pull/21257))
|
|
8
|
+
|
|
9
|
+
- Storage adapters now declare support for partial thread updates, letting newer @mastra/memory preserve existing thread titles instead of overwriting them, while remaining safe against older versions. ([#21257](https://github.com/mastra-ai/mastra/pull/21257))
|
|
10
|
+
|
|
11
|
+
- Updated dependencies [[`dc4a25d`](https://github.com/mastra-ai/mastra/commit/dc4a25d41af4e2fe97a816070eaec6aa963ab53b), [`dc4a25d`](https://github.com/mastra-ai/mastra/commit/dc4a25d41af4e2fe97a816070eaec6aa963ab53b)]:
|
|
12
|
+
- @mastra/core@1.58.0-alpha.15
|
|
13
|
+
|
|
3
14
|
## 1.20.0-alpha.3
|
|
4
15
|
|
|
5
16
|
### Patch Changes
|
package/dist/docs/SKILL.md
CHANGED
|
@@ -3,7 +3,7 @@ name: mastra-pg
|
|
|
3
3
|
description: Documentation for @mastra/pg. Use when working with @mastra/pg APIs, configuration, or implementation.
|
|
4
4
|
metadata:
|
|
5
5
|
package: "@mastra/pg"
|
|
6
|
-
version: "1.20.0-alpha.
|
|
6
|
+
version: "1.20.0-alpha.4"
|
|
7
7
|
---
|
|
8
8
|
|
|
9
9
|
## When to use
|
|
@@ -21,11 +21,11 @@ Read the individual reference documents for detailed explanations and code examp
|
|
|
21
21
|
- [Working memory](references/docs-memory-working-memory.md) - Learn how to configure working memory in Mastra to store persistent user data, preferences.
|
|
22
22
|
- [Storage overview](references/docs-storage-overview.md) - Configure storage for Mastra to persist runtime state across agents, workflows, observability, evals, schedules, and memory.
|
|
23
23
|
|
|
24
|
-
###
|
|
24
|
+
### Integrations
|
|
25
25
|
|
|
26
|
-
- [
|
|
27
|
-
- [
|
|
28
|
-
- [
|
|
26
|
+
- [DynamoDB](references/integrations-databases-dynamodb.md) - Documentation for the DynamoDB storage implementation in Mastra, using a single-table design with ElectroDB.
|
|
27
|
+
- [Neon Postgres](references/integrations-databases-neon.md) - Use Neon Postgres as a Mastra storage and vector backend.
|
|
28
|
+
- [PostgreSQL](references/integrations-databases-postgresql.md) - Documentation for the PostgreSQL storage implementation in Mastra.
|
|
29
29
|
|
|
30
30
|
### Reference
|
|
31
31
|
|
|
@@ -33,10 +33,12 @@ Read the individual reference documents for detailed explanations and code examp
|
|
|
33
33
|
- [Reference: MessageHistory](references/reference-processors-message-history-processor.md) - Documentation for the MessageHistory processor in Mastra, which handles retrieval and persistence of conversation history.
|
|
34
34
|
- [Reference: SemanticRecall](references/reference-processors-semantic-recall-processor.md) - Documentation for the SemanticRecall processor in Mastra, which enables semantic search over conversation history using vector embeddings.
|
|
35
35
|
- [Reference: WorkingMemory](references/reference-processors-working-memory-processor.md) - Documentation for the WorkingMemory processor in Mastra, which injects persistent user/context data as system instructions.
|
|
36
|
+
- [Chunking and embedding documents](references/reference-rag-chunking-and-embedding.md) - Guide on chunking and embedding documents in Mastra for efficient processing and retrieval.
|
|
36
37
|
- [Reference: Metadata filters](references/reference-rag-metadata-filters.md) - Documentation for metadata filtering capabilities in Mastra, which allow for precise querying of vector search results across different vector stores.
|
|
38
|
+
- [RAG (Retrieval-Augmented Generation) in Mastra](references/reference-rag-overview.md) - Overview of Retrieval-Augmented Generation (RAG) in Mastra, detailing its capabilities for enhancing LLM outputs with relevant context.
|
|
39
|
+
- [Retrieval, semantic search, reranking](references/reference-rag-retrieval.md) - Guide on retrieval processes in Mastra's RAG systems, including semantic search, filtering, and re-ranking.
|
|
40
|
+
- [Storing embeddings in a vector database](references/reference-rag-vector-databases.md) - Guide on vector storage options in Mastra, including embedded and dedicated vector databases for similarity search.
|
|
37
41
|
- [Reference: Composite storage](references/reference-storage-composite.md) - Documentation for combining multiple storage backends in Mastra.
|
|
38
|
-
- [Reference: DynamoDB storage](references/reference-storage-dynamodb.md) - Documentation for the DynamoDB storage implementation in Mastra, using a single-table design with ElectroDB.
|
|
39
|
-
- [Reference: PostgreSQL storage](references/reference-storage-postgresql.md) - Documentation for the PostgreSQL storage implementation in Mastra.
|
|
40
42
|
- [Reference: Storage retention (prune)](references/reference-storage-retention.md) - API reference for retention policies and prune() on Mastra storage.
|
|
41
43
|
- [Reference: createVectorQueryTool()](references/reference-tools-vector-query-tool.md) - Documentation for the Vector Query Tool in Mastra, which facilitates semantic search over vector stores with filtering and reranking capabilities.
|
|
42
44
|
- [Reference: PG vector store](references/reference-vectors-pg.md) - Documentation for the PgVector class in Mastra, which provides vector search using PostgreSQL with pgvector extension.
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
# Workers
|
|
4
4
|
|
|
5
|
-
> **Beta:**
|
|
5
|
+
> **Beta:** Breaking changes may occur without a major version bump until the API is stable. See [known limitations](#known-limitations) for current gaps.
|
|
6
6
|
|
|
7
7
|
Workers handle background processing outside the request-response cycle. Workflow step execution, cron-based scheduling, and long-running tool calls all run in workers, keeping the API responsive.
|
|
8
8
|
|
|
@@ -190,19 +190,20 @@ You can also route `observability` to a dedicated analytics backend. See the [ob
|
|
|
190
190
|
|
|
191
191
|
Each provider page includes installation instructions, configuration parameters, and usage examples:
|
|
192
192
|
|
|
193
|
-
- [
|
|
194
|
-
- [
|
|
195
|
-
- [
|
|
196
|
-
- [
|
|
197
|
-
- [
|
|
198
|
-
- [
|
|
199
|
-
- [
|
|
200
|
-
- [
|
|
201
|
-
- [
|
|
202
|
-
- [
|
|
203
|
-
- [
|
|
204
|
-
- [
|
|
205
|
-
- [
|
|
193
|
+
- [Cloudflare D1](https://mastra.ai/integrations/databases/cloudflare-d1)
|
|
194
|
+
- [Cloudflare KV & Durable Objects](https://mastra.ai/integrations/databases/cloudflare-kv)
|
|
195
|
+
- [Convex](https://mastra.ai/integrations/databases/convex)
|
|
196
|
+
- [DynamoDB](https://mastra.ai/integrations/databases/dynamodb)
|
|
197
|
+
- [Google Cloud Spanner](https://mastra.ai/integrations/databases/spanner)
|
|
198
|
+
- [LanceDB](https://mastra.ai/integrations/databases/lancedb)
|
|
199
|
+
- [libSQL](https://mastra.ai/integrations/databases/libsql)
|
|
200
|
+
- [Microsoft SQL Server](https://mastra.ai/integrations/databases/mssql)
|
|
201
|
+
- [MongoDB](https://mastra.ai/integrations/databases/mongodb)
|
|
202
|
+
- [Neon Postgres](https://mastra.ai/integrations/databases/neon)
|
|
203
|
+
- [OracleDB](https://mastra.ai/integrations/databases/oracledb)
|
|
204
|
+
- [PostgreSQL](https://mastra.ai/integrations/databases/postgresql)
|
|
205
|
+
- [Redis](https://mastra.ai/integrations/databases/redis)
|
|
206
|
+
- [Upstash](https://mastra.ai/integrations/databases/upstash)
|
|
206
207
|
|
|
207
208
|
> **Tip:** libSQL is the fastest path for local development because it doesn't require running a separate database server.
|
|
208
209
|
|
package/dist/docs/references/{reference-storage-dynamodb.md → integrations-databases-dynamodb.md}
RENAMED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
2
2
|
|
|
3
|
-
# DynamoDB
|
|
3
|
+
# DynamoDB
|
|
4
4
|
|
|
5
5
|
The DynamoDB storage implementation provides a high-capacity and performant NoSQL database solution for Mastra, using a single-table design pattern with [ElectroDB](https://electrodb.dev/).
|
|
6
6
|
|
|
@@ -0,0 +1,220 @@
|
|
|
1
|
+
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
2
|
+
|
|
3
|
+
# Neon Postgres
|
|
4
|
+
|
|
5
|
+
[Neon](https://neon.com) is a managed PostgreSQL service. Mastra connects to Neon through [`PostgresStore`](https://mastra.ai/integrations/databases/postgresql), which uses the Node.js `pg` driver. Neon doesn't require a separate Mastra storage package.
|
|
6
|
+
|
|
7
|
+
## Quickstart
|
|
8
|
+
|
|
9
|
+
Create a Neon project, then copy its [pooled connection string](https://neon.com/docs/connect/connect-from-any-app) into `DATABASE_URL`.
|
|
10
|
+
|
|
11
|
+
Install the PostgreSQL storage package:
|
|
12
|
+
|
|
13
|
+
**npm**:
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
npm install @mastra/pg@latest
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
**pnpm**:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
pnpm add @mastra/pg@latest
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
**Yarn**:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
yarn add @mastra/pg@latest
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
**Bun**:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
bun add @mastra/pg@latest
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Pass the pooled connection string to `PostgresStore`:
|
|
38
|
+
|
|
39
|
+
```typescript
|
|
40
|
+
import { Mastra } from '@mastra/core/mastra'
|
|
41
|
+
import { PostgresStore } from '@mastra/pg'
|
|
42
|
+
|
|
43
|
+
export const mastra = new Mastra({
|
|
44
|
+
storage: new PostgresStore({
|
|
45
|
+
id: 'neon-storage',
|
|
46
|
+
connectionString: process.env.DATABASE_URL!,
|
|
47
|
+
}),
|
|
48
|
+
})
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
On startup, Mastra calls `storage.init()` and creates its tables and indexes in Neon.
|
|
52
|
+
|
|
53
|
+
## Set up Neon with the CLI
|
|
54
|
+
|
|
55
|
+
The Neon CLI requires Node.js 20.19 or newer.
|
|
56
|
+
|
|
57
|
+
Install the [Neon CLI](https://neon.com/docs/cli/install):
|
|
58
|
+
|
|
59
|
+
**npm**:
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
npm install -g neon
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
**pnpm**:
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
pnpm add -g neon
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
**Yarn**:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
yarn global add neon
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
**Bun**:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
bun add --global neon
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
From your project directory, link an existing Neon project or create one:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
neon link
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
`neon link` signs you in and writes project context to `.neon`. It also pulls the current branch's environment variables. Run `neon env pull` again whenever you need to refresh them:
|
|
90
|
+
|
|
91
|
+
```bash
|
|
92
|
+
neon env pull
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
The command writes `DATABASE_URL` and `DATABASE_URL_UNPOOLED` to `.env.local` or to `.env` when that file already exists.
|
|
96
|
+
|
|
97
|
+
### Add `neon.ts`
|
|
98
|
+
|
|
99
|
+
To manage branch settings in the repository, create a [`neon.ts`](https://neon.com/docs/reference/neon-ts) file:
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
neon config init
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
This installs `@neon/config` and `@neon/env` and creates a starter config. For example, the following policy expires new non-default branches after seven days:
|
|
106
|
+
|
|
107
|
+
```typescript
|
|
108
|
+
import { defineConfig } from '@neon/config/v1'
|
|
109
|
+
|
|
110
|
+
export default defineConfig({
|
|
111
|
+
branch: branch => {
|
|
112
|
+
if (branch.isDefault) {
|
|
113
|
+
return {}
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
if (!branch.exists) {
|
|
117
|
+
return { ttl: '7d' }
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
return {}
|
|
121
|
+
},
|
|
122
|
+
})
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Apply the config:
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
neon deploy
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
## Install Neon agent skills
|
|
132
|
+
|
|
133
|
+
Install the [`neon` and `neon-postgres` agent skills](https://neon.com/docs/ai/agent-skills) from your project directory:
|
|
134
|
+
|
|
135
|
+
**npm**:
|
|
136
|
+
|
|
137
|
+
```bash
|
|
138
|
+
npx skills add neondatabase/agent-skills --skill neon --skill neon-postgres -y
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
**pnpm**:
|
|
142
|
+
|
|
143
|
+
```bash
|
|
144
|
+
pnpm dlx skills add neondatabase/agent-skills --skill neon --skill neon-postgres -y
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
**Yarn**:
|
|
148
|
+
|
|
149
|
+
```bash
|
|
150
|
+
yarn dlx skills add neondatabase/agent-skills --skill neon --skill neon-postgres -y
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
**Bun**:
|
|
154
|
+
|
|
155
|
+
```bash
|
|
156
|
+
bun x skills add neondatabase/agent-skills --skill neon --skill neon-postgres -y
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
The skills give compatible coding agents Neon setup instructions and Postgres-specific guidance.
|
|
160
|
+
|
|
161
|
+
## Production connections
|
|
162
|
+
|
|
163
|
+
Neon provides direct and pooled connection strings:
|
|
164
|
+
|
|
165
|
+
- `DATABASE_URL` uses Neon's pooled endpoint. Use it for normal application traffic.
|
|
166
|
+
- `DATABASE_URL_UNPOOLED` connects directly to Postgres. Use it for administrative tasks or schema changes that run outside the application.
|
|
167
|
+
|
|
168
|
+
For most applications, use the pooled `DATABASE_URL` shown in the quickstart and let Mastra initialize the schema at startup. `PostgresStore` supports initialization through a transaction-pooled connection.
|
|
169
|
+
|
|
170
|
+
### Separate schema initialization
|
|
171
|
+
|
|
172
|
+
If the runtime database role shouldn't create tables or indexes, initialize the schema during deployment with the direct connection:
|
|
173
|
+
|
|
174
|
+
```typescript
|
|
175
|
+
import { PostgresStore } from '@mastra/pg'
|
|
176
|
+
|
|
177
|
+
const storage = new PostgresStore({
|
|
178
|
+
id: 'neon-storage-init',
|
|
179
|
+
connectionString: process.env.DATABASE_URL_UNPOOLED!,
|
|
180
|
+
})
|
|
181
|
+
|
|
182
|
+
try {
|
|
183
|
+
await storage.init()
|
|
184
|
+
} finally {
|
|
185
|
+
await storage.close()
|
|
186
|
+
}
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
After the schema exists, use the pooled connection at runtime and disable automatic initialization:
|
|
190
|
+
|
|
191
|
+
```typescript
|
|
192
|
+
import { Mastra } from '@mastra/core/mastra'
|
|
193
|
+
import { PostgresStore } from '@mastra/pg'
|
|
194
|
+
|
|
195
|
+
export const mastra = new Mastra({
|
|
196
|
+
storage: new PostgresStore({
|
|
197
|
+
id: 'neon-storage',
|
|
198
|
+
connectionString: process.env.DATABASE_URL!,
|
|
199
|
+
disableInit: true,
|
|
200
|
+
}),
|
|
201
|
+
})
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
See [Neon connection pooling](https://neon.com/docs/connect/connection-pooling) for details.
|
|
205
|
+
|
|
206
|
+
## Vector search
|
|
207
|
+
|
|
208
|
+
Neon supports the [`pgvector` extension](https://neon.com/docs/extensions/pgvector). Use the direct `DATABASE_URL_UNPOOLED` connection while creating the extension and indexes. After setup, configure [`PgVector`](https://mastra.ai/reference/vectors/pg) with the pooled connection for runtime queries:
|
|
209
|
+
|
|
210
|
+
```typescript
|
|
211
|
+
import { PgVector } from '@mastra/pg'
|
|
212
|
+
|
|
213
|
+
const vector = new PgVector({
|
|
214
|
+
id: 'neon-vector',
|
|
215
|
+
connectionString: process.env.DATABASE_URL!,
|
|
216
|
+
disableInit: true,
|
|
217
|
+
})
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
Setting `disableInit: true` prevents the runtime process from creating or changing the extension, tables and indexes.
|
|
@@ -0,0 +1,182 @@
|
|
|
1
|
+
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
2
|
+
|
|
3
|
+
# Chunking and embedding documents
|
|
4
|
+
|
|
5
|
+
Before processing, create a MDocument instance from your content. You can initialize it from different formats:
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { MDocument } from '@mastra/rag'
|
|
9
|
+
|
|
10
|
+
const doc = MDocument.fromText('Your plain text content...')
|
|
11
|
+
const docFromHTML = MDocument.fromHTML('<html>Your HTML content...</html>')
|
|
12
|
+
const docFromMarkdown = MDocument.fromMarkdown('# Your Markdown content...')
|
|
13
|
+
const docFromJSON = MDocument.fromJSON(`{ "key": "value" }`)
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
## Document processing
|
|
17
|
+
|
|
18
|
+
Use `chunk` to split documents into manageable pieces. Mastra supports multiple chunking strategies optimized for different document types:
|
|
19
|
+
|
|
20
|
+
- `recursive`: Smart splitting based on content structure
|
|
21
|
+
- `character`: Simple character-based splits
|
|
22
|
+
- `token`: Token-aware splitting
|
|
23
|
+
- `markdown`: Markdown-aware splitting
|
|
24
|
+
- `semantic-markdown`: Markdown splitting based on related header families
|
|
25
|
+
- `html`: HTML structure-aware splitting
|
|
26
|
+
- `json`: JSON structure-aware splitting
|
|
27
|
+
- `latex`: LaTeX structure-aware splitting
|
|
28
|
+
- `sentence`: Sentence-aware splitting
|
|
29
|
+
|
|
30
|
+
> **Note:** Each strategy accepts different parameters optimized for its chunking approach.
|
|
31
|
+
|
|
32
|
+
Here's an example of how to use the `recursive` strategy:
|
|
33
|
+
|
|
34
|
+
```ts
|
|
35
|
+
const chunks = await doc.chunk({
|
|
36
|
+
strategy: 'recursive',
|
|
37
|
+
maxSize: 512,
|
|
38
|
+
overlap: 50,
|
|
39
|
+
separators: ['\n'],
|
|
40
|
+
extract: {
|
|
41
|
+
metadata: true, // Optionally extract metadata
|
|
42
|
+
},
|
|
43
|
+
})
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
For text where preserving sentence structure is important, here's an example of how to use the `sentence` strategy:
|
|
47
|
+
|
|
48
|
+
```ts
|
|
49
|
+
const chunks = await doc.chunk({
|
|
50
|
+
strategy: 'sentence',
|
|
51
|
+
maxSize: 450,
|
|
52
|
+
minSize: 50,
|
|
53
|
+
overlap: 0,
|
|
54
|
+
sentenceEnders: ['.'],
|
|
55
|
+
})
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
For markdown documents where preserving the semantic relationships between sections is important, here's an example of how to use the `semantic-markdown` strategy:
|
|
59
|
+
|
|
60
|
+
```ts
|
|
61
|
+
const chunks = await doc.chunk({
|
|
62
|
+
strategy: 'semantic-markdown',
|
|
63
|
+
joinThreshold: 500,
|
|
64
|
+
modelName: 'gpt-5',
|
|
65
|
+
})
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
> **Note:** Metadata extraction may use LLM calls, so ensure your API key is set.
|
|
69
|
+
|
|
70
|
+
We go deeper into chunking strategies in our [`chunk()` reference documentation](https://mastra.ai/reference/rag/chunk).
|
|
71
|
+
|
|
72
|
+
## Embedding generation
|
|
73
|
+
|
|
74
|
+
Transform chunks into embeddings using your preferred provider. Mastra supports embedding models through the model router.
|
|
75
|
+
|
|
76
|
+
### Using the Model Router
|
|
77
|
+
|
|
78
|
+
The simplest way is to use Mastra's model router with `provider/model` strings:
|
|
79
|
+
|
|
80
|
+
```ts
|
|
81
|
+
import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
|
|
82
|
+
import { embedMany } from 'ai'
|
|
83
|
+
|
|
84
|
+
const { embeddings } = await embedMany({
|
|
85
|
+
model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
|
|
86
|
+
values: chunks.map(chunk => chunk.text),
|
|
87
|
+
})
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Mastra supports OpenAI and Google embedding models. For a complete list of supported embedding models, see the [embeddings reference](https://mastra.ai/reference/rag/embeddings).
|
|
91
|
+
|
|
92
|
+
The model router automatically handles API key detection from environment variables.
|
|
93
|
+
|
|
94
|
+
The embedding functions return vectors, arrays of numbers representing the semantic meaning of your text, ready for similarity searches in your vector database.
|
|
95
|
+
|
|
96
|
+
### Configuring Embedding Dimensions
|
|
97
|
+
|
|
98
|
+
Embedding models typically output vectors with a fixed number of dimensions (e.g., 1536 for OpenAI's `text-embedding-3-small`). Some models support reducing this dimensionality, which can help:
|
|
99
|
+
|
|
100
|
+
- Decrease storage requirements in vector databases
|
|
101
|
+
- Reduce computational costs for similarity searches
|
|
102
|
+
|
|
103
|
+
Here are some supported models:
|
|
104
|
+
|
|
105
|
+
OpenAI (text-embedding-3 models):
|
|
106
|
+
|
|
107
|
+
```ts
|
|
108
|
+
import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
|
|
109
|
+
|
|
110
|
+
const { embeddings } = await embedMany({
|
|
111
|
+
model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
|
|
112
|
+
options: {
|
|
113
|
+
dimensions: 256, // Only supported in text-embedding-3 and later
|
|
114
|
+
},
|
|
115
|
+
values: chunks.map(chunk => chunk.text),
|
|
116
|
+
})
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Google (text-embedding-001):
|
|
120
|
+
|
|
121
|
+
```ts
|
|
122
|
+
const { embeddings } = await embedMany({
|
|
123
|
+
model: google('gemini-embedding-001', {
|
|
124
|
+
outputDimensionality: 256, // Truncates excessive values from the end
|
|
125
|
+
}),
|
|
126
|
+
values: chunks.map(chunk => chunk.text),
|
|
127
|
+
})
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
> **Vector Database Compatibility:** When storing embeddings, the vector database index must be configured to match the output size of your embedding model. If the dimensions don't match, you may get errors or data corruption.
|
|
131
|
+
|
|
132
|
+
## Example: Complete pipeline
|
|
133
|
+
|
|
134
|
+
Here's an example showing document processing, embedding generation, and vector storage:
|
|
135
|
+
|
|
136
|
+
```ts
|
|
137
|
+
import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
|
|
138
|
+
import { PgVector } from '@mastra/pg'
|
|
139
|
+
import { MDocument } from '@mastra/rag'
|
|
140
|
+
import { embedMany } from 'ai'
|
|
141
|
+
|
|
142
|
+
const doc = MDocument.fromText(`
|
|
143
|
+
Climate change poses significant challenges to global agriculture.
|
|
144
|
+
Rising temperatures and changing precipitation patterns affect crop yields.
|
|
145
|
+
`)
|
|
146
|
+
|
|
147
|
+
const chunks = await doc.chunk({
|
|
148
|
+
strategy: 'recursive',
|
|
149
|
+
maxSize: 256,
|
|
150
|
+
overlap: 50,
|
|
151
|
+
})
|
|
152
|
+
|
|
153
|
+
const { embeddings } = await embedMany({
|
|
154
|
+
model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
|
|
155
|
+
values: chunks.map(chunk => chunk.text),
|
|
156
|
+
})
|
|
157
|
+
|
|
158
|
+
const vectorStore = new PgVector({
|
|
159
|
+
id: 'pg-vector',
|
|
160
|
+
connectionString: process.env.POSTGRES_CONNECTION_STRING,
|
|
161
|
+
})
|
|
162
|
+
|
|
163
|
+
await vectorStore.createIndex({
|
|
164
|
+
indexName: 'embeddings',
|
|
165
|
+
dimension: 1536,
|
|
166
|
+
})
|
|
167
|
+
|
|
168
|
+
await vectorStore.upsert({
|
|
169
|
+
indexName: 'embeddings',
|
|
170
|
+
vectors: embeddings,
|
|
171
|
+
})
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
For more examples of different chunking strategies and embedding configurations, see:
|
|
175
|
+
|
|
176
|
+
- [Chunk Reference](https://mastra.ai/reference/rag/chunk)
|
|
177
|
+
- [Embeddings Reference](https://mastra.ai/reference/rag/embeddings)
|
|
178
|
+
|
|
179
|
+
For more details on vector databases and embeddings, see:
|
|
180
|
+
|
|
181
|
+
- [Vector Databases](https://mastra.ai/reference/rag/vector-databases)
|
|
182
|
+
- [Embedding API Reference](https://mastra.ai/reference/rag/embeddings)
|
|
@@ -63,11 +63,11 @@ This example shows the essentials. Initialize a document and create chunks, then
|
|
|
63
63
|
|
|
64
64
|
## Document processing
|
|
65
65
|
|
|
66
|
-
The basic building block of RAG is document processing. Documents can be chunked using strategies (recursive, sliding window, etc.) and enriched with metadata. See the [chunking and embedding doc](https://mastra.ai/
|
|
66
|
+
The basic building block of RAG is document processing. Documents can be chunked using strategies (recursive, sliding window, etc.) and enriched with metadata. See the [chunking and embedding doc](https://mastra.ai/reference/rag/chunking-and-embedding).
|
|
67
67
|
|
|
68
68
|
## Vector storage
|
|
69
69
|
|
|
70
|
-
Mastra supports multiple vector stores for embedding persistence and similarity search, including pgvector, OracleDB, Pinecone, Qdrant, and MongoDB. See the [vector database doc](https://mastra.ai/
|
|
70
|
+
Mastra supports multiple vector stores for embedding persistence and similarity search, including pgvector, OracleDB, Pinecone, Qdrant, and MongoDB. See the [vector database doc](https://mastra.ai/reference/rag/vector-databases).
|
|
71
71
|
|
|
72
72
|
## More resources
|
|
73
73
|
|
|
@@ -534,4 +534,4 @@ The re-ranked results combine vector similarity with semantic understanding to i
|
|
|
534
534
|
|
|
535
535
|
For more details about re-ranking, see the [rerank()](https://mastra.ai/reference/rag/rerankWithScorer) method.
|
|
536
536
|
|
|
537
|
-
For graph-based retrieval that follows connections between chunks, see the [GraphRAG](https://mastra.ai/
|
|
537
|
+
For graph-based retrieval that follows connections between chunks, see the [GraphRAG](https://mastra.ai/reference/rag/graph-rag-guide) documentation.
|
package/dist/docs/references/{guides-rag-vector-databases.md → reference-rag-vector-databases.md}
RENAMED
|
@@ -407,7 +407,7 @@ await store.createIndex({
|
|
|
407
407
|
|
|
408
408
|
The dimension size must match the output dimension of your chosen embedding model. Common dimension sizes are:
|
|
409
409
|
|
|
410
|
-
-
|
|
410
|
+
- OpenAI `text-embedding-3-small`: 1536 dimensions (or custom, e.g., 256)
|
|
411
411
|
- `Cohere embed-multilingual-v3`: 1024 dimensions
|
|
412
412
|
- `VoyageAI voyage-3.5`: 1024 dimensions (or custom: 256, 512, 1024, 2048)
|
|
413
413
|
- `Google gemini-embedding-001`: 768 dimensions (or custom)
|
|
@@ -338,7 +338,7 @@ const storage = new MastraCompositeStore({
|
|
|
338
338
|
|
|
339
339
|
Observability data can quickly overwhelm general-purpose databases in production. A single agent interaction can generate hundreds of spans, and high-traffic applications can produce thousands of traces per day.
|
|
340
340
|
|
|
341
|
-
**[ClickHouse](https://mastra.ai/
|
|
341
|
+
**[ClickHouse](https://mastra.ai/integrations/databases/clickhouse)** is recommended for production observability because it's optimized for high-volume, write-heavy analytics workloads. Use composite storage to route observability to ClickHouse while keeping other data in your primary database:
|
|
342
342
|
|
|
343
343
|
```typescript
|
|
344
344
|
import { MastraCompositeStore } from '@mastra/core/storage'
|
|
@@ -360,7 +360,7 @@ const storage = new MastraCompositeStore({
|
|
|
360
360
|
})
|
|
361
361
|
```
|
|
362
362
|
|
|
363
|
-
> **Note:** `ObservabilityStorageClickhouseVNext` is the current observability domain implementation. The legacy `ObservabilityStorageClickhouse` class is also exported and remains supported for projects that haven't migrated. See the [ClickHouse storage reference](https://mastra.ai/
|
|
363
|
+
> **Note:** `ObservabilityStorageClickhouseVNext` is the current observability domain implementation. The legacy `ObservabilityStorageClickhouse` class is also exported and remains supported for projects that haven't migrated. See the [ClickHouse storage reference](https://mastra.ai/integrations/databases/clickhouse) for details.
|
|
364
364
|
|
|
365
365
|
### Replicated ClickHouse for multi-replica clusters
|
|
366
366
|
|
|
@@ -392,6 +392,6 @@ const storage = new MastraCompositeStore({
|
|
|
392
392
|
})
|
|
393
393
|
```
|
|
394
394
|
|
|
395
|
-
Don't set `replication` on ClickHouse Cloud. Cloud rewrites `MergeTree` to `SharedMergeTree` server-side. See the [ClickHouse storage reference](https://mastra.ai/
|
|
395
|
+
Don't set `replication` on ClickHouse Cloud. Cloud rewrites `MergeTree` to `SharedMergeTree` server-side. See the [ClickHouse storage reference](https://mastra.ai/integrations/databases/clickhouse) for the full config shape and operator notes.
|
|
396
396
|
|
|
397
|
-
> **
|
|
397
|
+
> **Note:** This approach is also required when using storage providers that don't support observability (like Convex, DynamoDB, or Cloudflare). See the [MastraStorageExporter documentation](https://mastra.ai/docs/observability/integrations/exporters/mastra-storage) for the full list of supported providers.
|
|
@@ -8,7 +8,7 @@ Storage grows without bound by default. Retention is an opt-in, age-based cleanu
|
|
|
8
8
|
|
|
9
9
|
Retention covers **growth tables** only: tables that accumulate rows unbounded as a side effect of normal operation (conversation history, telemetry, job and run records, schedule fire history, event feeds). User-authored artifacts and config (agents, skills, workspaces, prompt blocks, datasets, schedule definitions, channel installations, and so on) grow with user intent and are edited or deleted explicitly, so they're not valid retention keys.
|
|
10
10
|
|
|
11
|
-
The reference implementations are [libSQL](https://mastra.ai/
|
|
11
|
+
The reference implementations are [libSQL](https://mastra.ai/integrations/databases/libsql), [PostgreSQL](https://mastra.ai/integrations/databases/postgresql), and [MongoDB](https://mastra.ai/integrations/databases/mongodb). Other adapters keep rows forever until they implement retention.
|
|
12
12
|
|
|
13
13
|
## Usage example
|
|
14
14
|
|
|
@@ -238,11 +238,11 @@ Handing that free space back to the OS is a separate concern that Mastra doesn't
|
|
|
238
238
|
|
|
239
239
|
For MongoDB, deleted documents are reused by future insertions. To reclaim disk space, run [`db.runCommand({ compact: "collection_name" })`](https://www.mongodb.com/docs/manual/reference/command/compact/) during a maintenance window.
|
|
240
240
|
|
|
241
|
-
> **LibSQL and Turso:** [Turso Cloud](https://mastra.ai/
|
|
241
|
+
> **LibSQL and Turso:** [Turso Cloud](https://mastra.ai/integrations/databases/libsql) manages storage compaction for you, so there's nothing to reclaim manually. This applies only to self-hosted libSQL files.
|
|
242
242
|
|
|
243
243
|
## Related
|
|
244
244
|
|
|
245
|
-
- [libSQL storage](https://mastra.ai/
|
|
246
|
-
- [PostgreSQL storage](https://mastra.ai/
|
|
245
|
+
- [libSQL storage](https://mastra.ai/integrations/databases/libsql)
|
|
246
|
+
- [PostgreSQL storage](https://mastra.ai/integrations/databases/postgresql)
|
|
247
247
|
- [Composite storage](https://mastra.ai/reference/storage/composite)
|
|
248
248
|
- [Storage overview](https://mastra.ai/reference/storage/overview)
|
package/dist/index.cjs
CHANGED
|
@@ -8728,6 +8728,7 @@ function dedupeMessagesForSave(messages) {
|
|
|
8728
8728
|
return Array.from(deduped.values());
|
|
8729
8729
|
}
|
|
8730
8730
|
var MemoryPG = class MemoryPG extends _mastra_core_storage.MemoryStorage {
|
|
8731
|
+
supportsPartialThreadUpdate = true;
|
|
8731
8732
|
supportsObservationalMemory = true;
|
|
8732
8733
|
/**
|
|
8733
8734
|
* Retention-eligible tables. `threads`, `messages`, and `resources` all anchor
|