@mastra/pg 1.20.0-alpha.2 → 1.20.0-alpha.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +30 -0
- package/dist/docs/SKILL.md +9 -7
- package/dist/docs/assets/SOURCE_MAP.json +1 -1
- package/dist/docs/references/docs-deployment-workers.md +1 -1
- package/dist/docs/references/docs-memory-semantic-recall.md +2 -1
- package/dist/docs/references/docs-memory-working-memory.md +1 -0
- package/dist/docs/references/docs-storage-overview.md +14 -12
- package/dist/docs/references/{reference-storage-dynamodb.md → integrations-databases-dynamodb.md} +1 -1
- package/dist/docs/references/integrations-databases-neon.md +220 -0
- package/dist/docs/references/{reference-storage-postgresql.md → integrations-databases-postgresql.md} +1 -1
- package/dist/docs/references/reference-rag-chunking-and-embedding.md +182 -0
- package/dist/docs/references/reference-rag-metadata-filters.md +13 -4
- package/dist/docs/references/{guides-rag-overview.md → reference-rag-overview.md} +2 -2
- package/dist/docs/references/{guides-rag-retrieval.md → reference-rag-retrieval.md} +18 -1
- package/dist/docs/references/{guides-rag-vector-databases.md → reference-rag-vector-databases.md} +42 -1
- package/dist/docs/references/reference-storage-composite.md +4 -4
- package/dist/docs/references/reference-storage-retention.md +4 -4
- package/dist/index.cjs +5 -4
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +5 -4
- package/dist/index.js.map +1 -1
- package/dist/storage/domains/memory/index.d.ts +3 -2
- package/dist/storage/domains/memory/index.d.ts.map +1 -1
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,35 @@
|
|
|
1
1
|
# @mastra/pg
|
|
2
2
|
|
|
3
|
+
## 1.20.0-alpha.4
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- Fixed a crash where updating a thread without a title (for example during observational memory buffering) could write a null title and violate the database's not-null constraint when running a newer @mastra/memory against an older storage package. Memory now checks whether the connected storage adapter supports partial thread updates and backfills the existing title for older adapters, so mixed-version deployments keep working. See #21041 for the original title-clobbering fix this makes backward compatible. ([#21257](https://github.com/mastra-ai/mastra/pull/21257))
|
|
8
|
+
|
|
9
|
+
- Storage adapters now declare support for partial thread updates, letting newer @mastra/memory preserve existing thread titles instead of overwriting them, while remaining safe against older versions. ([#21257](https://github.com/mastra-ai/mastra/pull/21257))
|
|
10
|
+
|
|
11
|
+
- Updated dependencies [[`dc4a25d`](https://github.com/mastra-ai/mastra/commit/dc4a25d41af4e2fe97a816070eaec6aa963ab53b), [`dc4a25d`](https://github.com/mastra-ai/mastra/commit/dc4a25d41af4e2fe97a816070eaec6aa963ab53b)]:
|
|
12
|
+
- @mastra/core@1.58.0-alpha.15
|
|
13
|
+
|
|
14
|
+
## 1.20.0-alpha.3
|
|
15
|
+
|
|
16
|
+
### Patch Changes
|
|
17
|
+
|
|
18
|
+
- Fixed generated thread titles being clobbered during a turn ([#21041](https://github.com/mastra-ai/mastra/pull/21041))
|
|
19
|
+
|
|
20
|
+
`updateThread` required both `title` and `metadata`, so callers that only needed to
|
|
21
|
+
change metadata (message persistence, working memory, observational memory, channel
|
|
22
|
+
subscriptions) had to read the thread and pass its title back. When title generation
|
|
23
|
+
finished between that read and the write, the freshly generated title was overwritten
|
|
24
|
+
with the stale one.
|
|
25
|
+
|
|
26
|
+
`title` and `metadata` are now independently optional: omitting one leaves that column
|
|
27
|
+
untouched. Callers that only change metadata no longer send a title, and message
|
|
28
|
+
persistence no longer rewrites a thread row it just read.
|
|
29
|
+
|
|
30
|
+
- Updated dependencies [[`1c75e32`](https://github.com/mastra-ai/mastra/commit/1c75e32f7fc0b9fb6f548b4407feaec8a1440212), [`c47165c`](https://github.com/mastra-ai/mastra/commit/c47165c983c87594c6952f1fd2fa51a90205034c), [`e08e789`](https://github.com/mastra-ai/mastra/commit/e08e789c1bf4cd2fe46363f7a4728536ceccc9bd), [`35cc901`](https://github.com/mastra-ai/mastra/commit/35cc90102cf834a84827acaf9eee0b6d6d1e2a3b), [`a8b4cf0`](https://github.com/mastra-ai/mastra/commit/a8b4cf02823cffebc4751a53337dfacf097c1ae1), [`f33264f`](https://github.com/mastra-ai/mastra/commit/f33264f517ae603279afd5c4251e2b40f6dd3618), [`689f2c4`](https://github.com/mastra-ai/mastra/commit/689f2c4b6c0835fe455702b01d21daa8abcd9331), [`eeae63e`](https://github.com/mastra-ai/mastra/commit/eeae63e7fbe8e1f237adc69bca6e2ac13c5ca907), [`4c186a0`](https://github.com/mastra-ai/mastra/commit/4c186a017275f45e6ed4c09de0f89550e2d09e8c), [`b0fa077`](https://github.com/mastra-ai/mastra/commit/b0fa077bcbc9b08551846fe372a0d3d15b71ed72)]:
|
|
31
|
+
- @mastra/core@1.58.0-alpha.8
|
|
32
|
+
|
|
3
33
|
## 1.20.0-alpha.2
|
|
4
34
|
|
|
5
35
|
### Patch Changes
|
package/dist/docs/SKILL.md
CHANGED
|
@@ -3,7 +3,7 @@ name: mastra-pg
|
|
|
3
3
|
description: Documentation for @mastra/pg. Use when working with @mastra/pg APIs, configuration, or implementation.
|
|
4
4
|
metadata:
|
|
5
5
|
package: "@mastra/pg"
|
|
6
|
-
version: "1.20.0-alpha.
|
|
6
|
+
version: "1.20.0-alpha.4"
|
|
7
7
|
---
|
|
8
8
|
|
|
9
9
|
## When to use
|
|
@@ -21,11 +21,11 @@ Read the individual reference documents for detailed explanations and code examp
|
|
|
21
21
|
- [Working memory](references/docs-memory-working-memory.md) - Learn how to configure working memory in Mastra to store persistent user data, preferences.
|
|
22
22
|
- [Storage overview](references/docs-storage-overview.md) - Configure storage for Mastra to persist runtime state across agents, workflows, observability, evals, schedules, and memory.
|
|
23
23
|
|
|
24
|
-
###
|
|
24
|
+
### Integrations
|
|
25
25
|
|
|
26
|
-
- [
|
|
27
|
-
- [
|
|
28
|
-
- [
|
|
26
|
+
- [DynamoDB](references/integrations-databases-dynamodb.md) - Documentation for the DynamoDB storage implementation in Mastra, using a single-table design with ElectroDB.
|
|
27
|
+
- [Neon Postgres](references/integrations-databases-neon.md) - Use Neon Postgres as a Mastra storage and vector backend.
|
|
28
|
+
- [PostgreSQL](references/integrations-databases-postgresql.md) - Documentation for the PostgreSQL storage implementation in Mastra.
|
|
29
29
|
|
|
30
30
|
### Reference
|
|
31
31
|
|
|
@@ -33,10 +33,12 @@ Read the individual reference documents for detailed explanations and code examp
|
|
|
33
33
|
- [Reference: MessageHistory](references/reference-processors-message-history-processor.md) - Documentation for the MessageHistory processor in Mastra, which handles retrieval and persistence of conversation history.
|
|
34
34
|
- [Reference: SemanticRecall](references/reference-processors-semantic-recall-processor.md) - Documentation for the SemanticRecall processor in Mastra, which enables semantic search over conversation history using vector embeddings.
|
|
35
35
|
- [Reference: WorkingMemory](references/reference-processors-working-memory-processor.md) - Documentation for the WorkingMemory processor in Mastra, which injects persistent user/context data as system instructions.
|
|
36
|
+
- [Chunking and embedding documents](references/reference-rag-chunking-and-embedding.md) - Guide on chunking and embedding documents in Mastra for efficient processing and retrieval.
|
|
36
37
|
- [Reference: Metadata filters](references/reference-rag-metadata-filters.md) - Documentation for metadata filtering capabilities in Mastra, which allow for precise querying of vector search results across different vector stores.
|
|
38
|
+
- [RAG (Retrieval-Augmented Generation) in Mastra](references/reference-rag-overview.md) - Overview of Retrieval-Augmented Generation (RAG) in Mastra, detailing its capabilities for enhancing LLM outputs with relevant context.
|
|
39
|
+
- [Retrieval, semantic search, reranking](references/reference-rag-retrieval.md) - Guide on retrieval processes in Mastra's RAG systems, including semantic search, filtering, and re-ranking.
|
|
40
|
+
- [Storing embeddings in a vector database](references/reference-rag-vector-databases.md) - Guide on vector storage options in Mastra, including embedded and dedicated vector databases for similarity search.
|
|
37
41
|
- [Reference: Composite storage](references/reference-storage-composite.md) - Documentation for combining multiple storage backends in Mastra.
|
|
38
|
-
- [Reference: DynamoDB storage](references/reference-storage-dynamodb.md) - Documentation for the DynamoDB storage implementation in Mastra, using a single-table design with ElectroDB.
|
|
39
|
-
- [Reference: PostgreSQL storage](references/reference-storage-postgresql.md) - Documentation for the PostgreSQL storage implementation in Mastra.
|
|
40
42
|
- [Reference: Storage retention (prune)](references/reference-storage-retention.md) - API reference for retention policies and prune() on Mastra storage.
|
|
41
43
|
- [Reference: createVectorQueryTool()](references/reference-tools-vector-query-tool.md) - Documentation for the Vector Query Tool in Mastra, which facilitates semantic search over vector stores with filtering and reranking capabilities.
|
|
42
44
|
- [Reference: PG vector store](references/reference-vectors-pg.md) - Documentation for the PgVector class in Mastra, which provides vector search using PostgreSQL with pgvector extension.
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
# Workers
|
|
4
4
|
|
|
5
|
-
> **Beta:**
|
|
5
|
+
> **Beta:** Breaking changes may occur without a major version bump until the API is stable. See [known limitations](#known-limitations) for current gaps.
|
|
6
6
|
|
|
7
7
|
Workers handle background processing outside the request-response cycle. Workflow step execution, cron-based scheduling, and long-running tool calls all run in workers, keeping the API responsive.
|
|
8
8
|
|
|
@@ -148,6 +148,7 @@ Each vector store page below includes installation instructions, configuration p
|
|
|
148
148
|
- [libSQL](https://mastra.ai/reference/vectors/libsql)
|
|
149
149
|
- [MongoDB](https://mastra.ai/reference/vectors/mongodb)
|
|
150
150
|
- [OpenSearch](https://mastra.ai/reference/vectors/opensearch)
|
|
151
|
+
- [OracleDB](https://mastra.ai/reference/vectors/oracledb)
|
|
151
152
|
- [Pinecone](https://mastra.ai/reference/vectors/pinecone)
|
|
152
153
|
- [PostgreSQL](https://mastra.ai/reference/vectors/pg)
|
|
153
154
|
- [Qdrant](https://mastra.ai/reference/vectors/qdrant)
|
|
@@ -180,7 +181,7 @@ const agent = new Agent({
|
|
|
180
181
|
})
|
|
181
182
|
```
|
|
182
183
|
|
|
183
|
-
> **Note:** `scope: 'resource'` is supported by the LibSQL, PostgreSQL, MongoDB, and Upstash storage adapters.
|
|
184
|
+
> **Note:** `scope: 'resource'` is supported by the LibSQL, OracleDB, PostgreSQL, MongoDB, and Upstash storage adapters.
|
|
184
185
|
|
|
185
186
|
### Metadata filtering
|
|
186
187
|
|
|
@@ -129,6 +129,7 @@ Resource-scoped working memory requires specific storage adapters that support t
|
|
|
129
129
|
|
|
130
130
|
- **libSQL** (`@mastra/libsql`)
|
|
131
131
|
- **PostgreSQL** (`@mastra/pg`)
|
|
132
|
+
- **OracleDB** (`@mastra/oracledb`)
|
|
132
133
|
- **Upstash** (`@mastra/upstash`)
|
|
133
134
|
- **MongoDB** (`@mastra/mongodb`)
|
|
134
135
|
|
|
@@ -190,18 +190,20 @@ You can also route `observability` to a dedicated analytics backend. See the [ob
|
|
|
190
190
|
|
|
191
191
|
Each provider page includes installation instructions, configuration parameters, and usage examples:
|
|
192
192
|
|
|
193
|
-
- [
|
|
194
|
-
- [
|
|
195
|
-
- [
|
|
196
|
-
- [
|
|
197
|
-
- [
|
|
198
|
-
- [
|
|
199
|
-
- [
|
|
200
|
-
- [
|
|
201
|
-
- [
|
|
202
|
-
- [
|
|
203
|
-
- [
|
|
204
|
-
- [
|
|
193
|
+
- [Cloudflare D1](https://mastra.ai/integrations/databases/cloudflare-d1)
|
|
194
|
+
- [Cloudflare KV & Durable Objects](https://mastra.ai/integrations/databases/cloudflare-kv)
|
|
195
|
+
- [Convex](https://mastra.ai/integrations/databases/convex)
|
|
196
|
+
- [DynamoDB](https://mastra.ai/integrations/databases/dynamodb)
|
|
197
|
+
- [Google Cloud Spanner](https://mastra.ai/integrations/databases/spanner)
|
|
198
|
+
- [LanceDB](https://mastra.ai/integrations/databases/lancedb)
|
|
199
|
+
- [libSQL](https://mastra.ai/integrations/databases/libsql)
|
|
200
|
+
- [Microsoft SQL Server](https://mastra.ai/integrations/databases/mssql)
|
|
201
|
+
- [MongoDB](https://mastra.ai/integrations/databases/mongodb)
|
|
202
|
+
- [Neon Postgres](https://mastra.ai/integrations/databases/neon)
|
|
203
|
+
- [OracleDB](https://mastra.ai/integrations/databases/oracledb)
|
|
204
|
+
- [PostgreSQL](https://mastra.ai/integrations/databases/postgresql)
|
|
205
|
+
- [Redis](https://mastra.ai/integrations/databases/redis)
|
|
206
|
+
- [Upstash](https://mastra.ai/integrations/databases/upstash)
|
|
205
207
|
|
|
206
208
|
> **Tip:** libSQL is the fastest path for local development because it doesn't require running a separate database server.
|
|
207
209
|
|
package/dist/docs/references/{reference-storage-dynamodb.md → integrations-databases-dynamodb.md}
RENAMED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
2
2
|
|
|
3
|
-
# DynamoDB
|
|
3
|
+
# DynamoDB
|
|
4
4
|
|
|
5
5
|
The DynamoDB storage implementation provides a high-capacity and performant NoSQL database solution for Mastra, using a single-table design pattern with [ElectroDB](https://electrodb.dev/).
|
|
6
6
|
|
|
@@ -0,0 +1,220 @@
|
|
|
1
|
+
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
2
|
+
|
|
3
|
+
# Neon Postgres
|
|
4
|
+
|
|
5
|
+
[Neon](https://neon.com) is a managed PostgreSQL service. Mastra connects to Neon through [`PostgresStore`](https://mastra.ai/integrations/databases/postgresql), which uses the Node.js `pg` driver. Neon doesn't require a separate Mastra storage package.
|
|
6
|
+
|
|
7
|
+
## Quickstart
|
|
8
|
+
|
|
9
|
+
Create a Neon project, then copy its [pooled connection string](https://neon.com/docs/connect/connect-from-any-app) into `DATABASE_URL`.
|
|
10
|
+
|
|
11
|
+
Install the PostgreSQL storage package:
|
|
12
|
+
|
|
13
|
+
**npm**:
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
npm install @mastra/pg@latest
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
**pnpm**:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
pnpm add @mastra/pg@latest
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
**Yarn**:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
yarn add @mastra/pg@latest
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
**Bun**:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
bun add @mastra/pg@latest
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Pass the pooled connection string to `PostgresStore`:
|
|
38
|
+
|
|
39
|
+
```typescript
|
|
40
|
+
import { Mastra } from '@mastra/core/mastra'
|
|
41
|
+
import { PostgresStore } from '@mastra/pg'
|
|
42
|
+
|
|
43
|
+
export const mastra = new Mastra({
|
|
44
|
+
storage: new PostgresStore({
|
|
45
|
+
id: 'neon-storage',
|
|
46
|
+
connectionString: process.env.DATABASE_URL!,
|
|
47
|
+
}),
|
|
48
|
+
})
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
On startup, Mastra calls `storage.init()` and creates its tables and indexes in Neon.
|
|
52
|
+
|
|
53
|
+
## Set up Neon with the CLI
|
|
54
|
+
|
|
55
|
+
The Neon CLI requires Node.js 20.19 or newer.
|
|
56
|
+
|
|
57
|
+
Install the [Neon CLI](https://neon.com/docs/cli/install):
|
|
58
|
+
|
|
59
|
+
**npm**:
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
npm install -g neon
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
**pnpm**:
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
pnpm add -g neon
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
**Yarn**:
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
yarn global add neon
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
**Bun**:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
bun add --global neon
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
From your project directory, link an existing Neon project or create one:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
neon link
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
`neon link` signs you in and writes project context to `.neon`. It also pulls the current branch's environment variables. Run `neon env pull` again whenever you need to refresh them:
|
|
90
|
+
|
|
91
|
+
```bash
|
|
92
|
+
neon env pull
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
The command writes `DATABASE_URL` and `DATABASE_URL_UNPOOLED` to `.env.local` or to `.env` when that file already exists.
|
|
96
|
+
|
|
97
|
+
### Add `neon.ts`
|
|
98
|
+
|
|
99
|
+
To manage branch settings in the repository, create a [`neon.ts`](https://neon.com/docs/reference/neon-ts) file:
|
|
100
|
+
|
|
101
|
+
```bash
|
|
102
|
+
neon config init
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
This installs `@neon/config` and `@neon/env` and creates a starter config. For example, the following policy expires new non-default branches after seven days:
|
|
106
|
+
|
|
107
|
+
```typescript
|
|
108
|
+
import { defineConfig } from '@neon/config/v1'
|
|
109
|
+
|
|
110
|
+
export default defineConfig({
|
|
111
|
+
branch: branch => {
|
|
112
|
+
if (branch.isDefault) {
|
|
113
|
+
return {}
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
if (!branch.exists) {
|
|
117
|
+
return { ttl: '7d' }
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
return {}
|
|
121
|
+
},
|
|
122
|
+
})
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Apply the config:
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
neon deploy
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
## Install Neon agent skills
|
|
132
|
+
|
|
133
|
+
Install the [`neon` and `neon-postgres` agent skills](https://neon.com/docs/ai/agent-skills) from your project directory:
|
|
134
|
+
|
|
135
|
+
**npm**:
|
|
136
|
+
|
|
137
|
+
```bash
|
|
138
|
+
npx skills add neondatabase/agent-skills --skill neon --skill neon-postgres -y
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
**pnpm**:
|
|
142
|
+
|
|
143
|
+
```bash
|
|
144
|
+
pnpm dlx skills add neondatabase/agent-skills --skill neon --skill neon-postgres -y
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
**Yarn**:
|
|
148
|
+
|
|
149
|
+
```bash
|
|
150
|
+
yarn dlx skills add neondatabase/agent-skills --skill neon --skill neon-postgres -y
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
**Bun**:
|
|
154
|
+
|
|
155
|
+
```bash
|
|
156
|
+
bun x skills add neondatabase/agent-skills --skill neon --skill neon-postgres -y
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
The skills give compatible coding agents Neon setup instructions and Postgres-specific guidance.
|
|
160
|
+
|
|
161
|
+
## Production connections
|
|
162
|
+
|
|
163
|
+
Neon provides direct and pooled connection strings:
|
|
164
|
+
|
|
165
|
+
- `DATABASE_URL` uses Neon's pooled endpoint. Use it for normal application traffic.
|
|
166
|
+
- `DATABASE_URL_UNPOOLED` connects directly to Postgres. Use it for administrative tasks or schema changes that run outside the application.
|
|
167
|
+
|
|
168
|
+
For most applications, use the pooled `DATABASE_URL` shown in the quickstart and let Mastra initialize the schema at startup. `PostgresStore` supports initialization through a transaction-pooled connection.
|
|
169
|
+
|
|
170
|
+
### Separate schema initialization
|
|
171
|
+
|
|
172
|
+
If the runtime database role shouldn't create tables or indexes, initialize the schema during deployment with the direct connection:
|
|
173
|
+
|
|
174
|
+
```typescript
|
|
175
|
+
import { PostgresStore } from '@mastra/pg'
|
|
176
|
+
|
|
177
|
+
const storage = new PostgresStore({
|
|
178
|
+
id: 'neon-storage-init',
|
|
179
|
+
connectionString: process.env.DATABASE_URL_UNPOOLED!,
|
|
180
|
+
})
|
|
181
|
+
|
|
182
|
+
try {
|
|
183
|
+
await storage.init()
|
|
184
|
+
} finally {
|
|
185
|
+
await storage.close()
|
|
186
|
+
}
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
After the schema exists, use the pooled connection at runtime and disable automatic initialization:
|
|
190
|
+
|
|
191
|
+
```typescript
|
|
192
|
+
import { Mastra } from '@mastra/core/mastra'
|
|
193
|
+
import { PostgresStore } from '@mastra/pg'
|
|
194
|
+
|
|
195
|
+
export const mastra = new Mastra({
|
|
196
|
+
storage: new PostgresStore({
|
|
197
|
+
id: 'neon-storage',
|
|
198
|
+
connectionString: process.env.DATABASE_URL!,
|
|
199
|
+
disableInit: true,
|
|
200
|
+
}),
|
|
201
|
+
})
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
See [Neon connection pooling](https://neon.com/docs/connect/connection-pooling) for details.
|
|
205
|
+
|
|
206
|
+
## Vector search
|
|
207
|
+
|
|
208
|
+
Neon supports the [`pgvector` extension](https://neon.com/docs/extensions/pgvector). Use the direct `DATABASE_URL_UNPOOLED` connection while creating the extension and indexes. After setup, configure [`PgVector`](https://mastra.ai/reference/vectors/pg) with the pooled connection for runtime queries:
|
|
209
|
+
|
|
210
|
+
```typescript
|
|
211
|
+
import { PgVector } from '@mastra/pg'
|
|
212
|
+
|
|
213
|
+
const vector = new PgVector({
|
|
214
|
+
id: 'neon-vector',
|
|
215
|
+
connectionString: process.env.DATABASE_URL!,
|
|
216
|
+
disableInit: true,
|
|
217
|
+
})
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
Setting `disableInit: true` prevents the runtime process from creating or changing the extension, tables and indexes.
|
|
@@ -0,0 +1,182 @@
|
|
|
1
|
+
> Discover all available pages from the documentation index: https://mastra.ai/llms.txt
|
|
2
|
+
|
|
3
|
+
# Chunking and embedding documents
|
|
4
|
+
|
|
5
|
+
Before processing, create a MDocument instance from your content. You can initialize it from different formats:
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { MDocument } from '@mastra/rag'
|
|
9
|
+
|
|
10
|
+
const doc = MDocument.fromText('Your plain text content...')
|
|
11
|
+
const docFromHTML = MDocument.fromHTML('<html>Your HTML content...</html>')
|
|
12
|
+
const docFromMarkdown = MDocument.fromMarkdown('# Your Markdown content...')
|
|
13
|
+
const docFromJSON = MDocument.fromJSON(`{ "key": "value" }`)
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
## Document processing
|
|
17
|
+
|
|
18
|
+
Use `chunk` to split documents into manageable pieces. Mastra supports multiple chunking strategies optimized for different document types:
|
|
19
|
+
|
|
20
|
+
- `recursive`: Smart splitting based on content structure
|
|
21
|
+
- `character`: Simple character-based splits
|
|
22
|
+
- `token`: Token-aware splitting
|
|
23
|
+
- `markdown`: Markdown-aware splitting
|
|
24
|
+
- `semantic-markdown`: Markdown splitting based on related header families
|
|
25
|
+
- `html`: HTML structure-aware splitting
|
|
26
|
+
- `json`: JSON structure-aware splitting
|
|
27
|
+
- `latex`: LaTeX structure-aware splitting
|
|
28
|
+
- `sentence`: Sentence-aware splitting
|
|
29
|
+
|
|
30
|
+
> **Note:** Each strategy accepts different parameters optimized for its chunking approach.
|
|
31
|
+
|
|
32
|
+
Here's an example of how to use the `recursive` strategy:
|
|
33
|
+
|
|
34
|
+
```ts
|
|
35
|
+
const chunks = await doc.chunk({
|
|
36
|
+
strategy: 'recursive',
|
|
37
|
+
maxSize: 512,
|
|
38
|
+
overlap: 50,
|
|
39
|
+
separators: ['\n'],
|
|
40
|
+
extract: {
|
|
41
|
+
metadata: true, // Optionally extract metadata
|
|
42
|
+
},
|
|
43
|
+
})
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
For text where preserving sentence structure is important, here's an example of how to use the `sentence` strategy:
|
|
47
|
+
|
|
48
|
+
```ts
|
|
49
|
+
const chunks = await doc.chunk({
|
|
50
|
+
strategy: 'sentence',
|
|
51
|
+
maxSize: 450,
|
|
52
|
+
minSize: 50,
|
|
53
|
+
overlap: 0,
|
|
54
|
+
sentenceEnders: ['.'],
|
|
55
|
+
})
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
For markdown documents where preserving the semantic relationships between sections is important, here's an example of how to use the `semantic-markdown` strategy:
|
|
59
|
+
|
|
60
|
+
```ts
|
|
61
|
+
const chunks = await doc.chunk({
|
|
62
|
+
strategy: 'semantic-markdown',
|
|
63
|
+
joinThreshold: 500,
|
|
64
|
+
modelName: 'gpt-5',
|
|
65
|
+
})
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
> **Note:** Metadata extraction may use LLM calls, so ensure your API key is set.
|
|
69
|
+
|
|
70
|
+
We go deeper into chunking strategies in our [`chunk()` reference documentation](https://mastra.ai/reference/rag/chunk).
|
|
71
|
+
|
|
72
|
+
## Embedding generation
|
|
73
|
+
|
|
74
|
+
Transform chunks into embeddings using your preferred provider. Mastra supports embedding models through the model router.
|
|
75
|
+
|
|
76
|
+
### Using the Model Router
|
|
77
|
+
|
|
78
|
+
The simplest way is to use Mastra's model router with `provider/model` strings:
|
|
79
|
+
|
|
80
|
+
```ts
|
|
81
|
+
import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
|
|
82
|
+
import { embedMany } from 'ai'
|
|
83
|
+
|
|
84
|
+
const { embeddings } = await embedMany({
|
|
85
|
+
model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
|
|
86
|
+
values: chunks.map(chunk => chunk.text),
|
|
87
|
+
})
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Mastra supports OpenAI and Google embedding models. For a complete list of supported embedding models, see the [embeddings reference](https://mastra.ai/reference/rag/embeddings).
|
|
91
|
+
|
|
92
|
+
The model router automatically handles API key detection from environment variables.
|
|
93
|
+
|
|
94
|
+
The embedding functions return vectors, arrays of numbers representing the semantic meaning of your text, ready for similarity searches in your vector database.
|
|
95
|
+
|
|
96
|
+
### Configuring Embedding Dimensions
|
|
97
|
+
|
|
98
|
+
Embedding models typically output vectors with a fixed number of dimensions (e.g., 1536 for OpenAI's `text-embedding-3-small`). Some models support reducing this dimensionality, which can help:
|
|
99
|
+
|
|
100
|
+
- Decrease storage requirements in vector databases
|
|
101
|
+
- Reduce computational costs for similarity searches
|
|
102
|
+
|
|
103
|
+
Here are some supported models:
|
|
104
|
+
|
|
105
|
+
OpenAI (text-embedding-3 models):
|
|
106
|
+
|
|
107
|
+
```ts
|
|
108
|
+
import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
|
|
109
|
+
|
|
110
|
+
const { embeddings } = await embedMany({
|
|
111
|
+
model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
|
|
112
|
+
options: {
|
|
113
|
+
dimensions: 256, // Only supported in text-embedding-3 and later
|
|
114
|
+
},
|
|
115
|
+
values: chunks.map(chunk => chunk.text),
|
|
116
|
+
})
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Google (text-embedding-001):
|
|
120
|
+
|
|
121
|
+
```ts
|
|
122
|
+
const { embeddings } = await embedMany({
|
|
123
|
+
model: google('gemini-embedding-001', {
|
|
124
|
+
outputDimensionality: 256, // Truncates excessive values from the end
|
|
125
|
+
}),
|
|
126
|
+
values: chunks.map(chunk => chunk.text),
|
|
127
|
+
})
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
> **Vector Database Compatibility:** When storing embeddings, the vector database index must be configured to match the output size of your embedding model. If the dimensions don't match, you may get errors or data corruption.
|
|
131
|
+
|
|
132
|
+
## Example: Complete pipeline
|
|
133
|
+
|
|
134
|
+
Here's an example showing document processing, embedding generation, and vector storage:
|
|
135
|
+
|
|
136
|
+
```ts
|
|
137
|
+
import { ModelRouterEmbeddingModel } from '@mastra/core/llm'
|
|
138
|
+
import { PgVector } from '@mastra/pg'
|
|
139
|
+
import { MDocument } from '@mastra/rag'
|
|
140
|
+
import { embedMany } from 'ai'
|
|
141
|
+
|
|
142
|
+
const doc = MDocument.fromText(`
|
|
143
|
+
Climate change poses significant challenges to global agriculture.
|
|
144
|
+
Rising temperatures and changing precipitation patterns affect crop yields.
|
|
145
|
+
`)
|
|
146
|
+
|
|
147
|
+
const chunks = await doc.chunk({
|
|
148
|
+
strategy: 'recursive',
|
|
149
|
+
maxSize: 256,
|
|
150
|
+
overlap: 50,
|
|
151
|
+
})
|
|
152
|
+
|
|
153
|
+
const { embeddings } = await embedMany({
|
|
154
|
+
model: new ModelRouterEmbeddingModel('openai/text-embedding-3-small'),
|
|
155
|
+
values: chunks.map(chunk => chunk.text),
|
|
156
|
+
})
|
|
157
|
+
|
|
158
|
+
const vectorStore = new PgVector({
|
|
159
|
+
id: 'pg-vector',
|
|
160
|
+
connectionString: process.env.POSTGRES_CONNECTION_STRING,
|
|
161
|
+
})
|
|
162
|
+
|
|
163
|
+
await vectorStore.createIndex({
|
|
164
|
+
indexName: 'embeddings',
|
|
165
|
+
dimension: 1536,
|
|
166
|
+
})
|
|
167
|
+
|
|
168
|
+
await vectorStore.upsert({
|
|
169
|
+
indexName: 'embeddings',
|
|
170
|
+
vectors: embeddings,
|
|
171
|
+
})
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
For more examples of different chunking strategies and embedding configurations, see:
|
|
175
|
+
|
|
176
|
+
- [Chunk Reference](https://mastra.ai/reference/rag/chunk)
|
|
177
|
+
- [Embeddings Reference](https://mastra.ai/reference/rag/embeddings)
|
|
178
|
+
|
|
179
|
+
For more details on vector databases and embeddings, see:
|
|
180
|
+
|
|
181
|
+
- [Vector Databases](https://mastra.ai/reference/rag/vector-databases)
|
|
182
|
+
- [Embedding API Reference](https://mastra.ai/reference/rag/embeddings)
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
# Metadata filters
|
|
4
4
|
|
|
5
|
-
Mastra provides a unified metadata filtering syntax across all vector stores, based on MongoDB/Sift query syntax. Each vector store translates these filters into
|
|
5
|
+
Mastra provides a unified metadata filtering syntax across all vector stores, based on MongoDB/Sift query syntax. Each vector store translates these filters into its native query format. For example, PgVector uses PostgreSQL JSONB predicates, while OracleDB stores metadata as Oracle JSON and compiles filters to `JSON_VALUE`, `JSON_EXISTS`, `REGEXP_LIKE`, and `LIKE` predicates with bound values.
|
|
6
6
|
|
|
7
7
|
## Basic example
|
|
8
8
|
|
|
@@ -34,11 +34,11 @@ const results = await store.query({
|
|
|
34
34
|
|
|
35
35
|
### Array Operators
|
|
36
36
|
|
|
37
|
-
`$in`Matches any value in array{ category: { $in: \["A", "B"] } }Supported by: All except Couchbase`$nin`Matches none of the values{ status: { $nin: \["deleted", "archived"] } }Supported by: All except Couchbase`$all`Matches arrays containing all elements{ tags: { $all: \["urgent", "high"] } }Supported by: Astra, Pinecone, Upstash, MongoDB`$elemMatch`Matches array elements meeting criteria{ scores: { $elemMatch: { $gt: 80 } } }Supported by: libSQL, PgVector, MongoDB
|
|
37
|
+
`$in`Matches any value in array{ category: { $in: \["A", "B"] } }Supported by: All except Couchbase`$nin`Matches none of the values{ status: { $nin: \["deleted", "archived"] } }Supported by: All except Couchbase`$all`Matches arrays containing all elements{ tags: { $all: \["urgent", "high"] } }Supported by: Astra, Pinecone, Upstash, MongoDB, OracleDB`$elemMatch`Matches array elements meeting criteria{ scores: { $elemMatch: { $gt: 80 } } }Supported by: libSQL, PgVector, MongoDB, OracleDB
|
|
38
38
|
|
|
39
39
|
### Logical Operators
|
|
40
40
|
|
|
41
|
-
`$and`Logical AND{ $and: \[{ price: { $gt: 100 } }, { stock: { $gt: 0 } }] }Supported by: All except Vectorize, Couchbase`$or`Logical OR{ $or: \[{ status: "active" }, { priority: "high" }] }Supported by: All except Vectorize, Couchbase`$not`Logical NOT{ price: { $not: { $lt: 100 } } }Supported by: Astra, Qdrant, Upstash, PgVector, libSQL, MongoDB`$nor`Logical NOR{ $nor: \[{ status: "deleted" }, { archived: true }] }Supported by: Qdrant, Upstash, PgVector, libSQL, MongoDB
|
|
41
|
+
`$and`Logical AND{ $and: \[{ price: { $gt: 100 } }, { stock: { $gt: 0 } }] }Supported by: All except Vectorize, Couchbase`$or`Logical OR{ $or: \[{ status: "active" }, { priority: "high" }] }Supported by: All except Vectorize, Couchbase`$not`Logical NOT{ price: { $not: { $lt: 100 } } }Supported by: Astra, Qdrant, Upstash, PgVector, libSQL, MongoDB, OracleDB`$nor`Logical NOR{ $nor: \[{ status: "deleted" }, { archived: true }] }Supported by: Qdrant, Upstash, PgVector, libSQL, MongoDB, OracleDB
|
|
42
42
|
|
|
43
43
|
### Element Operators
|
|
44
44
|
|
|
@@ -46,7 +46,7 @@ const results = await store.query({
|
|
|
46
46
|
|
|
47
47
|
### Custom Operators
|
|
48
48
|
|
|
49
|
-
`$contains`Text contains substring{ description: { $contains: "sale" } }Supported by: Upstash, libSQL, PgVector`$regex`Regular expression match{ name: { $regex: "^test" } }Supported by: Qdrant, PgVector, Upstash, MongoDB`$size`Array length check{ tags: { $size:
|
|
49
|
+
`$contains`Text contains substring{ description: { $contains: "sale" } }Supported by: Upstash, libSQL, PgVector, OracleDB`$regex`Regular expression match{ name: { $regex: "^test" } }Supported by: Qdrant, PgVector, Upstash, MongoDB, OracleDB`$size`Array length check{ tags: { $size: 3 } }Supported by: Astra, libSQL, PgVector, MongoDB, OracleDB`$geo`Geospatial query{ location: { $geo: { type: "radius", ... } } }Supported by: Qdrant`$datetime`Datetime range query{ created: { $datetime: { range: { gt: "2024-01-01" } } } }Supported by: Qdrant`$hasId`Vector ID existence check{ $hasId: \["id1", "id2"] }Supported by: Qdrant`$hasVector`Vector existence check{ $hasVector: true }Supported by: Qdrant
|
|
50
50
|
|
|
51
51
|
## Common rules and restrictions
|
|
52
52
|
|
|
@@ -124,6 +124,14 @@ const results = await store.query({
|
|
|
124
124
|
- Empty arrays in conditions are handled gracefully
|
|
125
125
|
- Metadata is stored in a JSONB column for efficient querying
|
|
126
126
|
|
|
127
|
+
### OracleDB
|
|
128
|
+
|
|
129
|
+
- Metadata is stored as Oracle JSON alongside each `VECTOR` row
|
|
130
|
+
- Scalar comparisons use `JSON_VALUE`, while array, existence, and element-match checks use `JSON_EXISTS`
|
|
131
|
+
- `$regex` uses Oracle `REGEXP_LIKE`; string `$contains` uses case-insensitive `LIKE`
|
|
132
|
+
- Nested fields are supported with dot notation and are converted to quoted Oracle JSON paths
|
|
133
|
+
- User-provided metadata values are bound as parameters instead of interpolated into SQL
|
|
134
|
+
|
|
127
135
|
### PgVector
|
|
128
136
|
|
|
129
137
|
- Full support for PostgreSQL's native JSON querying capabilities
|
|
@@ -211,6 +219,7 @@ const results = await store.query({
|
|
|
211
219
|
- [Cloudflare Vectorize](https://mastra.ai/reference/vectors/vectorize)
|
|
212
220
|
- [libSQL](https://mastra.ai/reference/vectors/libsql)
|
|
213
221
|
- [MongoDB](https://mastra.ai/reference/vectors/mongodb)
|
|
222
|
+
- [OracleDB](https://mastra.ai/reference/vectors/oracledb)
|
|
214
223
|
- [PgStore](https://mastra.ai/reference/vectors/pg)
|
|
215
224
|
- [Pinecone](https://mastra.ai/reference/vectors/pinecone)
|
|
216
225
|
- [Qdrant](https://mastra.ai/reference/vectors/qdrant)
|
|
@@ -63,11 +63,11 @@ This example shows the essentials. Initialize a document and create chunks, then
|
|
|
63
63
|
|
|
64
64
|
## Document processing
|
|
65
65
|
|
|
66
|
-
The basic building block of RAG is document processing. Documents can be chunked using strategies (recursive, sliding window, etc.) and enriched with metadata. See the [chunking and embedding doc](https://mastra.ai/
|
|
66
|
+
The basic building block of RAG is document processing. Documents can be chunked using strategies (recursive, sliding window, etc.) and enriched with metadata. See the [chunking and embedding doc](https://mastra.ai/reference/rag/chunking-and-embedding).
|
|
67
67
|
|
|
68
68
|
## Vector storage
|
|
69
69
|
|
|
70
|
-
Mastra supports multiple vector stores for embedding persistence and similarity search, including pgvector, Pinecone, Qdrant, and MongoDB. See the [vector database doc](https://mastra.ai/
|
|
70
|
+
Mastra supports multiple vector stores for embedding persistence and similarity search, including pgvector, OracleDB, Pinecone, Qdrant, and MongoDB. See the [vector database doc](https://mastra.ai/reference/rag/vector-databases).
|
|
71
71
|
|
|
72
72
|
## More resources
|
|
73
73
|
|