@venizia/ignis-docs 0.2.1-0 → 0.2.1-1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/content/best-practices/architectural-patterns.md +3 -3
- package/content/best-practices/code-style-standards/naming-conventions.md +1 -1
- package/content/best-practices/contribution-workflow.md +2 -2
- package/content/best-practices/error-handling.md +94 -89
- package/content/best-practices/security-guidelines.md +5 -5
- package/content/extensions/components/api-reference.md +22 -21
- package/content/extensions/components/authentication/api.md +64 -28
- package/content/extensions/components/authentication/errors.md +19 -5
- package/content/extensions/components/authentication/index.md +19 -18
- package/content/extensions/components/authentication/usage.md +55 -30
- package/content/extensions/components/authorization/api.md +351 -84
- package/content/extensions/components/authorization/errors.md +51 -17
- package/content/extensions/components/authorization/getting-started.md +227 -0
- package/content/extensions/components/authorization/index.md +24 -16
- package/content/extensions/components/authorization/usage.md +45 -21
- package/content/extensions/components/health-check.md +15 -9
- package/content/extensions/components/index.md +24 -90
- package/content/extensions/components/mail/api.md +105 -54
- package/content/extensions/components/mail/errors.md +12 -10
- package/content/extensions/components/mail/index.md +32 -13
- package/content/extensions/components/mail/usage.md +26 -18
- package/content/extensions/components/request-tracker.md +18 -14
- package/content/extensions/components/socket-io/api.md +377 -882
- package/content/extensions/components/socket-io/errors.md +49 -51
- package/content/extensions/components/socket-io/index.md +72 -88
- package/content/extensions/components/socket-io/usage.md +107 -117
- package/content/extensions/components/static-asset/api.md +83 -31
- package/content/extensions/components/static-asset/errors.md +18 -7
- package/content/extensions/components/static-asset/index.md +18 -11
- package/content/extensions/components/static-asset/usage.md +11 -6
- package/content/extensions/components/template/index.md +3 -3
- package/content/extensions/components/websocket/api.md +58 -27
- package/content/extensions/components/websocket/errors.md +3 -3
- package/content/extensions/components/websocket/index.md +8 -7
- package/content/extensions/components/websocket/usage.md +21 -8
- package/content/extensions/helpers/cron/index.md +8 -7
- package/content/extensions/helpers/crypto/index.md +16 -8
- package/content/extensions/helpers/crypto/reference.md +96 -24
- package/content/extensions/helpers/env/index.md +14 -10
- package/content/extensions/helpers/error/index.md +99 -23
- package/content/extensions/helpers/index.md +61 -47
- package/content/extensions/helpers/inversion/index.md +23 -6
- package/content/extensions/helpers/inversion/reference.md +30 -22
- package/content/extensions/helpers/kafka/admin.md +3 -2
- package/content/extensions/helpers/kafka/compile-binary.md +69 -44
- package/content/extensions/helpers/kafka/consumer.md +24 -21
- package/content/extensions/helpers/kafka/examples.md +22 -234
- package/content/extensions/helpers/kafka/index.md +30 -62
- package/content/extensions/helpers/kafka/producer.md +31 -26
- package/content/extensions/helpers/kafka/schema-registry.md +19 -14
- package/content/extensions/helpers/logger/hf-logger.md +49 -22
- package/content/extensions/helpers/logger/index.md +37 -12
- package/content/extensions/helpers/logger/pino.md +31 -11
- package/content/extensions/helpers/logger/reference.md +243 -52
- package/content/extensions/helpers/network/api.md +65 -30
- package/content/extensions/helpers/network/index.md +32 -11
- package/content/extensions/helpers/queue/index.md +17 -6
- package/content/extensions/helpers/queue/reference.md +52 -25
- package/content/extensions/helpers/redis/index.md +29 -11
- package/content/extensions/helpers/redis/reference.md +85 -28
- package/content/extensions/helpers/secrets/index.md +82 -12
- package/content/extensions/helpers/socket-io/api.md +40 -21
- package/content/extensions/helpers/socket-io/index.md +26 -10
- package/content/extensions/helpers/storage/api.md +42 -24
- package/content/extensions/helpers/storage/index.md +10 -7
- package/content/extensions/helpers/types/index.md +20 -7
- package/content/extensions/helpers/types/reference.md +53 -14
- package/content/extensions/helpers/uid/index.md +166 -10
- package/content/extensions/helpers/websocket/api.md +52 -13
- package/content/extensions/helpers/websocket/index.md +7 -4
- package/content/extensions/helpers/worker-thread/index.md +9 -5
- package/content/extensions/helpers/worker-thread/reference.md +20 -20
- package/content/extensions/index.md +38 -39
- package/content/extensions/src-details/mcp-server.md +96 -548
- package/content/guides/core-concepts/persistent/datasources.md +2 -2
- package/content/guides/core-concepts/persistent/index.md +5 -1
- package/content/guides/core-concepts/persistent/pglite.md +218 -0
- package/content/guides/core-concepts/persistent/postgres-drivers.md +21 -15
- package/content/guides/core-concepts/persistent/search-meilisearch.md +22 -12
- package/content/guides/core-concepts/persistent/search-typesense.md +26 -14
- package/content/guides/core-concepts/persistent/sqlite.md +296 -0
- package/content/guides/core-concepts/persistent/transactions.md +12 -11
- package/content/guides/get-started/5-minute-quickstart.md +59 -206
- package/content/guides/get-started/philosophy.md +135 -670
- package/content/guides/get-started/setup.md +53 -74
- package/content/guides/migrations/redis-helpers-migration.md +4 -3
- package/content/guides/migrations/scoped-rbac-migration.md +5 -5
- package/content/guides/migrations/unified-connectors-migration.md +7 -6
- package/content/guides/tutorials/realtime-chat.md +1 -1
- package/content/references/base/application.md +63 -21
- package/content/references/base/components.md +3 -3
- package/content/references/base/connectors.md +14 -14
- package/content/references/base/controllers.md +12 -12
- package/content/references/base/datasources-reference.md +32 -31
- package/content/references/base/datasources.md +6 -6
- package/content/references/base/dependency-injection.md +12 -11
- package/content/references/base/filter-system/application-usage.md +54 -30
- package/content/references/base/filter-system/array-operators.md +24 -46
- package/content/references/base/filter-system/comparison-operators.md +47 -67
- package/content/references/base/filter-system/default-filter.md +59 -53
- package/content/references/base/filter-system/fields-order-pagination.md +92 -146
- package/content/references/base/filter-system/index.md +25 -13
- package/content/references/base/filter-system/json-filtering.md +45 -184
- package/content/references/base/filter-system/list-operators.md +23 -53
- package/content/references/base/filter-system/logical-operators.md +63 -121
- package/content/references/base/filter-system/null-operators.md +34 -104
- package/content/references/base/filter-system/pattern-matching.md +40 -55
- package/content/references/base/filter-system/quick-reference.md +86 -198
- package/content/references/base/filter-system/range-operators.md +18 -46
- package/content/references/base/filter-system/tips.md +6 -6
- package/content/references/base/filter-system/use-cases.md +33 -15
- package/content/references/base/grpc-controllers.md +53 -17
- package/content/references/base/index.md +5 -3
- package/content/references/base/middlewares.md +11 -10
- package/content/references/base/models-reference.md +17 -17
- package/content/references/base/models.md +4 -3
- package/content/references/base/providers.md +8 -8
- package/content/references/base/repositories/advanced.md +224 -326
- package/content/references/base/repositories/index.md +19 -6
- package/content/references/base/repositories/mixins.md +5 -5
- package/content/references/base/repositories/relations.md +160 -293
- package/content/references/base/repositories/soft-deletable.md +16 -6
- package/content/references/base/secrets.md +17 -13
- package/content/references/base/services.md +6 -4
- package/content/references/configuration/environment-variables.md +31 -23
- package/content/references/configuration/index.md +6 -4
- package/content/references/index.md +1 -1
- package/content/references/utilities/duration.md +85 -0
- package/content/references/utilities/index.md +5 -1
- package/content/references/utilities/jsx-reference.md +11 -11
- package/content/references/utilities/jsx.md +2 -2
- package/content/references/utilities/module.md +78 -25
- package/content/references/utilities/request.md +2 -1
- package/content/references/utilities/retry.md +139 -0
- package/content/references/utilities/schema.md +2 -2
- package/content/references/utilities/statuses-reference.md +4 -4
- package/content/references/utilities/statuses.md +5 -5
- package/dist/mcp-server/index.js +0 -0
- package/dist/mcp-server/tools/github/list-project-files.tool.d.ts +1 -1
- package/dist/mcp-server/tools/github/list-project-files.tool.d.ts.map +1 -1
- package/dist/mcp-server/tools/github/list-project-files.tool.js +3 -3
- package/dist/mcp-server/tools/github/search-code.tool.js +1 -1
- package/dist/mcp-server/tools/github/verify-dependencies.tool.js +1 -1
- package/dist/mcp-server/tools/github/view-source-file.tool.js +1 -1
- package/package.json +17 -16
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
A DataSource manages database connections and supports **schema auto-discovery** from repositories.
|
|
4
4
|
|
|
5
5
|
> [!NOTE] Connectors
|
|
6
|
-
> This guide covers the **PostgreSQL connector** (`BasePostgresDataSource`, aliased as `BaseDataSource` for backward compatibility)
|
|
6
|
+
> This guide covers the **PostgreSQL connector** (`BasePostgresDataSource`, aliased as `BaseDataSource` for backward compatibility). It's the primary relational engine and the one used by most applications. IGNIS also ships a **SQLite connector** (`BaseSqliteDataSource`, see [SQLite](./sqlite)) and a **typesense connector** for full-text/vector search (see [Search & Typesense](./search-typesense)). All three implement the same engine-neutral `AbstractDataSource` contract - see [Connectors](/references/base/connectors) for the architecture.
|
|
7
7
|
|
|
8
8
|
## Creating a DataSource
|
|
9
9
|
|
|
@@ -57,7 +57,7 @@ export class PostgresDataSource extends BasePostgresDataSource<IDSConfigs> {
|
|
|
57
57
|
```
|
|
58
58
|
|
|
59
59
|
> [!NOTE] Driver seam: the raw client goes on `this.client`
|
|
60
|
-
> `this.client = new Pool(...)` is the short path: `configure()` builds only the client
|
|
60
|
+
> `this.client = new Pool(...)` is the short path: `configure()` builds only the client. `getConnector()`/`beginTransaction()` lazily instantiate the class named in `@datasource({ driver })` over it - `NodePostgresDriver` here. There is no `pool` field - the raw-client slot is `client`, whatever the client happens to be. Naming the driver class (rather than a driver-name string) is what carries `pg` into the app's bundle. A bundler only packages a real value reference, never text. The alternative is to wire a driver yourself for a custom or third-party driver: `configure()` calls `this.useDriver({ driver, schema? })`. That assigns `this.driver` **and** builds `this.connector` in one step (so the half-wired state cannot exist), bypassing `@datasource({ driver })` entirely. See [Postgres Drivers & Supabase](./postgres-drivers) for `postgres-js` and Supabase.
|
|
61
61
|
|
|
62
62
|
**How auto-discovery works:**
|
|
63
63
|
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
The persistent layer manages data using [Drizzle ORM](https://orm.drizzle.team/) for type-safe database access and the Repository pattern for data abstraction.
|
|
4
4
|
|
|
5
5
|
> [!NOTE] Connectors
|
|
6
|
-
> This page and the ones below it focus on the **PostgreSQL connector** (Drizzle + relational tables), the default and most common engine. The persistence layer also ships a **typesense connector** for search (see [Search & Typesense](./search-typesense)).
|
|
6
|
+
> This page and the ones below it focus on the **PostgreSQL connector** (Drizzle + relational tables), the default and most common engine. The persistence layer also ships a **SQLite connector** (see [SQLite](./sqlite)) and a **typesense connector** for search (see [Search & Typesense](./search-typesense)). All three share the same engine-neutral `AbstractRepository`/`AbstractDataSource`/`AbstractEntity` contracts - see [Connectors](/references/base/connectors) for the architecture.
|
|
7
7
|
|
|
8
8
|
## Architecture Overview
|
|
9
9
|
|
|
@@ -32,6 +32,8 @@ The persistent layer manages data using [Drizzle ORM](https://orm.drizzle.team/)
|
|
|
32
32
|
| **Repositories** | Provide type-safe CRUD operations | [Repositories Guide](./repositories.md) |
|
|
33
33
|
| **Transactions** | Handle atomic multi-step operations (PostgreSQL connector only) | [Transactions Guide](./transactions.md) |
|
|
34
34
|
| **Search & Typesense** | Full-text/faceted search over documents | [Search & Typesense Guide](./search-typesense.md) |
|
|
35
|
+
| **PGlite** | Postgres compiled to WebAssembly, running in-process | [PGlite Guide](./pglite.md) |
|
|
36
|
+
| **SQLite** | The second SQL engine, via libsql | [SQLite Guide](./sqlite.md) |
|
|
35
37
|
|
|
36
38
|
## Quick Example
|
|
37
39
|
|
|
@@ -107,6 +109,8 @@ export class Application extends BaseApplication {
|
|
|
107
109
|
- [DataSources](./datasources) - Database connections
|
|
108
110
|
- [Repositories](./repositories) - Data access layer
|
|
109
111
|
- [Transactions](./transactions) - Atomic operations
|
|
112
|
+
- [PGlite](./pglite) - Postgres in-process, for tests and embedded deployment
|
|
113
|
+
- [SQLite](./sqlite) - The second SQL engine, and what it refuses
|
|
110
114
|
- [Search & Typesense](./search-typesense) - The typesense connector
|
|
111
115
|
|
|
112
116
|
- **Related Concepts:**
|
|
@@ -0,0 +1,218 @@
|
|
|
1
|
+
# PGlite - Postgres In Your Process
|
|
2
|
+
|
|
3
|
+
PGlite is PostgreSQL compiled to WebAssembly. It runs inside your Node or Bun process, with no server, no port and no container. IGNIS reaches it through `PGliteDriver`, a third driver on the unchanged Postgres connector.
|
|
4
|
+
|
|
5
|
+
```typescript
|
|
6
|
+
import { PGlite } from '@electric-sql/pglite';
|
|
7
|
+
import { datasource } from '@venizia/ignis';
|
|
8
|
+
import type { TAnyDataSourceSchema, ValueOrPromise } from '@venizia/ignis';
|
|
9
|
+
import { BasePostgresDataSource } from '@venizia/ignis/postgres';
|
|
10
|
+
import { PGliteDriver } from '@venizia/ignis/postgres/pglite';
|
|
11
|
+
|
|
12
|
+
@datasource({ driver: PGliteDriver })
|
|
13
|
+
export class TestDataSource extends BasePostgresDataSource<{}, TAnyDataSourceSchema, {}, PGlite> {
|
|
14
|
+
constructor() {
|
|
15
|
+
super({ name: TestDataSource.name, config: {} });
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
override configure(): ValueOrPromise<void> {
|
|
19
|
+
this.client = new PGlite();
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
override getConnectionString(): ValueOrPromise<string> {
|
|
23
|
+
return 'memory://';
|
|
24
|
+
}
|
|
25
|
+
}
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Your models, repositories and filters do not change. PGlite reports `PostgreSQL 18.3`, so every statement the filter builder emits works unchanged - `#>>` JSON paths, `ILIKE`, regex operators, arrays, `SELECT ... FOR UPDATE`.
|
|
29
|
+
|
|
30
|
+
> [!IMPORTANT] Optional peer dependency
|
|
31
|
+
> `@electric-sql/pglite` is an optional peer. Only the driver class you import and name in `@datasource({ driver })` reaches your bundle.
|
|
32
|
+
>
|
|
33
|
+
> ```bash
|
|
34
|
+
> bun add @electric-sql/pglite
|
|
35
|
+
> ```
|
|
36
|
+
|
|
37
|
+
## What PGlite is good for
|
|
38
|
+
|
|
39
|
+
Two cases, and they are the only two worth the constraints below.
|
|
40
|
+
|
|
41
|
+
**An honest test database.** Your suite gets real Postgres semantics in-process - no Docker, no fixture server, no port collisions between parallel CI jobs. A SQLite test database would answer differently from production. PGlite answers the same. IGNIS runs its own relational conformance suite this way.
|
|
42
|
+
|
|
43
|
+
**Embedded deployment.** A CLI, a desktop app or an edge worker ships one data directory instead of a database service. `new PGlite('./pgdata')` creates and opens it. PGlite is a directory rather than a single file - `dumpDataDir()` is what packs it into one artifact.
|
|
44
|
+
|
|
45
|
+
Anything with concurrent writers belongs on a real Postgres server. Read [the constraints](#the-constraints) before you pick PGlite for a running service.
|
|
46
|
+
|
|
47
|
+
## Wiring a persistent instance
|
|
48
|
+
|
|
49
|
+
Point PGlite at a directory and it persists there:
|
|
50
|
+
|
|
51
|
+
```typescript
|
|
52
|
+
import { PGlite } from '@electric-sql/pglite';
|
|
53
|
+
import { datasource } from '@venizia/ignis';
|
|
54
|
+
import type { TAnyDataSourceSchema, ValueOrPromise } from '@venizia/ignis';
|
|
55
|
+
import { BasePostgresDataSource } from '@venizia/ignis/postgres';
|
|
56
|
+
import { PGliteDriver } from '@venizia/ignis/postgres/pglite';
|
|
57
|
+
|
|
58
|
+
interface IDataSourceConfigs {
|
|
59
|
+
dataDir: string;
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
@datasource({ driver: PGliteDriver })
|
|
63
|
+
export class EmbeddedDataSource extends BasePostgresDataSource<
|
|
64
|
+
IDataSourceConfigs,
|
|
65
|
+
TAnyDataSourceSchema,
|
|
66
|
+
{},
|
|
67
|
+
PGlite
|
|
68
|
+
> {
|
|
69
|
+
constructor() {
|
|
70
|
+
super({
|
|
71
|
+
name: EmbeddedDataSource.name,
|
|
72
|
+
config: { dataDir: process.env.APP_ENV_PGLITE_DATA_DIR ?? './pgdata' },
|
|
73
|
+
});
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
override configure(): ValueOrPromise<void> {
|
|
77
|
+
this.client = new PGlite(this.settings.dataDir);
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
override getConnectionString(): ValueOrPromise<string> {
|
|
81
|
+
return this.settings.dataDir;
|
|
82
|
+
}
|
|
83
|
+
}
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
`configure()` builds only the client. Naming `PGliteDriver` in the decorator is what wires the driver and the connector, lazily, on the first `getConnector()` or `beginTransaction()`.
|
|
87
|
+
|
|
88
|
+
The fourth type parameter pins `getClient()` to `PGlite`. Declare it whenever you reach for the raw client - `waitReady`, `exec()`, `dumpDataDir()`.
|
|
89
|
+
|
|
90
|
+
> [!NOTE] `getConnectionString()` returns a data directory, not a URL
|
|
91
|
+
> There is no host, port or user to spell. IGNIS never calls this method itself - it exists for your own tooling, so return whatever identifies the database. A directory path for a persistent instance, `memory://` for an ephemeral one.
|
|
92
|
+
|
|
93
|
+
## The constraints
|
|
94
|
+
|
|
95
|
+
These matter more than the setup. PGlite is a single embedded Postgres instance, and it has **exactly one session**.
|
|
96
|
+
|
|
97
|
+
### Transactions serialise
|
|
98
|
+
|
|
99
|
+
`beginTransaction()` borrows the one session from a 1-slot pool. A second caller waits for the first transaction to commit or roll back. There is no parallelism to lose - one session cannot run two transactions anyway.
|
|
100
|
+
|
|
101
|
+
A second `BEGIN` on that session would not nest and would not error. It joins the open transaction silently, and the outer `COMMIT` commits the inner writer's rows. The pool is what stops that.
|
|
102
|
+
|
|
103
|
+
### A write during an open transaction is swallowed by it
|
|
104
|
+
|
|
105
|
+
This is the trap. Repositories run through the pooled connector that `createConnector()` builds. On one session, that connector is inside whatever transaction happens to be open:
|
|
106
|
+
|
|
107
|
+
```typescript
|
|
108
|
+
const transaction = await dataSource.beginTransaction();
|
|
109
|
+
|
|
110
|
+
// Runs INSIDE the transaction above, because there is only one session.
|
|
111
|
+
await this.productRepository.create({ data: { name: 'widget' } });
|
|
112
|
+
|
|
113
|
+
await transaction.rollback(); // The product is gone.
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
Measured, not theorised: the row does not survive the rollback.
|
|
117
|
+
|
|
118
|
+
Pass the transaction explicitly, and the write is deliberate rather than accidental:
|
|
119
|
+
|
|
120
|
+
```typescript
|
|
121
|
+
await this.productRepository.create({
|
|
122
|
+
data: { name: 'widget' },
|
|
123
|
+
options: { transaction },
|
|
124
|
+
});
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
That is PGlite, not the IGNIS driver. On a server-backed Postgres each caller gets its own connection and the problem does not exist. Under concurrency, route every write through a transaction.
|
|
128
|
+
|
|
129
|
+
### The acquire timeout
|
|
130
|
+
|
|
131
|
+
A transaction that never commits holds the one slot forever. Without a bound, every later `acquire()` in the process hangs silently. So the wait is bounded:
|
|
132
|
+
|
|
133
|
+
| Option | Type | Default | Meaning |
|
|
134
|
+
| :--- | :--- | :--- | :--- |
|
|
135
|
+
| `acquireTimeoutMs` | `number` | `PGliteDriver.DEFAULT_ACQUIRE_TIMEOUT_MS` (30000) | Max ms to wait for the session before rejecting |
|
|
136
|
+
| `maxWaitingClients` | `number` | unlimited | Max queued waiters; over the limit, `acquire()` rejects immediately |
|
|
137
|
+
| `scope` | `string` | `'PGliteDriver'` | Logger scope |
|
|
138
|
+
| `client` | `PGlite` | required | The instance the driver borrows |
|
|
139
|
+
|
|
140
|
+
The timeout bounds the **wait for the slot**, never the transaction itself. It fires only under contention, so a migration or a bulk seed running alone never trips it. When it does fire you get a named error, not a dead process:
|
|
141
|
+
|
|
142
|
+
```
|
|
143
|
+
[PGliteDriver][acquire] Could not borrow the single PGlite session
|
|
144
|
+
| [acquire] Acquire timed out after 30000ms.
|
|
145
|
+
| An unreleased transaction still holds it - commit/rollback it, or raise `acquireTimeoutMs`
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
To change any of these, construct the driver yourself with `useDriver()`. That assigns the driver and builds the connector in one step, and bypasses `@datasource({ driver })`:
|
|
149
|
+
|
|
150
|
+
```typescript
|
|
151
|
+
override configure(): ValueOrPromise<void> {
|
|
152
|
+
const client = new PGlite(this.settings.dataDir);
|
|
153
|
+
|
|
154
|
+
this.useDriver({
|
|
155
|
+
driver: new PGliteDriver({ client, acquireTimeoutMs: 120_000 }),
|
|
156
|
+
schema: this.getSchema(),
|
|
157
|
+
});
|
|
158
|
+
}
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
`size` is not an option. One slot **is** the mutual exclusion PGlite lacks, so it is pinned at 1.
|
|
162
|
+
|
|
163
|
+
## Schema and migrations
|
|
164
|
+
|
|
165
|
+
drizzle-kit works here - `driver: 'pglite'` means it opens the data directory itself - but that is the catch: PGlite allows one writer on the directory at a time, so `drizzle-kit migrate` cannot reach a database your app has already opened. Generate the SQL offline, and let your own process apply it at boot.
|
|
166
|
+
|
|
167
|
+
Point `dbCredentials.url` at a folder path where a server config would carry a URL:
|
|
168
|
+
|
|
169
|
+
```typescript
|
|
170
|
+
// drizzle.config.ts
|
|
171
|
+
import { defineConfig } from 'drizzle-kit';
|
|
172
|
+
|
|
173
|
+
export default defineConfig({
|
|
174
|
+
dialect: 'postgresql',
|
|
175
|
+
driver: 'pglite',
|
|
176
|
+
schema: './src/models/**/*.model.ts',
|
|
177
|
+
out: './migrations',
|
|
178
|
+
dbCredentials: { url: './pgdata' },
|
|
179
|
+
});
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
```bash
|
|
183
|
+
bun run drizzle-kit generate
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
Apply them in-process, at boot, before the first repository call:
|
|
187
|
+
|
|
188
|
+
```typescript
|
|
189
|
+
import { drizzle } from 'drizzle-orm/pglite';
|
|
190
|
+
import { migrate } from 'drizzle-orm/pglite/migrator';
|
|
191
|
+
|
|
192
|
+
export const applyMigrations = async (opts: { dataSource: EmbeddedDataSource }): Promise<void> => {
|
|
193
|
+
const client = opts.dataSource.getClient();
|
|
194
|
+
await client.waitReady;
|
|
195
|
+
|
|
196
|
+
await migrate(drizzle({ client }), { migrationsFolder: './migrations' });
|
|
197
|
+
};
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
`migrate()` wants drizzle's own `PgliteDatabase`, which is narrower than the `getConnector()` return type. Build one over the same client - PGlite has a single session, so both handles talk to the same database.
|
|
201
|
+
|
|
202
|
+
## Runnable example
|
|
203
|
+
|
|
204
|
+
`examples/pglite-quickstart/` is this page as an app: a `pgTable` model with `uuid`, `jsonb` and `timestamptz`, a one-line repository, the standard CRUD controller, and a datasource that is the only file aware of PGlite.
|
|
205
|
+
|
|
206
|
+
```bash
|
|
207
|
+
cd examples/pglite-quickstart
|
|
208
|
+
bun install
|
|
209
|
+
bun run start # http://localhost:3000/api/notes
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
## See Also
|
|
213
|
+
|
|
214
|
+
- [Postgres Drivers & Supabase](./postgres-drivers) - the driver seam, `node-postgres` and `postgres-js`
|
|
215
|
+
- [SQLite](./sqlite) - the other embedded engine, and what it refuses
|
|
216
|
+
- [DataSources](./datasources) - schema auto-discovery and `configure()`
|
|
217
|
+
- [Transactions](./transactions) - commit, rollback and connection safety
|
|
218
|
+
- [Connectors](/references/base/connectors) - the engine-neutral contract
|
|
@@ -1,9 +1,10 @@
|
|
|
1
1
|
# Postgres Drivers & Supabase
|
|
2
2
|
|
|
3
|
-
IGNIS talks to PostgreSQL through a **driver seam
|
|
3
|
+
IGNIS talks to PostgreSQL through a **driver seam**. `IRelationalDriver` owns connection acquisition and the raw transaction control statements, and everything above it - repositories, transactions, the Casbin adapters - is driver-agnostic. Three drivers ship today:
|
|
4
4
|
|
|
5
5
|
- **node-postgres** (`pg`) - the default IGNIS has always used
|
|
6
6
|
- **postgres-js** (`postgres`) - required for Supabase's transaction pooler, and a faster option anywhere else
|
|
7
|
+
- **PGlite** (`@electric-sql/pglite`) - Postgres compiled to WebAssembly, running in your own process. It has one session and its own constraints, so it gets [its own page](./pglite)
|
|
7
8
|
|
|
8
9
|
Supabase is unmodified PostgreSQL, so it is not a separate connector: it varies the **driver**, not the SQL dialect. The `@venizia/ignis/postgres/supabase` submodule adds the two things Supabase deployments actually need - pooler presets and an RLS auth-context helper.
|
|
9
10
|
|
|
@@ -11,8 +12,9 @@ Supabase is unmodified PostgreSQL, so it is not a separate connector: it varies
|
|
|
11
12
|
> `pg` and `postgres` are both **optional peer dependencies**. The `@venizia/ignis/postgres` module pulls in neither - only the driver class you import and name in `@datasource({ driver })` reaches your bundle. Install the one your app uses:
|
|
12
13
|
>
|
|
13
14
|
> ```bash
|
|
14
|
-
> bun add pg
|
|
15
|
-
> bun add postgres
|
|
15
|
+
> bun add pg # node-postgres
|
|
16
|
+
> bun add postgres # postgres-js (>= 3.4.0)
|
|
17
|
+
> bun add @electric-sql/pglite # PGlite
|
|
16
18
|
> ```
|
|
17
19
|
|
|
18
20
|
## Import Paths
|
|
@@ -22,6 +24,7 @@ Supabase is unmodified PostgreSQL, so it is not a separate connector: it varies
|
|
|
22
24
|
| `@venizia/ignis/postgres` | `BasePostgresDataSource`, `IRelationalDriver`, repository hierarchy | no client library |
|
|
23
25
|
| `@venizia/ignis/postgres/node-postgres` | `NodePostgresDriver` | `pg` |
|
|
24
26
|
| `@venizia/ignis/postgres/postgres-js` | `PostgresJsDriver` | `postgres` |
|
|
27
|
+
| `@venizia/ignis/postgres/pglite` | `PGliteDriver` | `@electric-sql/pglite` |
|
|
25
28
|
| `@venizia/ignis/postgres/supabase` | `PoolerModes`, `buildPostgresJsOptions`, `withAuthContext`, Supabase role re-exports | `drizzle-orm/supabase` |
|
|
26
29
|
|
|
27
30
|
## Naming the Driver Class
|
|
@@ -42,12 +45,14 @@ export class PostgresDataSource extends BasePostgresDataSource<IDataSourceConfig
|
|
|
42
45
|
}
|
|
43
46
|
```
|
|
44
47
|
|
|
45
|
-
`configure()` only builds `this.client` - the raw `pg.Pool` (or postgres-js `Sql`) your app's connection settings produce. The base class wires the driver **and** the connector lazily, on first call to `getConnector()` or `beginTransaction()
|
|
48
|
+
`configure()` only builds `this.client` - the raw `pg.Pool` (or postgres-js `Sql`) your app's connection settings produce. The base class wires the driver **and** the connector lazily, on first call to `getConnector()` or `beginTransaction()`. It reads the class named in `@datasource({ driver })`, instantiates it over `this.client`, and builds the pooled Drizzle connector from that.
|
|
49
|
+
|
|
50
|
+
`getClient()` hands `this.client` back as the raw-client escape hatch. A datasource that sets neither `this.client` nor a driver (via `useDriver()`, below) throws `No driver and no client` on first use.
|
|
46
51
|
|
|
47
52
|
> [!IMPORTANT] Why a class, not a name
|
|
48
|
-
> A driver-name string cannot carry `pg` or `postgres` into your bundle - it is just text. A dynamic `import('./node-postgres.js')` keyed off that string would defer *execution*, not *packaging
|
|
53
|
+
> A driver-name string cannot carry `pg` or `postgres` into your bundle - it is just text. A dynamic `import('./node-postgres.js')` keyed off that string would defer *execution*, not *packaging*. Every bundler statically resolves a literal specifier and packages whatever it points to. A build that only used node-postgres would still fail with `Could not resolve: "postgres"`. The failure fires the moment postgres-js's import appears anywhere in the module graph reachable at build time. Naming the class instead makes the driver module a real value reference - the one thing a bundler is forced to keep. That's what lets `pg` and `postgres` stay genuinely optional peers. A bare side-effect import (`import '@venizia/ignis/postgres/node-postgres'`) would not work either. `@venizia/ignis` declares `sideEffects: false`, so a bundler is free to drop an import whose exports go unused.
|
|
49
54
|
>
|
|
50
|
-
> Two tests pin this from different angles
|
|
55
|
+
> Two tests pin this from different angles. `packages/core-server/src/__tests__/connectors/postgres/no-eager-driver-import.test.ts` proves no barrel **loads** a driver package in a fresh process (the runtime module graph). `packages/core-server/src/__tests__/connectors/postgres/bundle/optional-peers.test.ts` proves no barrel gets a driver package **packaged** by a real bundler.
|
|
51
56
|
|
|
52
57
|
## Using postgres-js
|
|
53
58
|
|
|
@@ -84,11 +89,11 @@ new NodePostgresDriver({ client: pool }); // client must expose connect() AND to
|
|
|
84
89
|
new PostgresJsDriver({ client: sql }); // client must expose reserve() AND unsafe()
|
|
85
90
|
```
|
|
86
91
|
|
|
87
|
-
`NodePostgresDriver` rejects a bare `pg.Client` - it exposes `connect()` too, but has no pool accounting and cannot hand out a dedicated connection per transaction. `PostgresJsDriver` rejects a `pg.Pool` the same way. You will not normally construct these yourself: `wireDriverFromMetadata()` does it for you from `this.client
|
|
92
|
+
`NodePostgresDriver` rejects a bare `pg.Client` - it exposes `connect()` too, but has no pool accounting and cannot hand out a dedicated connection per transaction. `PostgresJsDriver` rejects a `pg.Pool` the same way. You will not normally construct these yourself: `wireDriverFromMetadata()` does it for you from `this.client`. This validation fires the first time a datasource wired the wrong client behind the wrong `@datasource({ driver })` class.
|
|
88
93
|
|
|
89
94
|
## Custom or Third-Party Drivers: `useDriver()`
|
|
90
95
|
|
|
91
|
-
For a driver IGNIS does not ship, wire it explicitly with `useDriver()
|
|
96
|
+
For a driver IGNIS does not ship, wire it explicitly with `useDriver()`. It assigns the driver **and** builds the pooled connector in one step, so the half-wired state (driver set, connector forgotten) cannot exist:
|
|
92
97
|
|
|
93
98
|
```typescript
|
|
94
99
|
export class PostgresDataSource extends BasePostgresDataSource<IDataSourceConfigs> {
|
|
@@ -101,10 +106,10 @@ export class PostgresDataSource extends BasePostgresDataSource<IDataSourceConfig
|
|
|
101
106
|
}
|
|
102
107
|
```
|
|
103
108
|
|
|
104
|
-
`useDriver()` bypasses `@datasource({ driver })` entirely
|
|
109
|
+
`useDriver()` bypasses `@datasource({ driver })` entirely. You never need to name a class in the decorator when you wire the driver yourself in `configure()`.
|
|
105
110
|
|
|
106
111
|
> [!WARNING] postgres-js cannot destroy a poisoned connection
|
|
107
|
-
> After a failed `COMMIT` or `ROLLBACK`, node-postgres **destroys** the connection instead of pooling it
|
|
112
|
+
> After a failed `COMMIT` or `ROLLBACK`, node-postgres **destroys** the connection instead of pooling it. The session may still hold an open transaction that the next borrower would inherit. postgres-js has no destroy semantics (`ReservedSql.release()` takes no argument), so the connection is returned to the pool anyway. This asymmetry is real and IGNIS does not paper over it; it is pinned by the driver's own tests.
|
|
108
113
|
|
|
109
114
|
## The Driver Contract
|
|
110
115
|
|
|
@@ -125,9 +130,9 @@ interface IRelationalConnection<Schema> {
|
|
|
125
130
|
}
|
|
126
131
|
```
|
|
127
132
|
|
|
128
|
-
`acquire()` matters for transactions: `BEGIN` and `COMMIT` must land on the same backend
|
|
133
|
+
`acquire()` matters for transactions: `BEGIN` and `COMMIT` must land on the same backend. Each explicit transaction therefore gets a dedicated connection (`pool.connect()` for pg, `sql.reserve()` for postgres-js) - that's the reason for the `>= 3.4.0` floor.
|
|
129
134
|
|
|
130
|
-
The connection is checked out of the pool before Drizzle is constructed on top of it. If that constructor throws - a malformed discovered schema, a drizzle mismatch - both drivers catch the error
|
|
135
|
+
The connection is checked out of the pool before Drizzle is constructed on top of it. If that constructor throws - a malformed discovered schema, a drizzle mismatch - both drivers catch the error. They release the connection back to the pool first, and rethrow. Without this, every failed `acquire()` would strand a connection, and the pool would exhaust after enough of them.
|
|
131
136
|
|
|
132
137
|
`execute()` resolves to the neutral `IStatementResult` (`{ count }` - the same `count` the repository verbs speak). Each driver maps its native result shape at its own boundary; nothing above the seam ever inspects a driver-specific type.
|
|
133
138
|
|
|
@@ -143,7 +148,7 @@ Supabase exposes three ways in, and one of them silently breaks prepared stateme
|
|
|
143
148
|
| `PoolerModes.SESSION` | 5432 (pooler) | yes | pooled, one backend per client session |
|
|
144
149
|
| `PoolerModes.TRANSACTION` | 6543 | **no** | serverless / many short-lived connections |
|
|
145
150
|
|
|
146
|
-
The transaction pooler (Supavisor) rebinds the backend per transaction, so a server-side prepared statement created on one backend
|
|
151
|
+
The transaction pooler (Supavisor) rebinds the backend per transaction, so a server-side prepared statement created on one backend is not there next time. `buildPostgresJsOptions` encodes this so you cannot forget it:
|
|
147
152
|
|
|
148
153
|
```typescript
|
|
149
154
|
import { datasource } from '@venizia/ignis';
|
|
@@ -191,11 +196,12 @@ try {
|
|
|
191
196
|
Three properties make this safe under a transaction-mode pooler:
|
|
192
197
|
|
|
193
198
|
- `claims` is **bound as a query parameter**, never interpolated into SQL text.
|
|
194
|
-
- `role` must be a bare identifier (`/^[a-z_][a-z0-9_]*$/`) because `set local role $1` is not valid SQL
|
|
199
|
+
- `role` must be a bare identifier (`/^[a-z_][a-z0-9_]*$/`) because `set local role $1` is not valid SQL. A role taken from a JWT and interpolated unvalidated would be a privilege-escalation vector.
|
|
200
|
+
- Validation runs **before** any statement, so a rejected call leaves the session untouched.
|
|
195
201
|
- Everything is `SET LOCAL` / `set_config(..., true)` - transaction-scoped. A plain `SET` would leak the caller's identity to the next borrower of the pooled connection, and is deliberately not offered.
|
|
196
202
|
|
|
197
203
|
The submodule also re-exports Drizzle's Supabase helpers (`anonRole`, `authenticatedRole`, `serviceRole`, `authUid`, `authUsers`, ...) so RLS-aware schema files need one import.
|
|
198
204
|
|
|
199
205
|
## Adding a Driver
|
|
200
206
|
|
|
201
|
-
One file under `src/connectors/postgres/drivers/`, implementing the four verbs above
|
|
207
|
+
One file under `src/connectors/postgres/drivers/`, implementing the four verbs above. Add a fake client and a test that runs the shared conformance suite (`run({ driver, buildDriverProbe })` in `src/__tests__/connectors/postgres/drivers/conformance/`). Register a sub-path export and an optional peer dependency. Never re-export the driver from the drivers barrel - that is what would make its package load eagerly for everyone.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Search & Meilisearch
|
|
2
2
|
|
|
3
|
-
IGNIS ships a **meilisearch connector** under `@venizia/ignis/meilisearch`. It shares the entire search paradigm with the Typesense connector
|
|
3
|
+
IGNIS ships a **meilisearch connector** under `@venizia/ignis/meilisearch`. It shares the entire search paradigm with the Typesense connector: the same `@model`/`@repository`/`@datasource` decorators and the same `defineSearchCollection` field DSL. It also shares the same repository tiers and controller factory. All of that lives in the engine-neutral `@venizia/ignis/search` module.
|
|
4
4
|
|
|
5
5
|
What it does **not** share is the pretence that the two engines are the same. Where Meilisearch differs from Typesense, IGNIS surfaces the difference rather than papering over it.
|
|
6
6
|
|
|
@@ -53,7 +53,7 @@ Meilisearch is **schemaless**: it has no field schema at all. A collection there
|
|
|
53
53
|
| `synonyms` | a flat `synonyms` dictionary |
|
|
54
54
|
| `engineOverrides.meilisearch` | merged last onto settings |
|
|
55
55
|
|
|
56
|
-
The `searchable` and `filterable` flags are **load-bearing here** and are silently dropped by the Typesense compiler, which indexes every field by default. If no field declares `searchable`, IGNIS leaves `searchableAttributes` at Meilisearch's own `['*']` default
|
|
56
|
+
The `searchable` and `filterable` flags are **load-bearing here** and are silently dropped by the Typesense compiler, which indexes every field by default. If no field declares `searchable`, IGNIS leaves `searchableAttributes` at Meilisearch's own `['*']` default. Emitting an empty array instead would disable search entirely.
|
|
57
57
|
|
|
58
58
|
## The DataSource
|
|
59
59
|
|
|
@@ -93,7 +93,7 @@ await repository.create({ data: article });
|
|
|
93
93
|
// The document is retrievable here. No sleep, no retry loop.
|
|
94
94
|
```
|
|
95
95
|
|
|
96
|
-
The cost is real and worth knowing: each write polls until its task completes. The client SDK's own `waitForTask` defaults to a 5-second timeout, which is far too short for a bulk import
|
|
96
|
+
The cost is real and worth knowing: each write polls until its task completes. The client SDK's own `waitForTask` defaults to a 5-second timeout, which is far too short for a bulk import. IGNIS never relies on it - configure `taskTimeoutMs` instead. A task that fails raises the standard sanitized dependency error; one that never terminates raises a `core.search_engine.task_timeout`.
|
|
97
97
|
|
|
98
98
|
If you want fire-and-forget writes, reach the raw client:
|
|
99
99
|
|
|
@@ -108,7 +108,7 @@ Meilisearch's search route reports `estimatedTotalHits`, and its exhaustive `tot
|
|
|
108
108
|
|
|
109
109
|
IGNIS never counts through the search route. `count()` calls `POST /documents/fetch` with `limit: 0` and reads the exact `total`, which the engine's own documentation states is unaffected by `maxTotalHits`. A filtered count over 50,000 matching documents returns 50,000.
|
|
110
110
|
|
|
111
|
-
`ISearchResult.isFoundExact` reports the truth for the **search** route: always `true` on Typesense, and on Meilisearch `true` only in the exhaustive page mode. It travels all the way out through the generated REST response
|
|
111
|
+
`ISearchResult.isFoundExact` reports the truth for the **search** route: always `true` on Typesense, and on Meilisearch `true` only in the exhaustive page mode. It travels all the way out through the generated REST response. An API consumer paginating your endpoint can tell an estimate from an exhaustive count.
|
|
112
112
|
|
|
113
113
|
## Capability differences are compile-time, not runtime
|
|
114
114
|
|
|
@@ -141,25 +141,35 @@ class PortableRepository extends DefaultSearchRepository<TDocument> {
|
|
|
141
141
|
|
|
142
142
|
## Behaviours to know before you commit
|
|
143
143
|
|
|
144
|
-
**`create()` rejects duplicates, but the check is not atomic.** Meilisearch's `addDocuments` is add-or-replace; it has no conditional insert.
|
|
144
|
+
**`create()` rejects duplicates, but the check is not atomic.** Meilisearch's `addDocuments` is add-or-replace; it has no conditional insert. The neutral contract says `create()` returns 409 on a duplicate id, matching Typesense.
|
|
145
|
+
|
|
146
|
+
To honour it, IGNIS reads the id first and throws `409 core.search_engine.already_exists` if it exists. Two concurrent creates of the same id can both pass that check, and the second write wins. If you want last-write-wins, call `upsert()` and say so.
|
|
145
147
|
|
|
146
148
|
**`updateById` on a missing id throws `404`, it does not upsert.** Meilisearch's `updateDocuments` is add-or-update, so a missing id would silently fabricate a record. IGNIS checks existence before any write and throws `404 core.search_engine.not_found` instead - matching Typesense and the neutral contract. (This is the deliberate search-family divergence from the PostgreSQL connector, where a missing-id `updateById` is a silent `{ count: 0 }`.)
|
|
147
149
|
|
|
148
|
-
**`updateBy` is emulated and is not atomic.** Meilisearch has no update-by-filter endpoint. IGNIS pages primary keys out of `documents/fetch`, then issues merge-`PUT` batches. A concurrent write to a matched document may be overwritten.
|
|
150
|
+
**`updateBy` is emulated and is not atomic.** Meilisearch has no update-by-filter endpoint. IGNIS pages primary keys out of `documents/fetch`, then issues merge-`PUT` batches. A concurrent write to a matched document may be overwritten.
|
|
151
|
+
|
|
152
|
+
The experimental `documents/edit` route would do this server-side, but it is gated behind a feature flag and documented as breaking between minor versions. It also has an open correctness bug, so IGNIS does not use it.
|
|
153
|
+
|
|
154
|
+
**`import`/`createAll` is batch-atomic.** Documents are added in batches, each batch's task awaited before the next starts. A batch either lands whole or throws, so `count.fail` is structurally always `0`.
|
|
149
155
|
|
|
150
|
-
|
|
156
|
+
When a batch fails, the batches after the failure point did not land. The thrown error's `details` carry `{ totalCount, processedCount }`, so a caller can resume from `processedCount` rather than replaying the whole import.
|
|
151
157
|
|
|
152
|
-
**Meilisearch error classification reads the real SDK error shape.** A thrown `MeilisearchApiError` carries its body under `error.cause.code` and its HTTP status under `error.response.status`, not top-level `code`/`httpStatus`. Reading only the flat shape misclassifies a legitimate 404 (for example the existence probe inside `create()`) as an unrelated 503 dependency failure
|
|
158
|
+
**Meilisearch error classification reads the real SDK error shape.** A thrown `MeilisearchApiError` carries its body under `error.cause.code` and its HTTP status under `error.response.status`, not top-level `code`/`httpStatus`. Reading only the flat shape misclassifies a legitimate 404 (for example the existence probe inside `create()`) as an unrelated 503 dependency failure. The classifier checks both the flat shape (task-error responses) and the nested `cause`/`response` shape (thrown SDK errors).
|
|
153
159
|
|
|
154
|
-
**`createCollection`/`ensureCollection` are idempotent.** An `index_already_exists` failure is tolerated and logged rather than thrown
|
|
160
|
+
**`createCollection`/`ensureCollection` are idempotent.** An `index_already_exists` failure is tolerated and logged rather than thrown. Boot-time provisioning against an already-provisioned collection (a restart, a second app instance) succeeds instead of crashing.
|
|
155
161
|
|
|
156
|
-
**`hiddenProperties` is supported on Meilisearch via JS-side stripping, not engine-side exclusion.** Meilisearch has no per-query field-exclusion param (only the index-level `displayedAttributes` setting), so the dialect does not attempt to translate `hiddenFields
|
|
162
|
+
**`hiddenProperties` is supported on Meilisearch via JS-side stripping, not engine-side exclusion.** Meilisearch has no per-query field-exclusion param (only the index-level `displayedAttributes` setting), so the dialect does not attempt to translate `hiddenFields`. It deliberately passes them through untranslated.
|
|
163
|
+
|
|
164
|
+
The guarantee is upheld one layer up: `SearchBaseRepository` strips hidden fields from every document `find()`/`findOne()`/`search()` returns before it reaches the caller. A model with `hiddenProperties` works identically to Typesense from the caller's point of view - no throw, no leak.
|
|
157
165
|
|
|
158
166
|
**A geopoint field must be named `_geo`.** Meilisearch supports exactly one geo field, shaped `{ lat, lng }`, under that reserved name. The compiler throws on any other name, and on a second geo field.
|
|
159
167
|
|
|
160
168
|
**Vector distance is cosine only.** `field.vector(..., { distance: 'l2' })` throws.
|
|
161
169
|
|
|
162
|
-
**Engine-specific knobs never reach the neutral schema.** The neutral search input carries only fields every engine supports
|
|
170
|
+
**Engine-specific knobs never reach the neutral schema.** The neutral search input carries only fields every engine supports. Typesense-only tuning (`num_typos`, `pinned_hits`, `preset`, ...) travels through `engineParams` with wire names and is passed verbatim to the engine. Sending it to a Meilisearch datasource is a caller error.
|
|
171
|
+
|
|
172
|
+
The dialect still raises a named error for known-foreign knobs arriving via untyped input. Use `engineParams` (or `mode: 'raw'`) for Meilisearch-native tuning such as `rankingScoreThreshold`.
|
|
163
173
|
|
|
164
174
|
**`multiSearch` batches but does not merge.** `union: true` throws. Meilisearch merges results through its `federation` option, which this connector does not model, so `getCapabilities()` honestly reports `union: false`.
|
|
165
175
|
|
|
@@ -169,7 +179,7 @@ class PortableRepository extends DefaultSearchRepository<TDocument> {
|
|
|
169
179
|
|
|
170
180
|
Vector fields compile to `embedders`, and `mode: 'semantic'` / `mode: 'hybrid'` translate to Meilisearch's `hybrid: { semanticRatio, embedder }` plus an optional `vector`. The neutral `alpha` maps to `semanticRatio`; `mode: 'semantic'` pins it to `1.0`, and an omitted `alpha` on `hybrid` defaults to `0.5`.
|
|
171
181
|
|
|
172
|
-
This path is compiled and translated but is **not exercised end to end** by the framework's test suite
|
|
182
|
+
This path is compiled and translated but is **not exercised end to end** by the framework's test suite. Meilisearch has also moved its vector store in and out of `/experimental-features` across server versions. Verify against your target server version before depending on it.
|
|
173
183
|
|
|
174
184
|
## Adding another search engine
|
|
175
185
|
|
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Search & Typesense
|
|
2
2
|
|
|
3
|
-
IGNIS ships a **typesense connector** under `@venizia/ignis/typesense` for full-text and faceted search. It plugs into the same `@model`/`@repository`/`@datasource` decorators as the PostgreSQL connector
|
|
3
|
+
IGNIS ships a **typesense connector** under `@venizia/ignis/typesense` for full-text and faceted search. It plugs into the same `@model`/`@repository`/`@datasource` decorators as the PostgreSQL connector. Documents replace Drizzle tables, and the query surface is intentionally narrower - search engines are not relational databases.
|
|
4
4
|
|
|
5
5
|
> [!IMPORTANT] Subpath-only import
|
|
6
|
-
> `typesense` is an **optional peer dependency
|
|
6
|
+
> `typesense` is an **optional peer dependency**. Unlike `postgres`, the typesense connector is not re-exported from the `@venizia/ignis` root barrel, so apps that don't use search never pull in the `typesense` client. Always import from `@venizia/ignis/typesense`:
|
|
7
7
|
>
|
|
8
8
|
> ```typescript
|
|
9
9
|
> import {
|
|
@@ -77,7 +77,7 @@ The same `@model` settings you use with Drizzle entities - `hiddenProperties`, `
|
|
|
77
77
|
| `field.geopoint(name, flags?)` | `geopoint` (`[number, number]`) | |
|
|
78
78
|
| `field.vector(name, opts)` | `float[]` | Embedding vector for semantic/hybrid search. Its `opts` carry `dimensions`/`distance`/`embed` rather than the shared flags - see [Vector Fields](#vector-fields-for-semantic-search). |
|
|
79
79
|
|
|
80
|
-
Each `flags` object may set `searchable`, `filterable`, `facet`, `sortable`, `optional` (all `boolean`). Note: `searchable`/`filterable` have no direct Typesense wire equivalent
|
|
80
|
+
Each `flags` object may set `searchable`, `filterable`, `facet`, `sortable`, `optional` (all `boolean`). Note: `searchable`/`filterable` have no direct Typesense wire equivalent - Typesense indexes every field by default - so they're dropped at compile time. Only `facet`, `optional`, and `sortable` (mapped to Typesense's `sort`) actually reach the compiled collection schema. They're still worth setting for documentation/intent and for future engines that do distinguish them.
|
|
81
81
|
|
|
82
82
|
`defineSearchCollection` validates at call time: throws on an empty `name`, empty `fields`, duplicate field names, a non-`string` `id` field, or an unknown `defaultSort` field reference.
|
|
83
83
|
|
|
@@ -128,7 +128,7 @@ field.vector('embedding', {
|
|
|
128
128
|
});
|
|
129
129
|
```
|
|
130
130
|
|
|
131
|
-
The `model` config is camelCase and maps to Typesense's snake_case `model_config` at compile time (`name` becomes `model_name`, `apiKey` becomes `api_key`, `accessToken` becomes `access_token`, and so on)
|
|
131
|
+
The `model` config is camelCase and maps to Typesense's snake_case `model_config` at compile time (`name` becomes `model_name`, `apiKey` becomes `api_key`, `accessToken` becomes `access_token`, and so on). Any provider field not modeled is passed through unchanged. Never hardcode `apiKey` into a committed schema - source it from an environment variable.
|
|
132
132
|
|
|
133
133
|
**Client-provided vectors** - you compute the embedding yourself and send it with each document. Declare `dimensions` and a distance metric instead of `embed`:
|
|
134
134
|
|
|
@@ -160,7 +160,7 @@ static override schema = defineSearchCollection({
|
|
|
160
160
|
});
|
|
161
161
|
```
|
|
162
162
|
|
|
163
|
-
Declarative `synonyms` are provisioned as one **synonym set** per collection (named `<collection>_synonyms`) and linked to it
|
|
163
|
+
Declarative `synonyms` are provisioned as one **synonym set** per collection (named `<collection>_synonyms`) and linked to it. That matches Typesense v30+'s global synonym-sets model - the pre-v30 per-collection synonyms API was removed. For runtime management outside the declarative schema, the connector exposes the synonym-set verbs through the raw client escape hatch:
|
|
164
164
|
|
|
165
165
|
```typescript
|
|
166
166
|
const connector = repository.dataSource.getConnector();
|
|
@@ -241,26 +241,38 @@ export class ArticleRepository extends DefaultSearchRepository<TArticleDocument>
|
|
|
241
241
|
}
|
|
242
242
|
```
|
|
243
243
|
|
|
244
|
-
`find()`/`findOne()`/`findById()`/`count()`/`existsWith()` accept the same `TFilter`/`TWhere` shape as PostgreSQL repositories
|
|
244
|
+
`find()`/`findOne()`/`findById()`/`count()`/`existsWith()` accept the same `TFilter`/`TWhere` shape as PostgreSQL repositories. The `TypesenseQueryDialect` translates `where` into Typesense's `filter_by` syntax, `order` into `sort_by` (max 3 fields - a Typesense limit), `limit`/`skip` into `per_page`/`page`, and `fields` into `include_fields`. `filter.include` is **not supported** and throws - there is no relation model for documents.
|
|
245
|
+
|
|
246
|
+
Search engines index asynchronously, so a document can be missing right after a write. `find()`/`findOne()`/`findById()` accept the same `options.retry` as PostgreSQL - re-read until the document shows up:
|
|
247
|
+
|
|
248
|
+
```typescript
|
|
249
|
+
const article = await articleRepository.findById({
|
|
250
|
+
id,
|
|
251
|
+
options: { retry: { maxAttempts: 4 } },
|
|
252
|
+
});
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
Same options, same rules as PostgreSQL - see [Read Retry](/references/base/repositories/advanced#read-retry-replica-lag). The only difference is moot: retry skips inside a transaction, and search has no transactions to begin with.
|
|
245
256
|
|
|
246
257
|
### Envelope differences vs. PostgreSQL
|
|
247
258
|
|
|
248
259
|
| | PostgreSQL repositories | Search repositories |
|
|
249
260
|
| ------------------------------------------ | --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
250
261
|
| Range queries | Extra `COUNT(*)` query for `shouldQueryRange: true` | Single search call already returns `found` - no second query needed |
|
|
251
|
-
| `updateAll`/`deleteAll` with empty `where` | Requires `force: true` | `updateAll` **refuses** an unfiltered bulk update outright (throws, no override)
|
|
262
|
+
| `updateAll`/`deleteAll` with empty `where` | Requires `force: true` | `updateAll` **refuses** an unfiltered bulk update outright (throws, no override). `deleteAll` with no effective filter (no `where` and no `@model defaultFilter`) also throws. Passing `options.force: true` truncates the whole collection instead |
|
|
252
263
|
| Returning updated/deleted rows | Native SQL `RETURNING` | No `RETURNING` equivalent - `updateAll`/`deleteAll` are **count-only**: `data` is always `null` and `shouldReturn` is rejected at the type level (no hidden engine read is bolted on). Read the affected documents explicitly before the write when you need them |
|
|
253
264
|
| `createAll` | Native bulk insert, returns each row | Delegates to Typesense's bulk import; "created" rows are the input rows whose per-row response didn't report `success: false` |
|
|
254
265
|
|
|
255
266
|
### Write error semantics (shared by both search engines)
|
|
256
267
|
|
|
257
|
-
- **`create()` with a duplicate id throws `409
|
|
268
|
+
- **`create()` with a duplicate id throws `409`** (`normalized.code: 'core.search_engine.already_exists'`) on both Typesense and Meilisearch. A duplicate is a conflict, never a silent overwrite - call `upsert()` explicitly for last-write-wins.
|
|
269
|
+
- On Meilisearch, the duplicate pre-check is not atomic, so two concurrent creates of the same id can both pass it. See the [Meilisearch guide](./search-meilisearch) and use `upsert()` when that's possible.
|
|
258
270
|
- **`updateById()` against a missing id throws `404` (`normalized.code: 'core.search_engine.not_found'`)** on both engines. This is the deliberate divergence from the PostgreSQL connector, where a missing-id `updateById` is a silent `{ count: 0 }`.
|
|
259
271
|
- **`deleteById()` against a missing id is silent** (`{ count: 0, data: null }`), matching the relational connectors.
|
|
260
272
|
|
|
261
273
|
## Searching with `search({ mode })`
|
|
262
274
|
|
|
263
|
-
Beyond the `find()` tier (filter-only reads that return documents), `search()` is the search-engine entry point. It is one method whose shape is selected by `mode`, so TypeScript narrows the options to just that mode's fields
|
|
275
|
+
Beyond the `find()` tier (filter-only reads that return documents), `search()` is the search-engine entry point. It is one method whose shape is selected by `mode`, so TypeScript narrows the options to just that mode's fields. You never juggle every parameter at once. `mode` defaults to `keyword`.
|
|
264
276
|
|
|
265
277
|
The `keyword`/`semantic`/`hybrid` modes reuse the same filter translation as `find()` (`@model defaultFilter`, `hiddenProperties`, `where`/`order`/`limit`), then layer the search term or vector on top. The `raw` mode is a full passthrough with no translation.
|
|
266
278
|
|
|
@@ -299,7 +311,7 @@ await repository.search({
|
|
|
299
311
|
```
|
|
300
312
|
|
|
301
313
|
> [!NOTE]
|
|
302
|
-
> `semantic` and `hybrid` default `prefix=false`: prefix matching is meaningless for vector search
|
|
314
|
+
> `semantic` and `hybrid` default `prefix=false`: prefix matching is meaningless for vector search. Remote embedders (OpenAI/Google/...) reject it too (`Prefix search is not supported for remote embedders`). Pass `engineParams: { prefix: true }` to override.
|
|
303
315
|
|
|
304
316
|
**`raw`** - native Typesense params, no `TFilter`/`defaultFilter`/`hiddenProperties` translation. Use it for any engine feature the modes above do not model:
|
|
305
317
|
|
|
@@ -313,9 +325,9 @@ await repository.search({
|
|
|
313
325
|
> [!NOTE]
|
|
314
326
|
> `raw` skips `@model defaultFilter`, so it can surface documents the `find()`/`keyword` tiers would hide (for example non-`published` articles). Apply any tenant or visibility filtering explicitly in `params.filter_by` when exposing raw search over HTTP.
|
|
315
327
|
|
|
316
|
-
All non-`raw` modes accept the same optional camelCase tuning fields, mapped to Typesense's snake_case at the dialect boundary
|
|
328
|
+
All non-`raw` modes accept the same optional camelCase tuning fields, mapped to Typesense's snake_case at the dialect boundary. That covers faceting (`facetBy`, `facetQuery`, `maxFacetValues`), highlighting (`highlightFields`, `highlightFullFields`, `highlightStartTag`/`highlightEndTag`, `snippetThreshold`), grouping (`groupBy`, `groupLimit`, `groupMissingValues`), and `queryByWeights`. These are the ENGINE-NEUTRAL fields - every search engine supports them.
|
|
317
329
|
|
|
318
|
-
Typesense-specific tuning (`num_typos`, `prefix`, `infix`, `use_cache`, `cache_ttl`, `exhaustive_search`, `pinned_hits`, `hidden_hits`, `prioritize_exact_match`, `drop_tokens_threshold`, `preset`) goes through `engineParams`, passed verbatim to the engine with WIRE names
|
|
330
|
+
Typesense-specific tuning (`num_typos`, `prefix`, `infix`, `use_cache`, `cache_ttl`, `exhaustive_search`, `pinned_hits`, `hidden_hits`, `prioritize_exact_match`, `drop_tokens_threshold`, `preset`) goes through `engineParams`, passed verbatim to the engine with WIRE names. The neutral input schema no longer carries knobs only one engine understands:
|
|
319
331
|
|
|
320
332
|
```typescript
|
|
321
333
|
await repository.search({
|
|
@@ -381,7 +393,7 @@ await repository.dataSource.multiSearch({
|
|
|
381
393
|
Both `searches` entries and `commonParams` are camelCase (`filterBy`, `queryBy`, `perPage`); the datasource maps them to Typesense's snake_case wire format the same way single-collection `search()` does. For native snake_case access, `getConnector().multiSearch(...)` takes the engine's own params.
|
|
382
394
|
|
|
383
395
|
> [!NOTE]
|
|
384
|
-
> `multiSearch` injects each collection's `@model hiddenProperties` into that entry's `excludeFields` automatically
|
|
396
|
+
> `multiSearch` injects each collection's `@model hiddenProperties` into that entry's `excludeFields` automatically. Hidden fields never leak on the multi-search route, just as they don't on single-collection `search()`. An entry naming a collection IGNIS did not discover (not a bound `@repository` model) passes through untouched: the **caller owns exclusion** for unknown collections. The raw `getConnector().multiSearch(...)` escape hatch performs no injection at all.
|
|
385
397
|
|
|
386
398
|
## Search Controller Factory
|
|
387
399
|
|
|
@@ -403,7 +415,7 @@ const _SearchController = SearchControllerFactory.defineSearchController({
|
|
|
403
415
|
export const ArticleSearchController = _SearchController;
|
|
404
416
|
```
|
|
405
417
|
|
|
406
|
-
It registers `POST /search` (request body is the mode-discriminated schema, so validation and OpenAPI come for free and the handler dispatches to `repository.search`)
|
|
418
|
+
It registers `POST /search` (request body is the mode-discriminated schema, so validation and OpenAPI come for free, and the handler dispatches to `repository.search`). It also registers `POST /multi-search`, which dispatches to `dataSource.multiSearch`. For anything the factory does not cover, write a custom `BaseRestController` calling `repository.search()` or `getClient()` directly.
|
|
407
419
|
|
|
408
420
|
## Transactions and Locking
|
|
409
421
|
|