@voltro/sql-mysql 0.20.0 → 0.20.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +137 -0
- package/dist/index.d.ts +44 -0
- package/dist/index.js +157 -114
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -39,6 +39,143 @@ _Changes staged for the next release accumulate here (rolled up from
|
|
|
39
39
|
|
|
40
40
|
---
|
|
41
41
|
|
|
42
|
+
## [0.20.1] — 2026-07-30
|
|
43
|
+
|
|
44
|
+
### Changed
|
|
45
|
+
|
|
46
|
+
- **@voltro/database, @voltro/runtime, @voltro/plugin-versioning, @voltro/plugin-presence, @voltro/voltro** — Five framework-table indexes were holding GENERIC names in a namespace that is shared with your tables. Index names are unique per SCHEMA on every supported dialect, so `_voltro_row_history.index('byTrace')` reserved `byTrace` for the whole database — and `byTrace` is the first thing anyone reaches for when indexing a `traceId`. A consumer added `traceId` to their own audit table, indexed it the obvious way, and collided with ours; the framework's own error message even suggested renaming the framework's index as the fix.
|
|
47
|
+
|
|
48
|
+
Renamed: `_voltro_row_history` `byTrace` → `byRowHistoryTrace`, `bySubject` → `byRowHistorySubject`; `_voltro_api_keys` `byTenant` → `byApiKeyTenant`; `_voltro_presence` `byChannel` → `byPresenceChannel`; `_voltro_connections` `bySubject` → `byConnectionSubject`. These are `_voltro_*` tables, so the rename rides the declarative differ on `voltro db apply` / boot — no codemod. Adopters see a one-time index rebuild.
|
|
49
|
+
|
|
50
|
+
A test now enforces the rule that most framework tables already followed: a framework index name must MENTION its own table. Mechanical, so it cannot rot the way a curated list of "generic" names would, and it does not demand the full `_voltro_<table>_<name>` form — which would force renaming ~20 already-safe indexes for no benefit. It also asserts no two framework tables claim the same index name, since installing two such plugins together would fail at migrate time for a reason neither plugin's author could see.
|
|
51
|
+
|
|
52
|
+
### Fixed
|
|
53
|
+
|
|
54
|
+
- **@voltro/sql-mysql, @voltro/voltro** — A MariaDB table with a UNIQUE constraint on an UNBOUNDED text column can never be decoded from the binlog. The reader now says so ONCE — with the real cause and a remedy that works — and excludes the table, instead of looping on it forever.
|
|
55
|
+
|
|
56
|
+
**The mechanism.** MariaDB backs an unbounded UNIQUE with a **HASH long-unique index**, which adds a hidden `DB_ROW_HASH_n` column to the InnoDB row. That column IS in the binlog row image and is NOT in `information_schema.COLUMNS`, so the reader compares N+1 against N and throws on every write to that table:
|
|
57
|
+
|
|
58
|
+
```text
|
|
59
|
+
Table app.sessions schema changed between binlog event and metadata fetch:
|
|
60
|
+
the event has 9 columns, fetched metadata has 8
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Nothing is broken; the table is shaped that way, permanently. The previous recovery (skip to the current binlog end) recovered nothing, because the end is exactly where the next failing write appears — a loop a consumer measured at roughly every 9 seconds, re-signalling resync to the whole fleet each pass.
|
|
64
|
+
|
|
65
|
+
**The cause we shipped in the previous entry was WRONG, and this retracts it.** It blamed a `DROP COLUMN` that ran as `ALGORITHM=INSTANT` leaving a phantom column, and told people to run `ALTER TABLE … FORCE`. The same consumer measured that: 9 InnoDB columns before the rebuild, 9 after, hidden column still present — the rebuild recreates the index and therefore recreates the hidden column. The repair line sent readers in a circle. They also disproved the version theory, being on the same MariaDB 11.8 we had tested on and failed to reproduce a phantom column with.
|
|
66
|
+
|
|
67
|
+
**Now:** affected tables are found at CDC start by a privilege-free probe — the direct evidence in `INNODB_SYS_COLUMNS` needs `PROCESS`, which an app DB user does not have, so the constraint SHAPE is inferred from `information_schema.STATISTICS` + `COLUMNS` instead — reported once as an error naming `text().maxLength(n)` as the remedy and `ALTER TABLE FORCE` as explicitly not one, and EXCLUDED from the reader.
|
|
68
|
+
|
|
69
|
+
Excluding is what makes it converge, and that is measured rather than assumed: an excluded table with a hidden hash column produces no reader error at all, while the same table included throws on the first write. Cross-instance change events for such a table are lost until it is bounded; own-node reactivity is unaffected (writes still emit inline).
|
|
70
|
+
|
|
71
|
+
Framework `_voltro_*` tables cannot hit this — they are filtered out of the reader's include list before it reaches the replication client, and exclusion demonstrably shields the metadata fetch.
|
|
72
|
+
|
|
73
|
+
**Caveat worth reading if you are already affected:** on a table that ALREADY exists, adding `.maxLength(n)` currently changes nothing — the schema differ does not diff text length, so it plans 0 operations and reports "up to date". That is a separate defect, reported in the same round and not yet fixed; until it is, the remedy only applies to newly created tables.
|
|
74
|
+
- **@voltro/cli, @voltro/voltro** — `voltro codegen` no longer writes a silently plugin-less `rpcGroup.generated.ts`, and it now reports what it merged.
|
|
75
|
+
|
|
76
|
+
`loadApiConfig` swallows every failure into `null`, and `config?.plugins ?? []` turned that into "this app has no plugins". So an `app.config.ts` that threw while importing produced a generated file with **no plugin error union and no plugin routes** — followed by `voltro codegen: wrote rpcGroup.generated.ts`. The file typechecks, so nothing downstream catches it; the only symptom is a client branching on an error tag that never arrives.
|
|
77
|
+
|
|
78
|
+
A consumer with ~140 declarative `guards:` measured that file 2781 lines shorter after a version bump, with the `ScopeError` import and the whole `__voltroPluginErrors` union gone. For the record, since they were careful to separate measurement from conclusion: the generator did NOT drop the feature — the plugin-codegen path is byte-identical between 0.19.0 and 0.20.0, and the published `@voltro/cli@0.20.0` does contain the identifier they grepped for. Their `grep` came back empty because the bundled chunk contained a literal NUL byte, which makes a file binary to most search tools (fixed separately, and it had been hiding files from our own audits too). What was real is the artefact diff, and this is the path that produces it without a word.
|
|
79
|
+
|
|
80
|
+
Now: a config that EXISTS but fails to load is a refusal with a non-zero exit and the underlying cause, not a quiet downgrade. An app with no `app.config.ts` at all still generates — absence is legitimate, failure is not. And every run prints `(plugins N, error schemas N, plugin routes N)`, because a count that drops from 7 to 0 has to be visible in the success line or the next occurrence is found the same way: by diffing artefacts during a debugging session.
|
|
81
|
+
|
|
82
|
+
`loadApiConfigDiagnosed` is the new seam (`{ config, present, error }`); `loadApiConfig` is unchanged for every existing caller.
|
|
83
|
+
- **@voltro/cli, @voltro/voltro** — `ssr cold-compile` log lines now carry the compile's duration, and `voltro start` emits them at all.
|
|
84
|
+
|
|
85
|
+
The lines had a `start` and an `end` and no timing, which looks readable and is not: cold compiles run concurrently up to `VOLTRO_DEV_SSR_COMPILE_CONCURRENCY`, so the pairs INTERLEAVE. Subtracting adjacent timestamps names the wrong module, and above a limit of two they cannot be paired by eye at all — which is what a user reading a pod log actually hit, with three `start` lines before their `end`s:
|
|
86
|
+
|
|
87
|
+
```
|
|
88
|
+
…:59.704 ssr cold-compile start id=…/layout.tsx
|
|
89
|
+
…:59.704 ssr cold-compile start id=…/(main)/layout.tsx
|
|
90
|
+
…:03.447 ssr cold-compile end 3743ms id=…/layout.tsx
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
The gate had the number for free and threw it away. It is measured INSIDE the concurrency permit, so it is the module's own compile cost rather than the time it spent queued behind the limit — those are different numbers and only one of them is a property of the module. A slow first paint is usually one slow module, and this is the line that names it.
|
|
94
|
+
|
|
95
|
+
A failed compile now says `FAILED` instead of `end`. Without that, a 3.7-second line for a module that threw read exactly like a slow but successful compile.
|
|
96
|
+
|
|
97
|
+
`voltro start`'s middleware fallback constructed the same gate with NO callbacks, so an on-demand compile there produced no line whatsoever; it is wired now.
|
|
98
|
+
- **@voltro/cli, @voltro/voltro** — `voltro db plans`, `db drift` and `db restore-snapshot` worked on postgres only. On mysql/mariadb (and mssql and sqlite) all three died with:
|
|
99
|
+
|
|
100
|
+
```text
|
|
101
|
+
fatal unhandled cli error (FiberFailure) SqlError: Failed to execute statement
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
The cause is three `${sql('col')}::text AS ${sql('col')}` casts — POSTGRES syntax, in read paths whose helper is still called `buildPgLayer`. `db apply`, which WRITES the same ledger table, has no cast and worked, which is exactly the split a consumer reported: the commands that read were broken, the one that writes was fine.
|
|
105
|
+
|
|
106
|
+
The casts existed to stop a driver handing back a `jsonb` object or a `Date`. Normalising in JS gets the same result and cannot be dialect-specific, since drivers differ in whether a json column arrives parsed and whether a timestamp arrives as a `Date`.
|
|
107
|
+
|
|
108
|
+
Worth naming what it cost: `db drift` is the command whose whole job is "alert if live diverged from declared", and the consumer who found this had live divergence at the time. The specific detector and the general one were blind together.
|
|
109
|
+
|
|
110
|
+
**And the error now names the failing statement.** Their verdict was the actionable part of the report:
|
|
111
|
+
|
|
112
|
+
> *"the error names no statement … the statement text (or even the operation name) > would turn this from a dead end into a bug report. We would have sent you the > failing SQL if the error had contained it."*
|
|
113
|
+
|
|
114
|
+
Right twice over — they could not diagnose it, and neither could we from the report; it took reading our own source. `@effect/sql`'s `SqlError` carries the driver error in `cause`, and every supported driver puts the useful part there (mysql2: `code`, `errno`, `sqlState`, `sqlMessage`, usually `sql`; pg: `code`, `detail`, `hint`, `position`). The CLI's fatal reporter printed only the wrapper. It now walks the cause chain and prints the driver message, the codes and the statement — collapsed to one line, and saying `<not attached by the driver>` when there genuinely is none, because that is information too.
|
|
115
|
+
|
|
116
|
+
Shape-based rather than `instanceof SqlError`, deliberately: the CLI catches errors that have crossed the serve/start bundle boundary, where two copies of `@effect/sql` make `instanceof` silently false — the failure mode this repo has already paid for elsewhere.
|
|
117
|
+
- **@voltro/cli, @voltro/voltro** — `voltro doctor`'s `plaintext-secret` rule no longer flags metadata ABOUT a credential. An audit row denormalising the public facts of an api key — `apiKeyId`, `apiKeyKeyId`, `apiKeyType`, `apiKeyOwnerId`, `apiKeyName` — had three columns already excluded by the `*Id` suffix, while `apiKeyType` and `apiKeyName` fired. Telling a team to encrypt the LABEL of a credential is how a rule earns being ignored.
|
|
118
|
+
|
|
119
|
+
The exclusion now covers final words that cannot BE the credential — `Name`, `Type`, `Kind`, `Label`, `Prefix`, `Suffix`, `Status`, `State`, `Scope(s)`, `Version`, `Count`, `Provider`, `Format`, `Note`/`Description`/`Comment`, plus the existing `Id` and the hash family. Deliberately NOT on the list: `Value`, `Secret`, `Token`, `Key`, `Password` — the words that name the thing itself. A false negative from an over-wide list is silent, so that is the failure mode the list is built against, and a test pins the words that must still fire.
|
|
120
|
+
- **@voltro/database, @voltro/cli, @voltro/voltro** — `voltro doctor` no longer contradicts itself about the `.serverOnly()` wire audit. The same command on the same tree reported:
|
|
121
|
+
|
|
122
|
+
```
|
|
123
|
+
human: serverOnly: NOT CHECKED | json: {'checked': True, 'leaks': 0}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Two causes, both fixed. `registerRelations` refused a re-registration of the IDENTICAL relation object, so a process that executes a module twice looked like two conflicting declarations — it now mirrors `registerTable`'s `existing === table` tolerance (a DIFFERENT block claiming the same name still throws). And doctor loaded the app three times per run; it now loads once, so every report sees the same outcome instead of the first one succeeding and the next failing.
|
|
127
|
+
|
|
128
|
+
The consequence was worse than the noise: the throw aborted the wire audit, so the check that a token cannot reach a client had not run since the reporting app adopted the marker — and a CI gate written exactly as we documented (`fail on serverOnly.checked === false`) reported green on an app where the audit provably had not run. That is the "reads as coverage without being coverage" failure the `serverOnly` field was added to remove, reappearing in the field added to prevent it.
|
|
129
|
+
- **@voltro/sql-mysql, @voltro/voltro** — `insertIgnore` on MariaDB no longer reports a cause it cannot know, and no longer turns a REJECTED write into a silent "conflict". `INSERT IGNORE` downgrades EVERY error to a warning — foreign key, NOT NULL, CHECK, truncation — so the post-check's premise ("the insert was skipped ⇒ a unique constraint fired") does not hold on this dialect. It asserted a second unique index that did not exist; the real cause was an FK (an auto-stamped `createdBy` with no matching `actors` row), and a consumer spent the diagnosis looking for a phantom index.
|
|
130
|
+
|
|
131
|
+
The message now reads the real error from `SHOW WARNINGS` on the same connection — BEFORE the existing-row lookup, since that lookup is itself a statement and resets the warning list. A non-duplicate warning is reported as a rejection and throws, because returning there is data loss presented as a normal outcome: the row is not written and the caller is told it already was. A genuine duplicate on an unnamed constraint now names the constraint that fired. Outside a transaction the warning cannot be attributed to our own statement (each statement acquires from the pool independently), so the message says the constraint is unknown rather than guessing — framework mutations are auto-transactional, so the common path has the cause.
|
|
132
|
+
- **@voltro/logger, @voltro/cli, @voltro/database, @voltro/voltro** — `voltro doctor --json` and `voltro capabilities --json` now emit exactly one JSON document on stdout. A `log.warn` from module discovery landed there ahead of it, so:
|
|
133
|
+
|
|
134
|
+
```console
|
|
135
|
+
$ voltro doctor --json 2>/dev/null | python3 -c 'import json,sys; json.load(sys.stdin)'
|
|
136
|
+
JSONDecodeError: Extra data: line 2 column 1
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Note the `2>/dev/null` in that repro — stderr was already redirected, so there was no shell-side workaround. And it only happened when a warning fired, so a consumer's CI parsed the document correctly until one file out of 368 tripped one. That is the same failure the `serverOnly.checked` field was added to remove — an automat unable to separate the normal case from the special case — one layer out, in the surface added to fix it.
|
|
140
|
+
|
|
141
|
+
A command that owns stdout for machine output now calls `claimStdoutForJson()` before doing any work that could log, and every record goes to stderr from then on. The stream decision itself moved into ONE place (`@voltro/logger`'s `stream.ts`, exported as `routeDiagnosticsToStderr`): the Effect surface and the direct surface each carried their own copy of `level === 'error' ? stderr : stdout`, and two copies of one rule is how the rule failed to change.
|
|
142
|
+
|
|
143
|
+
**Also fixed, same report:** the warning that started it was itself wrong. A `*.relations.ts` whose `relations(...)` map is EMPTY was reported as *"no relations(...) export found"* — pointing the reader at a missing export that is right there. `isRelationsSpec` rejects an empty map (correctly — there is nothing to register), but the caller could not tell that apart from a module with no export at all. It now says the map is empty and names the export.
|
|
144
|
+
- **@voltro/cli, @voltro/voltro** — The SSR bundle build now externalises a bare specifier it cannot resolve instead of aborting, so an uninstalled OPTIONAL peer no longer makes `voltro build` impossible.
|
|
145
|
+
|
|
146
|
+
The SSR step runs with `ssr: { noExternal: true }` — inlining everything is what lets a production web image ship without a framework dependency tree — and that left no escape for a package that cannot be resolved at all. The commonest such package is an optional native peer reached through a library's Node entry:
|
|
147
|
+
|
|
148
|
+
```
|
|
149
|
+
Rolldown failed to resolve import "canvas"
|
|
150
|
+
from ".../konva/lib/index-node.js"
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
`konva`'s `main` is its Node build, which requires the optional native `canvas`; its `browser` field points at one that does not. An app that never renders to a canvas server-side has nothing to install.
|
|
154
|
+
|
|
155
|
+
A consumer measured that there was no way out from their side either, and each measurement is worth keeping: the import was ALREADY dynamic (rolldown must still resolve it to form the chunk), `renderMode: 'spa'` does not help (`.framework/app.tsx` imports every page statically for the router, so the module is in the SSR graph whatever the render mode), and an `ssr.external` passthrough in `app.config.ts` is not read. So `voltro build` — and with it the production image — was unavailable for that app.
|
|
156
|
+
|
|
157
|
+
The api serve bundle and the web start bundle already did exactly this; that plugin is esbuild's and this step is vite/rolldown, so it is the same probe behind a different interface. Framework packages (`@voltro/*`, `@effect/*`, `effect`) are never externalised, so the "needs nothing from node_modules" property still holds.
|
|
158
|
+
|
|
159
|
+
Every externalised specifier is NAMED in the `SSR bundle ready` line. Externalising is right for an uninstalled optional peer and wrong for a genuine missing dependency — it trades a loud build failure for a quiet runtime one — and only the reader can tell which, so it is reported rather than swallowed.
|
|
160
|
+
- **@voltro/sql-mysql, @voltro/sql-sqlite, @voltro/sql-mssql, @voltro/sql-postgres, @voltro/database, @voltro/voltro** — A typed error thrown inside a mutation now reaches the client TYPED, on every dialect. It arrived as an untagged `Die` defect on mysql/mariadb, sqlite and mssql: `transactional()` settled its program with `runPromise`, which rejects with Effect's `FiberFailure` wrapper, and the wrapper copies `message` and a decorated `name` but nothing else — no `_tag`, no payload, no prototype. So the rpc encoder could not match the failure against the mutation descriptor's `error:` union:
|
|
161
|
+
|
|
162
|
+
```
|
|
163
|
+
└─ ["error"] └─ ["_tag"] └─ is missing
|
|
164
|
+
Expected never, actual (FiberFailure) NotFoundError: …
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
Framework mutations are auto-transactional, so this was EVERY typed mutation error in an app. Nothing failed — `defineMutation({ error: … })` compiled, the client's type still said `NotFoundError`, and the `error._tag === 'NotFoundError'` branch was simply never taken at runtime. Actions, which are not auto-transactional, marshalled correctly the whole time, which is what made the transaction the discriminator. A hand-rolled error class lost its fields and its `instanceof` too; only `message` survived, which is why a workaround built on `error.message` looked like it worked and hid this.
|
|
168
|
+
|
|
169
|
+
Postgres already had the unwrap, with a comment describing this exact consequence, and the three sibling dialects kept the broken call — so the fix is now one shared `settleTransactionExit` in `@voltro/database` that all four import, plus a parity test that fails if any store's `transactional()` reaches `runtime.runPromise` again. Reported by a consumer on MariaDB who verified it against 0.19.0 too, so it is not a 0.20.0 regression.
|
|
170
|
+
|
|
171
|
+
### Internal (no consumer-facing effect)
|
|
172
|
+
|
|
173
|
+
- **@voltro/runtime, @voltro/database, @voltro/protocol, @voltro/cli, @voltro/plugin-billing, @voltro/plugin-mail, @voltro/plugin-sso-saml, @voltro/plugin-storage** — Fourteen source files carried a LITERAL NUL byte — the house idiom for a composite map key, written as the raw character instead of an escape. That makes the file BINARY to every text tool: `grep` skips it entirely and reports nothing, which is indistinguishable from a clean file. It was found because a new guard test scanning for framework index names came back clean on `runtime/src/connectionVault.ts` — 1020 lines that every previous grep-based audit in this repo had also silently skipped, including the one looking for exactly the index name that file declares.
|
|
174
|
+
|
|
175
|
+
Replaced with the JavaScript escape for U+0000. Identical runtime value, files are text again. No behaviour change.
|
|
176
|
+
|
|
177
|
+
---
|
|
178
|
+
|
|
42
179
|
## [0.20.0] — 2026-07-29
|
|
43
180
|
|
|
44
181
|
### ⚠ BREAKING
|
package/dist/index.d.ts
CHANGED
|
@@ -59,6 +59,18 @@ export declare interface BinlogCdcOptions {
|
|
|
59
59
|
* per-subscribe re-query already does this on the next change.
|
|
60
60
|
*/
|
|
61
61
|
readonly onResync?: () => void;
|
|
62
|
+
/**
|
|
63
|
+
* Names of tables whose binlog row image carries MORE columns than
|
|
64
|
+
* `information_schema.COLUMNS` reports, so the reader can NEVER decode them.
|
|
65
|
+
*
|
|
66
|
+
* Supplied by the store from a privilege-free probe (see
|
|
67
|
+
* `hashLongUniqueTablesQuery`). The reader subtracts them from `includeSchema`,
|
|
68
|
+
* which is what makes the condition converge: MEASURED against live MariaDB
|
|
69
|
+
* 11.8, an EXCLUDED table with a hidden hash column produces no error at all,
|
|
70
|
+
* while the same table included throws
|
|
71
|
+
* `the event has 3 columns, fetched metadata has 2` on the first write.
|
|
72
|
+
*/
|
|
73
|
+
readonly undecodableTables?: ReadonlyArray<string>;
|
|
62
74
|
/**
|
|
63
75
|
* Watchdog liveness cadence + stall threshold (ms). The reader tracks its
|
|
64
76
|
* last progress (any binlog event or a fresh attach); when nothing has
|
|
@@ -300,6 +312,38 @@ export declare class MysqlStore implements DataStore {
|
|
|
300
312
|
*/
|
|
301
313
|
private executeMariadbUpsert;
|
|
302
314
|
private executeInsertIgnore;
|
|
315
|
+
/**
|
|
316
|
+
* The warnings MariaDB raised for the statement that just ran on `txn`.
|
|
317
|
+
*
|
|
318
|
+
* Only meaningful inside a transaction, and that is why the parameter is not
|
|
319
|
+
* optional. `SHOW WARNINGS` reports the last statement on the CONNECTION, and
|
|
320
|
+
* outside a transaction each statement acquires from the pool independently —
|
|
321
|
+
* so a warning read there could belong to an unrelated statement on another
|
|
322
|
+
* caller's connection. Reporting that as this insert's cause would be a worse
|
|
323
|
+
* version of the bug this exists to fix, so `insertIgnore` passes `null` and
|
|
324
|
+
* says "unknown" rather than reading something it cannot attribute.
|
|
325
|
+
*
|
|
326
|
+
* Diagnostic-only: a failure to read warnings must never replace the caller's
|
|
327
|
+
* real problem with a problem about reading warnings.
|
|
328
|
+
*/
|
|
329
|
+
/**
|
|
330
|
+
* Tables among `includeTables` whose binlog row image carries a HIDDEN column,
|
|
331
|
+
* so the reader could never decode them.
|
|
332
|
+
*
|
|
333
|
+
* Privilege-free by construction: the direct evidence lives in
|
|
334
|
+
* `INNODB_SYS_COLUMNS`, which needs `PROCESS` — a grant an app DB user does not
|
|
335
|
+
* have (verified: `Access denied; you need … the PROCESS privilege(s)`). This
|
|
336
|
+
* infers the same thing from the CONSTRAINT shape, which any user can read: a
|
|
337
|
+
* UNIQUE index with no prefix length over a text/blob column can only be a
|
|
338
|
+
* MariaDB hash long-unique, and that is what adds the hidden column.
|
|
339
|
+
*
|
|
340
|
+
* Diagnostic-only — a probe that fails must never stop CDC from starting. An
|
|
341
|
+
* undiscovered table still gets caught by the reconnect path's repetition check;
|
|
342
|
+
* this just moves the diagnosis to boot, where it is one message instead of a
|
|
343
|
+
* loop.
|
|
344
|
+
*/
|
|
345
|
+
private findUndecodableCdcTables;
|
|
346
|
+
private readWarnings;
|
|
303
347
|
private findByConflict;
|
|
304
348
|
query(d: QueryDescriptor): Promise<readonly Readonly<Record<string, unknown>>[]>;
|
|
305
349
|
raw<T extends object = Row>(fragment: RawSqlFragment, _opts?: {
|
package/dist/index.js
CHANGED
|
@@ -3,16 +3,16 @@ import { Config as t, Effect as n, Layer as r, ManagedRuntime as i, Option as a,
|
|
|
3
3
|
import { EventEmitter as c } from "node:events";
|
|
4
4
|
import { createLogger as l } from "@voltro/logger";
|
|
5
5
|
import { SqlClient as u, TransactionConnection as d } from "@effect/sql/SqlClient";
|
|
6
|
-
import { CDC_OFFSETS_TABLE as f, EagerCardinalityError as p, _voltroCdcOffsetsTable as m, attachEagerLoads as h, attributionFields as g, attributionKey as _, claimPendingAttribution as v, compileEagerJson as y, compilePredicate as b, compileRawFragment as x, compileSelect as S, currentWriteAttribution as C, decodeRowsFromSchema as w, encodeRowForSchema as T, hasEagerLoads as E, isTableReactive as D, qualifyTable as
|
|
6
|
+
import { CDC_OFFSETS_TABLE as f, EagerCardinalityError as p, _voltroCdcOffsetsTable as m, attachEagerLoads as h, attributionFields as g, attributionKey as _, claimPendingAttribution as v, compileEagerJson as y, compilePredicate as b, compileRawFragment as x, compileSelect as S, currentWriteAttribution as C, decodeRowsFromSchema as w, encodeRowForSchema as T, hasEagerLoads as E, isTableReactive as D, qualifyTable as ee, raiseChangeListenerCeiling as O, recordsTable as te, registerPendingAttribution as ne, requireTable as re, runWithWriteAttribution as k, runWriteRecorders as A, settleTransactionExit as j, stampGeneratedId as M, stampGeneratedIds as N } from "@voltro/database";
|
|
7
7
|
//#region src/sqlLayer.ts
|
|
8
|
-
var
|
|
8
|
+
var P = (n) => e.layerConfig({
|
|
9
9
|
host: t.succeed(n.host),
|
|
10
10
|
port: t.succeed(n.port),
|
|
11
11
|
username: t.succeed(n.username),
|
|
12
12
|
password: t.succeed(o.make(n.password)),
|
|
13
13
|
database: t.succeed(n.database),
|
|
14
14
|
...n.maxConnections === void 0 ? {} : { maxConnections: t.succeed(n.maxConnections) }
|
|
15
|
-
}),
|
|
15
|
+
}), F = (e) => {
|
|
16
16
|
if (e.url) {
|
|
17
17
|
let t = new URL(e.url);
|
|
18
18
|
return {
|
|
@@ -32,63 +32,72 @@ var F = (n) => e.layerConfig({
|
|
|
32
32
|
database: e.database ?? "app",
|
|
33
33
|
...e.maxConnections === void 0 ? {} : { maxConnections: e.maxConnections }
|
|
34
34
|
};
|
|
35
|
-
},
|
|
35
|
+
}, I = (e) => P(F(e)), ie = (e, t) => e === null ? !1 : t === null ? !0 : e.filename === t.filename ? e.position > t.position : e.filename > t.filename, ae = (e) => e.msSinceProgress < e.stallThresholdMs ? "healthy" : e.primary !== null && !ie(e.primary, e.reader) ? "idle-caught-up" : "reconnect", L = (e) => (e instanceof Error ? e.message : String(e ?? "")).includes("schema changed between binlog event and metadata fetch"), R = (e) => {
|
|
36
|
+
let t = e instanceof Error ? e.message : String(e ?? "");
|
|
37
|
+
return /Table\s+[^\s.]+\.(\S+)\s+schema changed between binlog event and metadata fetch/.exec(t)?.[1] ?? null;
|
|
38
|
+
}, z = "\n SELECT DISTINCT s.TABLE_NAME AS tableName\n FROM information_schema.STATISTICS s\n JOIN information_schema.COLUMNS c\n ON c.TABLE_SCHEMA = s.TABLE_SCHEMA\n AND c.TABLE_NAME = s.TABLE_NAME\n AND c.COLUMN_NAME = s.COLUMN_NAME\n WHERE s.TABLE_SCHEMA = DATABASE()\n AND s.NON_UNIQUE = 0\n AND s.SUB_PART IS NULL\n AND c.DATA_TYPE IN ('text','tinytext','mediumtext','longtext','blob','tinyblob','mediumblob','longblob')\n", B = (e) => `cdc: table '${e}' is EXCLUDED from binlog capture — its row image carries a hidden column the reader cannot account for. Cause: a UNIQUE constraint on an UNBOUNDED text column, which MariaDB backs with a HASH long-unique index; that index adds a hidden DB_ROW_HASH_n column to the row, present in the binlog and absent from information_schema.COLUMNS. Remedy: bound the column — text().maxLength(n) — so the constraint becomes an ordinary B-tree index with no hidden column. ALTER TABLE FORCE does NOT help: the rebuild recreates the index and the hidden column. Until then, cross-instance change events for this table are lost; own-node reactivity is unaffected (writes still emit inline).`, V = 5 * 6e4, H = 3, U = (e, t) => {
|
|
39
|
+
let n = [...e.filter((e) => t - e < V), t];
|
|
40
|
+
return {
|
|
41
|
+
verdict: n.length >= H ? "persistent" : "backlog",
|
|
42
|
+
hits: n
|
|
43
|
+
};
|
|
44
|
+
}, W = /* @__PURE__ */ new Set([
|
|
36
45
|
"writerows",
|
|
37
46
|
"updaterows",
|
|
38
47
|
"deleterows"
|
|
39
|
-
]),
|
|
48
|
+
]), G = /\b(alter|rename|drop|create)\s+(table|column)?/i, K = (e) => new Promise((t) => setTimeout(t, e)), q = async (e) => {
|
|
40
49
|
let t = l({ scope: `voltro:${e.variant}:cdc` }), n;
|
|
41
50
|
try {
|
|
42
51
|
n = (await import("@vlasky/zongji")).default;
|
|
43
52
|
} catch (t) {
|
|
44
53
|
throw Error(`changeStrategy='cdc' on ${e.variant} requires the '@vlasky/zongji' optional dependency. Install it (it ships as an optionalDependency of @voltro/sql-mysql) to enable binlog CDC.`, { cause: t });
|
|
45
54
|
}
|
|
46
|
-
let r = e.includeTables ? new Set(e.includeTables) : null,
|
|
47
|
-
|
|
48
|
-
},
|
|
49
|
-
|
|
50
|
-
let
|
|
51
|
-
if (
|
|
52
|
-
|
|
55
|
+
let r = new Set(e.undecodableTables ?? []), i = e.includeTables ? new Set(e.includeTables.filter((e) => !r.has(e))) : null, a = e.connection.database, o = e.startPosition?.filename ?? null, s = e.startPosition ?? null, c = (t, n) => w([n], t, e.variant)[0], u = null, d = !1, f = 0, p = Date.now(), m = !1, h = /* @__PURE__ */ new Map(), g = /* @__PURE__ */ new Set(), _ = null, v = e.stallThresholdMs ?? 25e3, y = e.watchdogIntervalMs ?? 15e3, b = () => {
|
|
56
|
+
p = Date.now();
|
|
57
|
+
}, x = (n) => {
|
|
58
|
+
b();
|
|
59
|
+
let r = n.getEventName();
|
|
60
|
+
if (r === "rotate" && n.binlogName) {
|
|
61
|
+
o = n.binlogName;
|
|
53
62
|
return;
|
|
54
63
|
}
|
|
55
|
-
if (
|
|
56
|
-
|
|
64
|
+
if (r === "query" && n.query && G.test(n.query)) {
|
|
65
|
+
u && (u.tableMap = {});
|
|
57
66
|
return;
|
|
58
67
|
}
|
|
59
|
-
if (n.nextPosition &&
|
|
60
|
-
filename:
|
|
68
|
+
if (n.nextPosition && o && (s = {
|
|
69
|
+
filename: o,
|
|
61
70
|
position: n.nextPosition
|
|
62
|
-
}, e.onPosition?.(
|
|
63
|
-
let
|
|
64
|
-
if (!
|
|
65
|
-
let d =
|
|
66
|
-
if (!d.startsWith("_voltro_") && !(
|
|
67
|
-
if (
|
|
71
|
+
}, e.onPosition?.(s)), !W.has(r)) return;
|
|
72
|
+
let l = n.tableMap[n.tableId];
|
|
73
|
+
if (!l || l.parentSchema !== a) return;
|
|
74
|
+
let d = l.tableName;
|
|
75
|
+
if (!d.startsWith("_voltro_") && !(i && !i.has(d))) try {
|
|
76
|
+
if (r === "writerows") for (let t of n.rows) e.onChange({
|
|
68
77
|
table: d,
|
|
69
78
|
op: "insert",
|
|
70
79
|
old: null,
|
|
71
|
-
new:
|
|
80
|
+
new: c(d, t)
|
|
72
81
|
});
|
|
73
|
-
else if (
|
|
82
|
+
else if (r === "deleterows") for (let t of n.rows) e.onChange({
|
|
74
83
|
table: d,
|
|
75
84
|
op: "delete",
|
|
76
|
-
old:
|
|
85
|
+
old: c(d, t),
|
|
77
86
|
new: null
|
|
78
87
|
});
|
|
79
88
|
else for (let t of n.rows) e.onChange({
|
|
80
89
|
table: d,
|
|
81
90
|
op: "update",
|
|
82
|
-
old:
|
|
83
|
-
new:
|
|
91
|
+
old: c(d, t.before),
|
|
92
|
+
new: c(d, t.after)
|
|
84
93
|
});
|
|
85
94
|
} catch (n) {
|
|
86
95
|
t.warn("cdc: failed to map a row event", {
|
|
87
96
|
table: d,
|
|
88
|
-
name:
|
|
97
|
+
name: r
|
|
89
98
|
}, n), e.onError?.(n);
|
|
90
99
|
}
|
|
91
|
-
},
|
|
100
|
+
}, S = (t) => {
|
|
92
101
|
let n = {
|
|
93
102
|
serverId: e.serverId,
|
|
94
103
|
includeEvents: [
|
|
@@ -101,11 +110,11 @@ var F = (n) => e.layerConfig({
|
|
|
101
110
|
"query"
|
|
102
111
|
]
|
|
103
112
|
};
|
|
104
|
-
return e.includeTables && e.includeTables.length > 0 && (n.includeSchema = { [
|
|
105
|
-
},
|
|
113
|
+
return e.includeTables && e.includeTables.length > 0 && (n.includeSchema = { [a]: e.includeTables.filter((e) => !r.has(e)) }), t ? (n.filename = t.filename, n.position = t.position) : n.startAtEnd = !0, n;
|
|
114
|
+
}, C = (e) => {
|
|
106
115
|
let t = e;
|
|
107
116
|
return t?.errno === 1236 || String(t?.code ?? "").includes("ER_MASTER_FATAL_ERROR_READING_BINLOG");
|
|
108
|
-
},
|
|
117
|
+
}, T = (r) => new Promise((i, a) => {
|
|
109
118
|
let o = new n({
|
|
110
119
|
host: e.connection.host,
|
|
111
120
|
port: e.connection.port,
|
|
@@ -114,10 +123,10 @@ var F = (n) => e.layerConfig({
|
|
|
114
123
|
enableKeepAlive: !0,
|
|
115
124
|
keepAliveInitialDelay: e.keepAliveInitialDelayMs ?? 1e4
|
|
116
125
|
});
|
|
117
|
-
|
|
126
|
+
u = o;
|
|
118
127
|
let s = !1;
|
|
119
|
-
o.on("binlog",
|
|
120
|
-
|
|
128
|
+
o.on("binlog", x), o.on("ready", () => {
|
|
129
|
+
b(), !s && (s = !0, f = 0, t.info("cdc: binlog reader attached", {
|
|
121
130
|
serverId: e.serverId,
|
|
122
131
|
from: r ?? "current-end"
|
|
123
132
|
}), i());
|
|
@@ -126,78 +135,82 @@ var F = (n) => e.layerConfig({
|
|
|
126
135
|
s = !0, a(e instanceof Error ? e : Error(String(e)));
|
|
127
136
|
return;
|
|
128
137
|
}
|
|
129
|
-
|
|
130
|
-
}), o.start(
|
|
131
|
-
}),
|
|
132
|
-
if (!(
|
|
133
|
-
|
|
138
|
+
E("binlog reader error", e);
|
|
139
|
+
}), o.start(S(r));
|
|
140
|
+
}), E = async (n, i) => {
|
|
141
|
+
if (!(d || m)) {
|
|
142
|
+
m = !0, t.warn(`cdc: reconnecting — ${n}`, {}, i instanceof Error ? i : void 0), e.onError?.(i);
|
|
134
143
|
try {
|
|
135
|
-
let n =
|
|
136
|
-
for (; !
|
|
144
|
+
let n = i;
|
|
145
|
+
for (; !d;) {
|
|
137
146
|
try {
|
|
138
|
-
|
|
147
|
+
u?.stop();
|
|
139
148
|
} catch {}
|
|
140
|
-
if (
|
|
141
|
-
let
|
|
142
|
-
|
|
149
|
+
if (f++, await K(Math.min(3e4, 500 * 2 ** Math.min(f, 6))), d) return;
|
|
150
|
+
let i = s, a = C(n), c = !a && L(n), l = !1;
|
|
151
|
+
if (c) {
|
|
152
|
+
let e = R(n), i = e ?? "<unknown>", { verdict: a, hits: o } = U(h.get(i) ?? [], Date.now());
|
|
153
|
+
h.set(i, o), l = a === "persistent", l && !g.has(i) && (g.add(i), e !== null && r.add(e), t.error(B(i)));
|
|
154
|
+
}
|
|
155
|
+
(a || c) && (l || t.warn(c ? "cdc: un-replayable backlog event (schema moved past it) — jumping to current end + self-heal" : "cdc: binlog gap (purged/failover) — jumping to current end + self-heal"), i = e.resolveStartPosition ? await e.resolveStartPosition().catch(() => null) : null, o = i?.filename ?? null, l || e.onResync?.());
|
|
143
156
|
try {
|
|
144
|
-
await
|
|
157
|
+
await T(i), f = 0, b();
|
|
145
158
|
return;
|
|
146
159
|
} catch (e) {
|
|
147
160
|
n = e;
|
|
148
161
|
}
|
|
149
162
|
}
|
|
150
163
|
} finally {
|
|
151
|
-
|
|
164
|
+
m = !1;
|
|
152
165
|
}
|
|
153
166
|
}
|
|
154
167
|
};
|
|
155
168
|
try {
|
|
156
|
-
await
|
|
169
|
+
await T(e.startPosition ?? null);
|
|
157
170
|
} catch (n) {
|
|
158
|
-
if (
|
|
171
|
+
if (C(n) && e.startPosition) {
|
|
159
172
|
t.warn("cdc: persisted offset purged — starting at current end + self-heal");
|
|
160
173
|
try {
|
|
161
|
-
|
|
174
|
+
u?.stop();
|
|
162
175
|
} catch {}
|
|
163
176
|
let n = e.resolveStartPosition ? await e.resolveStartPosition().catch(() => null) : null;
|
|
164
|
-
|
|
177
|
+
o = n?.filename ?? null, e.onResync?.(), await K(500), await T(n);
|
|
165
178
|
} else throw n;
|
|
166
179
|
}
|
|
167
|
-
let
|
|
168
|
-
if (
|
|
180
|
+
let D = async () => {
|
|
181
|
+
if (d || m || Date.now() - p < v) return;
|
|
169
182
|
let n = null;
|
|
170
|
-
if (e.resolveStartPosition && (n = await e.resolveStartPosition().catch(() => null)),
|
|
171
|
-
let r =
|
|
172
|
-
msSinceProgress: Date.now() -
|
|
173
|
-
stallThresholdMs:
|
|
183
|
+
if (e.resolveStartPosition && (n = await e.resolveStartPosition().catch(() => null)), d || m) return;
|
|
184
|
+
let r = ae({
|
|
185
|
+
msSinceProgress: Date.now() - p,
|
|
186
|
+
stallThresholdMs: v,
|
|
174
187
|
primary: n,
|
|
175
|
-
reader:
|
|
188
|
+
reader: s
|
|
176
189
|
});
|
|
177
190
|
if (r !== "reconnect") {
|
|
178
|
-
r === "idle-caught-up" &&
|
|
191
|
+
r === "idle-caught-up" && b();
|
|
179
192
|
return;
|
|
180
193
|
}
|
|
181
194
|
t.warn("cdc: watchdog detected a stalled reader — forcing reconnect", {
|
|
182
|
-
reader:
|
|
195
|
+
reader: s,
|
|
183
196
|
primary: n,
|
|
184
|
-
stalledForMs: Date.now() -
|
|
185
|
-
}),
|
|
197
|
+
stalledForMs: Date.now() - p
|
|
198
|
+
}), E("watchdog: stream stalled (no events while primary advanced)", /* @__PURE__ */ Error("cdc watchdog stall"));
|
|
186
199
|
};
|
|
187
|
-
return
|
|
188
|
-
|
|
189
|
-
},
|
|
200
|
+
return _ = setInterval(() => {
|
|
201
|
+
D();
|
|
202
|
+
}, y), _.unref && _.unref(), {
|
|
190
203
|
stop: async () => {
|
|
191
|
-
|
|
204
|
+
d = !0, _ &&= (clearInterval(_), null);
|
|
192
205
|
try {
|
|
193
|
-
|
|
206
|
+
u?.stop();
|
|
194
207
|
} catch (e) {
|
|
195
208
|
t.warn("cdc: error stopping binlog reader", {}, e);
|
|
196
209
|
}
|
|
197
210
|
},
|
|
198
|
-
currentPosition: () =>
|
|
211
|
+
currentPosition: () => s
|
|
199
212
|
};
|
|
200
|
-
},
|
|
213
|
+
}, oe = /* @__PURE__ */ new Set(["1213", "1205"]), se = (e) => {
|
|
201
214
|
let t = e;
|
|
202
215
|
for (let e = 0; e < 5 && typeof t == "object" && t; e++) {
|
|
203
216
|
let e = t.errno;
|
|
@@ -206,31 +219,31 @@ var F = (n) => e.layerConfig({
|
|
|
206
219
|
if (typeof n == "string") return n;
|
|
207
220
|
t = t.cause;
|
|
208
221
|
}
|
|
209
|
-
},
|
|
210
|
-
let t =
|
|
211
|
-
return t !== void 0 &&
|
|
212
|
-
},
|
|
222
|
+
}, J = (e) => {
|
|
223
|
+
let t = se(e);
|
|
224
|
+
return t !== void 0 && oe.has(t);
|
|
225
|
+
}, Y = (e) => J(e) ? "retry" : "noRetry", X = (e) => {
|
|
213
226
|
if (e == null) return "null";
|
|
214
227
|
let t = typeof e;
|
|
215
228
|
if (t === "bigint") return `${e}n`;
|
|
216
229
|
if (t !== "object") return JSON.stringify(e);
|
|
217
230
|
if (e instanceof Date) return `"${e.toISOString()}"`;
|
|
218
|
-
if (Array.isArray(e)) return `[${e.map(
|
|
231
|
+
if (Array.isArray(e)) return `[${e.map(X).join(",")}]`;
|
|
219
232
|
let n = e;
|
|
220
|
-
return `{${Object.keys(n).sort().map((e) => `${JSON.stringify(e)}:${
|
|
221
|
-
},
|
|
222
|
-
let t =
|
|
233
|
+
return `{${Object.keys(n).sort().map((e) => `${JSON.stringify(e)}:${X(n[e])}`).join(",")}}`;
|
|
234
|
+
}, ce = (e) => {
|
|
235
|
+
let t = X(e), n = 2166136261;
|
|
223
236
|
for (let e = 0; e < t.length; e++) n ^= t.charCodeAt(e), n = Math.imul(n, 16777619);
|
|
224
237
|
return (n >>> 0).toString(36);
|
|
225
|
-
},
|
|
238
|
+
}, le = (e, t) => {
|
|
226
239
|
let n = setTimeout(e, t);
|
|
227
240
|
typeof n.unref == "function" && n.unref();
|
|
228
|
-
},
|
|
241
|
+
}, ue = class {
|
|
229
242
|
variant;
|
|
230
243
|
ttlMs;
|
|
231
244
|
schedule;
|
|
232
245
|
seen = /* @__PURE__ */ new Map();
|
|
233
|
-
constructor(e, t = 6e4, n =
|
|
246
|
+
constructor(e, t = 6e4, n = le) {
|
|
234
247
|
this.variant = e, this.ttlMs = t, this.schedule = n;
|
|
235
248
|
}
|
|
236
249
|
key(e) {
|
|
@@ -244,7 +257,7 @@ var F = (n) => e.layerConfig({
|
|
|
244
257
|
} catch {
|
|
245
258
|
r = t;
|
|
246
259
|
}
|
|
247
|
-
return `${e.table} ${e.op} ${String(n)} ${
|
|
260
|
+
return `${e.table} ${e.op} ${String(n)} ${ce(r)}`;
|
|
248
261
|
}
|
|
249
262
|
admit(e) {
|
|
250
263
|
let t = this.key(e);
|
|
@@ -259,15 +272,19 @@ var F = (n) => e.layerConfig({
|
|
|
259
272
|
get pending() {
|
|
260
273
|
return this.seen.size;
|
|
261
274
|
}
|
|
262
|
-
},
|
|
275
|
+
}, de = /* @__PURE__ */ new Set([
|
|
276
|
+
1022,
|
|
277
|
+
1062,
|
|
278
|
+
1586
|
|
279
|
+
]), Z = async (e) => {
|
|
263
280
|
let t = e.variant ?? "mysql", n = l({ scope: `voltro:${t}` }), a = e.changeStrategy ?? "inline", o = a;
|
|
264
281
|
a === "cdc" && !e.cdcConfig && (n.warn("changeStrategy='cdc' requires cdcConfig (serverId + connection); falling back to 'inline'."), o = "inline");
|
|
265
|
-
let s = e.tracerLayer ? r.mergeAll(e.sqlLayer, e.tracerLayer) : e.sqlLayer, c = i.make(s), d = new
|
|
282
|
+
let s = e.tracerLayer ? r.mergeAll(e.sqlLayer, e.tracerLayer) : e.sqlLayer, c = i.make(s), d = new fe(await c.runPromise(u), c, t, o);
|
|
266
283
|
return o === "cdc" && e.cdcConfig && await d.startCdcConsumer(e.cdcConfig), d;
|
|
267
284
|
}, Q = () => {
|
|
268
285
|
let e = C();
|
|
269
|
-
return e === void 0 ? (e) => e() : (t) =>
|
|
270
|
-
},
|
|
286
|
+
return e === void 0 ? (e) => e() : (t) => k(e, t);
|
|
287
|
+
}, fe = class e {
|
|
271
288
|
sql;
|
|
272
289
|
runtime;
|
|
273
290
|
variant;
|
|
@@ -283,13 +300,13 @@ var F = (n) => e.layerConfig({
|
|
|
283
300
|
cdcStreamName = "default";
|
|
284
301
|
cdcGate;
|
|
285
302
|
constructor(e, t, n, r = "inline", i = null, a, o) {
|
|
286
|
-
this.sql = e, this.runtime = t, this.variant = n, this.changeStrategy = r, this.namespace = i, this.log = l({ scope: `voltro:${n}` }), this.emitter = a ?? new c(),
|
|
303
|
+
this.sql = e, this.runtime = t, this.variant = n, this.changeStrategy = r, this.namespace = i, this.log = l({ scope: `voltro:${n}` }), this.emitter = a ?? new c(), O(this.emitter), this.cdcGate = o ?? new ue(n);
|
|
287
304
|
}
|
|
288
305
|
withNamespace(t) {
|
|
289
306
|
return t === this.namespace ? this : new e(this.sql, this.runtime, this.variant, this.changeStrategy, t, this.emitter, this.cdcGate);
|
|
290
307
|
}
|
|
291
308
|
nsT(e) {
|
|
292
|
-
return
|
|
309
|
+
return ee(this.namespace, e);
|
|
293
310
|
}
|
|
294
311
|
get dialectId() {
|
|
295
312
|
return this.variant;
|
|
@@ -314,7 +331,7 @@ var F = (n) => e.layerConfig({
|
|
|
314
331
|
return !1;
|
|
315
332
|
}
|
|
316
333
|
async executeInsert(e, t, r, i) {
|
|
317
|
-
t =
|
|
334
|
+
t = M(e, t);
|
|
318
335
|
let a = this.sql;
|
|
319
336
|
if (this.supportsInsertReturning) {
|
|
320
337
|
let o = a`INSERT INTO ${a(this.nsT(e))} ${a.insert(T(t, e))} RETURNING *`, s = r ? n.provideService(o, d, r) : o, c = (await this.runtime.runPromise(s))[0];
|
|
@@ -360,7 +377,7 @@ var F = (n) => e.layerConfig({
|
|
|
360
377
|
if (e) return this.runtime.runPromise(t(e));
|
|
361
378
|
this.inflightTxns++;
|
|
362
379
|
try {
|
|
363
|
-
let e = n.suspend(() => this.sql.withTransaction(n.flatMap(n.serviceOption(d), (e) => a.isNone(e) ? n.fail(/* @__PURE__ */ Error("MysqlStore.insert: TransactionConnection missing.")) : t(e.value)))), i = s.exponential("10 millis").pipe(s.compose(s.recurs(3)), s.whileInput(
|
|
380
|
+
let e = n.suspend(() => this.sql.withTransaction(n.flatMap(n.serviceOption(d), (e) => a.isNone(e) ? n.fail(/* @__PURE__ */ Error("MysqlStore.insert: TransactionConnection missing.")) : t(e.value)))), i = s.exponential("10 millis").pipe(s.compose(s.recurs(3)), s.whileInput(J)), o = e.pipe(n.retry(i), n.withSpan("store.insert", { attributes: {
|
|
364
381
|
"db.system": this.variant,
|
|
365
382
|
"db.operation": r
|
|
366
383
|
} }));
|
|
@@ -370,7 +387,7 @@ var F = (n) => e.layerConfig({
|
|
|
370
387
|
}
|
|
371
388
|
}
|
|
372
389
|
async executeInsertMany(e, t, r, i) {
|
|
373
|
-
if (t =
|
|
390
|
+
if (t = N(e, t), t.length === 0) return [];
|
|
374
391
|
let a = this.sql, o = t.map((t) => T(t, e));
|
|
375
392
|
if (this.supportsInsertReturning) {
|
|
376
393
|
let t = a`INSERT INTO ${a(this.nsT(e))} ${a.insert(o)} RETURNING *`, s = r ? n.provideService(t, d, r) : t, c = await this.runtime.runPromise(s);
|
|
@@ -481,7 +498,7 @@ var F = (n) => e.layerConfig({
|
|
|
481
498
|
if (e = {
|
|
482
499
|
...g(),
|
|
483
500
|
...e
|
|
484
|
-
},
|
|
501
|
+
}, te(e.table) && await A({
|
|
485
502
|
append: (e, t) => this.appendInTxn(e, t, n),
|
|
486
503
|
maxOf: (e, t, r) => this.maxInTxn(e, t, r, n)
|
|
487
504
|
}, {
|
|
@@ -493,7 +510,7 @@ var F = (n) => e.layerConfig({
|
|
|
493
510
|
subjectId: e.subjectId
|
|
494
511
|
}), this.changeStrategy === "cdc") {
|
|
495
512
|
let t = (e.op === "delete" ? e.old : e.new)?.id;
|
|
496
|
-
t != null &&
|
|
513
|
+
t != null && ne(_(e.table, e.op, t), {
|
|
497
514
|
...e.traceId === void 0 ? {} : { traceId: e.traceId },
|
|
498
515
|
...e.subjectId === void 0 ? {} : { subjectId: e.subjectId }
|
|
499
516
|
});
|
|
@@ -538,7 +555,7 @@ var F = (n) => e.layerConfig({
|
|
|
538
555
|
}, a, i), m;
|
|
539
556
|
}
|
|
540
557
|
async executeInsertIgnore(e, t, r, i, a) {
|
|
541
|
-
if (t =
|
|
558
|
+
if (t = M(e, t), this.variant === "mariadb") {
|
|
542
559
|
let o = this.sql, s = o`INSERT IGNORE INTO ${o(this.nsT(e))} ${o.insert(T(t, e))} RETURNING *`, c = i ? n.provideService(s, d, i) : s, l = (await this.runtime.runPromise(c))[0];
|
|
543
560
|
if (l) return await this.routeEvent({
|
|
544
561
|
table: e,
|
|
@@ -546,12 +563,36 @@ var F = (n) => e.layerConfig({
|
|
|
546
563
|
old: null,
|
|
547
564
|
new: l
|
|
548
565
|
}, a, i), l;
|
|
549
|
-
let u = await this.
|
|
550
|
-
if (
|
|
551
|
-
|
|
566
|
+
let u = await this.readWarnings(i), f = u.find((e) => !de.has(e.code));
|
|
567
|
+
if (f !== void 0) throw Error(`MysqlStore.insertIgnore: the insert into '${e}' was REJECTED, not skipped as a conflict. INSERT IGNORE downgrades every error to a warning, and the warning was: [${f.code}] ${f.message}. Nothing was written and nothing conflicted — fix the cause above.`);
|
|
568
|
+
let p = await this.findByConflict(e, t, r.conflictColumns, i);
|
|
569
|
+
if (p) return p;
|
|
570
|
+
let m = u[0];
|
|
571
|
+
throw Error(`MysqlStore.insertIgnore: the insert was skipped as a conflict, but no existing row matches conflictColumns [${r.conflictColumns.join(", ")}] on '${e}'. ` + (m === void 0 ? "The warning could not be read on this connection, so the constraint that fired is unknown — it may be a second unique index, or the primary key under another name. " : `The constraint that actually fired: [${m.code}] ${m.message}. `) + "insertIgnore models ONE conflict target: name the columns of the constraint that actually collides, or handle the violation yourself.");
|
|
552
572
|
}
|
|
553
573
|
return await this.findByConflict(e, t, r.conflictColumns, i) || Q()(() => this.executeInsert(e, t, i, a));
|
|
554
574
|
}
|
|
575
|
+
async findUndecodableCdcTables(e) {
|
|
576
|
+
if (this.variant !== "mariadb") return [];
|
|
577
|
+
try {
|
|
578
|
+
let t = (await this.runtime.runPromise(this.sql.unsafe(z))).map((e) => String(e.tableName ?? e.TABLE_NAME ?? "")).filter((e) => e !== "");
|
|
579
|
+
return e === void 0 ? t : t.filter((t) => e.includes(t));
|
|
580
|
+
} catch (e) {
|
|
581
|
+
return this.log.debug(`cdc: could not probe for undecodable tables — ${e?.message ?? String(e)}`), [];
|
|
582
|
+
}
|
|
583
|
+
}
|
|
584
|
+
async readWarnings(e) {
|
|
585
|
+
if (e === null) return [];
|
|
586
|
+
try {
|
|
587
|
+
let t = this.sql`SHOW WARNINGS`;
|
|
588
|
+
return (await this.runtime.runPromise(n.provideService(t, d, e))).map((e) => ({
|
|
589
|
+
code: Number(e.Code ?? e.code ?? 0),
|
|
590
|
+
message: String(e.Message ?? e.message ?? "")
|
|
591
|
+
}));
|
|
592
|
+
} catch {
|
|
593
|
+
return [];
|
|
594
|
+
}
|
|
595
|
+
}
|
|
555
596
|
async findByConflict(e, t, r, i) {
|
|
556
597
|
if (r.length === 0) return;
|
|
557
598
|
let a = this.sql, o = r.map((e) => a`${a(e)} = ${t[e]}`), s = a`SELECT * FROM ${a(this.nsT(e))} WHERE ${a.and(o)} LIMIT 1`, c = i ? n.provideService(s, d, i) : s;
|
|
@@ -574,7 +615,7 @@ var F = (n) => e.layerConfig({
|
|
|
574
615
|
if (e instanceof p) throw e;
|
|
575
616
|
this.log.warn(`${this.variant} JSON-agg eager-load failed; falling back to walker`, { err: e });
|
|
576
617
|
}
|
|
577
|
-
return h(await this.executeQuery(e, t), e.eager, e.sourceTable ??
|
|
618
|
+
return h(await this.executeQuery(e, t), e.eager, e.sourceTable ?? re(e.table), (e) => this.executeQuery(e, t));
|
|
578
619
|
}
|
|
579
620
|
getInternalRunWithEager() {
|
|
580
621
|
return this.runWithEager.bind(this);
|
|
@@ -622,7 +663,7 @@ var F = (n) => e.layerConfig({
|
|
|
622
663
|
let r = t.map((e) => e.id), a = i`SELECT * FROM ${i(this.nsT(e))} WHERE ${i("id")} IN ${i.in(r)}`;
|
|
623
664
|
return n.flatMap(n.provideService(l, d, s), () => n.provideService(a, d, s));
|
|
624
665
|
});
|
|
625
|
-
}))), c = s.exponential("10 millis").pipe(s.compose(s.recurs(3)), s.whileInput(
|
|
666
|
+
}))), c = s.exponential("10 millis").pipe(s.compose(s.recurs(3)), s.whileInput(J)), l = r.pipe(n.retry(c), n.withSpan("store.updateMany", { attributes: {
|
|
626
667
|
"db.system": this.variant,
|
|
627
668
|
"db.operation": "update"
|
|
628
669
|
} })), u = await this.runtime.runPromise(l);
|
|
@@ -657,7 +698,7 @@ var F = (n) => e.layerConfig({
|
|
|
657
698
|
if (a.isNone(t)) return n.fail(/* @__PURE__ */ Error("MysqlStore.deleteMany: TransactionConnection missing."));
|
|
658
699
|
let o = t.value, s = r`SELECT * FROM ${r(this.nsT(e))} WHERE ${i} FOR UPDATE`, c = r`DELETE FROM ${r(this.nsT(e))} WHERE ${i}`;
|
|
659
700
|
return n.flatMap(n.provideService(s, d, o), (e) => e.length === 0 ? n.succeed(e) : n.as(n.provideService(c, d, o), e));
|
|
660
|
-
}))), o = s.exponential("10 millis").pipe(s.compose(s.recurs(3)), s.whileInput(
|
|
701
|
+
}))), o = s.exponential("10 millis").pipe(s.compose(s.recurs(3)), s.whileInput(J)), c = t.pipe(n.retry(o), n.withSpan("store.deleteMany", { attributes: {
|
|
661
702
|
"db.system": this.variant,
|
|
662
703
|
"db.operation": "delete"
|
|
663
704
|
} })), l = await this.runtime.runPromise(c);
|
|
@@ -680,12 +721,14 @@ var F = (n) => e.layerConfig({
|
|
|
680
721
|
async startCdcConsumer(e) {
|
|
681
722
|
if (this.cdcHandle) return;
|
|
682
723
|
await this.assertBinlogConfig(), this.cdcReplicaId = e.replicaId;
|
|
683
|
-
let t = await this.readCdcOffset(e.replicaId) ?? await this.resolveBinlogEnd();
|
|
684
|
-
|
|
724
|
+
let t = await this.readCdcOffset(e.replicaId) ?? await this.resolveBinlogEnd(), n = await this.findUndecodableCdcTables(e.includeTables);
|
|
725
|
+
for (let e of n) this.log.error(B(e));
|
|
726
|
+
this.cdcHandle = await q({
|
|
685
727
|
connection: e.connection,
|
|
686
728
|
serverId: e.serverId,
|
|
687
729
|
variant: this.variant,
|
|
688
730
|
...e.includeTables ? { includeTables: e.includeTables } : {},
|
|
731
|
+
...n.length > 0 ? { undecodableTables: n } : {},
|
|
689
732
|
...e.watchdogIntervalMs === void 0 ? {} : { watchdogIntervalMs: e.watchdogIntervalMs },
|
|
690
733
|
...e.stallThresholdMs === void 0 ? {} : { stallThresholdMs: e.stallThresholdMs },
|
|
691
734
|
...e.keepAliveInitialDelayMs === void 0 ? {} : { keepAliveInitialDelayMs: e.keepAliveInitialDelayMs },
|
|
@@ -801,7 +844,7 @@ var F = (n) => e.layerConfig({
|
|
|
801
844
|
let i = ++r;
|
|
802
845
|
return this.sql.withTransaction(n.flatMap(n.serviceOption(d), (r) => {
|
|
803
846
|
if (a.isNone(r)) return n.fail(/* @__PURE__ */ Error("MysqlStore.transactional: TransactionConnection missing."));
|
|
804
|
-
let o = new
|
|
847
|
+
let o = new pe(this, r.value);
|
|
805
848
|
return n.tryPromise({
|
|
806
849
|
try: () => t(() => e(o)).then((e) => ({
|
|
807
850
|
result: e,
|
|
@@ -811,12 +854,12 @@ var F = (n) => e.layerConfig({
|
|
|
811
854
|
catch: (e) => e
|
|
812
855
|
});
|
|
813
856
|
}));
|
|
814
|
-
}), o = s.exponential("10 millis").pipe(s.compose(s.recurs(3)), s.whileInput(
|
|
857
|
+
}), o = s.exponential("10 millis").pipe(s.compose(s.recurs(3)), s.whileInput(J)), c = i.pipe(n.retry(o), n.withSpan("store.transactional", { attributes: {
|
|
815
858
|
"db.system": this.variant,
|
|
816
859
|
"db.operation": "transaction"
|
|
817
860
|
} }));
|
|
818
861
|
try {
|
|
819
|
-
let e = await this.runtime.
|
|
862
|
+
let e = j(await this.runtime.runPromiseExit(c));
|
|
820
863
|
return e.view.commitEvents(), e.result;
|
|
821
864
|
} finally {
|
|
822
865
|
this.inflightTxns--;
|
|
@@ -860,7 +903,7 @@ var F = (n) => e.layerConfig({
|
|
|
860
903
|
async ping() {
|
|
861
904
|
await this.runtime.runPromise(this.sql`SELECT 1`);
|
|
862
905
|
}
|
|
863
|
-
},
|
|
906
|
+
}, pe = class {
|
|
864
907
|
parent;
|
|
865
908
|
txn;
|
|
866
909
|
events = [];
|
|
@@ -932,7 +975,7 @@ var F = (n) => e.layerConfig({
|
|
|
932
975
|
this.events.length = 0;
|
|
933
976
|
}
|
|
934
977
|
}
|
|
935
|
-
}, $ = (e) => e.__mysqlReplicationFriend ?? null,
|
|
978
|
+
}, $ = (e) => e.__mysqlReplicationFriend ?? null, me = () => ({
|
|
936
979
|
async capturePrimaryPosition(e) {
|
|
937
980
|
let t = $(e);
|
|
938
981
|
if (t === null) throw Error("mysqlReplicationAdapter: primary is not a MysqlStore (missing __mysqlReplicationFriend).");
|
|
@@ -952,24 +995,24 @@ var F = (n) => e.layerConfig({
|
|
|
952
995
|
compare(e, t) {
|
|
953
996
|
return "behind";
|
|
954
997
|
}
|
|
955
|
-
}),
|
|
998
|
+
}), he = {
|
|
956
999
|
id: "mysql",
|
|
957
|
-
makeSqlLayer: (e) =>
|
|
1000
|
+
makeSqlLayer: (e) => I(e),
|
|
958
1001
|
makeStore: (e) => Z({
|
|
959
1002
|
...e,
|
|
960
1003
|
variant: "mysql"
|
|
961
1004
|
}),
|
|
962
1005
|
compileContains: (e, t, n) => n`${e} LIKE ${`%${t.replace(/[\\%_]/g, (e) => `\\${e}`)}%`} ESCAPE '\\'`,
|
|
963
|
-
retryFilter:
|
|
964
|
-
},
|
|
1006
|
+
retryFilter: Y
|
|
1007
|
+
}, ge = {
|
|
965
1008
|
id: "mariadb",
|
|
966
|
-
makeSqlLayer: (e) =>
|
|
1009
|
+
makeSqlLayer: (e) => I(e),
|
|
967
1010
|
makeStore: (e) => Z({
|
|
968
1011
|
...e,
|
|
969
1012
|
variant: "mariadb"
|
|
970
1013
|
}),
|
|
971
1014
|
compileContains: (e, t, n) => n`${e} LIKE ${`%${t.replace(/[\\%_]/g, (e) => `\\${e}`)}%`} ESCAPE '\\'`,
|
|
972
|
-
retryFilter:
|
|
1015
|
+
retryFilter: Y
|
|
973
1016
|
};
|
|
974
1017
|
//#endregion
|
|
975
|
-
export { f as CDC_OFFSETS_TABLE, e as MysqlClient, m as _voltroCdcOffsetsTable,
|
|
1018
|
+
export { f as CDC_OFFSETS_TABLE, e as MysqlClient, m as _voltroCdcOffsetsTable, F as connectionFromConfig, P as makeMysqlSqlLayer, I as makeMysqlSqlLayerFromConfig, Z as makeMysqlStore, ge as mariadbDialect, he as mysqlDialect, me as mysqlReplicationAdapter, Y as mysqlRetryFilter, q as startBinlogCdc };
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@voltro/sql-mysql",
|
|
3
|
-
"version": "0.20.
|
|
3
|
+
"version": "0.20.1",
|
|
4
4
|
"description": "MySQL/MariaDB dialect adapter for Voltro's cross-dialect DataStore (mariadb binlog CDC; mysql inline reactivity).",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"voltro",
|
|
@@ -34,8 +34,8 @@
|
|
|
34
34
|
"dependencies": {
|
|
35
35
|
"@effect/sql": "^0.51.1",
|
|
36
36
|
"@effect/sql-mysql2": "^0.52.0",
|
|
37
|
-
"@voltro/database": "0.20.
|
|
38
|
-
"@voltro/logger": "0.20.
|
|
37
|
+
"@voltro/database": "0.20.1",
|
|
38
|
+
"@voltro/logger": "0.20.1"
|
|
39
39
|
},
|
|
40
40
|
"optionalDependencies": {
|
|
41
41
|
"@vlasky/zongji": "^0.9.0"
|