@vlasky/zongji 0.7.1 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/README.md +83 -12
- package/index.d.ts +359 -9
- package/index.js +355 -28
- package/lib/binlog_event.js +892 -27
- package/lib/charset_map.js +86 -0
- package/lib/code_map.js +44 -1
- package/lib/common.js +384 -20
- package/lib/gtid_set.js +244 -0
- package/lib/mariadb_gtid.js +148 -0
- package/lib/packet/binlog.js +7 -1
- package/lib/rows_event.js +39 -7
- package/lib/sequence/binlog.js +178 -26
- package/package.json +6 -3
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,27 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to @vlasky/zongji since forking from nevill/zongji.
|
|
4
4
|
|
|
5
|
+
## [0.9.0] - 2026-07-06
|
|
6
|
+
|
|
7
|
+
- Export the `MariadbGtidPosition` class (`import { MariadbGtidPosition } from '@vlasky/zongji'`), the MariaDB counterpart of `GtidSet` and the model behind `zongji.gtidSet` on MariaDB servers: parse a persisted position (a `zongji.gtidSet` checkpoint or `@@gtid_binlog_pos`), build one from a `mariadbgtidlist` event with `fromGtidList()`, extend it with `add()`, and test transaction coverage with `covers(event.gtid, { inclusive })` instead of hand-rolling sequence-number comparators (which silently round above `Number.MAX_SAFE_INTEGER` when done with Number). The inclusive form (default) answers "was this transaction's commit at or before the position?" for snapshot barriers; redelivery watermarks must pass `{ inclusive: false }`, because all events of a transaction share one GTID and an inclusive check would drop every replayed event of the watermark transaction after the first. A server id differing from the position's entry for that domain reports not covered even for a lower sequence number (post-failover sequence numbers can regress under the new server id; for at-least-once consumers an idempotent redelivery is recoverable where a skip is not), as does a domain the position does not list.
|
|
8
|
+
- `GtidSet` and `MariadbGtidPosition` now use BigInt internally, making them exact over MySQL's full GNO range (1 to 2^63-1) and MariaDB's full u64 sequence range. Previously `GtidSet.parse`/`add` threw on GNOs above 2^53, so a snapshot barrier or checkpoint parse failed outright against a server carrying such GTIDs. Public APIs are unchanged: strings in, strings and booleans out.
|
|
9
|
+
- Fix Previous_gtids decoding silently rounding interval bounds above 2^53: the u64 bounds were combined with Number arithmetic, corrupting `event.gtidSet` - the exact text that seeds `zongji.gtidSet` when a dump starts at a binlog file boundary. Bounds are now decoded as BigInt, and the `event.sids` intervals follow the Number-or-exact-string convention used for other 64-bit values (typed `number | string`; previously typed `number` while silently wrong beyond 2^53).
|
|
10
|
+
- Document on the `seqNo` fields of `mariadbgtid` and `mariadbgtidlist` events that values above `Number.MAX_SAFE_INTEGER` arrive as exact strings and must never be compared numerically - use `event.gtid` with `MariadbGtidPosition`/`GtidSet` instead.
|
|
11
|
+
|
|
12
|
+
## [0.8.0] - 2026-07-06
|
|
13
|
+
|
|
14
|
+
- Export the `GtidSet` class (`import ZongJi, { GtidSet } from '@vlasky/zongji'`): parse a persisted set (e.g. a `zongji.gtidSet` checkpoint or `@@gtid_executed`), `add()` transactions, and test single-transaction membership with `contains(event.gtid)` - handy for snapshot drain barriers ("was this event already covered when I took my snapshot?") without maintaining a separate GTID parser. Fully tagged-GTID aware.
|
|
15
|
+
- Close the remaining resume-position gap for multi-table statements (multi-table `UPDATE`, foreign-key cascades). The server writes all of a statement's TableMap events before any row event, so a `filename`/`position` pair persisted while the first table's rows were being processed pointed past the later TableMaps; a consumer resuming there dropped the remaining tables' rows with no error. The resume position now freezes at a transaction's first TableMap and advances to the commit marker's end position, so a persisted position always replays whole transactions. Behaviour change: for row-logged transactions the position advances per transaction rather than per event, so resuming after a mid-transaction crash re-delivers that transaction's earlier row events (at-least-once, as already documented for the single-table case fixed in 0.7.1). Statement-logged content (`binlog_format=STATEMENT`/`MIXED`) still advances per event; its replay has no table-metadata dependency, so it never had the dropped-rows hazard.
|
|
16
|
+
- Tagged GTID support (MySQL 8.3+, `GTID_NEXT='AUTOMATIC:tag'`). Tagged transactions arrive as GTID_TAGGED_LOG_EVENT (code 42, a self-describing serialization format unrelated to the classic layout, previously `unknown`) and now decode as ordinary `gtid` events with `event.tag` set and `event.gtid` in `uuid:tag:gno` form. GTID sets accept and print the tagged grammar (`uuid:1-5:tag_a:1`), `zongji.gtidSet` tracks tagged transactions, `start({ gtidSet })` resumes from a tagged checkpoint (COM_BINLOG_DUMP_GTID's tagged payload encoding, understood by MySQL 8.3+ only), and `startAtEnd` seeds from a `gtid_executed` containing tags. This is also a compatibility fix: once a MySQL 8.3+ server has ever executed a tagged GTID it writes every Previous_gtids event in the tagged set encoding forever (surviving `RESET BINARY LOGS AND GTIDS`), which previously crashed parsing at the start of every dump, and any tag in `gtid_executed` broke `startAtEnd` seeding.
|
|
17
|
+
- New `rowsquery` event (MySQL ROWS_QUERY_LOG_EVENT, previously `unknown`): the SQL statement text behind the row events that follow, MySQL's analogue of MariaDB's `annotaterows`. The server sends it to every consumer while `binlog_rows_query_log_events=ON` (default `OFF`); no start option is involved, unlike MariaDB's opt-in `requestAnnotateRows`.
|
|
18
|
+
- `event.gtid` no longer leaks onto events that belong to no transaction: it is now cleared once a commit marker (Xid, XA Prepare, or a COMMIT/ROLLBACK/XA COMMIT/XA ROLLBACK query) has been delivered, so e.g. a rotate arriving between two transactions reports `gtid: undefined` instead of the previous transaction's GTID. The commit event itself still carries its transaction's GTID. A DDL statement has no commit marker of its own, so its GTID persists until the next transaction begins. New `zongji.lastGtid` getter exposes the same tracked value (the GTID of the transaction currently being delivered); it is only coherent read synchronously inside a `'binlog'` handler.
|
|
19
|
+
- Fix text CHAR and VARCHAR columns always decoding as utf8: they now decode via the column's own character set, as TEXT columns already did, so e.g. `latin1` and `greek` columns no longer produce mojibake. In the same pass, charsets whose stored byte order differs from the iconv-lite encoding of the same name are now mapped explicitly: MySQL's `latin1` is decoded as cp1252 (its real definition; `€`, `Ÿ` and friends in 0x80-0x9F previously came back as control characters from TEXT columns), and `ucs2`/`utf16`/`utf32` are decoded big-endian (previously byte-swapped garbage from TEXT columns). Behaviour change: consumers that stored the old garbled output will see correct strings after upgrading.
|
|
20
|
+
- MariaDB support (tested against MariaDB 11.8): ZongJi now detects the server flavour and follows MariaDB binlogs natively, announcing `@mariadb_slave_capability=4` so the server sends its real GTID events instead of rewriting them for legacy clients. MariaDB's GTID model (a `domain-server-sequence` watermark per replication domain) is fully supported: events carry `event.gtid` in MariaDB format, `zongji.gtidSet` exposes a MariaDB GTID position (seeded from `start({ gtidSet })`, from `@@gtid_current_pos` under `startAtEnd`, or from the GTID list event at the start of a binlog file), and `start({ gtidSet: '0-1-1234' })` resumes via `@slave_connect_state` with server-side skipping, including across failover. New event types: `mariadbgtid`, `mariadbgtidlist`, `binlogcheckpoint`, `startencryption`, plus `xaprepare` (also emitted by MySQL 5.7+, previously `unknown`). MariaDB data types decode to match mysql2 query results: `UUID`, `INET4` and `INET6` as canonical text, `VECTOR` as a raw Buffer, `JSON` as the LONGTEXT alias text it is, `COMPRESSED` columns transparently decompressed (with correct charsets, under all `binlog_row_metadata` settings), MariaDB 5.3-era "hires" temporals (`mysql56_temporal_format=OFF`) decoded correctly instead of desynchronising the row, and compressed binlog events (`log_bin_compress=ON`) decoded transparently as ordinary query/row events. `binlog_row_metadata=FULL` works on MariaDB 10.5+ with the same self-describing decode as MySQL; tables whose classic temporal columns could hide hires encodings automatically use `INFORMATION_SCHEMA` instead, and `UUID`/`INET*`/`VECTOR` values arrive as raw Buffers in FULL mode (the binlog cannot distinguish them from `BINARY`). An opt-in `requestAnnotateRows` start option asks the server for `annotaterows` events carrying the SQL statement text behind each row operation. The test suite runs against MariaDB 11.8 in CI alongside MySQL 5.7/8.0/8.4, and an offline MariaDB fixture pins the captured wire bytes of all of the above in the parser tests.
|
|
21
|
+
- Fix the resume position (`options.position`) going stale when the events carrying real positions are excluded by `includeEvents`: filtered events now advance it at the packet layer under the same safety rules as delivered ones (never past a TableMap, never a zero position). On MySQL the position merely lagged; on MariaDB, where events inside a transaction carry `end_log_pos=0`, common filter sets froze it entirely.
|
|
22
|
+
- Fix a filtered `rotate` event leaving an incoherent resume pair: the filename never updated while later events advanced the position into the new file. Rotates now update the `filename`/`position` pair before event filtering, whether delivered or not.
|
|
23
|
+
- GTID-based resume: `start({ gtidSet })` issues COM_BINLOG_DUMP_GTID, letting the server locate the correct binlog file and skip already-processed transactions itself; a persisted checkpoint therefore survives failover to another server in the same replication topology (requires `gtid_mode=ON`). `zongji.gtidSet` exposes the executed set for persisting: seeded exactly from the start set, from the server's `gtid_executed` under `startAtEnd`, or from the stream's Previous_gtids event when reading from the start of a binlog file, and extended only as observed transactions commit. Purged-GTID and GTID-mode errors surface through the `error` event. Heartbeat events (sent in place of server-side-skipped transactions and while idle) are now decoded instead of falling through to `unknown`, and Previous_gtids string formatting is canonical (single transactions print as `8`, not `8-8`)
|
|
24
|
+
- Support `binlog_row_metadata=FULL` (MySQL 8.0+): TableMap events now parse the optional metadata block (column names, signedness, character sets, enum/set value lists, primary key, column visibility), and when it is complete ZongJi decodes rows entirely from the binlog stream with no `INFORMATION_SCHEMA` queries and no connection pauses. Each TableMap event rebuilds the table's metadata, so `ALTER TABLE` can no longer leave stale column definitions behind. Enum/set values containing commas or quotes decode correctly in this mode (the `INFORMATION_SCHEMA` path cannot represent them). Under the default `binlog_row_metadata=MINIMAL`, integer signedness now comes from the binlog instead of being inferred from the `COLUMN_TYPE` string. TableMap events expose `columnNames`, `signedness`, `primaryKey` and `columnVisibility` where available, and column schemas gain `UNSIGNED`, `ENUM_VALUES` and `SET_VALUES`.
|
|
25
|
+
|
|
5
26
|
## [0.7.1] - 2026-07-04
|
|
6
27
|
|
|
7
28
|
- Fix `start()` calls made while a previous `start()` was still initialising silently discarding their filters. Since 0.7.0 filters are snapshotted and re-calling `start()` is the documented way to update them, but updates made between `start()` and the `ready` event were lost; consumers registering tables during boot (e.g. @vlasky/mysql-live-select) missed events for tables added in that window. Filters passed during initialisation now apply, exactly as when already running; stream options (filename/position/serverId) still come from the first call.
|
package/README.md
CHANGED
|
@@ -3,9 +3,9 @@
|
|
|
3
3
|
[](https://www.npmjs.com/package/@vlasky/zongji)
|
|
4
4
|
[](https://github.com/vlasky/zongji/blob/master/LICENSE)
|
|
5
5
|
|
|
6
|
-
MySQL binlog-based change data capture (CDC) for Node.js, [originally created by Nevill Dutt](https://github.com/nevill/zongji).
|
|
6
|
+
MySQL and MariaDB binlog-based change data capture (CDC) for Node.js, [originally created by Nevill Dutt](https://github.com/nevill/zongji).
|
|
7
7
|
|
|
8
|
-
[@vlasky/zongji](https://github.com/vlasky/zongji) has been tested working with MySQL 5.7, 8.0 and 8.4.
|
|
8
|
+
[@vlasky/zongji](https://github.com/vlasky/zongji) has been tested working with MySQL 5.7, 8.0 and 8.4, and MariaDB 11.8.
|
|
9
9
|
|
|
10
10
|
It leverages [`mysql2`](https://github.com/sidorares/node-mysql2) for connections and authentication, while using zongji's binlog parsing and event pipeline.
|
|
11
11
|
|
|
@@ -72,8 +72,9 @@ const { default: ZongJi } = await import('@vlasky/zongji');
|
|
|
72
72
|
$ npm install @vlasky/zongji
|
|
73
73
|
```
|
|
74
74
|
|
|
75
|
-
* Enable
|
|
76
|
-
> Binlog checksum is enabled by default in all supported MySQL versions. Zongji can work with it, but it doesn't verify the checksum.
|
|
75
|
+
* Enable the binlog in `my.cnf`, restart the server after making the changes.
|
|
76
|
+
> Binlog checksum is enabled by default in all supported MySQL and MariaDB versions. Zongji can work with it, but it doesn't verify the checksum.
|
|
77
|
+
> On MariaDB, `binlog_format` defaults to `MIXED`, so setting it to `row` explicitly matters there in particular.
|
|
77
78
|
|
|
78
79
|
```
|
|
79
80
|
# Must be unique integer from 1-2^32
|
|
@@ -87,6 +88,12 @@ const { default: ZongJi } = await import('@vlasky/zongji');
|
|
|
87
88
|
binlog_do_db = employees # Optional, limit which databases to log
|
|
88
89
|
expire_logs_days = 10 # Optional, purge old logs
|
|
89
90
|
max_binlog_size = 100M # Optional, limit log size
|
|
91
|
+
|
|
92
|
+
# Recommended on MySQL 8.0+ / MariaDB 10.5+: makes binlog events carry
|
|
93
|
+
# complete column metadata (names, signedness, charsets, enum/set
|
|
94
|
+
# values), so ZongJi decodes rows without querying INFORMATION_SCHEMA
|
|
95
|
+
# (see below)
|
|
96
|
+
binlog_row_metadata = FULL
|
|
90
97
|
```
|
|
91
98
|
* Create an account with replication privileges, e.g. given privileges to account `zongji` (or any account that you use to read binary logs)
|
|
92
99
|
|
|
@@ -112,6 +119,17 @@ Option | Effect on row values
|
|
|
112
119
|
`decimalNumbers` | `DECIMAL` columns are returned as exact strings by default (e.g. `'-123.4500'`); set `decimalNumbers: true` to receive Numbers (may lose precision beyond 15 significant digits).
|
|
113
120
|
`jsonStrings` | `JSON` columns are returned as parsed JavaScript values by default; set `jsonStrings: true` to receive JSON strings.
|
|
114
121
|
|
|
122
|
+
### Table metadata source
|
|
123
|
+
|
|
124
|
+
To decode row events, ZongJi needs each table's column metadata. It obtains this in one of two ways:
|
|
125
|
+
|
|
126
|
+
* **From the binlog itself** (MySQL 8.0+ or MariaDB 10.5+ with `binlog_row_metadata=FULL`): every `TableMap` event carries column names, signedness, character sets and enum/set value lists. ZongJi decodes rows entirely from the stream: no `INFORMATION_SCHEMA` queries, no pausing the binlog connection, and no risk of the metadata being stale after an `ALTER TABLE` (each event describes the schema in force when it was written). Enum/set values containing commas or quotes also decode correctly only in this mode.
|
|
127
|
+
* **From `INFORMATION_SCHEMA`** (all other cases, including MySQL 5.7, the MySQL default `binlog_row_metadata=MINIMAL` and the MariaDB default `NO_LOG`): on the first event for each table, ZongJi briefly pauses the stream and fetches column metadata over its control connection. If the table is altered between the binlog write and the fetch, decoded rows may be misinterpreted until the next `TableMap` refresh.
|
|
128
|
+
|
|
129
|
+
Setting `binlog_row_metadata=FULL` on the server is therefore recommended. It is a dynamic global variable (`SET GLOBAL binlog_row_metadata=FULL`) with negligible binlog size overhead for typical schemas. Even under `MINIMAL`, MySQL 8.0+ binlogs carry exact integer signedness, which ZongJi uses in preference to parsing the column definition.
|
|
130
|
+
|
|
131
|
+
MariaDB caveats under `FULL`: the binlog cannot distinguish `UUID`, `INET4`, `INET6` and `VECTOR` columns from `BINARY`/`VARBINARY`, so in this mode their values arrive as raw `Buffer`s rather than the formatted values the `INFORMATION_SCHEMA` path produces; and tables containing classic temporal type codes (which may hide MariaDB 5.3-era "hires" columns the binlog cannot describe) automatically fall back to the `INFORMATION_SCHEMA` path.
|
|
132
|
+
|
|
115
133
|
Each instance includes the following methods:
|
|
116
134
|
|
|
117
135
|
Method Name | Arguments | Description
|
|
@@ -139,6 +157,8 @@ Option Name | Type | Description
|
|
|
139
157
|
`startAtEnd` | `boolean` | Pass `true` to only emit binlog events that occur after ZongJi's instantiation. Must be used in `start()` method for effect.<br>**Default:** `false`
|
|
140
158
|
`filename` | `string` | Begin reading events from this binlog file. If specified together with `position`, will take precedence over `startAtEnd`.
|
|
141
159
|
`position` | `integer` | Begin reading events from this position. Must be included with `filename`.
|
|
160
|
+
`gtidSet` | `string` | Resume from executed GTIDs (e.g. a persisted `zongji.gtidSet`): a MySQL GTID set (`'uuid:1-27'`; tagged transactions print as `'uuid:1-27:tag_a:1'` on MySQL 8.3+) or a MariaDB GTID position (`'0-1-1234'`), matching the server flavour. The server locates the correct binlog file itself and skips transactions already processed, so a checkpoint remains valid across failover to another server in the same replication topology. Requires `gtid_mode=ON` on MySQL; always available on MariaDB. Pass `''` to stream the entire available history. Takes precedence over `filename`/`position` and `startAtEnd`.
|
|
161
|
+
`requestAnnotateRows` | `boolean` | MariaDB only: ask the server to send `annotaterows` events (the SQL statement text preceding each row operation's events). Off by default; note the statement text may contain sensitive literals that the row images alone would not expose. Ignored on MySQL.<br>**Default:** `false`
|
|
142
162
|
`includeEvents` | `[string]` | Array of event names to include<br>**Example:** `['writerows', 'updaterows', 'deleterows']`
|
|
143
163
|
`excludeEvents` | `[string]` | Array of event names to exclude<br>**Example:** `['rotate', 'tablemap']`
|
|
144
164
|
`includeSchema` | `object` | Object describing which databases and tables to include (Only for row events). Use database names as the key and pass an array of table names or `true` (for the entire database).<br>**Example:** ```{ 'my_database': ['allow_table', 'another_table'], 'another_db': true }```
|
|
@@ -156,20 +176,69 @@ Event name | Description
|
|
|
156
176
|
`unknown` | Catch any other events
|
|
157
177
|
`query` | [Insert/Update/Delete Query](https://dev.mysql.com/doc/internals/en/query-event.html)
|
|
158
178
|
`intvar` | [Autoincrement and LAST_INSERT_ID](https://dev.mysql.com/doc/internals/en/intvar-event.html)
|
|
159
|
-
`rotate` | [New Binlog file](https://dev.mysql.com/doc/internals/en/rotate-event.html)
|
|
179
|
+
`rotate` | [New Binlog file](https://dev.mysql.com/doc/internals/en/rotate-event.html). The `filename` and `position` resume properties stay updated across rotations whether or not this event is included; include it only if you want to observe rotations yourself.
|
|
160
180
|
`format` | [Format Description](https://dev.mysql.com/doc/internals/en/format-description-event.html)
|
|
161
181
|
`xid` | [Transaction ID](https://dev.mysql.com/doc/internals/en/xid-event.html)
|
|
162
|
-
`gtid` | GTID event with `gtid`, `sid`, `gno` properties
|
|
163
|
-
`anonymousgtid` |
|
|
164
|
-
`previousgtids` | Previous GTIDs event with `gtidSet` and `sids`
|
|
182
|
+
`gtid` | MySQL GTID event with `gtid`, `sid`, `gno` properties. Tagged GTIDs (MySQL 8.3+, `GTID_NEXT='AUTOMATIC:tag'`) arrive as the same event with `tag` set and `gtid` in `uuid:tag:gno` form
|
|
183
|
+
`anonymousgtid` | MySQL anonymous GTID event (same shape as `gtid`)
|
|
184
|
+
`previousgtids` | MySQL Previous GTIDs event with `gtidSet` and `sids`
|
|
185
|
+
`mariadbgtid` | MariaDB GTID event with `gtid` (`'domain-server-sequence'`), `domainId`, `serverId`, `seqNo` and flags (`standalone`, `isDdl`, XA states). Replaces the `BEGIN` query event of a transaction.
|
|
186
|
+
`mariadbgtidlist` | MariaDB GTID list event (start of every binlog file): the last GTID per domain and server seen so far, MariaDB's analogue of `previousgtids`
|
|
187
|
+
`binlogcheckpoint` | MariaDB binlog checkpoint event naming the oldest binlog file still needed for crash recovery (informational)
|
|
188
|
+
`annotaterows` | MariaDB: the SQL `statement` that produced the row events that follow (the analogue of MySQL's rows-query event). Only sent when `start()` was given `requestAnnotateRows: true`; useful for audit trails and statement attribution in CDC records
|
|
189
|
+
`rowsquery` | MySQL: the SQL `statement` that produced the row events that follow (the analogue of MariaDB's `annotaterows`). Sent to every consumer while the server runs with `binlog_rows_query_log_events=ON` (default `OFF`); no `start()` option is involved. The same sensitivity note applies: statement text may contain literals
|
|
190
|
+
`startencryption` | MariaDB encrypted-binlog marker (informational: the server decrypts before sending, the stream itself is cleartext)
|
|
191
|
+
`xaprepare` | XA PREPARE event (MySQL 5.7+ and MariaDB) terminating an XA-prepared event group, with `xidFormatId`, `xidGtrid`, `xidBqual` and `onePhase`
|
|
192
|
+
`heartbeat` | Sent by the server instead of real events: while the connection idles, and in place of transactions skipped server-side during a GTID resume. Carries `binlogName`; `nextPosition` holds the advanced position.
|
|
165
193
|
`tablemap` | Before any row event (must be included for any other row events)
|
|
166
194
|
`writerows` | Rows inserted, row data array available as `rows` property on event object
|
|
167
195
|
`updaterows` | Rows changed, row data array available as `rows` property on event object
|
|
168
196
|
`deleterows` | Rows deleted, row data array available as `rows` property on event object
|
|
169
|
-
`transactionpayload` | Compressed transaction from MySQL 8.0.20+ servers with `binlog_transaction_compression=ON`. ZongJi cannot decode the row events inside it, so it emits an `error` (once per instance) naming the server setting responsible.
|
|
197
|
+
`transactionpayload` | Compressed transaction from MySQL 8.0.20+ servers with `binlog_transaction_compression=ON`. ZongJi cannot decode the row events inside it, so it emits an `error` (once per instance) naming the server setting responsible. (MariaDB's equivalent, `log_bin_compress=ON`, IS supported: its compressed events decode transparently as ordinary `query` and row events.)
|
|
170
198
|
`partialupdaterows` | Partial JSON update from MySQL 8.0+ servers with `binlog_row_value_options=PARTIAL_JSON`. ZongJi cannot decode the JSON diff format, so it emits an `error` (once per instance) naming the server setting responsible.
|
|
171
199
|
|
|
172
|
-
When the server runs with `gtid_mode=ON
|
|
200
|
+
When the server runs with `gtid_mode=ON` (MySQL), or always on MariaDB, every emitted event carries a `gtid` property (`'uuid:sequence'` on MySQL, `'domain-server-sequence'` on MariaDB) identifying the transaction it belongs to, tracked internally even if GTID events are excluded by `includeEvents`. It is `undefined` for anonymous transactions, for events seen before the stream's first GTID event, and for events that belong to no transaction (e.g. a rotate arriving between one transaction's commit and the next transaction's GTID event; the commit event itself still carries its transaction's `gtid`). One exception: a DDL statement has no commit marker of its own, so its `gtid` persists on such trailing events until the next transaction begins. Use it for checkpointing and deduplication. The same value is available as `zongji.lastGtid` while an event is being delivered; it is only coherent when read synchronously inside a `'binlog'` handler, so checkpointing code should prefer `event.gtid`.
|
|
201
|
+
|
|
202
|
+
### File and position resume
|
|
203
|
+
|
|
204
|
+
`zongji.options.filename` and `zongji.options.position` always hold a safe resume point for the connected server. The position advances only at transaction boundaries: it freezes when a transaction's first `TableMap` arrives and jumps to the commit marker's end position, so a persisted pair never points inside a transaction (where row events could be separated from the table metadata they need and silently dropped). Resuming from a pair persisted mid-transaction therefore replays that whole transaction: consumers should treat delivery as at-least-once and make their processing idempotent. Read the pair synchronously inside a `'binlog'` handler when checkpointing per event.
|
|
205
|
+
|
|
206
|
+
### GTID-based resume
|
|
207
|
+
|
|
208
|
+
`zongji.gtidSet` exposes the executed GTIDs as a string: a MySQL GTID set (e.g. `'3e11fa47-…:1-27'`) or a MariaDB GTID position (e.g. `'0-1-1234'`), depending on the connected server. It is the value ZongJi started from, extended as each observed transaction *commits*, so a persisted value never claims a transaction whose row events were still in flight. Persist it alongside (or instead of) `filename`/`position` and resume with:
|
|
209
|
+
|
|
210
|
+
```javascript
|
|
211
|
+
zongji.start({ gtidSet: savedGtidSet, serverId: 5 });
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
Unlike file+position checkpoints, a GTID checkpoint is valid on any server in the same replication topology, which makes resuming across a failover possible. The server performs the skipping itself, so already-processed transactions are not resent.
|
|
215
|
+
|
|
216
|
+
The `GtidSet` class behind it is exported for consumers that need to work with GTID sets themselves, e.g. testing whether an incoming `event.gtid` was already covered by a snapshot:
|
|
217
|
+
|
|
218
|
+
```javascript
|
|
219
|
+
import { GtidSet } from '@vlasky/zongji';
|
|
220
|
+
|
|
221
|
+
const snapshot = GtidSet.parse(savedGtidSet); // or @@gtid_executed
|
|
222
|
+
if (!snapshot.contains(event.gtid)) {
|
|
223
|
+
applyChange(event);
|
|
224
|
+
}
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
Its MariaDB counterpart `MariadbGtidPosition` is exported too, with a `covers()` method so consumers do not have to compare sequence numbers themselves (`seqNo` values above `Number.MAX_SAFE_INTEGER` arrive as exact strings, and comparing them as Numbers silently rounds; internally `GtidSet` is exact over MySQL's full GNO range of 1 to 2^63-1 and `MariadbGtidPosition` over MariaDB's full u64 sequence range):
|
|
228
|
+
|
|
229
|
+
```javascript
|
|
230
|
+
import { MariadbGtidPosition } from '@vlasky/zongji';
|
|
231
|
+
|
|
232
|
+
const snapshot = MariadbGtidPosition.parse(savedGtidPos); // or @@gtid_binlog_pos
|
|
233
|
+
snapshot.covers(event.gtid); // at or before the position?
|
|
234
|
+
snapshot.covers(event.gtid, { inclusive: false }); // strictly before it?
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
The default (inclusive) form answers "was this transaction's commit at or before the position?", which is what snapshot-barrier checks want: a recorded position covers the whole of its final transaction. Redelivery watermarks must pass `{ inclusive: false }`: all events of a transaction share one GTID, so an inclusive check would treat the watermark transaction's own replayed events as already seen and drop every one after the first. A server id different from the position's entry for that domain reports **not covered**, even for a lower sequence number: after a failover, sequence numbers can regress under the new server id, so ordering against another server's watermark is not meaningful, and for at-least-once consumers an idempotent redelivery is recoverable where a skip is not. A domain the position does not list is likewise not covered.
|
|
238
|
+
|
|
239
|
+
`zongji.gtidSet` is available when `start()` was given a `gtidSet`, with `startAtEnd` (seeded from the server's `gtid_executed` on MySQL, `gtid_current_pos` on MariaDB), or when reading from the start of a binlog file (seeded from its Previous_gtids or MariaDB GTID list event). It is `undefined` when resuming from an arbitrary mid-file file+position checkpoint, where no exact value can be known. Tagged GTIDs (MySQL 8.3+) are fully supported: tagged transactions appear in the set as `uuid:…:tag:intervals` sections, and a checkpoint containing tags resumes correctly (its wire encoding is only understood by MySQL 8.3+ servers).
|
|
240
|
+
|
|
241
|
+
On MySQL, GTID resume requires `gtid_mode=ON` and a set is a per-source-UUID collection of intervals. On MariaDB, GTIDs are always on and a position is a single watermark per replication domain (`domain-server-sequence`): resuming replays each listed domain after its watermark, and domains not listed replay from the beginning. If the server has purged binlogs your checkpoint still needs, `start()` emits an error (code 1236) naming the problem; MariaDB additionally reports a checkpoint ahead of its binlog as error 1945/1955. A fresh snapshot is then required.
|
|
173
242
|
|
|
174
243
|
**Event Methods**
|
|
175
244
|
|
|
@@ -183,6 +252,7 @@ Name | Description
|
|
|
183
252
|
## Important Notes
|
|
184
253
|
|
|
185
254
|
* :star2: All MySQL column types are supported, with type casting similar to [mysql2](https://github.com/sidorares/node-mysql2).
|
|
255
|
+
* MariaDB-specific types and features are supported: `UUID`, `INET4` and `INET6` decode to their canonical text forms and `VECTOR` to a raw `Buffer` of little-endian float32s (matching mysql2 query results); MariaDB `JSON` is a `LONGTEXT` alias and yields strings; `COMPRESSED` columns are decompressed transparently; compressed binlog events (`log_bin_compress=ON`) decode as ordinary query/row events; and MariaDB 5.3-era "hires" temporal columns (tables written under `mysql56_temporal_format=OFF`) decode correctly.
|
|
186
256
|
* :speak_no_evil: 64-bit integers are decoded exactly using native BigInt (see #108). If an integer is within the safe range of JS numbers (-2^53, 2^53), a Number is returned, otherwise an exact String. This also applies to 64-bit integers inside JSON columns.
|
|
187
257
|
* :point_right: `TRUNCATE` statement does not cause corresponding `DeleteRows` event. Use unqualified `DELETE FROM` for same effect.
|
|
188
258
|
* When using fractional seconds with `DATETIME` and `TIMESTAMP` data types, only millisecond precision is available due to the limit of Javascript's `Date` object.
|
|
@@ -191,8 +261,9 @@ Name | Description
|
|
|
191
261
|
## Run Tests
|
|
192
262
|
|
|
193
263
|
* Install [Docker](https://www.docker.com/community-edition#download)
|
|
194
|
-
* Run `docker-compose up -d` to start MySQL containers
|
|
195
|
-
* Run `npm test` to execute the test suite
|
|
264
|
+
* Run `docker-compose up -d` to start the MySQL and MariaDB containers
|
|
265
|
+
* Run `npm test` to execute the test suite against MySQL 8.4 (port 3306)
|
|
266
|
+
* Run `TEST_MYSQL_PORT=3308 npm test` to execute it against MariaDB 11.8; MySQL-only tests skip automatically, and the MariaDB-specific suite in `test/mariadb.js` runs (it no-ops against MySQL)
|
|
196
267
|
|
|
197
268
|
## References
|
|
198
269
|
|