@vlasky/zongji 0.6.1 → 0.7.1

package/CHANGELOG.md

@@ -2,6 +2,38 @@
 
 All notable changes to @vlasky/zongji since forking from nevill/zongji.
 
+## [0.7.1] - 2026-07-04
+
+- Fix `start()` calls made while a previous `start()` was still initialising silently discarding their filters. Since 0.7.0 filters are snapshotted and re-calling `start()` is the documented way to update them, but updates made between `start()` and the `ready` event were lost; consumers registering tables during boot (e.g. @vlasky/mysql-live-select) missed events for tables added in that window. Filters passed during initialisation now apply, exactly as when already running; stream options (filename/position/serverId) still come from the first call.
+- Fix a resume-position gap that could silently drop row events. `options.position` was advanced past TableMap events on the cached-metadata path, so a consumer persisting `filename`/`position` for reconnect could resume between a TableMap and its row events; the resumed instance had no metadata for the table id and dropped those rows with no error. TableMap events no longer advance the resume position, closing the gap for single-table statements (the common case). A narrower window remains for multi-table statements (multi-table UPDATE, foreign-key cascades), where the server writes all TableMap events before any row events: emitting the first table's rows still advances the position past the later TableMaps. Rows already processed before a crash may be re-delivered after resume (at-least-once), which is recoverable where dropping is not.
+- Fix rotate events corrupting the `filename`/`position` resume pair. A rotate's header position refers to the old binlog file (0 for the artificial rotate at the start of every dump), yet it was written into `options.position` alongside the new file's name; a consumer resuming from that pair after a real rotation could get "position > file size" or a mid-event read, and the artificial rotate silently reset the start position to 0. The rotate's payload position (the start of the new file) is now used, and the filename update is unconditional. Present in every zongji release since the original upstream project.
+- Fix a corrupt GTID event's parse error being swallowed when `gtid` is excluded by `includeEvents`; the whole following transaction was then silently mislabelled as anonymous. The error now reaches the `error` event regardless of filtering.
+- Schema drift between a binlog event being written and the metadata fetch (e.g. a column dropped in between) now emits a descriptive error naming the table and column counts, instead of throwing a bare TypeError from inside event parsing; the affected table's rows are skipped until its next TableMap event refreshes the metadata.
+
+## [0.7.0] - 2026-07-04
+
+### Breaking changes
+
+- DECIMAL columns now emit exact string values (e.g. `'-123.4500'`) instead of lossy floats, matching mysql2 query results. Migration: set `decimalNumbers: true` on the connection options passed to ZongJi to restore Numbers.
+- JSON columns now emit parsed JavaScript values instead of JSON strings, matching mysql2 query results. Migration: set `jsonStrings: true` on the connection options passed to ZongJi to restore strings. In string mode, output now uses MySQL's own formatting (spaces after `:` and `,`) and 64-bit integers appear as exact raw numerals rather than lossy doubles.
+- Event and schema filters are snapshotted when `start()` is called; mutating the arrays or objects you passed in no longer changes filtering afterwards. Migration: call `start()` again with the new filters (the documented way to update them).
+
+### Other changes
+
+- Fix SQL injection in the table metadata query: schema and table names from TableMap events are now bound via a cached prepared statement (`execute()`) instead of being spliced into SQL text
+- Replace the big-integer dependency with native BigInt (one fewer dependency); also fixes silent corruption of 64-bit integers inside JSON columns beyond 2^53, which now follow the same exact Number-or-string rule as BIGINT columns
+- Emit an error (once per instance per type) when the server sends undecodable TRANSACTION_PAYLOAD_EVENT (`binlog_transaction_compression=ON`) or PARTIAL_UPDATE_ROWS_EVENT (`binlog_row_value_options=PARTIAL_JSON`) events, instead of silently dropping the row changes; remaining MySQL 8 event codes (TRANSACTION_CONTEXT, VIEW_CHANGE, XA_PREPARE, HEARTBEAT_V2) are now named in the code map
+- Lifecycle hardening: emit an explicit error instead of hanging silently when the control connection dies during a metadata fetch; a duplicate `start()` while one is still initialising is ignored, while stop-then-restart during initialisation now works (exactly one binlog dump command is ever enqueued); errors from connections deliberately destroyed by `stop()` are no longer forwarded as teardown noise; errors buffered before an `error` listener attaches are thrown if no listener ever appears, restoring Node's default unhandled `'error'` behaviour
+- DECIMAL parsing no longer mutates the shared network packet buffer when flipping the sign bit
+- Fix `stop()` destroying the control connection of a subsequent `start()`: the asynchronous KILL cleanup now only touches the connections that particular `stop()` owned, so immediate stop-then-restart no longer wedges the new stream on its first metadata fetch
+- The `nonBlock` option declared in the TypeScript definitions is now actually passed through by `start()`; previously it was dropped and the dump command always ran in blocking mode
+- Remove dead code left over from the mysql.js protocol layer (ComBinlog, EofPacket/ErrorPacket, BufferReader)
+- Compile event and schema filters into Sets and Maps for O(1) per-event filtering; only own keys of schema filter objects are considered
+- Add a package.json `exports` map with `types` and `default` conditions
+- All emitted events carry an `event.gtid` property (`'uuid:sequence'`) identifying their transaction when the server runs with `gtid_mode=ON`, tracked at the packet layer so it works even when `gtid` events are excluded by `includeEvents`; `undefined` for anonymous transactions
+- Update mysql2 to ^3.22.5; the internal APIs zongji relies on (addCommand, handlePacket, packet sequence validation) were verified unchanged, and a new regression test covers the binlog stream over a compressed connection
+- Continuous integration now tests Node.js 22, 24 and 26 against MySQL 5.7, 8.0 and 8.4. Node.js 18 and 20 are end-of-life: they remain allowed by `engines` (nothing in the code requires anything newer) and 0.7.0 passed the full test suite on both at release time, but they are no longer tested and future releases may break on them
+
 ## [0.6.1] - 2026-02-13
 
 - Updated .gitignore and .npmignore to exclude AI tool and build/test files

package/README.md

@@ -9,9 +9,11 @@ MySQL binlog-based change data capture (CDC) for Node.js, [originally created by
 
 It leverages [`mysql2`](https://github.com/sidorares/node-mysql2) for connections and authentication, while using zongji's binlog parsing and event pipeline.
 
-# Latest Release
+# Release Notes
 
-Version 0.6.0 is a major modernisation that rewrites the codebase to use the mysql2 module, ES6 syntax and ESM exports, adds TypeScript definitions, and adds official support for MySQL 8.4.
+See the [CHANGELOG](CHANGELOG.md) for the full release history.
+
+Version 0.6.0 was a major modernisation that rewrote the codebase to use the mysql2 module, ES6 syntax and ESM exports, added TypeScript definitions, and added official support for MySQL 8.4.
 
 Version 0.5.9 is the last release that supports Node.js versions below 18 and CommonJS.
 
@@ -54,9 +56,17 @@ import ZongJi from '@vlasky/zongji';
 
 For a complete implementation see [`example.js`](example.js)...
 
+## Module format
+
+Since v0.6.0 this package is ESM-only. CommonJS projects can load it on Node.js >= 20.17 or >= 22.12, where `require()` of ES modules is supported, or on any supported version via dynamic `import()`:
+
+```javascript
+const { default: ZongJi } = await import('@vlasky/zongji');
+```
+
 ## Installation
 
-* Requires Node.js v18+
+* Requires Node.js v18+. Actively supported and tested on Node.js 22, 24 and 26; version 0.7.0 was also verified on Node.js 18 and 20 at release, but those lines are end-of-life and no longer tested.
 
   ```bash
   $ npm install @vlasky/zongji
@@ -93,7 +103,14 @@ The `ZongJi` constructor accepts one argument of either:
 
 If a `Connection` or `Pool` object is passed to the constructor, it will not be destroyed/ended by Zongji's `stop()` method.
 
-The `dateStrings` and `timezone` configuration options from the connection details are respected for date/time value handling.
+Binlog row values follow the same [mysql2 connection options](https://sidorares.github.io/node-mysql2/docs/api-and-configurations) as query results, so CDC events and queries on the same connection agree:
+
+Option | Effect on row values
+-------|---------------------
+`dateStrings` | `DATE`, `DATETIME` and `TIMESTAMP` columns are returned as strings instead of `Date` objects.
+`timezone` | Applied when converting `DATETIME` and `TIMESTAMP` values to `Date` objects.
+`decimalNumbers` | `DECIMAL` columns are returned as exact strings by default (e.g. `'-123.4500'`); set `decimalNumbers: true` to receive Numbers (may lose precision beyond 15 significant digits).
+`jsonStrings` | `JSON` columns are returned as parsed JavaScript values by default; set `jsonStrings: true` to receive JSON strings.
 
 Each instance includes the following methods:
 
@@ -112,6 +129,8 @@ Event Name | Description
 `error`    | Every error will be caught by this event.
 `stopped`  | Emitted when ZongJi connection is stopped (ZongJi#stop is called).
 
+Always attach an `error` listener. Errors that occur before a listener attaches (for example, a connection failure in the same tick as construction) are buffered and re-delivered to the first `error` listener. If no listener is ever attached, the buffered errors are thrown, following Node's default behaviour for unhandled `'error'` events.
+
 **Options available:**
 
 Option Name | Type | Description
@@ -127,6 +146,8 @@ Option Name | Type | Description
 
 * By default, all events and schema are emitted.
 * `excludeSchema` and `excludeEvents` take precedence over `includeSchema` and `includeEvents`, respectively.
+* Calling `start()` while a previous `start()` is still initialising is ignored and the first call completes. The exception is after an intervening `stop()`: the new `start()` then restarts cleanly, and exactly one binlog stream is opened.
+* Calling `start()` while ZongJi is already running does not reconnect; it only updates the event and schema filters from the given options.
 
 **Supported Binlog Events:**
 
@@ -145,6 +166,10 @@ Event name  | Description
 `writerows` | Rows inserted, row data array available as `rows` property on event object
 `updaterows` | Rows changed, row data array available as `rows` property on event object
 `deleterows` | Rows deleted, row data array available as `rows` property on event object
+`transactionpayload` | Compressed transaction from MySQL 8.0.20+ servers with `binlog_transaction_compression=ON`. ZongJi cannot decode the row events inside it, so it emits an `error` (once per instance) naming the server setting responsible.
+`partialupdaterows` | Partial JSON update from MySQL 8.0+ servers with `binlog_row_value_options=PARTIAL_JSON`. ZongJi cannot decode the JSON diff format, so it emits an `error` (once per instance) naming the server setting responsible.
+
+When the server runs with `gtid_mode=ON`, every emitted event carries a `gtid` property (`'uuid:sequence'`) identifying the transaction it belongs to, tracked internally even if `gtid` events are excluded by `includeEvents`. It is `undefined` for anonymous transactions and for events seen before the stream's first GTID event. Note that non-row events arriving between a transaction's commit and the next transaction's GTID event (e.g. a rotate) still carry the previous transaction's `gtid`. Use it for checkpointing and deduplication.
 
 **Event Methods**
 
@@ -158,7 +183,7 @@ Name   | Description
 ## Important Notes
 
 * :star2: All MySQL column types are supported, with type casting similar to [mysql2](https://github.com/sidorares/node-mysql2).
-* :speak_no_evil: 64-bit integer is supported via package big-integer(see #108). If an integer is within the safe range of JS number (-2^53, 2^53), a Number object will returned, otherwise, will return as String.
+* :speak_no_evil: 64-bit integers are decoded exactly using native BigInt (see #108). If an integer is within the safe range of JS numbers (-2^53, 2^53), a Number is returned, otherwise an exact String. This also applies to 64-bit integers inside JSON columns.
 * :point_right: `TRUNCATE` statement does not cause corresponding `DeleteRows` event. Use unqualified `DELETE FROM` for same effect.
 * When using fractional seconds with `DATETIME` and `TIMESTAMP` data types, only millisecond precision is available due to the limit of Javascript's `Date` object.
 * Binlog checksums (e.g. `CRC32`) are supported; zongji will detect and ignore the checksum bytes at the end of row events.

package/example.js

@@ -12,6 +12,11 @@ zongji.on('binlog', function(evt) {
   evt.dump();
 });
 
+// Always attach an error listener; without one, errors are thrown
+zongji.on('error', function(err) {
+  console.error('zongji error:', err);
+});
+
 zongji.start({
   includeEvents: ['tablemap', 'writerows', 'updaterows', 'deleterows']
 });

package/index.d.ts

@@ -56,6 +56,13 @@ export interface BinlogEvent {
   nextPosition: number;
   /** Size of the event in bytes */
   size: number;
+  /**
+   * GTID of the transaction this event belongs to ('uuid:sequence'),
+   * when the server runs with gtid_mode=ON. Undefined for anonymous
+   * transactions and for events seen before the first GTID event of the
+   * stream. On Gtid/AnonymousGtid events this is their own parsed value.
+   */
+  gtid?: string;
   /** Get the lowercase event name */
   getEventName(): string;
   /** Get the event class name */
@@ -66,6 +73,8 @@ export interface BinlogEvent {
 
 // Rotate event - indicates binlog file change
 export interface RotateEvent extends BinlogEvent {
+  getTypeName(): 'Rotate';
+  getEventName(): 'rotate';
   /** Position in the new binlog file */
   position: number;
   /** Name of the new binlog file */
@@ -73,10 +82,15 @@ export interface RotateEvent extends BinlogEvent {
 }
 
 // Format Description event
-export interface FormatEvent extends BinlogEvent {}
+export interface FormatEvent extends BinlogEvent {
+  getTypeName(): 'Format';
+  getEventName(): 'format';
+}
 
 // GTID event
 export interface GtidEvent extends BinlogEvent {
+  getTypeName(): 'Gtid';
+  getEventName(): 'gtid';
   /** GTID flags */
   flags: number;
   /** Source UUID */
@@ -89,6 +103,8 @@ export interface GtidEvent extends BinlogEvent {
 
 // Anonymous GTID event
 export interface AnonymousGtidEvent extends BinlogEvent {
+  getTypeName(): 'AnonymousGtid';
+  getEventName(): 'anonymousgtid';
   /** GTID flags */
   flags: number;
   /** Source UUID */
@@ -111,6 +127,8 @@ export interface GtidSidEntry {
 }
 
 export interface PreviousGtidsEvent extends BinlogEvent {
+  getTypeName(): 'PreviousGtids';
+  getEventName(): 'previousgtids';
   /** Array of SID entries with their intervals */
   sids: GtidSidEntry[];
   /** GTID set as a formatted string */
@@ -119,12 +137,16 @@ export interface PreviousGtidsEvent extends BinlogEvent {
 
 // XID (commit) event
 export interface XidEvent extends BinlogEvent {
+  getTypeName(): 'Xid';
+  getEventName(): 'xid';
   /** Transaction ID for 2PC */
   xid: number | string;
 }
 
 // Query event
 export interface QueryEvent extends BinlogEvent {
+  getTypeName(): 'Query';
+  getEventName(): 'query';
   /** Slave proxy ID */
   slaveProxyId: number;
   /** Time in seconds the query took to execute */
@@ -145,6 +167,8 @@ export interface QueryEvent extends BinlogEvent {
 
 // IntVar event (for statement-based replication)
 export interface IntVarEvent extends BinlogEvent {
+  getTypeName(): 'IntVar';
+  getEventName(): 'intvar';
   /** Variable type: 1=LAST_INSERT_ID, 2=INSERT_ID */
   type: number;
   /** The integer value */
@@ -155,6 +179,8 @@ export interface IntVarEvent extends BinlogEvent {
 
 // TableMap event
 export interface TableMapEvent extends BinlogEvent {
+  getTypeName(): 'TableMap';
+  getEventName(): 'tablemap';
   /** Internal table ID */
   tableId: number;
   /** Table flags */
@@ -200,24 +226,47 @@ export interface RowsEvent extends BinlogEvent {
 
 // WriteRows event (INSERT)
 export interface WriteRowsEvent extends RowsEvent {
+  getTypeName(): 'WriteRows';
+  getEventName(): 'writerows';
   /** Array of inserted rows */
   rows: RowData[];
 }
 
 // DeleteRows event (DELETE)
 export interface DeleteRowsEvent extends RowsEvent {
+  getTypeName(): 'DeleteRows';
+  getEventName(): 'deleterows';
   /** Array of deleted rows */
   rows: RowData[];
 }
 
 // UpdateRows event (UPDATE)
 export interface UpdateRowsEvent extends RowsEvent {
+  getTypeName(): 'UpdateRows';
+  getEventName(): 'updaterows';
   /** Array of updated rows with before/after values */
   rows: UpdateRowData[];
 }
 
+// Transaction Payload event (MySQL 8.0.20+, binlog_transaction_compression=ON)
+// The compressed transaction body cannot be decoded and is skipped.
+export interface TransactionPayloadEvent extends BinlogEvent {
+  getTypeName(): 'TransactionPayload';
+  getEventName(): 'transactionpayload';
+}
+
+// Partial Update Rows event (MySQL 8.0+, binlog_row_value_options=PARTIAL_JSON)
+// The partial JSON diff body cannot be decoded and is skipped.
+export interface PartialUpdateRowsEvent extends BinlogEvent {
+  getTypeName(): 'PartialUpdateRows';
+  getEventName(): 'partialupdaterows';
+}
+
 // Unknown event type
-export interface UnknownEvent extends BinlogEvent {}
+export interface UnknownEvent extends BinlogEvent {
+  getTypeName(): 'Unknown';
+  getEventName(): 'unknown';
+}
 
 // Union type of all event types
 export type AnyBinlogEvent =
@@ -233,8 +282,40 @@ export type AnyBinlogEvent =
   | WriteRowsEvent
   | DeleteRowsEvent
   | UpdateRowsEvent
+  | TransactionPayloadEvent
+  | PartialUpdateRowsEvent
   | UnknownEvent;
 
+/** Union of all possible getTypeName() return values (e.g. 'Rotate', 'WriteRows') */
+export type BinlogEventTypeName = ReturnType<AnyBinlogEvent['getTypeName']>;
+
+/** Union of all possible getEventName() return values (e.g. 'rotate', 'writerows') */
+export type BinlogEventName = ReturnType<AnyBinlogEvent['getEventName']>;
+
+/**
+ * Look up the concrete event type for a getTypeName() value.
+ * Enables narrowing AnyBinlogEvent via getTypeName(), e.g.:
+ *
+ *   function isEventType<N extends BinlogEventTypeName>(
+ *     event: AnyBinlogEvent, name: N
+ *   ): event is BinlogEventByTypeName<N> {
+ *     return event.getTypeName() === name;
+ *   }
+ *
+ * (A plain `switch (event.getTypeName())` cannot narrow `event` because
+ * TypeScript does not narrow unions on method-call discriminants.)
+ */
+export type BinlogEventByTypeName<N extends BinlogEventTypeName> = Extract<
+  AnyBinlogEvent,
+  { getTypeName(): N }
+>;
+
+/** Look up the concrete event type for a getEventName() value */
+export type BinlogEventByName<N extends BinlogEventName> = Extract<
+  AnyBinlogEvent,
+  { getEventName(): N }
+>;
+
 // Connection options - can be mysql2 options, Connection, or Pool
 export type ZongJiDsn = string | ConnectionOptions | PoolOptions | Connection | Pool;
 
@@ -254,6 +335,7 @@ declare class ZongJi extends EventEmitter {
     filename?: string;
     position?: number;
     startAtEnd?: boolean;
+    nonBlock?: boolean;
   };
   /** Current filter settings */
   filters: {