@vlasky/zongji 0.5.9 → 0.6.1

package/CHANGELOG.md

@@ -0,0 +1,60 @@
+# Changelog
+
+All notable changes to @vlasky/zongji since forking from nevill/zongji.
+
+## [0.6.1] - 2026-02-13
+
+- Updated .gitignore and .npmignore to exclude AI tool and build/test files
+- Added npm version, downloads, node version, and licence badges to README
+
+## [0.6.0] - 2026-02-13
+
+- Migrate from @vlasky/mysql to mysql2
+- Convert codebase to ES modules
+- Add TypeScript definitions
+- Add official support for MySQL 8.4
+- Fix sequence ID warnings when using compression with binlog streams
+- Fix connection cleanup in stop() to prevent reuse of destroyed connections
+- Add stopped flag to support dynamic filter updates and safe stop during init
+- Fix flaky error test by handling all error events instead of just the first
+
+## [0.5.9] - 2023-01-11
+
+- Allow BLOB columns with utf8mb3 charset
+
+## [0.5.8] - 2021-09-19
+
+- Internal version bump
+
+## [0.5.7] - 2021-07-08
+
+- Fix connection when binlog_checksum is NONE
+
+## [0.5.6] - 2021-04-30
+
+- Update to @vlasky/mysql 2.18.5 with keepalive probe packet support
+
+## [0.5.5] - 2021-04-17
+
+- Update to @vlasky/mysql 2.18.4 to support additional charset collations in MySQL 8
+
+## [0.5.4] - 2021-04-17
+
+- Update to @vlasky/mysql 2.18.3 to support caching_sha2_password authentication plugin (MySQL 8 default)
+
+## [0.5.3] - 2021-03-24
+
+- Update to @vlasky/mysql 2.18.2 to support new MySQL 8 error codes
+- Handle table map events that change table IDs (from YousefED)
+- Fix null value in JSON column causing buffer RangeError (from YousefED)
+- MySQL 8 compatibility fix for column mapping query order
+
+## [0.5.2] - 2020-11-11
+
+- Fix IEEE754 conversion error using DataView (from jefbarn)
+- Update dependencies
+
+## [0.5.1] - 2019-11-09
+
+- Initial fork from nevill/zongji
+- Add binlog_row_image support

package/README.md

@@ -1,23 +1,26 @@
-A MySQL 8.0-compatible fork of ZongJi - a MySQL binlog listener for Node.js, [originally created by Nevill Dutt](https://github.com/nevill/zongji).
+[![npm version](https://img.shields.io/npm/v/@vlasky/zongji.svg)](https://www.npmjs.com/package/@vlasky/zongji)
+[![npm downloads](https://img.shields.io/npm/dm/@vlasky/zongji.svg)](https://www.npmjs.com/package/@vlasky/zongji)
+[![node version](https://img.shields.io/node/v/@vlasky/zongji.svg)](https://www.npmjs.com/package/@vlasky/zongji)
+[![license](https://img.shields.io/npm/l/@vlasky/zongji.svg)](https://github.com/vlasky/zongji/blob/master/LICENSE)
 
-[@vlasky/zongji](https://github.com/vlasky/zongji) has been tested working with MySQL 5.5, 5.6, 5.7 and 8.0.
+MySQL binlog-based change data capture (CDC) for Node.js, [originally created by Nevill Dutt](https://github.com/nevill/zongji).
 
-It leverages [`@vlasky/mysql`](https://github.com/vlasky/mysql), a fork of [`mysql`](https://github.com/mysqljs/mysql) with the following enhancements:
+[@vlasky/zongji](https://github.com/vlasky/zongji) has been tested working with MySQL 5.7, 8.0 and 8.4.
 
-* Support for authentication using the caching_sha2_password plugin, the new default authentication method in MySQL 8.0
-* Partial support for the MySQL compressed protocol (reads compressed data sent by server)
-* Optional sending of keepalive probe packets to check the state of the connection to the MySQL server and help keep the connection open when the network socket is idle
+It leverages [`mysql2`](https://github.com/sidorares/node-mysql2) for connections and authentication, while using zongji's binlog parsing and event pipeline.
 
 # Latest Release
 
-ZongJi release versions since 0.5.0 only support Node.js version 8 and above.
+Version 0.6.0 is a major modernisation that rewrites the codebase to use the mysql2 module, ES6 syntax and ESM exports, adds TypeScript definitions, and adds official support for MySQL 8.4.
 
-Version 0.4.7 is the last release that supports Node.js version 4.x.
+Version 0.5.9 is the last release that supports Node.js versions below 18 and CommonJS.
 
 ## Quick Start
 
 ```javascript
-let zongji = new ZongJi({ /* ... MySQL Connection Settings ... */ });
+import ZongJi from '@vlasky/zongji';
+
+const zongji = new ZongJi({ /* ... MySQL Connection Settings ... */ });
 
 // Each change to the replication log results in an event
 zongji.on('binlog', function(evt) {
@@ -30,18 +33,37 @@ zongji.start({
 });
 ```
 
+### GTID example
+
+```javascript
+zongji.on('binlog', function(evt) {
+  const type = evt.getTypeName();
+  if (type === 'Gtid' || type === 'AnonymousGtid') {
+    console.log('GTID:', evt.gtid, 'SID:', evt.sid, 'GNO:', evt.gno);
+  }
+});
+```
+
+### TypeScript
+
+TypeScript definitions are included:
+
+```typescript
+import ZongJi from '@vlasky/zongji';
+```
+
 For a complete implementation see [`example.js`](example.js)...
 
 ## Installation
 
-* Requires Node.js v8+
+* Requires Node.js v18+
 
   ```bash
   $ npm install @vlasky/zongji
   ```
 
 * Enable MySQL binlog in `my.cnf`, restart MySQL server after making the changes.
-  > From [MySQL 5.6](https://dev.mysql.com/doc/refman/5.6/en/replication-options-binary-log.html), binlog checksum is enabled by default. Zongji can work with it, but it doesn't really verify it.
+  > Binlog checksum is enabled by default in all supported MySQL versions. Zongji can work with it, but it doesn't verify the checksum.
 
   ```
   # Must be unique integer from 1-2^32
@@ -66,12 +88,12 @@ For a complete implementation see [`example.js`](example.js)...
 
 The `ZongJi` constructor accepts one argument of either:
 
-* An object containing MySQL connection details in the same format as used by [package mysql](https://npm.im/mysql)
-* Or, a [mysql](https://npm.im/mysql) `Connection` or `Pool` object that will be used for querying column information.
+* An object containing MySQL connection details in the same format as used by [package mysql2](https://npm.im/mysql2)
+* Or, a [mysql2](https://npm.im/mysql2) `Connection` or `Pool` object that will be used for querying column information.
 
 If a `Connection` or `Pool` object is passed to the constructor, it will not be destroyed/ended by Zongji's `stop()` method.
 
-If there is a `dateStrings` `mysql` configuration option in the connection details or connection, `ZongJi` will follow it.
+The `dateStrings` and `timezone` configuration options from the connection details are respected for date/time value handling.
 
 Each instance includes the following methods:
 
@@ -85,7 +107,7 @@ Some events can be emitted in different phases:
 
 Event Name | Description
 -----------|------------------------
-`ready`    | This event is occurred right after ZongJi successfully established a connection, setup slave status, and set binlog position.
+`ready`    | This event occurs right after ZongJi successfully establishes a connection, sets up replica (slave) status, and sets the binlog position.
 `binlog`   | Once a binlog is received and passes the filter, it will bubble up with this event.
 `error`    | Every error will be caught by this event.
 `stopped`  | Emitted when ZongJi connection is stopped (ZongJi#stop is called).
@@ -116,6 +138,9 @@ Event name  | Description
 `rotate`    | [New Binlog file](https://dev.mysql.com/doc/internals/en/rotate-event.html) Not required to be included to rotate to new files, but it is required to be included in order to keep the `filename` and `position` properties updated with current values for [graceful restarting on errors](https://gist.github.com/numtel/5b37b2a7f47b380c1a099596c6f3db2f).
 `format`    | [Format Description](https://dev.mysql.com/doc/internals/en/format-description-event.html)
 `xid`       | [Transaction ID](https://dev.mysql.com/doc/internals/en/xid-event.html)
+`gtid`      | GTID event with `gtid`, `sid`, `gno` properties
+`anonymousgtid` | Anonymous GTID event (same shape as `gtid`)
+`previousgtids` | Previous GTIDs event with `gtidSet` and `sids`
 `tablemap`  | Before any row event (must be included for any other row events)
 `writerows` | Rows inserted, row data array available as `rows` property on event object
 `updaterows` | Rows changed, row data array available as `rows` property on event object
@@ -132,20 +157,23 @@ Name   | Description
 
 ## Important Notes
 
-* :star2: [All types allowed by `mysql`](https://github.com/mysqljs/mysql#type-casting) are supported by this package.
+* :star2: All MySQL column types are supported, with type casting similar to [mysql2](https://github.com/sidorares/node-mysql2).
 * :speak_no_evil: 64-bit integer is supported via package big-integer(see #108). If an integer is within the safe range of JS number (-2^53, 2^53), a Number object will returned, otherwise, will return as String.
 * :point_right: `TRUNCATE` statement does not cause corresponding `DeleteRows` event. Use unqualified `DELETE FROM` for same effect.
-* When using fractional seconds with `DATETIME` and `TIMESTAMP` data types in MySQL > 5.6.4, only millisecond precision is available due to the limit of Javascript's `Date` object.
+* When using fractional seconds with `DATETIME` and `TIMESTAMP` data types, only millisecond precision is available due to the limit of Javascript's `Date` object.
+* Binlog checksums (e.g. `CRC32`) are supported; zongji will detect and ignore the checksum bytes at the end of row events.
 
 ## Run Tests
 
-* install [Docker](https://www.docker.com/community-edition#download)
-* run `docker-compose up` and then `./docker-test.sh`
+* Install [Docker](https://www.docker.com/community-edition#download)
+* Run `docker-compose up -d` to start MySQL containers
+* Run `npm test` to execute the test suite
 
 ## References
 
 The following resources provided valuable information that greatly assisted in creating ZongJi:
 
+* https://github.com/sidorares/node-mysql2
 * https://github.com/mysqljs/mysql
 * https://github.com/felixge/faster-than-c/
 * https://web.archive.org/web/20130117004733/https://intuitive-search.blogspot.co.uk/2011/07/binary-log-api-and-replication-listener.html
@@ -153,7 +181,7 @@ The following resources provided valuable information that greatly assisted in c
 * https://kkaefer.com/node-cpp-modules/
 * https://dev.mysql.com/doc/internals/en/replication-protocol.html
 * https://web.archive.org/web/20200201195450/https://www.cs.wichita.edu/~chang/lecture/cs742/program/how-mysql-c-api.html
-* https://github.com/jeremycole/mysql_binlog (Ruby implemenation of MySQL binlog parser)
+* https://github.com/jeremycole/mysql_binlog (Ruby implementation of MySQL binlog parser)
 * https://dev.mysql.com/doc/internals/en/date-and-time-data-type-representation.html
 
 ## License

package/docker-compose.yml

@@ -1,32 +1,33 @@
 ---
-version: '2'
 services:
-  mysql55:
-    image: mysql:5.5
+  mysql57:
+    image: mysql:5.7
     command: [ "--server-id=1", "--log-bin=/var/lib/mysql/mysql-bin.log", "--binlog-format=row"]
     networks:
       default:
         aliases:
-          - mysql55
+          - mysql57
     environment:
-      MYSQL_ROOT_PASSWORD: numtel
+      MYSQL_ROOT_PASSWORD: secret
 
-  mysql56:
-    image: mysql:5.6
+  mysql80:
+    image: mysql:8.0
     command: [ "--server-id=1", "--log-bin=/var/lib/mysql/mysql-bin.log", "--binlog-format=row"]
     networks:
       default:
         aliases:
-          - mysql56
+          - mysql80
     environment:
-      MYSQL_ROOT_PASSWORD: numtel
+      MYSQL_ROOT_PASSWORD: secret
 
-  mysql57:
-    image: mysql:5.7
+  mysql84:
+    image: mysql:8.4
     command: [ "--server-id=1", "--log-bin=/var/lib/mysql/mysql-bin.log", "--binlog-format=row"]
+    ports:
+      - "3306:3306"
     networks:
       default:
         aliases:
-          - mysql57
+          - mysql84
     environment:
-      MYSQL_ROOT_PASSWORD: numtel
+      MYSQL_ROOT_PASSWORD: secret

package/docker-test.sh

@@ -1,11 +1,11 @@
 #!/bin/bash
-MYSQL_HOSTS="mysql55 mysql56 mysql57"
+MYSQL_HOSTS="mysql57 mysql80 mysql84"
 
 for hostname in ${MYSQL_HOSTS}; do
-  echo $hostname + node 8
-  docker run -it --network=zongji_default -e MYSQL_HOST=$hostname -w /build -v $PWD:/build node:8 npm test
-  echo $hostname + node 10
-  docker run -it --network=zongji_default -e MYSQL_HOST=$hostname -w /build -v $PWD:/build node:10 npm test
-  echo $hostname + node 12
-  docker run -it --network=zongji_default -e MYSQL_HOST=$hostname -w /build -v $PWD:/build node:12 npm test
+  echo $hostname + node 18
+  docker run -it --network=zongji_default -e MYSQL_HOST=$hostname -w /build -v $PWD:/build node:18 npm test
+  echo $hostname + node 20
+  docker run -it --network=zongji_default -e MYSQL_HOST=$hostname -w /build -v $PWD:/build node:20 npm test
+  echo $hostname + node 22
+  docker run -it --network=zongji_default -e MYSQL_HOST=$hostname -w /build -v $PWD:/build node:22 npm test
 done

package/eslint.config.js

@@ -0,0 +1,35 @@
+import js from '@eslint/js';
+
+export default [
+  { ignores: ['.tap/'] },
+  js.configs.recommended,
+  {
+    languageOptions: {
+      ecmaVersion: 2022,
+      sourceType: 'module',
+      globals: {
+        Buffer: 'readonly',
+        clearImmediate: 'readonly',
+        clearInterval: 'readonly',
+        clearTimeout: 'readonly',
+        console: 'readonly',
+        process: 'readonly',
+        setImmediate: 'readonly',
+        setInterval: 'readonly',
+        setTimeout: 'readonly',
+      },
+    },
+    rules: {
+      'comma-dangle': ['warn', 'only-multiline'],
+      'eol-last': ['error'],
+      'keyword-spacing': ['error', { before: true }],
+      'no-console': 'off',
+      'no-trailing-spaces': ['error', { skipBlankLines: true }],
+      'no-unused-vars': 'warn',
+      'no-var': 'warn',
+      'quotes': ['warn', 'single', 'avoid-escape'],
+      'semi': ['error', 'always'],
+      'space-before-blocks': 'error',
+    },
+  },
+];

package/example.js

@@ -1,5 +1,5 @@
 // Client code
-const ZongJi = require('./');
+import ZongJi from './index.js';
 
 const zongji = new ZongJi({
   host     : 'localhost',

package/index.d.ts

@@ -0,0 +1,314 @@
+import { EventEmitter } from 'events';
+import { Connection, Pool, ConnectionOptions, PoolOptions } from 'mysql2';
+
+// Configuration options for ZongJi
+export interface ZongJiOptions {
+  /** Unique server ID for the binlog connection (default: 1) */
+  serverId?: number;
+  /** If true, start reading from the end of the binlog */
+  startAtEnd?: boolean;
+  /** Binlog filename to start from */
+  filename?: string;
+  /** Binlog position to start from */
+  position?: number;
+  /** If true, use non-blocking mode */
+  nonBlock?: boolean;
+  /** List of event types to include (e.g., ['tablemap', 'writerows']) */
+  includeEvents?: string[];
+  /** List of event types to exclude */
+  excludeEvents?: string[];
+  /** Schema/table filter for inclusion. Use `true` for all tables in a schema, or array of table names */
+  includeSchema?: Record<string, string[] | true>;
+  /** Schema/table filter for exclusion */
+  excludeSchema?: Record<string, string[] | true>;
+}
+
+// Column metadata from INFORMATION_SCHEMA
+export interface ColumnSchema {
+  COLUMN_NAME: string;
+  COLLATION_NAME: string | null;
+  CHARACTER_SET_NAME: string | null;
+  COLUMN_COMMENT: string;
+  COLUMN_TYPE: string;
+}
+
+// Column information from TableMap event
+export interface ColumnInfo {
+  name: string;
+  charset: string | null;
+  type: number;
+  metadata: Record<string, unknown>;
+}
+
+// Table metadata cached by ZongJi
+export interface TableMapEntry {
+  columnSchemas: ColumnSchema[];
+  parentSchema: string;
+  tableName: string;
+  columns?: ColumnInfo[];
+}
+
+// Base binlog event class
+export interface BinlogEvent {
+  /** Timestamp of the event in milliseconds */
+  timestamp: number;
+  /** Position in the binlog after this event */
+  nextPosition: number;
+  /** Size of the event in bytes */
+  size: number;
+  /** Get the lowercase event name */
+  getEventName(): string;
+  /** Get the event class name */
+  getTypeName(): string;
+  /** Print event details to console */
+  dump(): void;
+}
+
+// Rotate event - indicates binlog file change
+export interface RotateEvent extends BinlogEvent {
+  /** Position in the new binlog file */
+  position: number;
+  /** Name of the new binlog file */
+  binlogName: string;
+}
+
+// Format Description event
+export interface FormatEvent extends BinlogEvent {}
+
+// GTID event
+export interface GtidEvent extends BinlogEvent {
+  /** GTID flags */
+  flags: number;
+  /** Source UUID */
+  sid: string;
+  /** Group number */
+  gno: number;
+  /** Full GTID string (sid:gno) */
+  gtid: string;
+}
+
+// Anonymous GTID event
+export interface AnonymousGtidEvent extends BinlogEvent {
+  /** GTID flags */
+  flags: number;
+  /** Source UUID */
+  sid: string;
+  /** Group number */
+  gno: number;
+  /** Full GTID string (sid:gno) */
+  gtid: string;
+}
+
+// Previous GTIDs event
+export interface GtidInterval {
+  start: number;
+  end: number;
+}
+
+export interface GtidSidEntry {
+  sid: string;
+  intervals: GtidInterval[];
+}
+
+export interface PreviousGtidsEvent extends BinlogEvent {
+  /** Array of SID entries with their intervals */
+  sids: GtidSidEntry[];
+  /** GTID set as a formatted string */
+  gtidSet: string;
+}
+
+// XID (commit) event
+export interface XidEvent extends BinlogEvent {
+  /** Transaction ID for 2PC */
+  xid: number | string;
+}
+
+// Query event
+export interface QueryEvent extends BinlogEvent {
+  /** Slave proxy ID */
+  slaveProxyId: number;
+  /** Time in seconds the query took to execute */
+  executionTime: number;
+  /** Length of the schema name */
+  schemaLength: number;
+  /** Error code (0 for success) */
+  errorCode: number;
+  /** Length of status variables */
+  statusVarsLength: number;
+  /** Status variables */
+  statusVars: string;
+  /** Database/schema name */
+  schema: string;
+  /** The SQL query */
+  query: string;
+}
+
+// IntVar event (for statement-based replication)
+export interface IntVarEvent extends BinlogEvent {
+  /** Variable type: 1=LAST_INSERT_ID, 2=INSERT_ID */
+  type: number;
+  /** The integer value */
+  value: number | string;
+  /** Get the variable type name */
+  getIntTypeName(): string;
+}
+
+// TableMap event
+export interface TableMapEvent extends BinlogEvent {
+  /** Internal table ID */
+  tableId: number;
+  /** Table flags */
+  flags: number;
+  /** Database/schema name */
+  schemaName: string;
+  /** Table name */
+  tableName: string;
+  /** Number of columns */
+  columnCount: number;
+  /** Array of MySQL type codes for each column */
+  columnTypes: number[];
+  /** Column metadata */
+  columnsMetadata: Record<string, unknown>[];
+  /** Reference to the tableMap cache */
+  tableMap: Record<number, TableMapEntry>;
+  /** Update column info after fetching from INFORMATION_SCHEMA */
+  updateColumnInfo(): void;
+}
+
+// Row data type - key is column name, value is column value
+export type RowData = Record<string, unknown>;
+
+// Update row with before/after values
+export interface UpdateRowData {
+  before: RowData;
+  after: RowData;
+}
+
+// Base rows event
+export interface RowsEvent extends BinlogEvent {
+  /** Internal table ID */
+  tableId: number;
+  /** Event flags */
+  flags: number;
+  /** Number of columns */
+  numberOfColumns: number;
+  /** Reference to the tableMap cache */
+  tableMap: Record<number, TableMapEntry>;
+  /** Whether checksum is enabled */
+  useChecksum: boolean;
+}
+
+// WriteRows event (INSERT)
+export interface WriteRowsEvent extends RowsEvent {
+  /** Array of inserted rows */
+  rows: RowData[];
+}
+
+// DeleteRows event (DELETE)
+export interface DeleteRowsEvent extends RowsEvent {
+  /** Array of deleted rows */
+  rows: RowData[];
+}
+
+// UpdateRows event (UPDATE)
+export interface UpdateRowsEvent extends RowsEvent {
+  /** Array of updated rows with before/after values */
+  rows: UpdateRowData[];
+}
+
+// Unknown event type
+export interface UnknownEvent extends BinlogEvent {}
+
+// Union type of all event types
+export type AnyBinlogEvent =
+  | RotateEvent
+  | FormatEvent
+  | GtidEvent
+  | AnonymousGtidEvent
+  | PreviousGtidsEvent
+  | XidEvent
+  | QueryEvent
+  | IntVarEvent
+  | TableMapEvent
+  | WriteRowsEvent
+  | DeleteRowsEvent
+  | UpdateRowsEvent
+  | UnknownEvent;
+
+// Connection options - can be mysql2 options, Connection, or Pool
+export type ZongJiDsn = string | ConnectionOptions | PoolOptions | Connection | Pool;
+
+// Main ZongJi class
+declare class ZongJi extends EventEmitter {
+  /** Cached table metadata */
+  tableMap: Record<number, TableMapEntry>;
+  /** Whether ZongJi is ready to receive events */
+  ready: boolean;
+  /** Whether ZongJi has been stopped */
+  stopped: boolean;
+  /** Whether binlog checksum is enabled */
+  useChecksum: boolean;
+  /** Current binlog options */
+  options: {
+    serverId?: number;
+    filename?: string;
+    position?: number;
+    startAtEnd?: boolean;
+  };
+  /** Current filter settings */
+  filters: {
+    includeEvents?: string[];
+    excludeEvents?: string[];
+    includeSchema?: Record<string, string[] | true>;
+    excludeSchema?: Record<string, string[] | true>;
+  };
+  /** The control connection (for metadata queries) */
+  ctrlConnection: Connection | null;
+  /** The binlog streaming connection */
+  connection: Connection | null;
+
+  /**
+   * Create a new ZongJi instance
+   * @param dsn - Connection string, options object, or existing Connection/Pool
+   */
+  constructor(dsn: ZongJiDsn);
+
+  /**
+   * Start listening to the binlog
+   * @param options - Configuration options
+   */
+  start(options?: ZongJiOptions): void;
+
+  /**
+   * Stop listening to the binlog
+   */
+  stop(): void;
+
+  /**
+   * Get current option value(s)
+   * @param name - Option name or array of names
+   */
+  get(name: string): unknown;
+  get(name: string[]): Record<string, unknown>;
+
+  // Event emitter overloads for type safety
+  on(event: 'binlog', listener: (event: AnyBinlogEvent) => void): this;
+  on(event: 'ready', listener: () => void): this;
+  on(event: 'stopped', listener: () => void): this;
+  on(event: 'error', listener: (error: Error) => void): this;
+  on(event: string, listener: (...args: unknown[]) => void): this;
+
+  once(event: 'binlog', listener: (event: AnyBinlogEvent) => void): this;
+  once(event: 'ready', listener: () => void): this;
+  once(event: 'stopped', listener: () => void): this;
+  once(event: 'error', listener: (error: Error) => void): this;
+  once(event: string, listener: (...args: unknown[]) => void): this;
+
+  emit(event: 'binlog', binlogEvent: AnyBinlogEvent): boolean;
+  emit(event: 'ready'): boolean;
+  emit(event: 'stopped'): boolean;
+  emit(event: 'error', error: Error): boolean;
+  emit(event: string, ...args: unknown[]): boolean;
+}
+
+export default ZongJi;