pgsync 6.0.0__tar.gz → 6.1.0__tar.gz

{pgsync-6.0.0 → pgsync-6.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pgsync
-Version: 6.0.0
+Version: 6.1.0
 Summary: Postgres/MySQL/MariaDB to Elasticsearch/OpenSearch sync
 Home-page: https://github.com/toluaina/pgsync
 Author: Tolu Aina
@@ -33,9 +33,9 @@ License-File: LICENSE
 License-File: AUTHORS.rst
 Requires-Dist: async-timeout==5.0.1
 Requires-Dist: backports-datetime-fromisoformat==2.0.3
-Requires-Dist: boto3==1.40.64
-Requires-Dist: botocore==1.40.64
-Requires-Dist: certifi==2025.10.5
+Requires-Dist: boto3==1.41.2
+Requires-Dist: botocore==1.41.2
+Requires-Dist: certifi==2025.11.12
 Requires-Dist: charset-normalizer==3.4.4
 Requires-Dist: click==8.1.8
 Requires-Dist: elastic-transport==9.1.0
@@ -46,7 +46,7 @@ Requires-Dist: events==0.5
 Requires-Dist: idna==3.11
 Requires-Dist: jmespath==1.0.1
 Requires-Dist: marshmallow==4.0.1
-Requires-Dist: mysql-replication==1.0.9
+Requires-Dist: mysql-replication==1.0.12
 Requires-Dist: opensearch-dsl==2.1.0
 Requires-Dist: opensearch-py==3.0.0
 Requires-Dist: packaging==25.0
@@ -57,7 +57,7 @@ Requires-Dist: python-dotenv==1.2.1
 Requires-Dist: redis==7.0.1
 Requires-Dist: requests==2.32.5
 Requires-Dist: requests-aws4auth==1.3.1
-Requires-Dist: s3transfer==0.14.0
+Requires-Dist: s3transfer==0.15.0
 Requires-Dist: six==1.17.0
 Requires-Dist: sqlalchemy==2.0.44
 Requires-Dist: sqlparse==0.5.3
@@ -79,7 +79,7 @@ Dynamic: requires-dist
 Dynamic: requires-python
 Dynamic: summary
 
-# PostgreSQL to Elasticsearch/OpenSearch sync
+# PostgreSQL/MySQL/MariaDB to Elasticsearch/OpenSearch sync
 
 
 - [PGSync](https://pgsync.com) is a middleware for syncing data from [Postgres](https://www.postgresql.org) to [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) or [OpenSearch](https://opensearch.org/).

{pgsync-6.0.0 → pgsync-6.1.0}/README.md

@@ -73,7 +73,7 @@ Of course, if your data never changed, then you could just take a snapshot in ti
 PGSync is appropriate for you if:
 - [Postgres](https://www.postgresql.org) or [MySQL](https://www.mysql.com/) or [MariaDB](https://mariadb.org/) is your read/write source of truth whilst [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) is your 
 read-only search layer.
-- You need to denormalize relational data into a NoSQL data source.
+- You need to denormalize relational data into a NoSQL data source like [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/).
 - Your data is constantly changing.
 - You have existing data in a relational database such as [Postgres](https://www.postgresql.org) or [MySQL](https://www.mysql.com/) or [MariaDB](https://mariadb.org/) and you need a secondary NoSQL database like [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) for text-based queries or autocomplete queries to mirror the existing data without having your application perform dual writes.
 - You want to keep your existing data untouched whilst taking advantage of
@@ -99,7 +99,8 @@ PGSync operates in an event-driven model by creating triggers for tables in your
 
 *This is the only time PGSync will ever make any changes to your database.*
 
-**NOTE**: **If you change the structure of your PGSync's schema config, you would need to rebuild your Elasticsearch/OpenSearch indices.**
+>**NOTE**: **If you change the structure of your PGSync schema config, it's recommended and in most cases necessary to rebuild your Elasticsearch/OpenSearch indices.**
+
 There are plans to support zero-downtime migrations to streamline this process.
 
 
@@ -187,7 +188,6 @@ Environment variable placeholders - full list [here](https://pgsync.com/env-vars
 ### MySQL / MariaDB setup
 
 - Enable binary logging in your MySQL / MariaDB setting.
-
 - You also need to set up the following parameters in your MySQL / MariaDB config my.cnf, then restart the database server.
 
   ```server-id = 1``` # any non-zero unique ID
@@ -195,10 +195,8 @@ Environment variable placeholders - full list [here](https://pgsync.com/env-vars
   ```log_bin = mysql-bin```
 
   ```binlog_row_image = FULL``` # recommended; if not supported on older MariaDB, omit
-
 - optional housekeeping:
   ```binlog_expire_logs_seconds = 604800```   # 7 days
-
 - You need to create a replication user with REPLICATION SLAVE and REPLICATION CLIENT privileges
     
   ```sql
@@ -224,17 +222,19 @@ Environment variable placeholders - full list [here](https://pgsync.com/env-vars
 
 Key features of PGSync are:
 
-- Easily denormalize relational data. 
-- Works with any PostgreSQL database (version 9.6 or later). 
-- Negligible impact on database performance.
-- Transactionally consistent output in Elasticsearch/OpenSearch. This means: writes appear only when they are committed to the database, insert, update and delete operations appear in the same order as they were committed (as opposed to eventual consistency).
-- Fault-tolerant: does not lose data, even if processes crash or a network interruption occurs, etc. The process can be recovered from the last checkpoint.
-- Returns the data directly as Postgres/MySQL/MariaDB JSON from the database for speed.
-- Supports composite primary and foreign keys.
-- Supports Views and Materialized views.
-- Supports an arbitrary depth of nested entities i.e Tables having long chain of relationship dependencies.
-- Supports PostgreSQL/MySQL/MariaDB JSON data fields. This means: we can extract JSON fields in a database table as a separate field in the resulting document.
-- Customizable document structure.
+- Easily denormalize relational data
+- Works with any PostgreSQL database (9.6 or later)
+- Negligible impact on database performance
+- Transactionally consistent output in Elasticsearch/OpenSearch:
+  - Writes appear only after they’re committed
+  - Inserts, updates, and deletes appear in commit order (not eventually)
+- Fault-tolerant: no data loss even on crashes or network issues; processing resumes from the last checkpoint
+- Returns data directly as PostgreSQL/MySQL/MariaDB JSON for speed
+- Supports composite primary and foreign keys
+- Supports views and materialized views
+- Handles arbitrarily deep nesting of related tables
+- Supports PostgreSQL/MySQL/MariaDB JSON fields, allowing JSON properties to be extracted as separate document fields
+- Customizable document structure
 
 
 #### Requirements
@@ -360,23 +360,26 @@ e.g
   }
 ```
 
-PGSync addresses the following challenges:
-- What if we update the author's name in the database?
-- What if we wanted to add another author for an existing book?
-- What if we have lots of documents already with the same author we wanted to change the author name?
-- What if we delete or update an author?
-- What if we truncate an entire table?
+PGSync addresses common data consistency challenges, such as:
+
+- Updating an author's name in the database
+- Adding an additional author to an existing book
+- Changing an author's name across many existing documents
+- Deleting or updating an author record
+- Truncating an entire table and keeping indexes in sync
 
 
 #### Benefits
 
-- PGSync is a simple to use out of the box solution for Change data capture.
-- PGSync handles data deletions.
-- PGSync requires little development effort. You simply define a schema config describing your data.
-- PGSync generates advanced queries matching your schema directly.
-- PGSync allows you to easily rebuild your indexes in case of a schema change.
-- You can expose only the data you require in Elasticsearch/OpenSearch.
-- Supports multiple Postgres/MySQL/MariaDB schemas for multi-tennant applications.
+PGSync is a simple, out-of-the-box solution for change data capture, designed to minimize development effort and keep your search indexes in sync.
+
+- Handles data deletions automatically.
+- Requires minimal setup. Just define a schema config that describes your data.
+- Generates advanced queries directly from your schema.
+- Makes it easy to rebuild indexes after schema changes.
+- Lets you expose only the data you need in Elasticsearch/OpenSearch.
+- Supports multiple Postgres/MySQL/MariaDB schemas for multi-tenant applications.
+
 
 
 #### Contributing

{pgsync-6.0.0 → pgsync-6.1.0}/README.rst

@@ -1,4 +1,4 @@
-# PostgreSQL to Elasticsearch/OpenSearch sync
+# PostgreSQL/MySQL/MariaDB to Elasticsearch/OpenSearch sync
 
 
 - [PGSync](https://pgsync.com) is a middleware for syncing data from [Postgres](https://www.postgresql.org) to [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) or [OpenSearch](https://opensearch.org/).

{pgsync-6.0.0 → pgsync-6.1.0}/pgsync/__init__.py

@@ -2,4 +2,4 @@
 
 __author__ = "Tolu Aina"
 __email__ = "tolu@pgsync.com"
-__version__ = "6.0.0"
+__version__ = "6.1.0"

{pgsync-6.0.0 → pgsync-6.1.0}/pgsync/base.py

@@ -8,8 +8,11 @@ import time
 import typing as t
 from contextlib import contextmanager
 
+import psycopg2
 import sqlalchemy as sa
+from psycopg2.extras import LogicalReplicationConnection
 from sqlalchemy.dialects import postgresql  # noqa
+from sqlalchemy.engine.url import make_url
 from sqlalchemy.orm import sessionmaker
 
 from .constants import (
@@ -746,6 +749,21 @@ class Base(object):
             )
         )[0]
 
+    def get_replication_connection(
+        self, engine: sa.engine.Engine
+    ) -> psycopg2.extensions.connection:
+        url: sa.engine.URL = make_url(str(engine.url))
+        # Build a libpq-style connection by keyword args
+        conn: psycopg2.extensions.connection = psycopg2.connect(
+            host=url.host,
+            port=url.port or 5432,
+            user=url.username,
+            password=url.password,
+            dbname=url.database,
+            connection_factory=LogicalReplicationConnection,
+        )
+        return conn
+
     def logical_slot_get_changes(
         self,
         slot_name: str,

{pgsync-6.0.0 → pgsync-6.1.0}/pgsync/helper.py

@@ -63,7 +63,7 @@ def teardown(
             drop_database(sync.database)
         if drop_index:
             sync.search_client.teardown(sync.index)
-        if delete_redis:
+        if delete_redis and sync.redis is not None:
             sync.redis.delete()
         if delete_checkpoint:
             try:

{pgsync-6.0.0 → pgsync-6.1.0}/pgsync/sync.py

@@ -95,6 +95,7 @@ class Sync(Base, metaclass=Singleton):
         producer: bool = True,
         consumer: bool = True,
         bootstrap: bool = False,
+        wal: bool = False,
         **kwargs,
     ) -> None:
         """Constructor."""
@@ -119,12 +120,13 @@ class Sync(Base, metaclass=Singleton):
         self.producer: bool = producer
         self.consumer: bool = consumer
         self.num_workers: int = num_workers
-        self.redis: RedisQueue = RedisQueue(self.__name)
+        # Redis not required in wal or polling mode
+        self._redis: t.Optional[RedisQueue] = None
         self.tree: Tree = Tree(
             self.models, nodes=self.nodes, database=doc["database"]
         )
         if bootstrap:
-            self.setup()
+            self.setup(wal, polling)
 
         if validate:
             self.validate(repl_slots=repl_slots, polling=polling)
@@ -137,6 +139,8 @@ class Sync(Base, metaclass=Singleton):
         self.count: dict = dict(xlog=0, db=0, redis=0)
         self.tasks: t.List[asyncio.Task] = []
         self.lock: threading.Lock = threading.Lock()
+        # holds Payload objects across multiple consume() calls
+        self._buffer: list["Payload"] = []
 
     @property
     def slot_name(self) -> str:
@@ -147,6 +151,16 @@ class Sync(Base, metaclass=Singleton):
     def checkpoint_file(self) -> str:
         return os.path.join(settings.CHECKPOINT_PATH, f".{self.__name}")
 
+    @property
+    def redis(self) -> t.Optional[RedisQueue]:
+        """Return the Redis queue instance."""
+        if self._redis is None:
+            try:
+                self._redis = RedisQueue(self.__name)
+            except Exception:
+                pass
+        return self._redis
+
     def validate(self, repl_slots: bool = True, polling: bool = False) -> None:
         """Perform all validation right away."""
 
@@ -161,45 +175,51 @@ class Sync(Base, metaclass=Singleton):
         if self.index is None:
             raise ValueError("Index is missing for doc")
 
-        if not self.is_mysql_compat:
-            if not polling:
-                max_replication_slots: t.Optional[str] = self.pg_settings(
-                    "max_replication_slots"
+        # replication slot not needed in polling or mysql
+        if not self.is_mysql_compat and not polling:
+            max_replication_slots: t.Optional[str] = self.pg_settings(
+                "max_replication_slots"
+            )
+            try:
+                if int(max_replication_slots) < 1:
+                    raise TypeError
+            except TypeError:
+                raise RuntimeError(
+                    "Ensure there is at least one replication slot defined "
+                    "by setting max_replication_slots = 1"
                 )
-                try:
-                    if int(max_replication_slots) < 1:
-                        raise TypeError
-                except TypeError:
-                    raise RuntimeError(
-                        "Ensure there is at least one replication slot defined "
-                        "by setting max_replication_slots = 1"
-                    )
 
-                wal_level: t.Optional[str] = self.pg_settings("wal_level")
-                if not wal_level or wal_level.lower() != "logical":
-                    raise RuntimeError(
-                        "Enable logical decoding by setting wal_level = logical"
-                    )
+            wal_level: t.Optional[str] = self.pg_settings("wal_level")
+            if not wal_level or wal_level.lower() != "logical":
+                raise RuntimeError(
+                    "Enable logical decoding by setting wal_level = logical"
+                )
 
-                self._can_create_replication_slot("_tmp_")
+            self._can_create_replication_slot("_tmp_")
 
-                rds_logical_replication: t.Optional[str] = self.pg_settings(
-                    "rds.logical_replication"
-                )
-                if (
-                    rds_logical_replication
-                    and rds_logical_replication.lower() == "off"
-                ):
-                    raise RDSError("rds.logical_replication is not enabled")
+            rds_logical_replication: t.Optional[str] = self.pg_settings(
+                "rds.logical_replication"
+            )
+            if (
+                rds_logical_replication
+                and rds_logical_replication.lower() == "off"
+            ):
+                raise RDSError("rds.logical_replication is not enabled")
 
-                # ensure we have run bootstrap and the replication slot exists
-                if repl_slots and not self.replication_slots(self.__name):
-                    raise RuntimeError(
-                        f'Replication slot "{self.__name}" does not exist.\n'
-                        f'Make sure you have run the "bootstrap" command.'
-                    )
+            # ensure we have run bootstrap and the replication slot exists
+            if repl_slots and not self.replication_slots(self.__name):
+                raise RuntimeError(
+                    f'Replication slot "{self.__name}" does not exist.\n'
+                    f'Make sure you have run the "bootstrap" command.'
+                )
 
-        if not settings.REDIS_CHECKPOINT:
+        if settings.REDIS_CHECKPOINT:
+            # ensure Redis is reachable
+            try:
+                self.redis.ping()
+            except Exception as e:
+                raise RuntimeError(f"Cannot reach Redis: {e}")
+        else:
             # ensure the checkpoint dirpath is valid
             if not Path(settings.CHECKPOINT_PATH).exists():
                 raise RuntimeError(
@@ -300,8 +320,12 @@ class Sync(Base, metaclass=Singleton):
             routing=self.routing,
         )
 
-    def setup(self, no_create: bool = False) -> None:
-        """Create the database triggers and replication slot."""
+    def setup(
+        self, no_create: bool = False, wal: bool = False, polling: bool = False
+    ) -> None:
+        """Create the database triggers and replication slot.
+        Generally bootstrap should not require Redis as it is optional in certain cases.
+        """
         if self.is_mysql_compat:
             raise NotImplementedError(
                 "Setup is not supported for MySQL-family backend (MySQL or MariaDB)"
@@ -318,75 +342,85 @@ class Sync(Base, metaclass=Singleton):
 
                 self.teardown(drop_view=False)
 
-            for schema in self.schemas:
-                # TODO: move if_not_exists to the function
-                if if_not_exists or not self.function_exists(schema):
-
-                    self.create_function(schema)
-
-                tables: t.Set = set()
-                # tables with user defined foreign keys
-                user_defined_fkey_tables: dict = {}
-                node_columns: dict = {}
+            if not polling:
+                for schema in self.schemas:
+                    # TODO: move if_not_exists to the function
+                    if if_not_exists or not self.function_exists(schema):
+
+                        self.create_function(schema)
+
+                    tables: t.Set = set()
+                    # tables with user defined foreign keys
+                    user_defined_fkey_tables: dict = {}
+                    node_columns: dict = {}
+
+                    for node in self.tree.traverse_breadth_first():
+                        if node.schema != schema:
+                            continue
+                        tables |= set(
+                            [
+                                through.table
+                                for through in node.relationship.throughs
+                            ]
+                        )
+                        tables |= set([node.table])
+                        # we also need to bootstrap the base tables
+                        tables |= set(node.base_tables)
+                        node_columns[node.table] = set(
+                            [
+                                re.split(
+                                    rf"\s*({'|'.join(re.escape(op) for op in JSONB_OPERATORS)})\s*",
+                                    c,
+                                    maxsplit=1,
+                                )[0]
+                                for c in node.column_names
+                            ]
+                        )
+                        # we want to get both the parent and the child keys here
+                        # even though only one of them is the foreign_key.
+                        # this is because we define both in the schema but
+                        # do not specify which table is the foreign key.
+                        columns: list = []
+                        if node.relationship.foreign_key.parent:
+                            columns.extend(
+                                node.relationship.foreign_key.parent
+                            )
+                        if node.relationship.foreign_key.child:
+                            columns.extend(node.relationship.foreign_key.child)
+                        if columns:
+                            user_defined_fkey_tables.setdefault(
+                                node.table, set()
+                            )
+                            user_defined_fkey_tables[node.table] |= set(
+                                columns
+                            )
+                    if tables:
+                        if if_not_exists or not self.view_exists(
+                            MATERIALIZED_VIEW, schema
+                        ):
+                            self.create_view(
+                                self.index,
+                                schema,
+                                tables,
+                                user_defined_fkey_tables,
+                                node_columns,
+                            )
 
-                for node in self.tree.traverse_breadth_first():
-                    if node.schema != schema:
-                        continue
-                    tables |= set(
-                        [
-                            through.table
-                            for through in node.relationship.throughs
-                        ]
-                    )
-                    tables |= set([node.table])
-                    # we also need to bootstrap the base tables
-                    tables |= set(node.base_tables)
-                    node_columns[node.table] = set(
-                        [
-                            re.split(
-                                rf"\s*({'|'.join(re.escape(op) for op in JSONB_OPERATORS)})\s*",
-                                c,
-                                maxsplit=1,
-                            )[0]
-                            for c in node.column_names
-                        ]
-                    )
-                    # we want to get both the parent and the child keys here
-                    # even though only one of them is the foreign_key.
-                    # this is because we define both in the schema but
-                    # do not specify which table is the foreign key.
-                    columns: list = []
-                    if node.relationship.foreign_key.parent:
-                        columns.extend(node.relationship.foreign_key.parent)
-                    if node.relationship.foreign_key.child:
-                        columns.extend(node.relationship.foreign_key.child)
-                    if columns:
-                        user_defined_fkey_tables.setdefault(node.table, set())
-                        user_defined_fkey_tables[node.table] |= set(columns)
-                if tables:
-                    if if_not_exists or not self.view_exists(
-                        MATERIALIZED_VIEW, schema
-                    ):
-                        self.create_view(
-                            self.index,
+                        self.create_triggers(
                             schema,
-                            tables,
-                            user_defined_fkey_tables,
-                            node_columns,
+                            tables=tables,
+                            join_queries=join_queries,
+                            if_not_exists=if_not_exists,
                         )
 
-                    self.create_triggers(
-                        schema,
-                        tables=tables,
-                        join_queries=join_queries,
-                        if_not_exists=if_not_exists,
-                    )
-
-            if if_not_exists or not self.replication_slots(self.__name):
+            if not wal:
+                if if_not_exists or not self.replication_slots(self.__name):
 
-                self.create_replication_slot(self.__name)
+                    self.create_replication_slot(self.__name)
 
-    def teardown(self, drop_view: bool = True) -> None:
+    def teardown(
+        self, drop_view: bool = True, polling: bool = False, wal: bool = False
+    ) -> None:
         """Drop the database triggers and replication slot."""
         if self.is_mysql_compat:
             raise NotImplementedError(
@@ -405,28 +439,35 @@ class Sync(Base, metaclass=Singleton):
                     f"Checkpoint file not found: {self.checkpoint_file}"
                 )
 
-            self.redis.delete()
+            try:
+                if self._redis is None:
+                    raise RuntimeError("Redis is not configured.")
+                self.redis.delete()
+            except Exception as e:
+                logger.warning(f"Could not clear Redis checkpoint queue: {e}")
 
-            for schema in self.schemas:
-                tables: t.Set = set()
-                for node in self.tree.traverse_breadth_first():
-                    tables |= set(
-                        [
-                            through.table
-                            for through in node.relationship.throughs
-                        ]
+            if not polling:
+                for schema in self.schemas:
+                    tables: t.Set = set()
+                    for node in self.tree.traverse_breadth_first():
+                        tables |= set(
+                            [
+                                through.table
+                                for through in node.relationship.throughs
+                            ]
+                        )
+                        tables |= set([node.table])
+                        # we also need to teardown the base tables
+                        tables |= set(node.base_tables)
+                    self.drop_triggers(
+                        schema=schema, tables=tables, join_queries=join_queries
                     )
-                    tables |= set([node.table])
-                    # we also need to teardown the base tables
-                    tables |= set(node.base_tables)
-                self.drop_triggers(
-                    schema=schema, tables=tables, join_queries=join_queries
-                )
-                if drop_view:
-                    self.drop_view(schema)
-                    self.drop_function(schema)
+                    if drop_view:
+                        self.drop_view(schema)
+                        self.drop_function(schema)
 
-            self.drop_replication_slot(self.__name)
+            if not wal:
+                self.drop_replication_slot(self.__name)
 
     def get_doc_id(self, primary_keys: t.List[str], table: str) -> str:
         """
@@ -571,7 +612,7 @@ class Sync(Base, metaclass=Singleton):
             is_mariadb: bool = getattr(conn.dialect, "is_mariadb", False)
 
         def _conn_settings_from_engine(engine: sa.Engine) -> dict:
-            url = engine.url
+            url: sa.engine.URL = engine.url
             return {
                 "host": url.host,
                 "port": int(url.port),
@@ -581,9 +622,9 @@ class Sync(Base, metaclass=Singleton):
                 "autocommit": True,
             }
 
-        base = _conn_settings_from_engine(self.engine)
-        connection_settings = dict(base)  # replication socket
-        ctl_connection_settings = dict(base)
+        base: dict = _conn_settings_from_engine(self.engine)
+        connection_settings: dict = dict(base)  # replication socket
+        ctl_connection_settings: dict = dict(base)
         ctl_connection_settings["cursorclass"] = (
             pymysql.cursors.Cursor
         )  # tuple rows
@@ -607,11 +648,11 @@ class Sync(Base, metaclass=Singleton):
             freeze_schema=False,
         )
 
-        current = 0
-        total = None
+        current: int = 0
+        total: t.Optional[int] = None
         batch: list = []
         last_key: t.Optional[tuple[str, str]] = None
-        batch_limit = limit
+        batch_limit: int = limit
 
         # Single-save checkpoint snapshot
         save_file: t.Optional[str] = start_log
@@ -652,7 +693,7 @@ class Sync(Base, metaclass=Singleton):
                                 self.engine, schema, table, row.get("values")
                             ),
                         )
-                        key = (payload.tg_op, payload.table)
+                        key: tuple[str, str] = (payload.tg_op, payload.table)
                         if last_key is None or key == last_key:
                             batch.append(payload)
                         else:
@@ -1836,6 +1877,97 @@ class Sync(Base, metaclass=Singleton):
 
         self._truncate = True
 
+    def _flush_buffer(
+        self,
+        cursor: t.Any,
+        flush_lsn: t.Optional[str] = None,
+        force_ack: bool = False,
+    ) -> None:
+        # If we have buffered docs, send them
+        if self._buffer:
+            logger.info(f"flushing buffer with {len(self._buffer)} docs")
+            docs: list = []
+            for (op, tbl), run in groupby(
+                self._buffer,
+                key=lambda payload: (payload.tg_op, payload.table),
+            ):
+                batch: list = list(run)
+                logger.info(f"bulk group op={op} tbl={tbl} size={len(batch)}")
+                docs.extend(self._payloads(batch))
+
+            if docs:
+                processed: int = len(self._buffer)
+                logger.info(f"sending bulk of {len(docs)} docs")
+                self.search_client.bulk(self.index, docs)
+                self.count["xlog"] += processed
+                logger.info(f"sent bulk of {len(docs)} docs")
+
+            # if caller didn't provide a flush_lsn, then fall back to last buffered row
+            if flush_lsn is None:
+                flush_lsn = self._buffer_last_lsn
+
+            # clear buffer after successful bulk
+            self._buffer.clear()
+            self._buffer_last_lsn = None
+
+        # Even if buffer was empty, we may want to ACK a COMMIT LSN
+        if flush_lsn is not None and (force_ack or not self._buffer):
+            cursor.send_feedback(flush_lsn=flush_lsn, force=True)
+            logger.info(f"sent feedback flush_lsn=P{flush_lsn}")
+
+    def consume(self, message: t.Any) -> None:
+        raw: t.Any = message.payload
+        lsn: t.Optional[str] = message.data_start
+        chunk_size: int = settings.LOGICAL_SLOT_CHUNK_SIZE
+
+        logger.debug(f"[LSN {lsn}] {raw}")
+
+        match = TX_BOUNDARY_RE.match(raw)
+        if match:
+            kind: str = match.group(1).upper()
+            if kind == "COMMIT":
+                # Flush any buffered docs, and ACK this COMMIT LSN
+                self._flush_buffer(
+                    cursor=message.cursor,
+                    flush_lsn=lsn,
+                    force_ack=True,  # ACK even if buffer empty
+                )
+            # BEGIN/COMMIT don't include rows by themselves
+            return
+
+        # Not BEGIN/COMMIT -> row change
+        try:
+            payload: Payload = self.parse_logical_slot(raw)
+        except Exception:
+            logger.exception(f"Error parsing row: {raw}")
+            raise
+
+        # Filter by schema
+        if payload.schema not in self.tree.schemas:
+            # we still saw this LSN; it will be ACKed at COMMIT
+            return
+
+        # Buffer across transactions
+        self._buffer.append(payload)
+        self._buffer_last_lsn = lsn
+
+        # Flush when big enough
+        if len(self._buffer) >= chunk_size:
+            self._flush_buffer(message.cursor)
+
+    def wal_consumer(self) -> None:
+        # open a replication‐mode connection
+        conn = self.get_replication_connection(self.engine)
+        cursor = conn.cursor()
+        # start streaming; include XIDs so you see BEGIN/COMMIT markers
+        cursor.start_replication(
+            slot_name=self.__name,
+            options={"include-xids": "1", "skip-empty-xacts": "1"},
+            decode=True,  # gets you str instead of bytes
+        )
+        logger.info("Starting logical replication stream (test_decoding)...")
+        cursor.consume_stream(self.consume)
+
     @threaded
     @exception
     def truncate_slots(self) -> None:
@@ -1962,7 +2094,7 @@ class Sync(Base, metaclass=Singleton):
     is_flag=True,
     help="Run as a daemon (Incompatible with --polling)",
     cls=MutuallyExclusiveOption,
-    mutually_exclusive=["polling"],
+    mutually_exclusive=["polling", "wal"],
 )
 @click.option(
     "--producer",
@@ -1985,7 +2117,19 @@ class Sync(Base, metaclass=Singleton):
     is_flag=True,
     help="Polling mode (Incompatible with -d)",
     cls=MutuallyExclusiveOption,
-    mutually_exclusive=["daemon"],
+    mutually_exclusive=["daemon", "wal"],
+)
+@click.option(
+    "--wal",
+    "-w",
+    is_flag=True,
+    default=False,
+    help="Use WAL for replication",
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=[
+        "daemon",
+        "polling",
+    ],
 )
 @click.option("--host", "-h", help="PG_HOST override")
 @click.option("--password", is_flag=True, help="Prompt for database password")
@@ -2066,6 +2210,7 @@ def main(
     producer: bool,
     consumer: bool,
     bootstrap: bool,
+    wal: bool,
 ) -> None:
     """Main application syncer."""
     if version:
@@ -2129,17 +2274,25 @@ def main(
             # In polling mode, the app can run without replication slots or triggers.
             # However, this is not the preferred mode of operation.
             # It should be considered a workaround for running on a read-only cluster.
-            kwargs["polling"] = True
             while True:
                 for doc in config_loader(
                     config=config,
                     schema_url=schema_url,
                     s3_schema_url=s3_schema_url,
                 ):
-                    sync: Sync = Sync(doc, verbose=verbose, **kwargs)
+                    sync: Sync = Sync(
+                        doc, verbose=verbose, polling=True, **kwargs
+                    )
                     sync.pull(polling=True)
                 time.sleep(settings.POLL_INTERVAL)
-
+        elif wal:
+            for doc in config_loader(
+                config=config,
+                schema_url=schema_url,
+                s3_schema_url=s3_schema_url,
+            ):
+                sync: Sync = Sync(doc, verbose=verbose, wal=True, **kwargs)
+                sync.wal_consumer()
         else:
             tasks: t.List[asyncio.Task] = []
             for doc in config_loader(
@@ -2162,7 +2315,7 @@ def main(
                     tasks.extend(sync.tasks)
 
             if settings.USE_ASYNC:
-                event_loop = asyncio.get_event_loop()
+                event_loop: asyncio.AbstractEventLoop = asyncio.get_event_loop()
                 event_loop.run_until_complete(asyncio.gather(*tasks))
                 event_loop.close()
 

{pgsync-6.0.0 → pgsync-6.1.0}/pgsync/utils.py

@@ -280,7 +280,10 @@ def config_loader(
 
     try:
         with open(config_path, "r") as f:
-            data = json.load(f)
+            try:
+                data = json.load(f)
+            except json.JSONDecodeError as e:
+                raise ValueError(f"{config_path} is not valid JSON: {e}") from e
             for doc in data:
                 for key, value in doc.items():
                     try:

{pgsync-6.0.0 → pgsync-6.1.0}/pgsync.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pgsync
-Version: 6.0.0
+Version: 6.1.0
 Summary: Postgres/MySQL/MariaDB to Elasticsearch/OpenSearch sync
 Home-page: https://github.com/toluaina/pgsync
 Author: Tolu Aina
@@ -33,9 +33,9 @@ License-File: LICENSE
 License-File: AUTHORS.rst
 Requires-Dist: async-timeout==5.0.1
 Requires-Dist: backports-datetime-fromisoformat==2.0.3
-Requires-Dist: boto3==1.40.64
-Requires-Dist: botocore==1.40.64
-Requires-Dist: certifi==2025.10.5
+Requires-Dist: boto3==1.41.2
+Requires-Dist: botocore==1.41.2
+Requires-Dist: certifi==2025.11.12
 Requires-Dist: charset-normalizer==3.4.4
 Requires-Dist: click==8.1.8
 Requires-Dist: elastic-transport==9.1.0
@@ -46,7 +46,7 @@ Requires-Dist: events==0.5
 Requires-Dist: idna==3.11
 Requires-Dist: jmespath==1.0.1
 Requires-Dist: marshmallow==4.0.1
-Requires-Dist: mysql-replication==1.0.9
+Requires-Dist: mysql-replication==1.0.12
 Requires-Dist: opensearch-dsl==2.1.0
 Requires-Dist: opensearch-py==3.0.0
 Requires-Dist: packaging==25.0
@@ -57,7 +57,7 @@ Requires-Dist: python-dotenv==1.2.1
 Requires-Dist: redis==7.0.1
 Requires-Dist: requests==2.32.5
 Requires-Dist: requests-aws4auth==1.3.1
-Requires-Dist: s3transfer==0.14.0
+Requires-Dist: s3transfer==0.15.0
 Requires-Dist: six==1.17.0
 Requires-Dist: sqlalchemy==2.0.44
 Requires-Dist: sqlparse==0.5.3
@@ -79,7 +79,7 @@ Dynamic: requires-dist
 Dynamic: requires-python
 Dynamic: summary
 
-# PostgreSQL to Elasticsearch/OpenSearch sync
+# PostgreSQL/MySQL/MariaDB to Elasticsearch/OpenSearch sync
 
 
 - [PGSync](https://pgsync.com) is a middleware for syncing data from [Postgres](https://www.postgresql.org) to [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) or [OpenSearch](https://opensearch.org/).

{pgsync-6.0.0 → pgsync-6.1.0}/pgsync.egg-info/requires.txt

@@ -1,8 +1,8 @@
 async-timeout==5.0.1
 backports-datetime-fromisoformat==2.0.3
-boto3==1.40.64
-botocore==1.40.64
-certifi==2025.10.5
+boto3==1.41.2
+botocore==1.41.2
+certifi==2025.11.12
 charset-normalizer==3.4.4
 click==8.1.8
 elastic-transport==9.1.0
@@ -13,7 +13,7 @@ events==0.5
 idna==3.11
 jmespath==1.0.1
 marshmallow==4.0.1
-mysql-replication==1.0.9
+mysql-replication==1.0.12
 opensearch-dsl==2.1.0
 opensearch-py==3.0.0
 packaging==25.0
@@ -24,7 +24,7 @@ python-dotenv==1.2.1
 redis==7.0.1
 requests==2.32.5
 requests-aws4auth==1.3.1
-s3transfer==0.14.0
+s3transfer==0.15.0
 six==1.17.0
 sqlalchemy==2.0.44
 sqlparse==0.5.3

{pgsync-6.0.0 → pgsync-6.1.0}/tests/test_sync.py

@@ -937,7 +937,7 @@ class TestSync(object):
             mock_teardown.assert_called_once_with(drop_view=False)
 
     @patch("pgsync.redisqueue.RedisQueue.delete")
-    def test_teardown(self, mock_redis, sync):
+    def test_teardown(self, mock_redis_delete, sync):
         with override_env_var(JOIN_QUERIES="False"):
             importlib.reload(settings)
 
@@ -960,16 +960,19 @@ class TestSync(object):
                         )
                     mock_drop_view.assert_called_once_with("public")
                 mock_drop_function.assert_called_once_with("public")
-            mock_redis.assert_called_once()
+            mock_redis_delete.assert_not_called()
             assert os.path.exists(sync.checkpoint_file) is False
 
         with patch("pgsync.sync.logger") as mock_logger:
             with patch("pgsync.sync.Base.drop_replication_slot"):
                 self.checkpoint_file = "foo"
                 sync.teardown()
-                mock_logger.warning.assert_called_once_with(
-                    "Checkpoint file not found: ./.testdb_testdb"
-                )
+                assert mock_logger.warning.call_args_list == [
+                    call("Checkpoint file not found: ./.testdb_testdb"),
+                    call(
+                        "Could not clear Redis checkpoint queue: Redis is not configured."
+                    ),
+                ]
 
     def test_root(self, sync):
         root = sync.tree.root