pgsync 7.0.5__tar.gz → 7.1.0__tar.gz

{pgsync-7.0.5 → pgsync-7.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pgsync
-Version: 7.0.5
+Version: 7.1.0
 Summary: Postgres/MySQL/MariaDB to Elasticsearch/OpenSearch sync
 Home-page: https://github.com/toluaina/pgsync
 Author: Tolu Aina
@@ -17,7 +17,6 @@ Keywords: change data capture,elasticsearch,opensearch,pgsync,postgres,mysql,mar
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Natural Language :: English
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
@@ -27,43 +26,46 @@ Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Programming Language :: Python :: Implementation :: PyPy
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.9.0
+Requires-Python: >=3.10.0
 Description-Content-Type: text/x-rst
 License-File: LICENSE
 License-File: AUTHORS.rst
 Requires-Dist: async-timeout==5.0.1
 Requires-Dist: backports-datetime-fromisoformat==2.0.3
-Requires-Dist: boto3==1.42.21
-Requires-Dist: botocore==1.42.21
-Requires-Dist: certifi==2026.1.4
-Requires-Dist: charset-normalizer==3.4.4
-Requires-Dist: click==8.1.8
-Requires-Dist: elastic-transport==9.1.0
-Requires-Dist: elasticsearch==7.17.12
+Requires-Dist: boto3==1.43.36
+Requires-Dist: botocore==1.43.36
+Requires-Dist: certifi==2026.6.17
+Requires-Dist: charset-normalizer==3.4.7
+Requires-Dist: click==8.4.2
+Requires-Dist: elastic-transport==9.4.2
+Requires-Dist: elasticsearch==7.17.13
 Requires-Dist: elasticsearch-dsl==7.4.1
-Requires-Dist: environs==14.4.0
+Requires-Dist: environs==15.0.1
 Requires-Dist: events==0.5
-Requires-Dist: greenlet==3.2.4
-Requires-Dist: idna==3.11
-Requires-Dist: jmespath==1.0.1
-Requires-Dist: marshmallow==4.0.1
-Requires-Dist: mysql-replication==1.0.12
+Requires-Dist: grpcio==1.81.1
+Requires-Dist: idna==3.18
+Requires-Dist: jmespath==1.1.0
+Requires-Dist: marshmallow==4.3.0
+Requires-Dist: mysql-replication==1.0.15
 Requires-Dist: opensearch-dsl==2.1.0
-Requires-Dist: opensearch-py==3.0.0
-Requires-Dist: packaging==25.0
-Requires-Dist: psycopg2-binary==2.9.11
-Requires-Dist: pymysql==1.1.2
+Requires-Dist: opensearch-protobufs==1.2.0
+Requires-Dist: opensearch-py==3.2.0
+Requires-Dist: packaging==26.2
+Requires-Dist: protobuf==7.35.1
+Requires-Dist: psycopg2-binary==2.9.12
+Requires-Dist: pymysql==1.2.0
 Requires-Dist: python-dateutil==2.9.0.post0
-Requires-Dist: python-dotenv==1.2.1
-Requires-Dist: redis==7.0.1
-Requires-Dist: requests==2.32.5
-Requires-Dist: requests-aws4auth==1.3.1
-Requires-Dist: s3transfer==0.16.0
+Requires-Dist: python-dotenv==1.2.2
+Requires-Dist: redis==8.0.1
+Requires-Dist: requests==2.34.2
+Requires-Dist: requests-aws4auth==1.3.2
+Requires-Dist: s3transfer==0.19.0
 Requires-Dist: six==1.17.0
-Requires-Dist: sqlalchemy==2.0.45
+Requires-Dist: sniffio==1.3.1
+Requires-Dist: sqlalchemy==2.0.51
 Requires-Dist: sqlparse==0.5.5
 Requires-Dist: typing-extensions==4.15.0
-Requires-Dist: urllib3==1.26.20
+Requires-Dist: urllib3==2.7.0
 Dynamic: author
 Dynamic: author-email
 Dynamic: classifier
@@ -121,7 +123,7 @@ Key Features
 Requirements
 ------------
 
-- `Python <https://www.python.org>`_ 3.9+
+- `Python <https://www.python.org>`_ 3.10+
 - `PostgreSQL <https://www.postgresql.org>`_ 9.6+ or `MySQL <https://www.mysql.com>`_ 8.0.0+ or `MariaDB <https://mariadb.org>`_ 12.0.0+
 - `Redis <https://redis.io>`_ 3.1.0+ or `Valkey <https://valkey.io>`_ 7.2.0+ (optional in WAL mode)
 - `Elasticsearch <https://www.elastic.co/products/elastic-stack>`_ 6.3.1+ or `OpenSearch <https://opensearch.org>`_ 1.3.7+

{pgsync-7.0.5 → pgsync-7.1.0}/README.md

@@ -27,6 +27,7 @@
 PGSync is a change data capture tool that syncs data from **PostgreSQL**, **MySQL**, or **MariaDB** to **Elasticsearch** or **OpenSearch** in real-time. Define your document structure in JSON, and PGSync handles the rest — no custom code required.
 
 ```mermaid
+%%{init: {'look': 'handDrawn', 'theme': 'neutral'}}%%
 flowchart LR
     subgraph Source["🗄️ Source Database"]
         DB[(PostgreSQL<br/>MySQL<br/>MariaDB)]
@@ -86,12 +87,28 @@ pgsync --config schema.json -d
 
 ### Using Docker Compose
 
+**Default (Elasticsearch + Kibana):**
 ```bash
 git clone https://github.com/toluaina/pgsync
 cd pgsync
 docker-compose up
 ```
 
+This starts PostgreSQL, Redis, Elasticsearch, Kibana, and PGSync configured for Elasticsearch.
+
+**For OpenSearch:**
+```bash
+docker-compose --profile opensearch up
+```
+
+This starts PostgreSQL, Redis, OpenSearch, and PGSync configured for OpenSearch.
+
+**Ports:**
+- PostgreSQL: `15432`
+- Elasticsearch: `9201` (default)
+- Kibana: `5601` (default)
+- OpenSearch: `9400` (OpenSearch profile)
+
 ---
 
 ## How It Works
@@ -137,7 +154,7 @@ Changes to any related table automatically update the document in Elasticsearch/
 
 | Component | Version |
 |-----------|---------|
-| ![Python](https://img.shields.io/badge/Python-3.9+-3776AB?logo=python&logoColor=white) | 3.9+ |
+| ![Python](https://img.shields.io/badge/Python-3.10+-3776AB?logo=python&logoColor=white) | 3.10+ |
 | ![PostgreSQL](https://img.shields.io/badge/PostgreSQL-9.6+-4169E1?logo=postgresql&logoColor=white) | 9.6+ (or MySQL 5.7.22+ / MariaDB 10.5+) |
 | ![Elasticsearch](https://img.shields.io/badge/Elasticsearch-6.3+-005571?logo=elasticsearch&logoColor=white) | 6.3.1+ (or OpenSearch 1.3.7+) |
 | ![Redis](https://img.shields.io/badge/Redis-3.1+-DC382D?logo=redis&logoColor=white) | 3.1+ (or Valkey 7.2+) — optional in WAL mode |
@@ -225,6 +242,96 @@ PGSync transforms this into search-ready documents:
 
 ---
 
+## Transforms
+
+PGSync supports built-in transforms to modify field values before indexing. Transforms are applied in order: `replace` → `rename` → `concat`.
+
+### Replace
+
+Find and replace substrings within field values:
+
+```json
+{
+  "table": "product",
+  "columns": ["code", "name"],
+  "transform": {
+    "replace": {
+      "code": {
+        "-": "/",
+        "_": " "
+      }
+    }
+  }
+}
+```
+
+| Before | After |
+|--------|-------|
+| `ABC-DEF_GHI` | `ABC/DEF GHI` |
+
+### Rename
+
+Rename fields in the output document:
+
+```json
+{
+  "table": "book",
+  "columns": ["id", "title"],
+  "transform": {
+    "rename": {
+      "id": "book_id",
+      "title": "book_title"
+    }
+  }
+}
+```
+
+### Concat
+
+Combine multiple fields into a new field:
+
+```json
+{
+  "table": "user",
+  "columns": ["first_name", "last_name"],
+  "transform": {
+    "concat": {
+      "columns": ["first_name", "last_name"],
+      "destination": "full_name",
+      "delimiter": " "
+    }
+  }
+}
+```
+
+### Combined Example
+
+Transforms can be combined and applied to nested children:
+
+```json
+{
+  "table": "book",
+  "columns": ["isbn", "title"],
+  "children": [{
+    "table": "publisher",
+    "columns": ["code", "name"],
+    "transform": {
+      "replace": { "code": { "-": "." } },
+      "rename": { "name": "publisher_name" }
+    }
+  }],
+  "transform": {
+    "concat": {
+      "columns": ["isbn", "title"],
+      "destination": "search_text",
+      "delimiter": " - "
+    }
+  }
+}
+```
+
+---
+
 ## Why PGSync?
 
 | Challenge | PGSync Solution |

{pgsync-7.0.5 → pgsync-7.1.0}/README.rst

@@ -39,7 +39,7 @@ Key Features
 Requirements
 ------------
 
-- `Python <https://www.python.org>`_ 3.9+
+- `Python <https://www.python.org>`_ 3.10+
 - `PostgreSQL <https://www.postgresql.org>`_ 9.6+ or `MySQL <https://www.mysql.com>`_ 8.0.0+ or `MariaDB <https://mariadb.org>`_ 12.0.0+
 - `Redis <https://redis.io>`_ 3.1.0+ or `Valkey <https://valkey.io>`_ 7.2.0+ (optional in WAL mode)
 - `Elasticsearch <https://www.elastic.co/products/elastic-stack>`_ 6.3.1+ or `OpenSearch <https://opensearch.org>`_ 1.3.7+

{pgsync-7.0.5 → pgsync-7.1.0}/bin/bootstrap

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 
 """PGSync bootstrap."""
+
 import logging
 
 import click

{pgsync-7.0.5 → pgsync-7.1.0}/bin/parallel_sync

@@ -31,6 +31,7 @@ The sync process works as follows:
 This parallel approach significantly improves synchronization performance,
 especially in high-latency network environments.
 """
+
 import asyncio
 import multiprocessing
 import os

{pgsync-7.0.5 → pgsync-7.1.0}/bin/pgsync

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 
 """PGSync runtime."""
+
 from pgsync import sync
 
 if __name__ == "__main__":

{pgsync-7.0.5 → pgsync-7.1.0}/pgsync/__init__.py

@@ -2,4 +2,4 @@
 
 __author__ = "Tolu Aina"
 __email__ = "tolu@pgsync.com"
-__version__ = "7.0.5"
+__version__ = "7.1.0"

{pgsync-7.0.5 → pgsync-7.1.0}/pgsync/base.py

@@ -43,6 +43,7 @@ from .settings import (
     PG_SSLROOTCERT,
     PG_URL_RO,
     PG_USER_RO,
+    PG_WORK_MEM,
     QUERY_CHUNK_SIZE,
     SQLALCHEMY_MAX_OVERFLOW,
     SQLALCHEMY_POOL_PRE_PING,
@@ -242,6 +243,10 @@ class Base(object):
         self.verbose: bool = verbose
         self._conn = None
         self._session = None
+        # Per-thread set of advisory lock keys currently held, to make
+        # advisory_lock() re-entrant without requiring the nested call to
+        # acquire a second backend connection.
+        self._advisory_locks_held = threading.local()
 
     def connect(self) -> None:
         """Connect to database."""
@@ -567,50 +572,6 @@ class Base(object):
         )
         return row[0]
 
-    def pg_try_advisory_lock(
-        self, key: t.Union[int, str], timeout: int = 0
-    ) -> bool:
-        """
-        Attempts to acquire an dvisory/named lock based on a hashed slot name without blocking.
-
-        PostgreSQL: integer key -> PG_TRY_ADVISORY_LOCK(key) -> bool
-        MySQL/MariaDB: string name -> GET_LOCK(name, timeout) -> 1 on success
-                    (timeout defaults to 0 = non-blocking)
-
-        Returns:
-            bool: True if the lock was acquired, False otherwise.
-        """
-        if self.is_mysql_compat:
-            row = self.fetchone(
-                sa.text("SELECT GET_LOCK(:name, :timeout)").bindparams(
-                    name=str(key), timeout=int(timeout)
-                )
-            )
-            return bool(row and row[0] == 1)
-
-        row = self.fetchone(
-            sa.text("SELECT PG_TRY_ADVISORY_LOCK(:key)").bindparams(key=key)
-        )
-        return bool(row and row[0])
-
-    def pg_advisory_unlock(self, key: t.Union[int, str]) -> bool:
-        """
-        Releases an advisory lock associated with the hashed slot name.
-
-        Returns:
-            bool: True if the lock was released, False if it was not held.
-        """
-        if self.is_mysql_compat:
-            row = self.fetchone(
-                sa.text("SELECT RELEASE_LOCK(:name)").bindparams(name=str(key))
-            )
-            return bool(row and row[0] == 1)
-
-        row = self.fetchone(
-            sa.text("SELECT PG_ADVISORY_UNLOCK(:key)").bindparams(key=key)
-        )
-        return bool(row and row[0])
-
     @contextmanager
     def advisory_lock(
         self,
@@ -626,58 +587,111 @@ class Base(object):
         Context manager to acquire a PostgreSQL advisory lock with optional retries.
         Acquire a PostgreSQL advisory lock with retries, backoff, and jitter.
         Jitter reduces lock-step contention so callers don't starve.
+
+        A single connection is held for the entire lifetime of the context
+        so that lock and unlock always run on the same PostgreSQL backend,
+        preventing advisory lock leaks. Re-entrant within the same thread:
+        nested calls for the same slot are no-ops.
         """
         key: int = self.advisory_key(slot_name)
+
+        held: set = getattr(self._advisory_locks_held, "keys", None)
+        if held is None:
+            held = set()
+            self._advisory_locks_held.keys = held
+
+        if key in held:
+            # Already held in this thread; nested acquires are a no-op.
+            yield
+            return
+
         attempt: int = 0
 
         base_delay: float = float(retry_interval)
         # current backoff window (seconds)
         delay: float = base_delay
 
-        while True:
-            if self.pg_try_advisory_lock(key):
-                break
+        with self.engine.connect() as conn:
+            while True:
+                if self._try_lock_on(conn, key):
+                    break
 
-            if (max_retries is not None) and (attempt >= max_retries):
-                raise RuntimeError(
-                    f"Failed to acquire advisory lock for '{slot_name}' after {max_retries} retries."
-                )
+                if (max_retries is not None) and (attempt >= max_retries):
+                    raise RuntimeError(
+                        f"Failed to acquire advisory lock for '{slot_name}' after {max_retries} retries."
+                    )
 
-            # Compute sleep using jitter strategy
-            if jitter == "decorrelated":
-                # Decorrelated jitter chooses the *next* delay first.
-                delay = min(max_delay, random.uniform(base_delay, delay * 3))
-                sleep_for = delay
-            else:
-                # For other modes, sleep is derived from current delay.
-                if jitter == "full":
-                    sleep_for = random.uniform(0.0, delay)
-                elif jitter == "equal":
-                    sleep_for = (delay / 2.0) + random.uniform(
-                        0.0, delay / 2.0
+                # Compute sleep using jitter strategy
+                if jitter == "decorrelated":
+                    # Decorrelated jitter chooses the *next* delay first.
+                    delay = min(
+                        max_delay, random.uniform(base_delay, delay * 3)
                     )
-                elif jitter == "none":
                     sleep_for = delay
                 else:
-                    # Fallback to full jitter if an unknown option is passed
-                    sleep_for = random.uniform(0.0, delay)
+                    # For other modes, sleep is derived from current delay.
+                    if jitter == "full":
+                        sleep_for = random.uniform(0.0, delay)
+                    elif jitter == "equal":
+                        sleep_for = (delay / 2.0) + random.uniform(
+                            0.0, delay / 2.0
+                        )
+                    elif jitter == "none":
+                        sleep_for = delay
+                    else:
+                        # Fallback to full jitter if an unknown option is passed
+                        sleep_for = random.uniform(0.0, delay)
 
-            time.sleep(max(0.0, sleep_for))
+                time.sleep(max(0.0, sleep_for))
 
-            # Increase delay for next attempt (except decorrelated which already advanced)
-            if backoff_type == "exponential" and jitter != "decorrelated":
-                delay = min(max_delay, delay * backoff_factor)
-            # For fixed backoff, 'delay' stays at base_delay unless decorrelated changed it.
+                # Increase delay for next attempt (except decorrelated which already advanced)
+                if backoff_type == "exponential" and jitter != "decorrelated":
+                    delay = min(max_delay, delay * backoff_factor)
+                # For fixed backoff, 'delay' stays at base_delay unless decorrelated changed it.
 
-            attempt += 1
+                attempt += 1
 
-        try:
-            yield
-        finally:
+            held.add(key)
             try:
-                self.pg_advisory_unlock(key)
-            except Exception:
-                pass
+                yield
+            finally:
+                held.discard(key)
+                try:
+                    self._unlock_on(conn, key)
+                except Exception:
+                    pass
+
+    # ------------------------------------------------------------------
+    # Internal helpers that run lock/unlock on an explicit connection
+    # ------------------------------------------------------------------
+
+    def _try_lock_on(
+        self, conn: sa.engine.Connection, key: t.Union[int, str]
+    ) -> bool:
+        """Acquire an advisory lock on *conn* without blocking."""
+        if self.is_mysql_compat:
+            row = conn.execute(
+                sa.text("SELECT GET_LOCK(:name, 0)").bindparams(name=str(key))
+            ).fetchone()
+            return bool(row and row[0] == 1)
+        row = conn.execute(
+            sa.text("SELECT PG_TRY_ADVISORY_LOCK(:key)").bindparams(key=key)
+        ).fetchone()
+        return bool(row and row[0])
+
+    def _unlock_on(
+        self, conn: sa.engine.Connection, key: t.Union[int, str]
+    ) -> bool:
+        """Release an advisory lock on *conn*."""
+        if self.is_mysql_compat:
+            row = conn.execute(
+                sa.text("SELECT RELEASE_LOCK(:name)").bindparams(name=str(key))
+            ).fetchone()
+            return bool(row and row[0] == 1)
+        row = conn.execute(
+            sa.text("SELECT PG_ADVISORY_UNLOCK(:key)").bindparams(key=key)
+        ).fetchone()
+        return bool(row and row[0])
 
     def _logical_slot_changes(
         self,
@@ -1050,15 +1064,13 @@ class Base(object):
             if not xid8s:
                 return {}
             # TODO: use the SQLAlchemy ORM to handle this query
-            statement = sa.text(
-                """
+            statement = sa.text("""
                 SELECT xid AS xid8,
                 PG_VISIBLE_IN_SNAPSHOT(xid::xid8, PG_CURRENT_SNAPSHOT()) AS visible
                 FROM UNNEST(CAST(:xid8s AS text[]))
                 WITH ORDINALITY AS t(xid, ord)
                 ORDER BY t.ord
-            """
-            )
+            """)
             if self.verbose:
                 compiled_query(
                     statement,
@@ -1397,23 +1409,39 @@ def _pg_engine(
     if SQLALCHEMY_USE_NULLPOOL:
         from sqlalchemy.pool import NullPool
 
-        return sa.create_engine(
+        engine = sa.create_engine(
             url,
             echo=echo,
             connect_args=connect_args,
             poolclass=NullPool,
         )
+    else:
+        engine = sa.create_engine(
+            url,
+            echo=echo,
+            connect_args=connect_args,
+            pool_size=SQLALCHEMY_POOL_SIZE,
+            max_overflow=SQLALCHEMY_MAX_OVERFLOW,
+            pool_pre_ping=SQLALCHEMY_POOL_PRE_PING,
+            pool_recycle=SQLALCHEMY_POOL_RECYCLE,
+            pool_timeout=SQLALCHEMY_POOL_TIMEOUT,
+        )
 
-    return sa.create_engine(
-        url,
-        echo=echo,
-        connect_args=connect_args,
-        pool_size=SQLALCHEMY_POOL_SIZE,
-        max_overflow=SQLALCHEMY_MAX_OVERFLOW,
-        pool_pre_ping=SQLALCHEMY_POOL_PRE_PING,
-        pool_recycle=SQLALCHEMY_POOL_RECYCLE,
-        pool_timeout=SQLALCHEMY_POOL_TIMEOUT,
-    )
+    # Set work_mem on each connection if configured.
+    # This prevents temp file creation during complex queries with
+    # LATERAL JOINs and JSON aggregation (typically needs 12-16MB).
+    if PG_WORK_MEM and not IS_MYSQL_COMPAT:
+        from sqlalchemy import event
+
+        @event.listens_for(engine, "connect")
+        def set_work_mem(dbapi_conn, connection_record):
+            cursor = dbapi_conn.cursor()
+            cursor.execute(f"SET work_mem = '{PG_WORK_MEM}'")
+            cursor.close()
+
+        logger.debug(f"Configured work_mem={PG_WORK_MEM} for new connections")
+
+    return engine
 
 
 def pg_logical_repl_conn(

{pgsync-7.0.5 → pgsync-7.1.0}/pgsync/node.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 import re
 import threading
 import typing as t
+from collections import deque
 from dataclasses import dataclass
 
 import sqlalchemy as sa
@@ -342,12 +343,12 @@ class Node(object):
             child.display(prefix, leaf)
 
     def traverse_breadth_first(self) -> t.Generator:
-        stack: t.List[Node] = [self]
-        while stack:
-            node: Node = stack.pop(0)
+        queue: deque = deque([self])
+        while queue:
+            node: Node = queue.popleft()
             yield node
             for child in node.children:
-                stack.append(child)
+                queue.append(child)
 
     def traverse_post_order(self) -> t.Generator:
         for child in self.children:

{pgsync-7.0.5 → pgsync-7.1.0}/pgsync/querybuilder.py

@@ -210,7 +210,8 @@ class QueryBuilder(threading.local):
                 getattr(getattr(node, "model", None), "original", None)
             )
 
-        # merge relationship provided hints (if any); do NOT short-circuit
+        # merge relationship provided hints (if any)
+        has_explicit_fk = False
         for node in (node_a, node_b):
             rel_fk = getattr(
                 getattr(node, "relationship", None), "foreign_key", None
@@ -218,6 +219,14 @@ class QueryBuilder(threading.local):
             if not rel_fk:
                 continue
 
+            # node.relationship.foreign_key describes the FK between node and
+            # node.parent (in the tree). Only honor it when the other node IS
+            # node's tree parent; otherwise it can leak a middle node's FK to
+            # its grandparent into a grandchild join's resolution dict.
+            other = node_b if node is node_a else node_a
+            if getattr(node, "parent", None) is not other:
+                continue
+
             parent_tbl_key = node_table_key(node, prefer_parent=True)
             child_tbl_key = node_table_key(node, prefer_parent=False)
 
@@ -241,7 +250,15 @@ class QueryBuilder(threading.local):
             # child table cols
             merge_side(getattr(rel_fk, "child", None), child_tbl_key)
 
+            if getattr(rel_fk, "parent", None) and getattr(
+                rel_fk, "child", None
+            ):
+                has_explicit_fk = True
+
         # SQLAlchemy introspection in both directions (A -> B and B -> A)
+        # Skip when an explicit foreign_key was provided in the schema,
+        # otherwise auto discovery adds ALL FKs between the tables which
+        # causes mismatches when a child has multiple FKs to the parent.
         A = getattr(getattr(node_a, "model", None), "original", None)
         B = getattr(getattr(node_b, "model", None), "original", None)
 
@@ -252,7 +269,7 @@ class QueryBuilder(threading.local):
         def same_table(t1: t.Any, t2: t.Any) -> bool:
             return qname(t1) is not None and qname(t1) == qname(t2)
 
-        if A is not None and B is not None:
+        if not has_explicit_fk and A is not None and B is not None:
             for fk in getattr(A, "foreign_keys", []):
                 # does A have an FK pointing to B?
                 if same_table(getattr(fk, "column", None).table, B):