pgsync 4.1.0__tar.gz → 4.2.1__tar.gz

{pgsync-4.1.0 → pgsync-4.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pgsync
-Version: 4.1.0
+Version: 4.2.1
 Summary: Postgres to Elasticsearch/OpenSearch sync
 Home-page: https://github.com/toluaina/pgsync
 Author: Tolu Aina
@@ -32,33 +32,33 @@ License-File: LICENSE
 License-File: AUTHORS.rst
 Requires-Dist: async-timeout==5.0.1
 Requires-Dist: backports-datetime-fromisoformat==2.0.3
-Requires-Dist: boto3==1.38.44
-Requires-Dist: botocore==1.38.44
-Requires-Dist: certifi==2025.6.15
-Requires-Dist: charset-normalizer==3.4.2
+Requires-Dist: boto3==1.40.35
+Requires-Dist: botocore==1.40.35
+Requires-Dist: certifi==2025.8.3
+Requires-Dist: charset-normalizer==3.4.3
 Requires-Dist: click==8.1.8
 Requires-Dist: elastic-transport==8.17.1
-Requires-Dist: elasticsearch==8.18.1
-Requires-Dist: elasticsearch-dsl==8.18.0
-Requires-Dist: environs==14.2.0
+Requires-Dist: elasticsearch==8.19.1
+Requires-Dist: elasticsearch-dsl==8.15.4
+Requires-Dist: environs==14.3.0
 Requires-Dist: events==0.5
-Requires-Dist: greenlet==3.2.3
+Requires-Dist: greenlet==3.2.4
 Requires-Dist: idna==3.10
 Requires-Dist: jmespath==1.0.1
-Requires-Dist: marshmallow==4.0.0
+Requires-Dist: marshmallow==4.0.1
 Requires-Dist: opensearch-dsl==2.1.0
 Requires-Dist: opensearch-py==3.0.0
 Requires-Dist: psycopg2-binary==2.9.10
 Requires-Dist: python-dateutil==2.9.0.post0
 Requires-Dist: python-dotenv==1.1.1
-Requires-Dist: redis==6.2.0
-Requires-Dist: requests==2.32.4
+Requires-Dist: redis==6.4.0
+Requires-Dist: requests==2.32.5
 Requires-Dist: requests-aws4auth==1.3.1
-Requires-Dist: s3transfer==0.13.0
+Requires-Dist: s3transfer==0.14.0
 Requires-Dist: six==1.17.0
-Requires-Dist: sqlalchemy==2.0.41
+Requires-Dist: sqlalchemy==2.0.43
 Requires-Dist: sqlparse==0.5.3
-Requires-Dist: typing-extensions==4.14.0
+Requires-Dist: typing-extensions==4.15.0
 Requires-Dist: urllib3==1.26.20
 Dynamic: author
 Dynamic: author-email
@@ -88,9 +88,9 @@ expose structured denormalized documents in [Elasticsearch](https://www.elastic.
 
 - [Python](https://www.python.org) 3.9+
 - [Postgres](https://www.postgresql.org) 9.6+
-- [Redis](https://redis.io) 3.1.0+
+- [Redis](https://redis.io) 3.1.0+ or [Valkey](https://valkey.io) 7.2.0+
 - [Elasticsearch](https://www.elastic.co/products/elastic-stack) 6.3.1+ or [OpenSearch](https://opensearch.org/) 1.3.7+
-- [SQlAlchemy](https://www.sqlalchemy.org) 1.3.4+
+- [SQLAlchemy](https://www.sqlalchemy.org) 1.3.4+
 
 ### Postgres setup
   

{pgsync-4.1.0 → pgsync-4.2.1}/README.md

@@ -40,7 +40,7 @@ of engineering and development.
 Other benefits of PGSync include:
 - Real-time analytics
 - Reliable primary datastore/source of truth
-- Scale on-demand
+- Scale on-demand (multiple consumers)
 - Easily join multiple nested tables
 
 #### Why?
@@ -66,7 +66,7 @@ the search capabilities of [Elasticsearch](https://www.elastic.co/products/elast
 
 #### How it works
 
-PGSync is written in Python (supporting version 3.9 onwards) and the stack is composed of: [Redis](https://redis.io), [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/), [Postgres](https://www.postgresql.org), and [SQlAlchemy](https://www.sqlalchemy.org).
+PGSync is written in Python (supporting version 3.9 onwards) and the stack is composed of: [Redis](https://redis.io)/[Valkey](https://valkey.io), [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/), [Postgres](https://www.postgresql.org), and [SQLAlchemy](https://www.sqlalchemy.org).
 
 PGSync leverages the [logical decoding](https://www.postgresql.org/docs/current/logicaldecoding.html) feature of [Postgres](https://www.postgresql.org) (introduced in PostgreSQL 9.4) to capture a continuous stream of change events.
 This feature needs to be enabled in your [Postgres](https://www.postgresql.org) configuration file by setting in the postgresql.conf file:
@@ -93,9 +93,14 @@ There are several ways of installing and trying PGSync
  - [Manual configuration](#manual-configuration) 
 
 
-##### Running in Docker
+##### Running in Docker (Using Github Repository)
 
 To startup all services with docker.
+
+```
+$ git checkout https://github.com/toluaina/pgsync
+```
+
 Run:
 ```
 $ docker-compose up
@@ -106,6 +111,39 @@ Show the content in Elasticsearch/OpenSearch
 $ curl -X GET http://[Elasticsearch/OpenSearch host]:9201/reservations/_search?pretty=true
 ```
 
+
+##### Running with Docker (Using Image Repository)
+
+To start all services with Docker, follow these steps:
+
+1. Pull the Docker image:
+
+  ```
+  $ docker pull toluaina1/pgsync:latest
+  ```
+
+2. Run the container:
+
+  ```
+  $ docker run --rm -it \
+  -e REDIS_CHECKPOINT=true \
+  -e REDIS_HOST=<redis_host_address> \
+  -e PG_URL=postgres://<username>:<password>@<postgres_host>/<database> \
+  -e ELASTICSEARCH_URL=http://<elasticsearch_host>:9200 \
+  -v "$(pwd)/schema.json:/app/schema.json" \
+  toluaina1/pgsync:latest -c schema.json -d -b
+  ```
+
+Environment variable placeholders - full list [here](https://pgsync.com/env-vars):
+
+- redis_host_address — Address of the Redis/Valkey server (e.g., host.docker.internal for local Docker setup)
+- username — PostgreSQL username
+- password — PostgreSQL password
+- postgres_host — Host address for PostgreSQL instance (e.g., host.docker.internal)
+- database — Name of PostgreSQL database
+- elasticsearch_host — Address of Elasticsearch/OpenSearch instance (e.g., host.docker.internal)
+
+
 ##### Manual configuration
 
 - Setup
@@ -154,9 +192,9 @@ Key features of PGSync are:
 
 - [Python](https://www.python.org) 3.9+
 - [Postgres](https://www.postgresql.org) 9.6+
-- [Redis](https://redis.io) 3.1.0+
+- [Redis](https://redis.io) 3.1.0+ or [Valkey](https://valkey.io) 7.2.0+
 - [Elasticsearch](https://www.elastic.co/products/elastic-stack) 6.3.1+ or [OpenSearch](https://opensearch.org/) 1.3.7+
-- [SQlAlchemy](https://www.sqlalchemy.org) 1.3.4+
+- [SQLAlchemy](https://www.sqlalchemy.org) 1.3.4+
 
 
 #### Example

{pgsync-4.1.0 → pgsync-4.2.1}/README.rst

@@ -10,9 +10,9 @@ expose structured denormalized documents in [Elasticsearch](https://www.elastic.
 
 - [Python](https://www.python.org) 3.9+
 - [Postgres](https://www.postgresql.org) 9.6+
-- [Redis](https://redis.io) 3.1.0+
+- [Redis](https://redis.io) 3.1.0+ or [Valkey](https://valkey.io) 7.2.0+
 - [Elasticsearch](https://www.elastic.co/products/elastic-stack) 6.3.1+ or [OpenSearch](https://opensearch.org/) 1.3.7+
-- [SQlAlchemy](https://www.sqlalchemy.org) 1.3.4+
+- [SQLAlchemy](https://www.sqlalchemy.org) 1.3.4+
 
 ### Postgres setup
   

{pgsync-4.1.0 → pgsync-4.2.1}/bin/bootstrap

@@ -5,8 +5,14 @@ import logging
 
 import click
 
+from pgsync import settings
 from pgsync.sync import Sync
-from pgsync.utils import config_loader, get_config, show_settings
+from pgsync.utils import (
+    config_loader,
+    MutuallyExclusiveOption,
+    show_settings,
+    validate_config,
+)
 
 logger = logging.getLogger(__name__)
 
@@ -17,6 +23,19 @@ logger = logging.getLogger(__name__)
     "-c",
     help="Schema config",
     type=click.Path(exists=True),
+    default=settings.SCHEMA,
+    show_default=True,
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=["s3_schema_url"],
+)
+@click.option(
+    "--s3_schema_url",
+    help="S3 URL for schema config",
+    type=click.STRING,
+    default=settings.S3_SCHEMA_URL,
+    show_default=True,
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=["config"],
 )
 @click.option("--host", "-h", help="PG_HOST override")
 @click.option("--password", is_flag=True, help="Prompt for database password")
@@ -48,6 +67,7 @@ logger = logging.getLogger(__name__)
 def main(
     teardown: bool,
     config: str,
+    s3_schema_url: str,
     user: str,
     password: bool,
     host: str,
@@ -69,13 +89,13 @@ def main(
         )
     kwargs = {key: value for key, value in kwargs.items() if value is not None}
 
-    config: str = get_config(config)
+    validate_config(config=config, s3_schema_url=s3_schema_url)
 
-    show_settings(config)
+    show_settings(config=config, s3_schema_url=s3_schema_url)
 
     validate: bool = False if teardown else True
 
-    for doc in config_loader(config):
+    for doc in config_loader(config=config, s3_schema_url=s3_schema_url):
         sync: Sync = Sync(
             doc,
             verbose=verbose,

{pgsync-4.1.0 → pgsync-4.2.1}/bin/parallel_sync

@@ -47,16 +47,21 @@ import sys
 import typing as t
 from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
 from dataclasses import dataclass
-from pathlib import Path
 from queue import Queue
 from threading import Thread
 
 import click
 import sqlalchemy as sa
 
-from pgsync.settings import BLOCK_SIZE, CHECKPOINT_PATH
+from pgsync.settings import BLOCK_SIZE, CHECKPOINT_PATH, S3_SCHEMA_URL, SCHEMA
 from pgsync.sync import Sync
-from pgsync.utils import config_loader, get_config, show_settings, timeit
+from pgsync.utils import (
+    config_loader,
+    MutuallyExclusiveOption,
+    show_settings,
+    timeit,
+    validate_config,
+)
 
 
 def save_ctid(page: int, row: int, filename: str) -> None:
@@ -378,6 +383,19 @@ def run_task(
     "-c",
     help="Schema config",
     type=click.Path(exists=True),
+    default=SCHEMA,
+    show_default=True,
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=["s3_schema_url"],
+)
+@click.option(
+    "--s3_schema_url",
+    help="S3 URL for schema config",
+    type=click.STRING,
+    default=S3_SCHEMA_URL,
+    show_default=True,
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=["config"],
 )
 @click.option(
     "--verbose",
@@ -409,17 +427,20 @@ def run_task(
     ),
     default="multiprocess_async",
 )
-def main(config: str, nprocs: int, mode: str, verbose: bool) -> None:
+def main(
+    config: str, s3_schema_url: str, nprocs: int, mode: str, verbose: bool
+) -> None:
     """
     TODO:
     - Track progress across cpus/threads
     - Handle KeyboardInterrupt Exception
     """
-    config: str = get_config(config)
 
-    show_settings(config)
+    validate_config(config=config, s3_schema_url=s3_schema_url)
+
+    show_settings(config=config, s3_schema_url=s3_schema_url)
 
-    for doc in config_loader(config):
+    for doc in config_loader(config=config, s3_schema_url=s3_schema_url):
         tasks: t.Generator = fetch_tasks(doc)
         if mode == "synchronous":
             synchronous(tasks, doc, verbose=verbose)

{pgsync-4.1.0 → pgsync-4.2.1}/pgsync/__init__.py

@@ -2,4 +2,4 @@
 
 __author__ = "Tolu Aina"
 __email__ = "tolu@pgsync.com"
-__version__ = "4.1.0"
+__version__ = "4.2.1"

{pgsync-4.1.0 → pgsync-4.2.1}/pgsync/base.py

@@ -2,6 +2,7 @@
 
 import logging
 import os
+import threading
 import time
 import typing as t
 from contextlib import contextmanager
@@ -28,8 +29,13 @@ from .exc import (
     TableNotFoundError,
 )
 from .settings import (
+    PG_HOST_RO,
+    PG_PASSWORD_RO,
+    PG_PORT_RO,
     PG_SSLMODE,
     PG_SSLROOTCERT,
+    PG_URL_RO,
+    PG_USER_RO,
     QUERY_CHUNK_SIZE,
     STREAM_RESULTS,
 )
@@ -48,7 +54,6 @@ try:
 except ImportError:
     pass
 
-
 logger = logging.getLogger(__name__)
 
 SSL_MODES = (
@@ -153,6 +158,8 @@ class TupleIdentifierType(sa.types.UserDefinedType):
 
 
 class Base(object):
+    _thread_local = threading.local()
+
     INT_TYPES = (
         "bigint",
         "bigserial",
@@ -190,6 +197,26 @@ class Base(object):
         self.__engine: sa.engine.Engine = _pg_engine(
             database, echo=False, **kwargs
         )
+        self.__engine_ro: t.Optional[sa.engine.Engine] = None
+        if (
+            PG_USER_RO
+            or PG_HOST_RO
+            or PG_PASSWORD_RO
+            or PG_PORT_RO
+            or PG_URL_RO
+        ):
+            kwargs.update(
+                {
+                    "user": PG_USER_RO,
+                    "host": PG_HOST_RO,
+                    "password": PG_PASSWORD_RO,
+                    "port": PG_PORT_RO,
+                    "url": PG_URL_RO,
+                }
+            )
+            self.__engine_ro: sa.engine.Engine = _pg_engine(
+                database, echo=False, **kwargs
+            )
         self.__schemas: t.Optional[dict] = None
         # models is a dict of f'{schema}.{table}'
         self.__models: dict = {}
@@ -307,6 +334,8 @@ class Base(object):
     @property
     def engine(self) -> sa.engine.Engine:
         """Get the database engine."""
+        if getattr(self._thread_local, "read_only", False):
+            return self.__engine_ro
         return self.__engine
 
     @property
@@ -910,6 +939,37 @@ class Base(object):
             label="txid_current",
         )[0]
 
+    def pg_visible_in_snapshot(
+        self, literal_binds: bool = False
+    ) -> t.Callable[[t.List[int]], dict]:
+        def _pg_visible_in_snapshot(xid8s: t.List[int]) -> dict:
+            if not xid8s:
+                return {}
+            # TODO: use the SQLAlchemy ORM to handle this query
+            statement = sa.text(
+                """
+                SELECT xid AS xid8,
+                PG_VISIBLE_IN_SNAPSHOT(xid::xid8, PG_CURRENT_SNAPSHOT()) AS visible
+                FROM UNNEST(CAST(:xid8s AS text[]))
+                WITH ORDINALITY AS t(xid, ord)
+                ORDER BY t.ord
+            """
+            )
+            if self.verbose:
+                compiled_query(
+                    statement,
+                    label="xmin_visibility",
+                    literal_binds=literal_binds,
+                )
+
+            # xid8s = list of xid8 strings
+            params: dict = {"xid8s": list(map(str, xid8s))}
+            with self.__engine_ro.connect() as conn:
+                result = conn.execute(statement, params)
+                return {int(row.xid8): row.visible for row in result}
+
+        return _pg_visible_in_snapshot
+
     def parse_value(self, type_: str, value: str) -> t.Optional[str]:
         """
         Parse datatypes from db.
@@ -1168,6 +1228,7 @@ def _pg_engine(
     echo: bool = False,
     sslmode: t.Optional[str] = None,
     sslrootcert: t.Optional[str] = None,
+    url: t.Optional[str] = None,
 ) -> sa.engine.Engine:
     connect_args: dict = {}
     sslmode = sslmode or PG_SSLMODE
@@ -1187,13 +1248,14 @@ def _pg_engine(
             )
         connect_args["sslrootcert"] = sslrootcert
 
-    url: str = get_postgres_url(
-        database,
-        user=user,
-        host=host,
-        password=password,
-        port=port,
-    )
+    if url is None:
+        url: str = get_postgres_url(
+            database,
+            user=user,
+            host=host,
+            password=password,
+            port=port,
+        )
     return sa.create_engine(url, echo=echo, connect_args=connect_args)
 
 

{pgsync-4.1.0 → pgsync-4.2.1}/pgsync/helper.py

@@ -8,7 +8,7 @@ import sqlalchemy as sa
 
 from .base import database_exists, drop_database
 from .sync import Sync
-from .utils import config_loader, get_config
+from .utils import config_loader, validate_config
 
 logger = logging.getLogger(__name__)
 
@@ -20,6 +20,7 @@ def teardown(
     drop_index: bool = True,
     delete_checkpoint: bool = True,
     config: t.Optional[str] = None,
+    s3_schema_url: t.Optional[str] = None,
     validate: bool = False,
 ) -> None:
     """
@@ -28,15 +29,15 @@ def teardown(
     Args:
         drop_db (bool, optional): Whether to drop the database. Defaults to True.
         truncate_db (bool, optional): Whether to truncate the database. Defaults to True.
-        delete_redis (bool, optional): Whether to delete Redis. Defaults to True.
+        delete_redis (bool, optional): Whether to delete Redis/Valkey. Defaults to True.
         drop_index (bool, optional): Whether to drop the index. Defaults to True.
         delete_checkpoint (bool, optional): Whether to delete the checkpoint. Defaults to True.
         config (Optional[str], optional): The configuration file path. Defaults to None.
         validate (bool, optional): Whether to validate the configuration. Defaults to False.
     """
-    config: str = get_config(config)
+    validate_config(config=config, s3_schema_url=s3_schema_url)
 
-    for doc in config_loader(config):
+    for doc in config_loader(config=config, s3_schema_url=s3_schema_url):
         if not database_exists(doc["database"]):
             logger.warning(f'Database {doc["database"]} does not exist')
             continue

{pgsync-4.1.0 → pgsync-4.2.1}/pgsync/node.py

@@ -133,6 +133,7 @@ class Node(object):
     relationship: t.Optional[dict] = None
     parent: t.Optional[Node] = None
     base_tables: t.Optional[list] = None
+    is_through: bool = False
 
     def __post_init__(self):
         self.model: sa.sql.Alias = self.models(self.table, self.schema)
@@ -328,8 +329,9 @@ class Tree(threading.local):
             self.root = node
 
         self.tables.add(node.table)
-        for through in node.relationship.throughs:
-            self.tables.add(through.table)
+        for through_node in node.relationship.throughs:
+            through_node.is_through = True
+            self.tables.add(through_node.table)
 
         for child in nodes.get("children", []):
             node.add_child(self.build(child))