pgsync 4.1.0__tar.gz → 4.2.0__tar.gz

{pgsync-4.1.0 → pgsync-4.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pgsync
-Version: 4.1.0
+Version: 4.2.0
 Summary: Postgres to Elasticsearch/OpenSearch sync
 Home-page: https://github.com/toluaina/pgsync
 Author: Tolu Aina
@@ -32,15 +32,15 @@ License-File: LICENSE
 License-File: AUTHORS.rst
 Requires-Dist: async-timeout==5.0.1
 Requires-Dist: backports-datetime-fromisoformat==2.0.3
-Requires-Dist: boto3==1.38.44
-Requires-Dist: botocore==1.38.44
-Requires-Dist: certifi==2025.6.15
+Requires-Dist: boto3==1.40.1
+Requires-Dist: botocore==1.40.1
+Requires-Dist: certifi==2025.8.3
 Requires-Dist: charset-normalizer==3.4.2
 Requires-Dist: click==8.1.8
 Requires-Dist: elastic-transport==8.17.1
-Requires-Dist: elasticsearch==8.18.1
-Requires-Dist: elasticsearch-dsl==8.18.0
-Requires-Dist: environs==14.2.0
+Requires-Dist: elasticsearch==8.19.0
+Requires-Dist: elasticsearch-dsl==8.15.4
+Requires-Dist: environs==14.3.0
 Requires-Dist: events==0.5
 Requires-Dist: greenlet==3.2.3
 Requires-Dist: idna==3.10
@@ -54,11 +54,11 @@ Requires-Dist: python-dotenv==1.1.1
 Requires-Dist: redis==6.2.0
 Requires-Dist: requests==2.32.4
 Requires-Dist: requests-aws4auth==1.3.1
-Requires-Dist: s3transfer==0.13.0
+Requires-Dist: s3transfer==0.13.1
 Requires-Dist: six==1.17.0
-Requires-Dist: sqlalchemy==2.0.41
+Requires-Dist: sqlalchemy==2.0.42
 Requires-Dist: sqlparse==0.5.3
-Requires-Dist: typing-extensions==4.14.0
+Requires-Dist: typing-extensions==4.14.1
 Requires-Dist: urllib3==1.26.20
 Dynamic: author
 Dynamic: author-email
@@ -90,7 +90,7 @@ expose structured denormalized documents in [Elasticsearch](https://www.elastic.
 - [Postgres](https://www.postgresql.org) 9.6+
 - [Redis](https://redis.io) 3.1.0+
 - [Elasticsearch](https://www.elastic.co/products/elastic-stack) 6.3.1+ or [OpenSearch](https://opensearch.org/) 1.3.7+
-- [SQlAlchemy](https://www.sqlalchemy.org) 1.3.4+
+- [SQLAlchemy](https://www.sqlalchemy.org) 1.3.4+
 
 ### Postgres setup
   

{pgsync-4.1.0 → pgsync-4.2.0}/README.md

@@ -40,7 +40,7 @@ of engineering and development.
 Other benefits of PGSync include:
 - Real-time analytics
 - Reliable primary datastore/source of truth
-- Scale on-demand
+- Scale on-demand (multiple consumers)
 - Easily join multiple nested tables
 
 #### Why?
@@ -66,7 +66,7 @@ the search capabilities of [Elasticsearch](https://www.elastic.co/products/elast
 
 #### How it works
 
-PGSync is written in Python (supporting version 3.9 onwards) and the stack is composed of: [Redis](https://redis.io), [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/), [Postgres](https://www.postgresql.org), and [SQlAlchemy](https://www.sqlalchemy.org).
+PGSync is written in Python (supporting version 3.9 onwards) and the stack is composed of: [Redis](https://redis.io), [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/), [Postgres](https://www.postgresql.org), and [SQLAlchemy](https://www.sqlalchemy.org).
 
 PGSync leverages the [logical decoding](https://www.postgresql.org/docs/current/logicaldecoding.html) feature of [Postgres](https://www.postgresql.org) (introduced in PostgreSQL 9.4) to capture a continuous stream of change events.
 This feature needs to be enabled in your [Postgres](https://www.postgresql.org) configuration file by setting in the postgresql.conf file:
@@ -93,9 +93,14 @@ There are several ways of installing and trying PGSync
  - [Manual configuration](#manual-configuration) 
 
 
-##### Running in Docker
+##### Running in Docker (Using Github Repository)
 
 To startup all services with docker.
+
+```
+$ git checkout https://github.com/toluaina/pgsync
+```
+
 Run:
 ```
 $ docker-compose up
@@ -106,6 +111,39 @@ Show the content in Elasticsearch/OpenSearch
 $ curl -X GET http://[Elasticsearch/OpenSearch host]:9201/reservations/_search?pretty=true
 ```
 
+
+##### Running with Docker (Using Image Repository)
+
+To start all services with Docker, follow these steps:
+
+1. Pull the Docker image:
+
+  ```
+  $ docker pull toluaina1/pgsync:latest
+  ```
+
+2. Run the container:
+
+  ```
+  $ docker run --rm -it \
+  -e REDIS_CHECKPOINT=true \
+  -e REDIS_HOST=<redis_host_address> \
+  -e PG_URL=postgres://<username>:<password>@<postgres_host>/<database> \
+  -e ELASTICSEARCH_URL=http://<elasticsearch_host>:9200 \
+  -v "$(pwd)/schema.json:/app/schema.json" \
+  toluaina1/pgsync:latest -c schema.json -d -b
+  ```
+
+Environment variable placeholders - full list [here](https://pgsync.com/env-vars):
+
+- redis_host_address — Address of the Redis server (e.g., host.docker.internal for local Docker setup)
+- username — PostgreSQL username
+- password — PostgreSQL password
+- postgres_host — Host address for PostgreSQL instance (e.g., host.docker.internal)
+- database — Name of PostgreSQL database
+- elasticsearch_host — Address of Elasticsearch/OpenSearch instance (e.g., host.docker.internal)
+
+
 ##### Manual configuration
 
 - Setup
@@ -156,7 +194,7 @@ Key features of PGSync are:
 - [Postgres](https://www.postgresql.org) 9.6+
 - [Redis](https://redis.io) 3.1.0+
 - [Elasticsearch](https://www.elastic.co/products/elastic-stack) 6.3.1+ or [OpenSearch](https://opensearch.org/) 1.3.7+
-- [SQlAlchemy](https://www.sqlalchemy.org) 1.3.4+
+- [SQLAlchemy](https://www.sqlalchemy.org) 1.3.4+
 
 
 #### Example

{pgsync-4.1.0 → pgsync-4.2.0}/README.rst

@@ -12,7 +12,7 @@ expose structured denormalized documents in [Elasticsearch](https://www.elastic.
 - [Postgres](https://www.postgresql.org) 9.6+
 - [Redis](https://redis.io) 3.1.0+
 - [Elasticsearch](https://www.elastic.co/products/elastic-stack) 6.3.1+ or [OpenSearch](https://opensearch.org/) 1.3.7+
-- [SQlAlchemy](https://www.sqlalchemy.org) 1.3.4+
+- [SQLAlchemy](https://www.sqlalchemy.org) 1.3.4+
 
 ### Postgres setup
   

{pgsync-4.1.0 → pgsync-4.2.0}/bin/bootstrap

@@ -5,8 +5,14 @@ import logging
 
 import click
 
+from pgsync import settings
 from pgsync.sync import Sync
-from pgsync.utils import config_loader, get_config, show_settings
+from pgsync.utils import (
+    config_loader,
+    MutuallyExclusiveOption,
+    show_settings,
+    validate_config,
+)
 
 logger = logging.getLogger(__name__)
 
@@ -17,6 +23,19 @@ logger = logging.getLogger(__name__)
     "-c",
     help="Schema config",
     type=click.Path(exists=True),
+    default=settings.SCHEMA,
+    show_default=True,
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=["s3_schema_url"],
+)
+@click.option(
+    "--s3_schema_url",
+    help="S3 URL for schema config",
+    type=click.STRING,
+    default=settings.S3_SCHEMA_URL,
+    show_default=True,
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=["config"],
 )
 @click.option("--host", "-h", help="PG_HOST override")
 @click.option("--password", is_flag=True, help="Prompt for database password")
@@ -48,6 +67,7 @@ logger = logging.getLogger(__name__)
 def main(
     teardown: bool,
     config: str,
+    s3_schema_url: str,
     user: str,
     password: bool,
     host: str,
@@ -69,13 +89,13 @@ def main(
         )
     kwargs = {key: value for key, value in kwargs.items() if value is not None}
 
-    config: str = get_config(config)
+    validate_config(config=config, s3_schema_url=s3_schema_url)
 
-    show_settings(config)
+    show_settings(config=config, s3_schema_url=s3_schema_url)
 
     validate: bool = False if teardown else True
 
-    for doc in config_loader(config):
+    for doc in config_loader(config=config, s3_schema_url=s3_schema_url):
         sync: Sync = Sync(
             doc,
             verbose=verbose,

{pgsync-4.1.0 → pgsync-4.2.0}/bin/parallel_sync

@@ -47,16 +47,21 @@ import sys
 import typing as t
 from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
 from dataclasses import dataclass
-from pathlib import Path
 from queue import Queue
 from threading import Thread
 
 import click
 import sqlalchemy as sa
 
-from pgsync.settings import BLOCK_SIZE, CHECKPOINT_PATH
+from pgsync.settings import BLOCK_SIZE, CHECKPOINT_PATH, S3_SCHEMA_URL, SCHEMA
 from pgsync.sync import Sync
-from pgsync.utils import config_loader, get_config, show_settings, timeit
+from pgsync.utils import (
+    config_loader,
+    MutuallyExclusiveOption,
+    show_settings,
+    timeit,
+    validate_config,
+)
 
 
 def save_ctid(page: int, row: int, filename: str) -> None:
@@ -378,6 +383,19 @@ def run_task(
     "-c",
     help="Schema config",
     type=click.Path(exists=True),
+    default=SCHEMA,
+    show_default=True,
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=["s3_schema_url"],
+)
+@click.option(
+    "--s3_schema_url",
+    help="S3 URL for schema config",
+    type=click.STRING,
+    default=S3_SCHEMA_URL,
+    show_default=True,
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=["config"],
 )
 @click.option(
     "--verbose",
@@ -409,17 +427,20 @@ def run_task(
     ),
     default="multiprocess_async",
 )
-def main(config: str, nprocs: int, mode: str, verbose: bool) -> None:
+def main(
+    config: str, s3_schema_url: str, nprocs: int, mode: str, verbose: bool
+) -> None:
     """
     TODO:
     - Track progress across cpus/threads
     - Handle KeyboardInterrupt Exception
     """
-    config: str = get_config(config)
 
-    show_settings(config)
+    validate_config(config=config, s3_schema_url=s3_schema_url)
+
+    show_settings(config=config, s3_schema_url=s3_schema_url)
 
-    for doc in config_loader(config):
+    for doc in config_loader(config=config, s3_schema_url=s3_schema_url):
         tasks: t.Generator = fetch_tasks(doc)
         if mode == "synchronous":
             synchronous(tasks, doc, verbose=verbose)

{pgsync-4.1.0 → pgsync-4.2.0}/pgsync/__init__.py

@@ -2,4 +2,4 @@
 
 __author__ = "Tolu Aina"
 __email__ = "tolu@pgsync.com"
-__version__ = "4.1.0"
+__version__ = "4.2.0"

{pgsync-4.1.0 → pgsync-4.2.0}/pgsync/base.py

@@ -2,6 +2,7 @@
 
 import logging
 import os
+import threading
 import time
 import typing as t
 from contextlib import contextmanager
@@ -28,8 +29,13 @@ from .exc import (
     TableNotFoundError,
 )
 from .settings import (
+    PG_HOST_RO,
+    PG_PASSWORD_RO,
+    PG_PORT_RO,
     PG_SSLMODE,
     PG_SSLROOTCERT,
+    PG_URL_RO,
+    PG_USER_RO,
     QUERY_CHUNK_SIZE,
     STREAM_RESULTS,
 )
@@ -48,7 +54,6 @@ try:
 except ImportError:
     pass
 
-
 logger = logging.getLogger(__name__)
 
 SSL_MODES = (
@@ -153,6 +158,8 @@ class TupleIdentifierType(sa.types.UserDefinedType):
 
 
 class Base(object):
+    _thread_local = threading.local()
+
     INT_TYPES = (
         "bigint",
         "bigserial",
@@ -190,6 +197,26 @@ class Base(object):
         self.__engine: sa.engine.Engine = _pg_engine(
             database, echo=False, **kwargs
         )
+        self.__engine_ro: t.Optional[sa.engine.Engine] = None
+        if (
+            PG_USER_RO
+            or PG_HOST_RO
+            or PG_PASSWORD_RO
+            or PG_PORT_RO
+            or PG_URL_RO
+        ):
+            kwargs.update(
+                {
+                    "user": PG_USER_RO,
+                    "host": PG_HOST_RO,
+                    "password": PG_PASSWORD_RO,
+                    "port": PG_PORT_RO,
+                    "url": PG_URL_RO,
+                }
+            )
+            self.__engine_ro: sa.engine.Engine = _pg_engine(
+                database, echo=False, **kwargs
+            )
         self.__schemas: t.Optional[dict] = None
         # models is a dict of f'{schema}.{table}'
         self.__models: dict = {}
@@ -307,6 +334,8 @@ class Base(object):
     @property
     def engine(self) -> sa.engine.Engine:
         """Get the database engine."""
+        if getattr(self._thread_local, "read_only", False):
+            return self.__engine_ro
         return self.__engine
 
     @property
@@ -910,6 +939,37 @@ class Base(object):
             label="txid_current",
         )[0]
 
+    def pg_visible_in_snapshot(
+        self, literal_binds: bool = False
+    ) -> t.Callable[[t.List[int]], dict]:
+        def _pg_visible_in_snapshot(xid8s: t.List[int]) -> dict:
+            if not xid8s:
+                return {}
+            # TODO: use the SQLAlchemy ORM to handle this query
+            statement = sa.text(
+                """
+                SELECT xid AS xid8,
+                PG_VISIBLE_IN_SNAPSHOT(xid::xid8, PG_CURRENT_SNAPSHOT()) AS visible
+                FROM UNNEST(CAST(:xid8s AS text[]))
+                WITH ORDINALITY AS t(xid, ord)
+                ORDER BY t.ord
+            """
+            )
+            if self.verbose:
+                compiled_query(
+                    statement,
+                    label="xmin_visibility",
+                    literal_binds=literal_binds,
+                )
+
+            # xid8s = list of xid8 strings
+            params: dict = {"xid8s": list(map(str, xid8s))}
+            with self.__engine_ro.connect() as conn:
+                result = conn.execute(statement, params)
+                return {int(row.xid8): row.visible for row in result}
+
+        return _pg_visible_in_snapshot
+
     def parse_value(self, type_: str, value: str) -> t.Optional[str]:
         """
         Parse datatypes from db.
@@ -1168,6 +1228,7 @@ def _pg_engine(
     echo: bool = False,
     sslmode: t.Optional[str] = None,
     sslrootcert: t.Optional[str] = None,
+    url: t.Optional[str] = None,
 ) -> sa.engine.Engine:
     connect_args: dict = {}
     sslmode = sslmode or PG_SSLMODE
@@ -1187,13 +1248,14 @@ def _pg_engine(
             )
         connect_args["sslrootcert"] = sslrootcert
 
-    url: str = get_postgres_url(
-        database,
-        user=user,
-        host=host,
-        password=password,
-        port=port,
-    )
+    if url is None:
+        url: str = get_postgres_url(
+            database,
+            user=user,
+            host=host,
+            password=password,
+            port=port,
+        )
     return sa.create_engine(url, echo=echo, connect_args=connect_args)
 
 

{pgsync-4.1.0 → pgsync-4.2.0}/pgsync/helper.py

@@ -8,7 +8,7 @@ import sqlalchemy as sa
 
 from .base import database_exists, drop_database
 from .sync import Sync
-from .utils import config_loader, get_config
+from .utils import config_loader, validate_config
 
 logger = logging.getLogger(__name__)
 
@@ -20,6 +20,7 @@ def teardown(
     drop_index: bool = True,
     delete_checkpoint: bool = True,
     config: t.Optional[str] = None,
+    s3_schema_url: t.Optional[str] = None,
     validate: bool = False,
 ) -> None:
     """
@@ -34,9 +35,9 @@ def teardown(
         config (Optional[str], optional): The configuration file path. Defaults to None.
         validate (bool, optional): Whether to validate the configuration. Defaults to False.
     """
-    config: str = get_config(config)
+    validate_config(config=config, s3_schema_url=s3_schema_url)
 
-    for doc in config_loader(config):
+    for doc in config_loader(config=config, s3_schema_url=s3_schema_url):
         if not database_exists(doc["database"]):
             logger.warning(f'Database {doc["database"]} does not exist')
             continue

{pgsync-4.1.0 → pgsync-4.2.0}/pgsync/redisqueue.py

@@ -52,6 +52,34 @@ class RedisQueue(object):
             logger.debug(f"pop size: {len(items[0])}")
             return list(map(lambda value: json.loads(value), items[0]))
 
+    def pop_visible_in_snapshot(
+        self,
+        pg_visible_in_snapshot: t.Callable[[t.List[int]], dict],
+        chunk_size: t.Optional[int] = None,
+    ) -> t.List[dict]:
+        """
+        Pop items in the queue that are visible in the current snapshot.
+        Uses the provided pg_visible_in_snapshot function to determine visibility.
+        This function is useful for read-only consumers that need to process items
+        that are visible in the current PostgreSQL snapshot.
+        """
+        chunk_size = chunk_size or REDIS_READ_CHUNK_SIZE
+        items: t.List = self.__db.lrange(self.key, 0, chunk_size - 1)
+        if not items:
+            return []
+        payloads = [json.loads(i) for i in items]
+        visible_map: dict = pg_visible_in_snapshot()(
+            [payload["xmin"] for payload in payloads]
+        )
+        visible: t.List[dict] = []
+        for item, payload in zip(items, payloads):
+            if visible_map.get(payload["xmin"]):
+                # Claim atomically
+                removed = self.__db.lrem(self.key, 1, item)
+                if removed:
+                    visible.append(payload)
+        return visible
+
     def push(self, items: t.List) -> None:
         """Push multiple items onto the queue."""
         self.__db.rpush(self.key, *map(json.dumps, items))

{pgsync-4.1.0 → pgsync-4.2.0}/pgsync/settings.py

@@ -43,6 +43,7 @@ REPLICATION_SLOT_CLEANUP_INTERVAL = env.float(
 )
 # path to the application schema config
 SCHEMA = env.str("SCHEMA", default=None)
+S3_SCHEMA_URL = env.str("S3_SCHEMA_URL", default=None)
 USE_ASYNC = env.bool("USE_ASYNC", default=False)
 STREAM_RESULTS = env.bool("STREAM_RESULTS", default=True)
 # db polling interval
@@ -173,9 +174,26 @@ if PG_URL:
     PG_USER = env.str("PG_USER", default=None)
 else:
     # If PG_URL is not set, we need to use the other PG_* variables
-    PG_URL = None
     PG_USER = env.str("PG_USER")
 
+# Read-only Postgres:
+# This is used for read-only consumers that do not require replication slots or triggers.
+# full database url including user, password, host, port and dbname
+PG_URL_RO = env.str("PG_URL_RO", default=None)
+PG_HOST_RO = env.str("PG_HOST_RO", default=None)
+PG_PASSWORD_RO = env.str("PG_PASSWORD_RO", default=None)
+PG_PORT_RO = env.int("PG_PORT_RO", default=None)
+PG_SSLMODE_RO = env.str("PG_SSLMODE_RO", default=None)
+PG_SSLROOTCERT_RO = env.str("PG_SSLROOTCERT_RO", default=None)
+PG_USER_RO = env.str("PG_USER_RO", default=None)
+if PG_URL_RO:
+    # If PG_URL_RO is set, we don't need to use the other PG_*_RO variables
+    PG_HOST_RO = None
+    PG_PASSWORD_RO = None
+    PG_PORT_RO = None
+    PG_SSLMODE_RO = None
+    PG_SSLROOTCERT_RO = None
+
 # Redis:
 REDIS_AUTH = env.str("REDIS_AUTH", default=None)
 REDIS_USER = env.str("REDIS_USER", default=None)