pgsync 5.0.0__tar.gz → 6.1.0__tar.gz

{pgsync-5.0.0 → pgsync-6.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pgsync
-Version: 5.0.0
+Version: 6.1.0
 Summary: Postgres/MySQL/MariaDB to Elasticsearch/OpenSearch sync
 Home-page: https://github.com/toluaina/pgsync
 Author: Tolu Aina
@@ -22,6 +22,7 @@ Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Programming Language :: Python :: Implementation :: PyPy
 Classifier: License :: OSI Approved :: MIT License
@@ -32,31 +33,31 @@ License-File: LICENSE
 License-File: AUTHORS.rst
 Requires-Dist: async-timeout==5.0.1
 Requires-Dist: backports-datetime-fromisoformat==2.0.3
-Requires-Dist: boto3==1.40.50
-Requires-Dist: botocore==1.40.50
-Requires-Dist: certifi==2025.10.5
-Requires-Dist: charset-normalizer==3.4.3
+Requires-Dist: boto3==1.41.2
+Requires-Dist: botocore==1.41.2
+Requires-Dist: certifi==2025.11.12
+Requires-Dist: charset-normalizer==3.4.4
 Requires-Dist: click==8.1.8
 Requires-Dist: elastic-transport==9.1.0
 Requires-Dist: elasticsearch==7.17.12
 Requires-Dist: elasticsearch-dsl==7.4.1
-Requires-Dist: environs==14.3.0
+Requires-Dist: environs==14.4.0
 Requires-Dist: events==0.5
-Requires-Dist: idna==3.10
+Requires-Dist: idna==3.11
 Requires-Dist: jmespath==1.0.1
 Requires-Dist: marshmallow==4.0.1
-Requires-Dist: mysql-replication==1.0.9
+Requires-Dist: mysql-replication==1.0.12
 Requires-Dist: opensearch-dsl==2.1.0
 Requires-Dist: opensearch-py==3.0.0
 Requires-Dist: packaging==25.0
 Requires-Dist: psycopg2-binary==2.9.11
 Requires-Dist: pymysql==1.1.2
 Requires-Dist: python-dateutil==2.9.0.post0
-Requires-Dist: python-dotenv==1.1.1
-Requires-Dist: redis==6.4.0
+Requires-Dist: python-dotenv==1.2.1
+Requires-Dist: redis==7.0.1
 Requires-Dist: requests==2.32.5
 Requires-Dist: requests-aws4auth==1.3.1
-Requires-Dist: s3transfer==0.14.0
+Requires-Dist: s3transfer==0.15.0
 Requires-Dist: six==1.17.0
 Requires-Dist: sqlalchemy==2.0.44
 Requires-Dist: sqlparse==0.5.3
@@ -78,7 +79,7 @@ Dynamic: requires-dist
 Dynamic: requires-python
 Dynamic: summary
 
-# PostgreSQL to Elasticsearch/OpenSearch sync
+# PostgreSQL/MySQL/MariaDB to Elasticsearch/OpenSearch sync
 
 
 - [PGSync](https://pgsync.com) is a middleware for syncing data from [Postgres](https://www.postgresql.org) to [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) or [OpenSearch](https://opensearch.org/).

{pgsync-5.0.0 → pgsync-6.1.0}/README.md

@@ -5,6 +5,7 @@
 [![Python versions](https://img.shields.io/pypi/pyversions/pgsync)](https://pypi.org/project/pgsync)
 [![Downloads](https://img.shields.io/pypi/dm/pgsync)](https://pypi.org/project/pgsync)
 [![codecov](https://codecov.io/gh/toluaina/pgsync/branch/main/graph/badge.svg?token=cvQzYkz6CV)](https://codecov.io/gh/toluaina/pgsync)
+[![Sponsored by DigitalOcean](https://img.shields.io/badge/Sponsored%20by-DigitalOcean-0080FF?logo=digitalocean&logoColor=white)](https://www.digitalocean.com/?utm_medium=opensource&utm_source=pgsync)
 
 
 ## PostgreSQL/MySQL/MariaDB to Elasticsearch/OpenSearch sync
@@ -43,6 +44,23 @@ Other benefits of PGSync include:
 - Scale on-demand (multiple consumers)
 - Easily join multiple nested tables
 
+## Sponsors
+
+[PGSync](https://pgsync.com) is made possible with support from [DigitalOcean](https://www.digitalocean.com/?utm_medium=opensource&utm_source=pgsync).
+
+<p>
+  <a href="https://www.digitalocean.com/?utm_medium=opensource&utm_source=pgsync" rel="sponsored noopener noreferrer">
+    <img
+      src="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/SVG/DO_Logo_horizontal_blue.svg"
+      alt="DigitalOcean"
+      width="210"
+      loading="lazy"
+      decoding="async"
+    >
+  </a>
+</p>
+
+
 #### Why?
 
 At a high level, you have data in a PostgreSQL/MySQL/MariaDB database and you want to mirror it in Elasticsearch/OpenSearch.  
@@ -55,7 +73,7 @@ Of course, if your data never changed, then you could just take a snapshot in ti
 PGSync is appropriate for you if:
 - [Postgres](https://www.postgresql.org) or [MySQL](https://www.mysql.com/) or [MariaDB](https://mariadb.org/) is your read/write source of truth whilst [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) is your 
 read-only search layer.
-- You need to denormalize relational data into a NoSQL data source.
+- You need to denormalize relational data into a NoSQL data source like [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/).
 - Your data is constantly changing.
 - You have existing data in a relational database such as [Postgres](https://www.postgresql.org) or [MySQL](https://www.mysql.com/) or [MariaDB](https://mariadb.org/) and you need a secondary NoSQL database like [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) for text-based queries or autocomplete queries to mirror the existing data without having your application perform dual writes.
 - You want to keep your existing data untouched whilst taking advantage of
@@ -81,7 +99,8 @@ PGSync operates in an event-driven model by creating triggers for tables in your
 
 *This is the only time PGSync will ever make any changes to your database.*
 
-**NOTE**: **If you change the structure of your PGSync's schema config, you would need to rebuild your Elasticsearch/OpenSearch indices.**
+>**NOTE**: **If you change the structure of your PGSync schema config, it's recommended and in most cases necessary to rebuild your Elasticsearch/OpenSearch indices.**
+
 There are plans to support zero-downtime migrations to streamline this process.
 
 
@@ -91,6 +110,14 @@ There are several ways of installing and trying PGSync
  - [Running in Docker](#running-in-docker) is the easiest way to get up and running.
  - [Manual configuration](#manual-configuration) 
 
+#### Book Demo Example (requires a DigitalOcean account)
+
+[![Deploy to DO](https://www.deploytodo.com/do-btn-blue.svg)](https://cloud.digitalocean.com/apps/new?repo=https://github.com/toluaina/pgsync/tree/main)
+
+Fill in the following during the setup
+ - `ELASTICSEARCH_URL` e.g. https://user:pass@os-host:443
+ - `REDIS_URL` e.g. rediss://default:pass@host:port/0
+
 
 ##### Running in Docker (Using Github Repository)
 
@@ -161,7 +188,6 @@ Environment variable placeholders - full list [here](https://pgsync.com/env-vars
 ### MySQL / MariaDB setup
 
 - Enable binary logging in your MySQL / MariaDB setting.
-
 - You also need to set up the following parameters in your MySQL / MariaDB config my.cnf, then restart the database server.
 
   ```server-id = 1``` # any non-zero unique ID
@@ -169,10 +195,8 @@ Environment variable placeholders - full list [here](https://pgsync.com/env-vars
   ```log_bin = mysql-bin```
 
   ```binlog_row_image = FULL``` # recommended; if not supported on older MariaDB, omit
-
 - optional housekeeping:
   ```binlog_expire_logs_seconds = 604800```   # 7 days
-
 - You need to create a replication user with REPLICATION SLAVE and REPLICATION CLIENT privileges
     
   ```sql
@@ -198,17 +222,19 @@ Environment variable placeholders - full list [here](https://pgsync.com/env-vars
 
 Key features of PGSync are:
 
-- Easily denormalize relational data. 
-- Works with any PostgreSQL database (version 9.6 or later). 
-- Negligible impact on database performance.
-- Transactionally consistent output in Elasticsearch/OpenSearch. This means: writes appear only when they are committed to the database, insert, update and delete operations appear in the same order as they were committed (as opposed to eventual consistency).
-- Fault-tolerant: does not lose data, even if processes crash or a network interruption occurs, etc. The process can be recovered from the last checkpoint.
-- Returns the data directly as Postgres/MySQL/MariaDB JSON from the database for speed.
-- Supports composite primary and foreign keys.
-- Supports Views and Materialized views.
-- Supports an arbitrary depth of nested entities i.e Tables having long chain of relationship dependencies.
-- Supports PostgreSQL/MySQL/MariaDB JSON data fields. This means: we can extract JSON fields in a database table as a separate field in the resulting document.
-- Customizable document structure.
+- Easily denormalize relational data
+- Works with any PostgreSQL database (9.6 or later)
+- Negligible impact on database performance
+- Transactionally consistent output in Elasticsearch/OpenSearch:
+  - Writes appear only after they’re committed
+  - Inserts, updates, and deletes appear in commit order (not eventually)
+- Fault-tolerant: no data loss even on crashes or network issues; processing resumes from the last checkpoint
+- Returns data directly as PostgreSQL/MySQL/MariaDB JSON for speed
+- Supports composite primary and foreign keys
+- Supports views and materialized views
+- Handles arbitrarily deep nesting of related tables
+- Supports PostgreSQL/MySQL/MariaDB JSON fields, allowing JSON properties to be extracted as separate document fields
+- Customizable document structure
 
 
 #### Requirements
@@ -334,23 +360,26 @@ e.g
   }
 ```
 
-PGSync addresses the following challenges:
-- What if we update the author's name in the database?
-- What if we wanted to add another author for an existing book?
-- What if we have lots of documents already with the same author we wanted to change the author name?
-- What if we delete or update an author?
-- What if we truncate an entire table?
+PGSync addresses common data consistency challenges, such as:
+
+- Updating an author's name in the database
+- Adding an additional author to an existing book
+- Changing an author's name across many existing documents
+- Deleting or updating an author record
+- Truncating an entire table and keeping indexes in sync
 
 
 #### Benefits
 
-- PGSync is a simple to use out of the box solution for Change data capture.
-- PGSync handles data deletions.
-- PGSync requires little development effort. You simply define a schema config describing your data.
-- PGSync generates advanced queries matching your schema directly.
-- PGSync allows you to easily rebuild your indexes in case of a schema change.
-- You can expose only the data you require in Elasticsearch/OpenSearch.
-- Supports multiple Postgres/MySQL/MariaDB schemas for multi-tennant applications.
+PGSync is a simple, out-of-the-box solution for change data capture, designed to minimize development effort and keep your search indexes in sync.
+
+- Handles data deletions automatically.
+- Requires minimal setup. Just define a schema config that describes your data.
+- Generates advanced queries directly from your schema.
+- Makes it easy to rebuild indexes after schema changes.
+- Lets you expose only the data you need in Elasticsearch/OpenSearch.
+- Supports multiple Postgres/MySQL/MariaDB schemas for multi-tenant applications.
+
 
 
 #### Contributing

{pgsync-5.0.0 → pgsync-6.1.0}/README.rst

@@ -1,4 +1,4 @@
-# PostgreSQL to Elasticsearch/OpenSearch sync
+# PostgreSQL/MySQL/MariaDB to Elasticsearch/OpenSearch sync
 
 
 - [PGSync](https://pgsync.com) is a middleware for syncing data from [Postgres](https://www.postgresql.org) to [Elasticsearch](https://www.elastic.co/products/elastic-stack)/[OpenSearch](https://opensearch.org/) or [OpenSearch](https://opensearch.org/).

{pgsync-5.0.0 → pgsync-6.1.0}/bin/bootstrap

@@ -26,7 +26,16 @@ logger = logging.getLogger(__name__)
     default=settings.SCHEMA,
     show_default=True,
     cls=MutuallyExclusiveOption,
-    mutually_exclusive=["s3_schema_url"],
+    mutually_exclusive=["s3_schema_url", "schema_url"],
+)
+@click.option(
+    "--schema_url",
+    help="URL for schema config",
+    type=click.STRING,
+    default=settings.SCHEMA_URL,
+    show_default=True,
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=["config", "s3_schema_url"],
 )
 @click.option(
     "--s3_schema_url",
@@ -35,7 +44,7 @@ logger = logging.getLogger(__name__)
     default=settings.S3_SCHEMA_URL,
     show_default=True,
     cls=MutuallyExclusiveOption,
-    mutually_exclusive=["config"],
+    mutually_exclusive=["config", "schema_url"],
 )
 @click.option("--host", "-h", help="PG_HOST override")
 @click.option("--password", is_flag=True, help="Prompt for database password")
@@ -67,6 +76,7 @@ logger = logging.getLogger(__name__)
 def main(
     teardown: bool,
     config: str,
+    schema_url: str,
     s3_schema_url: str,
     user: str,
     password: bool,
@@ -89,13 +99,24 @@ def main(
         )
     kwargs = {key: value for key, value in kwargs.items() if value is not None}
 
-    validate_config(config=config, s3_schema_url=s3_schema_url)
+    validate_config(
+        config=config,
+        schema_url=schema_url,
+        s3_schema_url=s3_schema_url,
+    )
 
-    show_settings(config=config, s3_schema_url=s3_schema_url)
+    show_settings(
+        config=config,
+        schema_url=schema_url,
+        s3_schema_url=s3_schema_url,
+        **kwargs,
+    )
 
     validate: bool = False if teardown else True
 
-    for doc in config_loader(config=config, s3_schema_url=s3_schema_url):
+    for doc in config_loader(
+        config=config, schema_url=schema_url, s3_schema_url=s3_schema_url
+    ):
         sync: Sync = Sync(
             doc,
             verbose=verbose,

{pgsync-5.0.0 → pgsync-6.1.0}/bin/parallel_sync

@@ -53,7 +53,13 @@ from threading import Thread
 import click
 import sqlalchemy as sa
 
-from pgsync.settings import BLOCK_SIZE, CHECKPOINT_PATH, S3_SCHEMA_URL, SCHEMA
+from pgsync.settings import (
+    BLOCK_SIZE,
+    CHECKPOINT_PATH,
+    S3_SCHEMA_URL,
+    SCHEMA,
+    SCHEMA_URL,
+)
 from pgsync.sync import Sync
 from pgsync.utils import (
     config_loader,
@@ -386,7 +392,16 @@ def run_task(
     default=SCHEMA,
     show_default=True,
     cls=MutuallyExclusiveOption,
-    mutually_exclusive=["s3_schema_url"],
+    mutually_exclusive=["s3_schema_url", "schema_url"],
+)
+@click.option(
+    "--schema_url",
+    help="URL for schema config",
+    type=click.STRING,
+    default=SCHEMA_URL,
+    show_default=True,
+    cls=MutuallyExclusiveOption,
+    mutually_exclusive=["config", "s3_schema_url"],
 )
 @click.option(
     "--s3_schema_url",
@@ -395,7 +410,7 @@ def run_task(
     default=S3_SCHEMA_URL,
     show_default=True,
     cls=MutuallyExclusiveOption,
-    mutually_exclusive=["config"],
+    mutually_exclusive=["config", "schema_url"],
 )
 @click.option(
     "--verbose",
@@ -428,7 +443,12 @@ def run_task(
     default="multiprocess_async",
 )
 def main(
-    config: str, s3_schema_url: str, nprocs: int, mode: str, verbose: bool
+    config: str,
+    schema_url: str,
+    s3_schema_url: str,
+    nprocs: int,
+    mode: str,
+    verbose: bool,
 ) -> None:
     """
     TODO:
@@ -436,11 +456,17 @@ def main(
     - Handle KeyboardInterrupt Exception
     """
 
-    validate_config(config=config, s3_schema_url=s3_schema_url)
+    validate_config(
+        config=config, schema_url=schema_url, s3_schema_url=s3_schema_url
+    )
 
-    show_settings(config=config, s3_schema_url=s3_schema_url)
+    show_settings(
+        config=config, schema_url=schema_url, s3_schema_url=s3_schema_url
+    )
 
-    for doc in config_loader(config=config, s3_schema_url=s3_schema_url):
+    for doc in config_loader(
+        config=config, schema_url=schema_url, s3_schema_url=s3_schema_url
+    ):
         tasks: t.Generator = fetch_tasks(doc)
         if mode == "synchronous":
             synchronous(tasks, doc, verbose=verbose)

{pgsync-5.0.0 → pgsync-6.1.0}/pgsync/__init__.py

@@ -2,4 +2,4 @@
 
 __author__ = "Tolu Aina"
 __email__ = "tolu@pgsync.com"
-__version__ = "5.0.0"
+__version__ = "6.1.0"

{pgsync-5.0.0 → pgsync-6.1.0}/pgsync/base.py

@@ -8,8 +8,11 @@ import time
 import typing as t
 from contextlib import contextmanager
 
+import psycopg2
 import sqlalchemy as sa
+from psycopg2.extras import LogicalReplicationConnection
 from sqlalchemy.dialects import postgresql  # noqa
+from sqlalchemy.engine.url import make_url
 from sqlalchemy.orm import sessionmaker
 
 from .constants import (
@@ -31,6 +34,8 @@ from .exc import (
 )
 from .settings import (
     IS_MYSQL_COMPAT,
+    MYSQL_DATABASE,
+    PG_DATABASE,
     PG_HOST_RO,
     PG_PASSWORD_RO,
     PG_PORT_RO,
@@ -39,6 +44,12 @@ from .settings import (
     PG_URL_RO,
     PG_USER_RO,
     QUERY_CHUNK_SIZE,
+    SQLALCHEMY_MAX_OVERFLOW,
+    SQLALCHEMY_POOL_PRE_PING,
+    SQLALCHEMY_POOL_RECYCLE,
+    SQLALCHEMY_POOL_SIZE,
+    SQLALCHEMY_POOL_TIMEOUT,
+    SQLALCHEMY_USE_NULLPOOL,
     STREAM_RESULTS,
 )
 from .trigger import CREATE_TRIGGER_TEMPLATE
@@ -230,6 +241,7 @@ class Base(object):
         self.__columns: dict = {}
         self.verbose: bool = verbose
         self._conn = None
+        self._session = None
 
     def connect(self) -> None:
         """Connect to database."""
@@ -338,8 +350,19 @@ class Base(object):
 
     @property
     def session(self) -> sessionmaker:
-        Session = sessionmaker(bind=self.engine.connect(), autoflush=True)
-        return Session()
+        if self._session is None:
+            Session = sessionmaker(bind=self.engine, autoflush=True)
+            self._session = Session()
+        return self._session
+
+    def close_session(self) -> None:
+        """Close the cached session and reset it."""
+        if self._session is not None:
+            try:
+                self._session.close()
+            except Exception:
+                pass
+            self._session = None
 
     @property
     def engine(self) -> sa.engine.Engine:
@@ -726,6 +749,21 @@ class Base(object):
             )
         )[0]
 
+    def get_replication_connection(
+        self, engine: sa.engine.Engine
+    ) -> psycopg2.extensions.connection:
+        url: sa.engine.URL = make_url(str(engine.url))
+        # Build a libpq-style connection by keyword args
+        conn: psycopg2.extensions.connection = psycopg2.connect(
+            host=url.host,
+            port=url.port or 5432,
+            user=url.username,
+            password=url.password,
+            dbname=url.database,
+            connection_factory=LogicalReplicationConnection,
+        )
+        return conn
+
     def logical_slot_get_changes(
         self,
         slot_name: str,
@@ -822,6 +860,7 @@ class Base(object):
         schema: str,
         tables: t.Set,
         user_defined_fkey_tables: dict,
+        node_columns: dict,
     ) -> None:
         create_view(
             self.engine,
@@ -832,6 +871,7 @@ class Base(object):
             tables,
             user_defined_fkey_tables,
             self._materialized_views(schema),
+            node_columns,
         )
 
     def drop_view(self, schema: str) -> None:
@@ -946,9 +986,9 @@ class Base(object):
         """Check if the trigger function exists."""
         return self.exists(
             sa.text(
-                f"SELECT 1 FROM pg_proc WHERE proname = :name "
-                f"AND pronamespace = (SELECT oid FROM pg_namespace "
-                f"WHERE nspname = :schema)"
+                "SELECT 1 FROM pg_proc WHERE proname = :name "
+                "AND pronamespace = (SELECT oid FROM pg_namespace "
+                "WHERE nspname = :schema)"
             ).bindparams(name=TRIGGER_FUNC, schema=schema),
         )
 
@@ -989,7 +1029,7 @@ class Base(object):
             self.disable_trigger(schema, table)
             logger.debug(f"Disabled trigger on table: {schema}.{table}")
 
-    def enable_trigger(self, schema: str, table, str) -> None:
+    def enable_trigger(self, schema: str, table: str) -> None:
         """Enable a pgsync defined trigger."""
         for name in ("notify", "truncate"):
             self.execute(
@@ -1107,20 +1147,21 @@ class Base(object):
         # including trailing space below is deliberate
         suffix: str = f"{row[span[1]:]} "
 
-        if "old-key" and "new-tuple" in suffix:
+        if "old-key" in suffix and "new-tuple" in suffix:
             # this can only be an UPDATE operation
             if payload.tg_op != UPDATE:
                 msg = f"Unknown {payload.tg_op} operation for row: {row}"
                 raise LogicalSlotParseError(msg)
 
-            i: int = suffix.index("old-key:")
+            i: int = suffix.find("old-key:")
             if i > -1:
-                j: int = suffix.index("new-tuple:")
-                s: str = suffix[i + len("old-key:") : j]
-                for key, value in _parse_logical_slot(s):
-                    payload.old[key] = value
+                j: int = suffix.find("new-tuple:")
+                if j > -1:
+                    s: str = suffix[i + len("old-key:") : j]
+                    for key, value in _parse_logical_slot(s):
+                        payload.old[key] = value
 
-            i = suffix.index("new-tuple:")
+            i = suffix.find("new-tuple:")
             if i > -1:
                 s = suffix[i + len("new-tuple:") :]
                 for key, value in _parse_logical_slot(s):
@@ -1336,7 +1377,28 @@ def _pg_engine(
             password=password,
             port=port,
         )
-    return sa.create_engine(url, echo=echo, connect_args=connect_args)
+
+    # Use NullPool for testing to avoid connection exhaustion
+    if SQLALCHEMY_USE_NULLPOOL:
+        from sqlalchemy.pool import NullPool
+
+        return sa.create_engine(
+            url,
+            echo=echo,
+            connect_args=connect_args,
+            poolclass=NullPool,
+        )
+
+    return sa.create_engine(
+        url,
+        echo=echo,
+        connect_args=connect_args,
+        pool_size=SQLALCHEMY_POOL_SIZE,
+        max_overflow=SQLALCHEMY_MAX_OVERFLOW,
+        pool_pre_ping=SQLALCHEMY_POOL_PRE_PING,
+        pool_recycle=SQLALCHEMY_POOL_RECYCLE,
+        pool_timeout=SQLALCHEMY_POOL_TIMEOUT,
+    )
 
 
 def pg_execute(
@@ -1367,7 +1429,7 @@ def create_database(database: str, echo: bool = False) -> None:
     """Create a database."""
     logger.debug(f"Creating database: {database}")
     with pg_engine(
-        "information_schema" if IS_MYSQL_COMPAT else "postgres",
+        MYSQL_DATABASE if IS_MYSQL_COMPAT else PG_DATABASE,
         echo=echo,
     ) as engine:
         pg_execute(
@@ -1382,8 +1444,7 @@ def drop_database(database: str, echo: bool = False) -> None:
     """Drop a database."""
     logger.debug(f"Dropping database: {database}")
     with pg_engine(
-        "information_schema" if IS_MYSQL_COMPAT else "postgres",
-        echo=echo,
+        MYSQL_DATABASE if IS_MYSQL_COMPAT else PG_DATABASE, echo=echo
     ) as engine:
         pg_execute(
             engine,
@@ -1396,7 +1457,7 @@ def drop_database(database: str, echo: bool = False) -> None:
 def database_exists(database: str, echo: bool = False) -> bool:
     """Check if database is present."""
     with pg_engine(
-        "information_schema" if IS_MYSQL_COMPAT else "postgres",
+        MYSQL_DATABASE if IS_MYSQL_COMPAT else PG_DATABASE,
         echo=echo,
     ) as engine:
         with engine.connect() as conn:

{pgsync-5.0.0 → pgsync-6.1.0}/pgsync/constants.py

@@ -135,6 +135,7 @@ ELASTICSEARCH_MAPPING_PARAMETERS = [
     "boost",
     "coerce",
     "copy_to",
+    "dimension",
     "doc_values",
     "dynamic",
     "eager_global_ordinals",
@@ -198,6 +199,7 @@ MATERIALIZED_VIEW_COLUMNS = [
     "indices",
     "primary_keys",
     "table_name",
+    "columns",
 ]
 
 # Primary key delimiter

{pgsync-5.0.0 → pgsync-6.1.0}/pgsync/helper.py

@@ -20,6 +20,7 @@ def teardown(
     drop_index: bool = True,
     delete_checkpoint: bool = True,
     config: t.Optional[str] = None,
+    schema_url: t.Optional[str] = None,
     s3_schema_url: t.Optional[str] = None,
     validate: bool = False,
 ) -> None:
@@ -33,11 +34,17 @@ def teardown(
         drop_index (bool, optional): Whether to drop the index. Defaults to True.
         delete_checkpoint (bool, optional): Whether to delete the checkpoint. Defaults to True.
         config (Optional[str], optional): The configuration file path. Defaults to None.
+        schema_url (Optional[str], optional): The schema URL. Defaults to None.
+        s3_schema_url (Optional[str], optional): The S3 schema URL. Defaults to
         validate (bool, optional): Whether to validate the configuration. Defaults to False.
     """
-    validate_config(config=config, s3_schema_url=s3_schema_url)
+    validate_config(
+        config=config, schema_url=schema_url, s3_schema_url=s3_schema_url
+    )
 
-    for doc in config_loader(config=config, s3_schema_url=s3_schema_url):
+    for doc in config_loader(
+        config=config, schema_url=schema_url, s3_schema_url=s3_schema_url
+    ):
         if not database_exists(doc["database"]):
             logger.warning(f'Database {doc["database"]} does not exist')
             continue
@@ -56,7 +63,7 @@ def teardown(
             drop_database(sync.database)
         if drop_index:
             sync.search_client.teardown(sync.index)
-        if delete_redis:
+        if delete_redis and sync.redis is not None:
             sync.redis.delete()
         if delete_checkpoint:
             try:

{pgsync-5.0.0 → pgsync-6.1.0}/pgsync/node.py

@@ -294,7 +294,7 @@ class Node(object):
                 self.columns.append(self.model.c[column_name])
 
     @property
-    def primary_keys(self):
+    def primary_keys(self) -> t.List[sa.sql.ColumnElement]:
         return [
             self.model.c[str(sa.text(primary_key))]
             for primary_key in self.model.primary_keys
@@ -311,7 +311,7 @@ class Node(object):
 
     def add_child(self, node: Node) -> None:
         """All nodes except the root node must have a relationship defined."""
-        node.parent: Node = self
+        node.parent = self
         if not node.is_root and (
             not node.relationship.type or not node.relationship.variant
         ):
@@ -388,7 +388,9 @@ class Tree(threading.local):
             raise TableNotInNodeError(f"Table not specified in node: {nodes}")
 
         if not set(nodes.keys()).issubset(set(NODE_ATTRIBUTES)):
-            attrs = set(nodes.keys()).difference(set(NODE_ATTRIBUTES))
+            attrs: t.Set[str] = set(nodes.keys()).difference(
+                set(NODE_ATTRIBUTES)
+            )
             raise NodeAttributeError(f"Unknown node attribute(s): {attrs}")
 
         node: Node = Node(