pgsync 7.0.2__tar.gz → 7.1.0__tar.gz

{pgsync-7.0.2 → pgsync-7.1.0}/MANIFEST.in

@@ -4,6 +4,7 @@ include HISTORY.rst
 include LICENSE
 include README.md
 include README.rst
+recursive-include requirements *.txt
 
 recursive-include tests *
 recursive-exclude * __pycache__

pgsync-7.1.0/PKG-INFO

@@ -0,0 +1,263 @@
+Metadata-Version: 2.4
+Name: pgsync
+Version: 7.1.0
+Summary: Postgres/MySQL/MariaDB to Elasticsearch/OpenSearch sync
+Home-page: https://github.com/toluaina/pgsync
+Author: Tolu Aina
+Author-email: tolu@pgsync.com
+Maintainer: Tolu Aina
+Maintainer-email: tolu@pgsync.com
+License: MIT
+Project-URL: Bug Reports, https://github.com/toluaina/pgsync/issues
+Project-URL: Funding, https://github.com/sponsors/toluaina
+Project-URL: Source, https://github.com/toluaina/pgsync
+Project-URL: Web, https://pgsync.com
+Project-URL: Documentation, https://pgsync.com
+Keywords: change data capture,elasticsearch,opensearch,pgsync,postgres,mysql,mariadb
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Natural Language :: English
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10.0
+Description-Content-Type: text/x-rst
+License-File: LICENSE
+License-File: AUTHORS.rst
+Requires-Dist: async-timeout==5.0.1
+Requires-Dist: backports-datetime-fromisoformat==2.0.3
+Requires-Dist: boto3==1.43.36
+Requires-Dist: botocore==1.43.36
+Requires-Dist: certifi==2026.6.17
+Requires-Dist: charset-normalizer==3.4.7
+Requires-Dist: click==8.4.2
+Requires-Dist: elastic-transport==9.4.2
+Requires-Dist: elasticsearch==7.17.13
+Requires-Dist: elasticsearch-dsl==7.4.1
+Requires-Dist: environs==15.0.1
+Requires-Dist: events==0.5
+Requires-Dist: grpcio==1.81.1
+Requires-Dist: idna==3.18
+Requires-Dist: jmespath==1.1.0
+Requires-Dist: marshmallow==4.3.0
+Requires-Dist: mysql-replication==1.0.15
+Requires-Dist: opensearch-dsl==2.1.0
+Requires-Dist: opensearch-protobufs==1.2.0
+Requires-Dist: opensearch-py==3.2.0
+Requires-Dist: packaging==26.2
+Requires-Dist: protobuf==7.35.1
+Requires-Dist: psycopg2-binary==2.9.12
+Requires-Dist: pymysql==1.2.0
+Requires-Dist: python-dateutil==2.9.0.post0
+Requires-Dist: python-dotenv==1.2.2
+Requires-Dist: redis==8.0.1
+Requires-Dist: requests==2.34.2
+Requires-Dist: requests-aws4auth==1.3.2
+Requires-Dist: s3transfer==0.19.0
+Requires-Dist: six==1.17.0
+Requires-Dist: sniffio==1.3.1
+Requires-Dist: sqlalchemy==2.0.51
+Requires-Dist: sqlparse==0.5.5
+Requires-Dist: typing-extensions==4.15.0
+Requires-Dist: urllib3==2.7.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: license
+Dynamic: license-file
+Dynamic: maintainer
+Dynamic: maintainer-email
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+======
+PGSync
+======
+
+.. image:: https://img.shields.io/pypi/v/pgsync.svg
+   :target: https://pypi.org/project/pgsync/
+   :alt: PyPI Version
+
+.. image:: https://img.shields.io/pypi/pyversions/pgsync.svg
+   :target: https://pypi.org/project/pgsync/
+   :alt: Python Versions
+
+.. image:: https://img.shields.io/pypi/l/pgsync.svg
+   :target: https://opensource.org/licenses/MIT
+   :alt: License
+
+.. image:: https://img.shields.io/pypi/dm/pgsync.svg
+   :target: https://pypi.org/project/pgsync/
+   :alt: Downloads
+
+|
+
+**PostgreSQL/MySQL/MariaDB to Elasticsearch/OpenSearch sync**
+
+`PGSync <https://pgsync.com>`_ is a middleware for syncing data from `PostgreSQL <https://www.postgresql.org>`_, `MySQL <https://www.mysql.com>`_, or `MariaDB <https://mariadb.org>`_ to `Elasticsearch <https://www.elastic.co/products/elastic-stack>`_ or `OpenSearch <https://opensearch.org>`_.
+
+Keep your relational database as the source of truth and expose structured denormalized documents in your search engine.
+
+Key Features
+------------
+
+- Real-time sync via logical decoding (PostgreSQL) or binary log (MySQL/MariaDB)
+- Denormalize complex relational data into nested search documents
+- JSON schema-based configuration
+- Support for one-to-one, one-to-many relationships
+- Plugin system for document transformation
+- Multiple operation modes: daemon, polling, or direct WAL streaming
+
+Requirements
+------------
+
+- `Python <https://www.python.org>`_ 3.10+
+- `PostgreSQL <https://www.postgresql.org>`_ 9.6+ or `MySQL <https://www.mysql.com>`_ 8.0.0+ or `MariaDB <https://mariadb.org>`_ 12.0.0+
+- `Redis <https://redis.io>`_ 3.1.0+ or `Valkey <https://valkey.io>`_ 7.2.0+ (optional in WAL mode)
+- `Elasticsearch <https://www.elastic.co/products/elastic-stack>`_ 6.3.1+ or `OpenSearch <https://opensearch.org>`_ 1.3.7+
+
+Installation
+------------
+
+Install from `PyPI <https://pypi.org/project/pgsync/>`_:
+
+.. code-block:: bash
+
+   pip install pgsync
+
+Database Setup
+--------------
+
+PostgreSQL
+~~~~~~~~~~
+
+Enable `logical decoding <https://www.postgresql.org/docs/current/logicaldecoding.html>`_ in your PostgreSQL configuration (``postgresql.conf``):
+
+.. code-block:: ini
+
+   wal_level = logical
+   max_replication_slots = 1
+
+MySQL / MariaDB
+~~~~~~~~~~~~~~~
+
+Enable binary logging in your MySQL/MariaDB configuration (``my.cnf``):
+
+.. code-block:: ini
+
+   server-id = 1
+   log_bin = mysql-bin
+   binlog_row_image = FULL
+   binlog_expire_logs_seconds = 604800
+
+Create a replication user:
+
+.. code-block:: sql
+
+   CREATE USER 'replicator'@'%' IDENTIFIED WITH mysql_native_password BY 'password';
+   GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'replicator'@'%';
+   FLUSH PRIVILEGES;
+
+Configuration
+-------------
+
+Create a JSON schema file (e.g., ``schema.json``) defining your sync mapping:
+
+.. code-block:: json
+
+   [
+       {
+           "database": "book",
+           "index": "book",
+           "nodes": {
+               "table": "book",
+               "columns": ["isbn", "title", "description"],
+               "children": [
+                   {
+                       "table": "publisher",
+                       "columns": ["name"],
+                       "relationship": {
+                           "variant": "object",
+                           "type": "one_to_one"
+                       }
+                   },
+                   {
+                       "table": "author",
+                       "label": "authors",
+                       "columns": ["name", "date_of_birth"],
+                       "relationship": {
+                           "variant": "object",
+                           "type": "one_to_many",
+                           "through_tables": ["book_author"]
+                       }
+                   }
+               ]
+           }
+       }
+   ]
+
+See the `examples directory <https://github.com/toluaina/pgsync/tree/main/examples>`_ for more schema examples (airbnb, social, rental, etc.).
+
+Environment Variables
+---------------------
+
+Configure PGSync via environment variables:
+
+.. code-block:: bash
+
+   # Schema
+   SCHEMA='/path/to/schema.json'
+
+   # PostgreSQL
+   PG_HOST=localhost
+   PG_PORT=5432
+   PG_USER=postgres
+   PG_PASSWORD=*****
+
+   # Elasticsearch / OpenSearch
+   ELASTICSEARCH_HOST=localhost
+   ELASTICSEARCH_PORT=9200
+
+   # Redis (optional in WAL mode)
+   REDIS_HOST=localhost
+   REDIS_PORT=6379
+
+Running
+-------
+
+**Bootstrap** (run once to set up triggers and replication slots):
+
+.. code-block:: bash
+
+   bootstrap --config schema.json
+
+**Run as daemon**:
+
+.. code-block:: bash
+
+   pgsync --config schema.json --daemon
+
+Links
+-----
+
+- **Documentation**: https://pgsync.com
+- **Source Code**: https://github.com/toluaina/pgsync
+- **Bug Reports**: https://github.com/toluaina/pgsync/issues
+- **Sponsor**: https://github.com/sponsors/toluaina
+
+License
+-------
+
+MIT License - see `LICENSE <https://github.com/toluaina/pgsync/blob/main/LICENSE>`_ for details.

pgsync-7.1.0/README.md

@@ -0,0 +1,380 @@
+<div align="center">
+
+# PGSync
+
+### Real-time PostgreSQL to Elasticsearch/OpenSearch sync
+
+**Keep your relational database as the source of truth while powering lightning-fast search**
+
+[![PyPI version](https://img.shields.io/pypi/v/pgsync?color=blue&logo=pypi&logoColor=white)](https://pypi.org/project/pgsync)
+[![Python versions](https://img.shields.io/pypi/pyversions/pgsync?logo=python&logoColor=white)](https://pypi.org/project/pgsync)
+[![Build status](https://img.shields.io/github/actions/workflow/status/toluaina/pgsync/python-build.yml?branch=main&logo=github)](https://github.com/toluaina/pgsync/actions)
+[![codecov](https://img.shields.io/codecov/c/github/toluaina/pgsync?logo=codecov&logoColor=white)](https://codecov.io/gh/toluaina/pgsync)
+[![Downloads](https://img.shields.io/pypi/dm/pgsync?color=brightgreen&logo=pypi&logoColor=white)](https://pypi.org/project/pgsync)
+[![License](https://img.shields.io/github/license/toluaina/pgsync?color=blue)](LICENSE)
+
+[![Docker](https://img.shields.io/badge/docker-ready-2496ED?logo=docker&logoColor=white)](https://hub.docker.com/r/toluaina1/pgsync)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
+[Website](https://pgsync.com) · [Documentation](https://pgsync.com) · [Examples](examples/) · [Report Bug](https://github.com/toluaina/pgsync/issues)
+
+</div>
+
+---
+
+## What is PGSync?
+
+PGSync is a change data capture tool that syncs data from **PostgreSQL**, **MySQL**, or **MariaDB** to **Elasticsearch** or **OpenSearch** in real-time. Define your document structure in JSON, and PGSync handles the rest — no custom code required.
+
+```mermaid
+%%{init: {'look': 'handDrawn', 'theme': 'neutral'}}%%
+flowchart LR
+    subgraph Source["🗄️ Source Database"]
+        DB[(PostgreSQL<br/>MySQL<br/>MariaDB)]
+    end
+
+    subgraph CDC["⚡ Change Data Capture"]
+        P[PGSync]
+    end
+
+    subgraph Search["🔍 Search Engine"]
+        ES[(Elasticsearch<br/>OpenSearch)]
+    end
+
+    DB -->|WAL / Binlog| P
+    P -->|Bulk Index| ES
+```
+
+### Key Features
+
+| Feature | Description |
+|---------|-------------|
+| **Real-time sync** | Changes propagate instantly via logical replication |
+| **Zero code** | Define mappings in JSON — no ETL pipelines to build |
+| **Nested documents** | Automatically denormalize complex relationships |
+| **Fault tolerant** | Resumes from checkpoints after crashes |
+| **Transactionally consistent** | Documents appear in commit order |
+| **Minimal overhead** | Lightweight CDC with negligible database impact |
+
+---
+
+## Quick Start
+
+### Using Docker (Fastest)
+
+```bash
+docker run --rm -it \
+  -e PG_URL=postgres://user:pass@host/db \
+  -e ELASTICSEARCH_URL=http://localhost:9200 \
+  -e REDIS_HOST=localhost \
+  -v "$(pwd)/schema.json:/app/schema.json" \
+  toluaina1/pgsync:latest -c schema.json -d -b
+```
+
+### Using pip
+
+```bash
+pip install pgsync
+```
+
+```bash
+# Bootstrap (one-time setup)
+bootstrap --config schema.json
+
+# Run sync
+pgsync --config schema.json -d
+```
+
+### Using Docker Compose
+
+**Default (Elasticsearch + Kibana):**
+```bash
+git clone https://github.com/toluaina/pgsync
+cd pgsync
+docker-compose up
+```
+
+This starts PostgreSQL, Redis, Elasticsearch, Kibana, and PGSync configured for Elasticsearch.
+
+**For OpenSearch:**
+```bash
+docker-compose --profile opensearch up
+```
+
+This starts PostgreSQL, Redis, OpenSearch, and PGSync configured for OpenSearch.
+
+**Ports:**
+- PostgreSQL: `15432`
+- Elasticsearch: `9201` (default)
+- Kibana: `5601` (default)
+- OpenSearch: `9400` (OpenSearch profile)
+
+---
+
+## How It Works
+
+**1. Define your schema** — Map tables to document structure:
+
+```json
+{
+  "table": "book",
+  "columns": ["isbn", "title", "description"],
+  "children": [{
+    "table": "author",
+    "columns": ["name"]
+  }]
+}
+```
+
+**2. PGSync generates optimized queries** — Complex JOINs handled automatically:
+
+```sql
+SELECT JSON_BUILD_OBJECT(
+  'isbn', book.isbn,
+  'title', book.title,
+  'authors', (SELECT JSON_AGG(author.name) FROM author ...)
+) FROM book
+```
+
+**3. Get denormalized documents** — Ready for search:
+
+```json
+{
+  "isbn": "9785811243570",
+  "title": "Charlie and the Chocolate Factory",
+  "authors": ["Roald Dahl"]
+}
+```
+
+Changes to any related table automatically update the document in Elasticsearch/OpenSearch.
+
+---
+
+## Requirements
+
+| Component | Version |
+|-----------|---------|
+| ![Python](https://img.shields.io/badge/Python-3.10+-3776AB?logo=python&logoColor=white) | 3.10+ |
+| ![PostgreSQL](https://img.shields.io/badge/PostgreSQL-9.6+-4169E1?logo=postgresql&logoColor=white) | 9.6+ (or MySQL 5.7.22+ / MariaDB 10.5+) |
+| ![Elasticsearch](https://img.shields.io/badge/Elasticsearch-6.3+-005571?logo=elasticsearch&logoColor=white) | 6.3.1+ (or OpenSearch 1.3.7+) |
+| ![Redis](https://img.shields.io/badge/Redis-3.1+-DC382D?logo=redis&logoColor=white) | 3.1+ (or Valkey 7.2+) — optional in WAL mode |
+
+---
+
+## Database Setup
+
+<details>
+<summary><b>PostgreSQL</b></summary>
+
+Enable logical decoding in `postgresql.conf`:
+
+```ini
+wal_level = logical
+max_replication_slots = 1
+```
+
+Optionally limit WAL size:
+
+```ini
+max_slot_wal_keep_size = 100GB
+```
+
+</details>
+
+<details>
+<summary><b>MySQL / MariaDB</b></summary>
+
+Enable binary logging in `my.cnf`:
+
+```ini
+server-id = 1
+log_bin = mysql-bin
+binlog_row_image = FULL
+binlog_expire_logs_seconds = 604800
+```
+
+Create replication user:
+
+```sql
+CREATE USER 'replicator'@'%' IDENTIFIED WITH mysql_native_password BY 'password';
+GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'replicator'@'%';
+FLUSH PRIVILEGES;
+```
+
+</details>
+
+---
+
+## Example
+
+Consider a book library with related authors:
+
+| **Book** | | | |
+|----------|--|--|--|
+| isbn *(PK)* | title | description |
+| 9785811243570 | Charlie and the Chocolate Factory | Willy Wonka's famous... |
+| 9781471331435 | 1984 | George Orwell's chilling... |
+
+| **Author** | |
+|------------|--|
+| id *(PK)* | name |
+| 1 | Roald Dahl |
+| 4 | George Orwell |
+
+PGSync transforms this into search-ready documents:
+
+```json
+[
+  {
+    "isbn": "9785811243570",
+    "title": "Charlie and the Chocolate Factory",
+    "authors": ["Roald Dahl"]
+  },
+  {
+    "isbn": "9781471331435",
+    "title": "1984",
+    "authors": ["George Orwell"]
+  }
+]
+```
+
+**Any change** — updating an author's name, adding a new book, deleting a relationship — is automatically synced.
+
+---
+
+## Transforms
+
+PGSync supports built-in transforms to modify field values before indexing. Transforms are applied in order: `replace` → `rename` → `concat`.
+
+### Replace
+
+Find and replace substrings within field values:
+
+```json
+{
+  "table": "product",
+  "columns": ["code", "name"],
+  "transform": {
+    "replace": {
+      "code": {
+        "-": "/",
+        "_": " "
+      }
+    }
+  }
+}
+```
+
+| Before | After |
+|--------|-------|
+| `ABC-DEF_GHI` | `ABC/DEF GHI` |
+
+### Rename
+
+Rename fields in the output document:
+
+```json
+{
+  "table": "book",
+  "columns": ["id", "title"],
+  "transform": {
+    "rename": {
+      "id": "book_id",
+      "title": "book_title"
+    }
+  }
+}
+```
+
+### Concat
+
+Combine multiple fields into a new field:
+
+```json
+{
+  "table": "user",
+  "columns": ["first_name", "last_name"],
+  "transform": {
+    "concat": {
+      "columns": ["first_name", "last_name"],
+      "destination": "full_name",
+      "delimiter": " "
+    }
+  }
+}
+```
+
+### Combined Example
+
+Transforms can be combined and applied to nested children:
+
+```json
+{
+  "table": "book",
+  "columns": ["isbn", "title"],
+  "children": [{
+    "table": "publisher",
+    "columns": ["code", "name"],
+    "transform": {
+      "replace": { "code": { "-": "." } },
+      "rename": { "name": "publisher_name" }
+    }
+  }],
+  "transform": {
+    "concat": {
+      "columns": ["isbn", "title"],
+      "destination": "search_text",
+      "delimiter": " - "
+    }
+  }
+}
+```
+
+---
+
+## Why PGSync?
+
+| Challenge | PGSync Solution |
+|-----------|-----------------|
+| Dual writes are error-prone | Captures changes from WAL — single source of truth |
+| Complex JOIN queries | Auto-generates optimized SQL from your schema |
+| Nested document updates | Detects changes in any related table |
+| Data consistency | Transactionally consistent, ordered delivery |
+| Crash recovery | Checkpoint-based resumption |
+
+---
+
+## Environment Variables
+
+Full list at [pgsync.com/env-vars](https://pgsync.com/env-vars)
+
+| Variable | Description |
+|----------|-------------|
+| `PG_URL` | PostgreSQL connection string |
+| `ELASTICSEARCH_URL` | Elasticsearch/OpenSearch URL |
+| `REDIS_HOST` | Redis/Valkey host |
+| `REDIS_CHECKPOINT` | Use Redis for checkpoints (recommended for production) |
+
+---
+
+## One-Click Deploy
+
+[![Deploy to DigitalOcean](https://www.deploytodo.com/do-btn-blue.svg)](https://cloud.digitalocean.com/apps/new?repo=https://github.com/toluaina/pgsync/tree/main)
+
+---
+
+## Sponsors
+
+<a href="https://www.digitalocean.com/?utm_medium=opensource&utm_source=pgsync">
+  <img src="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/SVG/DO_Logo_horizontal_blue.svg" alt="DigitalOcean" width="200">
+</a>
+
+---
+
+## Contributing
+
+Contributions welcome! See [CONTRIBUTING.rst](CONTRIBUTING.rst) for guidelines.
+
+## License
+
+[MIT](LICENSE) — use it freely in your projects.