mkpipe-loader-postgres 0.1.34__tar.gz → 0.3.0__tar.gz

mkpipe_loader_postgres-0.3.0/PKG-INFO

@@ -0,0 +1,50 @@
+Metadata-Version: 2.4
+Name: mkpipe-loader-postgres
+Version: 0.3.0
+Summary: PostgreSQL loader for mkpipe.
+Author: Metin Karakus
+Author-email: metin_karakus@yahoo.com
+License: Apache License 2.0
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: Apache Software License
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: mkpipe
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: license
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# MkPipe
+
+**MkPipe** is a modular, open-source ETL (Extract, Transform, Load) tool that allows you to integrate various data sources and sinks easily. It is designed to be extensible with a plugin-based architecture that supports extractors, transformers, and loaders.
+
+## Documentation
+
+For more detailed documentation, please visit the [GitHub repository](https://github.com/mkpipe-etl/mkpipe).
+
+## License
+
+This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.
+
+
+## mkpipe_project.yaml Variables
+```yaml
+...
+  connections:
+    source:
+      host: 'XXX'
+      port: 'XXX'
+      database: 'XXX'
+      schema: 'XXX' 
+      user: 'XXX'
+      password: 'XXX'
+...
+```

mkpipe_loader_postgres-0.3.0/README.md

@@ -0,0 +1,26 @@
+# MkPipe
+
+**MkPipe** is a modular, open-source ETL (Extract, Transform, Load) tool that allows you to integrate various data sources and sinks easily. It is designed to be extensible with a plugin-based architecture that supports extractors, transformers, and loaders.
+
+## Documentation
+
+For more detailed documentation, please visit the [GitHub repository](https://github.com/mkpipe-etl/mkpipe).
+
+## License
+
+This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.
+
+
+## mkpipe_project.yaml Variables
+```yaml
+...
+  connections:
+    source:
+      host: 'XXX'
+      port: 'XXX'
+      database: 'XXX'
+      schema: 'XXX' 
+      user: 'XXX'
+      password: 'XXX'
+...
+```

mkpipe_loader_postgres-0.3.0/mkpipe_loader_postgres/__init__.py

@@ -0,0 +1,15 @@
+from mkpipe.spark import JdbcLoader
+
+
+class PostgresLoader(JdbcLoader, variant='postgresql'):
+    driver_name = 'postgresql'
+    driver_jdbc = 'org.postgresql.Driver'
+
+    def build_jdbc_url(self):
+        url = (
+            f'jdbc:{self.driver_name}://{self.host}:{self.port}/{self.database}'
+            f'?user={self.username}&password={self.password}'
+        )
+        if self.schema:
+            url += f'&currentSchema={self.schema}'
+        return url

mkpipe_loader_postgres-0.3.0/mkpipe_loader_postgres.egg-info/PKG-INFO

@@ -0,0 +1,50 @@
+Metadata-Version: 2.4
+Name: mkpipe-loader-postgres
+Version: 0.3.0
+Summary: PostgreSQL loader for mkpipe.
+Author: Metin Karakus
+Author-email: metin_karakus@yahoo.com
+License: Apache License 2.0
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: Apache Software License
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: mkpipe
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: license
+Dynamic: license-file
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# MkPipe
+
+**MkPipe** is a modular, open-source ETL (Extract, Transform, Load) tool that allows you to integrate various data sources and sinks easily. It is designed to be extensible with a plugin-based architecture that supports extractors, transformers, and loaders.
+
+## Documentation
+
+For more detailed documentation, please visit the [GitHub repository](https://github.com/mkpipe-etl/mkpipe).
+
+## License
+
+This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.
+
+
+## mkpipe_project.yaml Variables
+```yaml
+...
+  connections:
+    source:
+      host: 'XXX'
+      port: 'XXX'
+      database: 'XXX'
+      schema: 'XXX' 
+      user: 'XXX'
+      password: 'XXX'
+...
+```

{mkpipe_loader_postgres-0.1.34 → mkpipe_loader_postgres-0.3.0}/setup.py

@@ -2,7 +2,7 @@ from setuptools import setup, find_packages
 
 setup(
     name='mkpipe-loader-postgres',
-    version='0.1.34',
+    version='0.3.0',
     license='Apache License 2.0',
     packages=find_packages(exclude=['tests', 'scripts', 'deploy', 'install_jars.py']),
     install_requires=['mkpipe'],

mkpipe_loader_postgres-0.1.34/PKG-INFO

@@ -1,73 +0,0 @@
-Metadata-Version: 2.1
-Name: mkpipe-loader-postgres
-Version: 0.1.34
-Summary: PostgreSQL loader for mkpipe.
-Author: Metin Karakus
-Author-email: metin_karakus@yahoo.com
-License: Apache License 2.0
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: Apache Software License
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: mkpipe
-
-# MkPipe
-
-**MkPipe** is a modular, open-source ETL (Extract, Transform, Load) tool that allows you to integrate various data sources and sinks easily. It is designed to be extensible with a plugin-based architecture that supports extractors, transformers, and loaders.
-
-## Features
-
-- Extract data from multiple sources (e.g., PostgreSQL, MongoDB).
-- Transform data using custom Python logic and Apache Spark.
-- Load data into various sinks (e.g., ClickHouse, PostgreSQL, Parquet).
-- Plugin-based architecture that supports future extensions.
-- Cloud-native architecture, can be deployed on Kubernetes and other environments.
-
-## Installation
-
-You can install the core package and extractors using pip:
-
-### Install the core package:
-```bash
-pip install mkpipe
-```
-
-### Install the Postgres extractor:
-```bash
-pip install mkpipe-loader-postgres
-```
-
-### Install additional extractors or loaders as needed:
-You can find or contribute new extractors and loaders in the future.
-
-## Usage
-
-To run the ETL process, use the following command:
-
-```py
-from mkpipe_core.plugins.registry import EXTRACTORS
-
-def test_postgres_extractor():
-    postgres_extractor = EXTRACTORS.get("postgresql")
-    if not postgres_extractor:
-        print("Postgres extractor not found!")
-        return
-    instance = postgres_extractor()
-    instance.extract()
-
-if __name__ == "__main__":
-    test_postgres_extractor()
-
-```
-
-Where `elt.yaml` is your configuration file that specifies the extractors, transformers, and loaders.
-
-## Documentation
-
-For more detailed documentation, please visit the [GitHub repository](https://github.com/mkpipe-etl/mkpipe).
-
-## License
-
-This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.
-

mkpipe_loader_postgres-0.1.34/README.md

@@ -1,59 +0,0 @@
-# MkPipe
-
-**MkPipe** is a modular, open-source ETL (Extract, Transform, Load) tool that allows you to integrate various data sources and sinks easily. It is designed to be extensible with a plugin-based architecture that supports extractors, transformers, and loaders.
-
-## Features
-
-- Extract data from multiple sources (e.g., PostgreSQL, MongoDB).
-- Transform data using custom Python logic and Apache Spark.
-- Load data into various sinks (e.g., ClickHouse, PostgreSQL, Parquet).
-- Plugin-based architecture that supports future extensions.
-- Cloud-native architecture, can be deployed on Kubernetes and other environments.
-
-## Installation
-
-You can install the core package and extractors using pip:
-
-### Install the core package:
-```bash
-pip install mkpipe
-```
-
-### Install the Postgres extractor:
-```bash
-pip install mkpipe-loader-postgres
-```
-
-### Install additional extractors or loaders as needed:
-You can find or contribute new extractors and loaders in the future.
-
-## Usage
-
-To run the ETL process, use the following command:
-
-```py
-from mkpipe_core.plugins.registry import EXTRACTORS
-
-def test_postgres_extractor():
-    postgres_extractor = EXTRACTORS.get("postgresql")
-    if not postgres_extractor:
-        print("Postgres extractor not found!")
-        return
-    instance = postgres_extractor()
-    instance.extract()
-
-if __name__ == "__main__":
-    test_postgres_extractor()
-
-```
-
-Where `elt.yaml` is your configuration file that specifies the extractors, transformers, and loaders.
-
-## Documentation
-
-For more detailed documentation, please visit the [GitHub repository](https://github.com/mkpipe-etl/mkpipe).
-
-## License
-
-This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.
-

mkpipe_loader_postgres-0.1.34/mkpipe_loader_postgres/__init__.py

@@ -1,169 +0,0 @@
-import time
-from pathlib import Path
-from urllib.parse import quote_plus
-from pyspark.sql import functions as F
-from pyspark.sql.types import TimestampType
-from mkpipe.config import load_config
-from mkpipe.functions_db import get_db_connector
-from mkpipe.functions_spark import remove_partitioned_parquet, get_parser
-from mkpipe.utils import log_container, Logger
-from mkpipe.utils.base_class import PipeSettings
-
-
-class PostgresLoader:
-    def __init__(self, config, settings):
-        if isinstance(settings, dict):
-            self.settings = PipeSettings(**settings)
-        else:
-            self.settings = settings
-        self.connection_params = config['connection_params']
-        self.host = self.connection_params['host']
-        self.port = self.connection_params['port']
-        self.username = self.connection_params['user']
-        self.password = quote_plus(str(self.connection_params['password']))
-        self.database = self.connection_params['database']
-        self.schema = self.connection_params['schema']
-
-        self.driver_name = 'postgresql'
-        self.driver_jdbc = 'org.postgresql.Driver'
-
-        self.settings.driver_name = self.driver_name 
-        self.jdbc_url = f'jdbc:{self.driver_name}://{self.host}:{self.port}/{self.database}?user={self.username}&password={self.password}&currentSchema={self.schema}'
-
-        config = load_config()
-        connection_params = config['settings']['backend']
-        db_type = connection_params['database_type']
-        self.backend = get_db_connector(db_type)(connection_params)
-
-    def add_custom_columns(self, df, elt_start_time):
-        if 'etl_time' in df.columns:
-            df = df.drop('etl_time')
-
-        df = df.withColumn('etl_time', F.lit(elt_start_time))
-        return df
-
-    @log_container(__file__)
-    def load(self, data, elt_start_time):
-        try:
-            logger = Logger(__file__)
-            start_time = time.time()
-            name = data['table_name']
-
-            write_mode = data.get('write_mode', None)
-            file_type = data.get('file_type', None)
-            last_point_value = data.get('last_point_value', None)
-            iterate_column_type = data.get('iterate_column_type', None)
-            replication_method = data.get('replication_method', 'full')
-            batchsize = data.get('fetchsize', 100_000)
-            pass_on_error = data.get('pass_on_error', None)
-
-            if not file_type:
-                'means that the data fetched before no new data'
-                self.backend.manifest_table_update(
-                    name=name,
-                    value=None,  # Last point remains unchanged
-                    value_type=None,  # Type remains unchanged
-                    status='completed',  # ('completed', 'failed', 'extracting', 'loading')
-                    replication_method=replication_method,  # ('incremental', 'full')
-                    error_message='',
-                )
-                return
-
-            self.backend.manifest_table_update(
-                name=name,
-                value=None,  # Last point remains unchanged
-                value_type=None,  # Type remains unchanged
-                status='loading',  # ('completed', 'failed', 'extracting', 'loading')
-                replication_method=replication_method,  # ('incremental', 'full')
-                error_message='',
-            )
-
-            df = get_parser(file_type)(data, self.settings)
-            df = self.add_custom_columns(df, elt_start_time)
-            message = dict(
-                table_name=name,
-                status='loading',
-                total_partition_count=df.rdd.getNumPartitions(),
-            )
-            logger.info(message)
-
-            (
-                df.write.format('jdbc')
-                .mode(
-                    write_mode
-                )  # Use write_mode for the first iteration, 'append' for others
-                .option('url', self.jdbc_url)
-                .option('dbtable', name)
-                .option('driver', self.driver_jdbc)
-                .option('batchsize', batchsize)
-                .save()
-            )
-
-            """
-            for index in range(df.rdd.getNumPartitions()):
-                # Filter the DataFrame by partition index
-                partition_df = df.filter(F.spark_partition_id() == index)
-
-                # Determine the correct write mode (use write_mode for the first iteration, 'append' for others)
-                current_write_mode = write_mode if index == 0 else 'append'
-
-                # Write the partitioned DataFrame to PostgreSQL
-                (
-                    partition_df.write.format('jdbc')
-                    .mode(
-                        current_write_mode
-                    )  # Use write_mode for the first iteration, 'append' for others
-                    .option('url', self.jdbc_url)
-                    .option('dbtable', name)
-                    .option('driver', self.driver_jdbc)
-                    .option('batchsize', batchsize)
-                    .save()
-                )
-            """
-
-            # Update last point in the mkpipe_manifest table if applicable
-            self.backend.manifest_table_update(
-                name=name,
-                value=last_point_value,
-                value_type=iterate_column_type,
-                status='completed',
-                replication_method=replication_method,
-                error_message='',
-            )
-
-            message = dict(table_name=name, status=write_mode)
-            logger.info(message)
-
-            # remove the parquet to reduce the storage
-            remove_partitioned_parquet(data['path'])
-
-            run_time = time.time() - start_time
-            message = dict(table_name=name, status='success', run_time=run_time)
-            logger.info(message)
-
-        except Exception as e:
-            # Log the error message and update the mkpipe_manifest with the error details
-            message = dict(
-                table_name=name,
-                status='failed',
-                type='loading',
-                error_message=str(e),
-                etl_start_time=str(elt_start_time),
-            )
-
-            self.backend.manifest_table_update(
-                name=name,
-                value=None,  # Last point remains unchanged
-                value_type=None,  # Type remains unchanged
-                status='failed',
-                replication_method=replication_method,
-                error_message=str(e),
-            )
-
-            if pass_on_error:
-                logger.warning(message)
-                return
-            else:
-                logger.error(message)
-                raise Exception(message) from e
-        return

mkpipe_loader_postgres-0.1.34/mkpipe_loader_postgres.egg-info/PKG-INFO

@@ -1,73 +0,0 @@
-Metadata-Version: 2.1
-Name: mkpipe-loader-postgres
-Version: 0.1.34
-Summary: PostgreSQL loader for mkpipe.
-Author: Metin Karakus
-Author-email: metin_karakus@yahoo.com
-License: Apache License 2.0
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: Apache Software License
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: mkpipe
-
-# MkPipe
-
-**MkPipe** is a modular, open-source ETL (Extract, Transform, Load) tool that allows you to integrate various data sources and sinks easily. It is designed to be extensible with a plugin-based architecture that supports extractors, transformers, and loaders.
-
-## Features
-
-- Extract data from multiple sources (e.g., PostgreSQL, MongoDB).
-- Transform data using custom Python logic and Apache Spark.
-- Load data into various sinks (e.g., ClickHouse, PostgreSQL, Parquet).
-- Plugin-based architecture that supports future extensions.
-- Cloud-native architecture, can be deployed on Kubernetes and other environments.
-
-## Installation
-
-You can install the core package and extractors using pip:
-
-### Install the core package:
-```bash
-pip install mkpipe
-```
-
-### Install the Postgres extractor:
-```bash
-pip install mkpipe-loader-postgres
-```
-
-### Install additional extractors or loaders as needed:
-You can find or contribute new extractors and loaders in the future.
-
-## Usage
-
-To run the ETL process, use the following command:
-
-```py
-from mkpipe_core.plugins.registry import EXTRACTORS
-
-def test_postgres_extractor():
-    postgres_extractor = EXTRACTORS.get("postgresql")
-    if not postgres_extractor:
-        print("Postgres extractor not found!")
-        return
-    instance = postgres_extractor()
-    instance.extract()
-
-if __name__ == "__main__":
-    test_postgres_extractor()
-
-```
-
-Where `elt.yaml` is your configuration file that specifies the extractors, transformers, and loaders.
-
-## Documentation
-
-For more detailed documentation, please visit the [GitHub repository](https://github.com/mkpipe-etl/mkpipe).
-
-## License
-
-This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details.
-