bp-condenser-postgresql 0.2.1__tar.gz

bp_condenser_postgresql-0.2.1/LICENSE

@@ -0,0 +1,9 @@
+Copyright 2019, Tonic AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+

bp_condenser_postgresql-0.2.1/PKG-INFO

@@ -0,0 +1,119 @@
+Metadata-Version: 2.4
+Name: bp-condenser-postgresql
+Version: 0.2.1
+Summary: Config-driven Postgres database subsetting tool.
+Author: Brightpick
+License-Expression: MIT
+Keywords: database,subset,subsetting,postgres,sampling,ETL
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Operating System :: OS Independent
+Classifier: Environment :: Console
+Classifier: Topic :: Database
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: toposort
+Requires-Dist: psycopg2-binary
+Dynamic: license-file
+
+# Condenser
+
+Condenser is a config-driven database subsetting tool for Postgres.
+
+Subsetting data is the process of taking a representative sample of your data in a manner that preserves the integrity of your database, e.g., give me 5% of my users. If you do this naively, e.g., just grab 5% of all the tables in your database, most likely, your database will break foreign key constraints. At best, you’ll end up with a statistically non-representative data sample.
+
+One common use-case is to scale down a production database to a more reasonable size so that it can be used in staging, test, and development environments. This can be done to save costs and, when used in tandem with PII removal, can be quite powerful as a productivity enhancer. Another example is copying specific rows from one database and placing them into another while maintaining referential integrity.
+
+You can find more details about how we built this [here](https://www.tonic.ai/blog/condenser-a-database-subsetting-tool) and [here](https://www.tonic.ai/blog/condenser-v2/).
+
+## Need to Subset a Large Database?
+
+Our open-source tool can subset databases up to 10GB, but it will struggle with larger databases. Our premium database subsetter can, among other things (graphical UI, job scheduling, fancy algorithms), subset multi-TB databases with ease. If you're interested find us at [hello@tonic.ai](mailto:hello@tonic.ai).
+
+# Installation
+
+Five steps to install, assuming Python 3.8+:
+
+1. Download the required Python modules. You can use [`pip`](https://pypi.org/project/pip/) for easy installation. The required modules are `toposort` and `psycopg2-binary`.
+```
+$ pip install toposort
+$ pip install psycopg2-binary
+```
+2. Install Postgres database tools. We need `pg_dump` and `psql`; they need to be on your `$PATH` or point to them with `$POSTGRES_PATH`.
+3. Download this repo. You can clone the repo or Download it as a zip. Scroll up, it's the green button that says "Clone or download".
+4. Setup your configuration and save it in `config.json`. The provided `config.json.example` has the skeleton of what you need to provide: source and destination database connection details, as well as subsetting goals in `initial_targets`. Here's an example that will collect 10% of a table named `public.target_table`.
+```
+"initial_targets": [
+    {
+        "table": "public.target_table",
+        "percent": 10
+    }
+]
+```
+There may be more required configuration depending on your database, but simple databases should be easy. See the Config section for more details, and `config.json.example_all` for all of the options in a single config file.
+
+5. Run! `$ python direct_subset.py`
+
+# Config
+
+Configuration must exist in `config.json`. There is an example configuration provided in `config.json.example`. Most of the configuration is straightforward: source and destination DB connection details and subsetting settings. There are three fields that deserve some additional attention.
+
+The first is `initial_targets`. This is where you tell the subsetter to begin the subset. You can specify any number of tables as an initial target, and provide either a percent goal (e.g. 5% of the `users` table) or a WHERE clause.
+
+Next is `dependency_breaks`. The best way to get a full understanding of this is to read our [blog post](https://www.tonic.ai/blog/condenser-a-database-subsetting-tool). But if you want a TLDR, it's this: The subsetting tool cannot operate on databases with cycles in their foreign key relationships. (Example: Table `events` references `users`, which references `company`, which references `events`, a cycle exists if you think about the foreign keys as a directed graph.) If your database has a foreign key cycle (any many do), have no fear! This field lets you tell the subsetter to ignore certain foreign keys, and essentially remove the cycle. You'll have to know a bit about your database to use this field effectively. The tool will warn you if you have a cycle that you haven't broken.
+
+The last is `fk_augmentation`. Databases frequently have foreign keys that are not codified as constraints on the database, these are implicit foreign keys. For a subsetter to create useful subsets if needs to know about this implicit constraints. This field lets you essentially add foreign keys to the subsetter that the DB doesn't have listed as a constraint.
+
+Below we describe the use of all configuration parameters, but the best place to start for the exact format is `config.json.example`.
+
+`db_type`: Required database type selector. The only supported value is `"postgres"`.
+
+`source_db_connection_info`: Source database connection details. These are recorded as a JSON object with the fields `user_name`, `host`, `db_name`, `ssl_mode`, `password` (optional), and `port`. If `password` is omitted, then you will be prompted for a password. See `config.json.example` for details.
+
+`destination_db_connection_info`: Destination database connection details. Same fields as `source_db_connection_info`.
+
+`initial_targets`: JSON array of JSON objects. The inner object must contain a `table` field, which is a target table, and either a `where` field or a `percent` field. The `where` field is used to specify a WHERE clause for the subsetting. The `percent` field indicates we want a specific percentage of the target table; it is equivalent to `"where": "random() < <percent>/100.0"`.
+
+`passthrough_tables`: Tables that will be copied to the destination database in whole. The value is a JSON array of strings in the form `"<schema>.<table>"`.
+
+`excluded_tables`: Tables that will be excluded from the subset. The table will exist in the output, but contain no rows. The value is a JSON array of strings in the form `"<schema>.<table>"`.
+
+`upstream_filters`: Additional filtering to be applied to tables during upstream subsetting. Upstream subsetting happens when a row is imported, and there are rows with foreign keys to that row. The subsetter then greedily grabs as many rows from the database as it can, based on the rows already imported. If you don't want such greedy behavior you can impose additional filters with this option. This is an advanced feature, you probably won't need for your first subsets. The value is a JSON array of JSON objects. See `example-config.json` for details.
+
+`fk_augmentation`: Additional foreign keys that, while not represented as constraints in the database, are logically present in the data. Foreign keys listed in `fk_augmentation` are unioned with the foreign keys provided by constraints in the database. `fk_augmentation` is useful when there are foreign keys existing in the data, but not represented in the database. The value is a JSON array of JSON objects. See `example-config.json` for details.
+
+`dependency_breaks`: An array containing JSON objects with *"fk_table"* and *"target_table"* fields of table relationships to be ignored in order to break cycles
+
+`keep_disconnected_tables`: If `true` tables that the subset target(s) don't reach, when following foreign keys, will be copied 100% over. If it's `false` then their schema will be copied but the table contents will be empty. Put more mathematically, the tables and foreign keys create a graph (where tables are nodes and foreign keys are directed edges) disconnected tables are the tables in components that don't contain any targets. This setting decides how to import those tables.
+
+`max_rows_per_table`: This is interpreted as a limit on all of the tables to be copied. Useful if you have some very large tables that you want a sampling from. For an unlimited dataset (recommended) set this parameter to `ALL`.
+
+`pre_constraint_sql`: An array of SQL commands that will be issued on the destination database after subsetting is complete, but before the database constraints have been applied. Useful to perform tasks that will clean up any data that would otherwise violate the database constraints. `post_subset_sql` is the preferred option for any general purpose queries.
+
+`post_subset_sql`: An array of SQL commands that will be issued on the destination database after subsetting is complete, and after the database constraints have been applied. Useful to perform additional adhoc tasks after subsetting.
+
+# Running
+
+Almost all the configuration is in the `config.json` file, so running is as simple as
+
+```
+$ python direct_subset.py
+```
+
+Two commandline arguments are supported:
+
+`-v`: Verbose output. Useful for performance debugging. Lists almost every query made, and it's speed.
+
+`--no-constraints`: Do not add constraints found in the source database to the destination database.
+
+# Requirements
+
+Reference the `requirements.txt` file for a list of required Python packages. Python 3.8+ is required.

bp_condenser_postgresql-0.2.1/README.md

@@ -0,0 +1,93 @@
+# Condenser
+
+Condenser is a config-driven database subsetting tool for Postgres.
+
+Subsetting data is the process of taking a representative sample of your data in a manner that preserves the integrity of your database, e.g., give me 5% of my users. If you do this naively, e.g., just grab 5% of all the tables in your database, most likely, your database will break foreign key constraints. At best, you’ll end up with a statistically non-representative data sample.
+
+One common use-case is to scale down a production database to a more reasonable size so that it can be used in staging, test, and development environments. This can be done to save costs and, when used in tandem with PII removal, can be quite powerful as a productivity enhancer. Another example is copying specific rows from one database and placing them into another while maintaining referential integrity.
+
+You can find more details about how we built this [here](https://www.tonic.ai/blog/condenser-a-database-subsetting-tool) and [here](https://www.tonic.ai/blog/condenser-v2/).
+
+## Need to Subset a Large Database?
+
+Our open-source tool can subset databases up to 10GB, but it will struggle with larger databases. Our premium database subsetter can, among other things (graphical UI, job scheduling, fancy algorithms), subset multi-TB databases with ease. If you're interested find us at [hello@tonic.ai](mailto:hello@tonic.ai).
+
+# Installation
+
+Five steps to install, assuming Python 3.8+:
+
+1. Download the required Python modules. You can use [`pip`](https://pypi.org/project/pip/) for easy installation. The required modules are `toposort` and `psycopg2-binary`.
+```
+$ pip install toposort
+$ pip install psycopg2-binary
+```
+2. Install Postgres database tools. We need `pg_dump` and `psql`; they need to be on your `$PATH` or point to them with `$POSTGRES_PATH`.
+3. Download this repo. You can clone the repo or Download it as a zip. Scroll up, it's the green button that says "Clone or download".
+4. Setup your configuration and save it in `config.json`. The provided `config.json.example` has the skeleton of what you need to provide: source and destination database connection details, as well as subsetting goals in `initial_targets`. Here's an example that will collect 10% of a table named `public.target_table`.
+```
+"initial_targets": [
+    {
+        "table": "public.target_table",
+        "percent": 10
+    }
+]
+```
+There may be more required configuration depending on your database, but simple databases should be easy. See the Config section for more details, and `config.json.example_all` for all of the options in a single config file.
+
+5. Run! `$ python direct_subset.py`
+
+# Config
+
+Configuration must exist in `config.json`. There is an example configuration provided in `config.json.example`. Most of the configuration is straightforward: source and destination DB connection details and subsetting settings. There are three fields that deserve some additional attention.
+
+The first is `initial_targets`. This is where you tell the subsetter to begin the subset. You can specify any number of tables as an initial target, and provide either a percent goal (e.g. 5% of the `users` table) or a WHERE clause.
+
+Next is `dependency_breaks`. The best way to get a full understanding of this is to read our [blog post](https://www.tonic.ai/blog/condenser-a-database-subsetting-tool). But if you want a TLDR, it's this: The subsetting tool cannot operate on databases with cycles in their foreign key relationships. (Example: Table `events` references `users`, which references `company`, which references `events`, a cycle exists if you think about the foreign keys as a directed graph.) If your database has a foreign key cycle (any many do), have no fear! This field lets you tell the subsetter to ignore certain foreign keys, and essentially remove the cycle. You'll have to know a bit about your database to use this field effectively. The tool will warn you if you have a cycle that you haven't broken.
+
+The last is `fk_augmentation`. Databases frequently have foreign keys that are not codified as constraints on the database, these are implicit foreign keys. For a subsetter to create useful subsets if needs to know about this implicit constraints. This field lets you essentially add foreign keys to the subsetter that the DB doesn't have listed as a constraint.
+
+Below we describe the use of all configuration parameters, but the best place to start for the exact format is `config.json.example`.
+
+`db_type`: Required database type selector. The only supported value is `"postgres"`.
+
+`source_db_connection_info`: Source database connection details. These are recorded as a JSON object with the fields `user_name`, `host`, `db_name`, `ssl_mode`, `password` (optional), and `port`. If `password` is omitted, then you will be prompted for a password. See `config.json.example` for details.
+
+`destination_db_connection_info`: Destination database connection details. Same fields as `source_db_connection_info`.
+
+`initial_targets`: JSON array of JSON objects. The inner object must contain a `table` field, which is a target table, and either a `where` field or a `percent` field. The `where` field is used to specify a WHERE clause for the subsetting. The `percent` field indicates we want a specific percentage of the target table; it is equivalent to `"where": "random() < <percent>/100.0"`.
+
+`passthrough_tables`: Tables that will be copied to the destination database in whole. The value is a JSON array of strings in the form `"<schema>.<table>"`.
+
+`excluded_tables`: Tables that will be excluded from the subset. The table will exist in the output, but contain no rows. The value is a JSON array of strings in the form `"<schema>.<table>"`.
+
+`upstream_filters`: Additional filtering to be applied to tables during upstream subsetting. Upstream subsetting happens when a row is imported, and there are rows with foreign keys to that row. The subsetter then greedily grabs as many rows from the database as it can, based on the rows already imported. If you don't want such greedy behavior you can impose additional filters with this option. This is an advanced feature, you probably won't need for your first subsets. The value is a JSON array of JSON objects. See `example-config.json` for details.
+
+`fk_augmentation`: Additional foreign keys that, while not represented as constraints in the database, are logically present in the data. Foreign keys listed in `fk_augmentation` are unioned with the foreign keys provided by constraints in the database. `fk_augmentation` is useful when there are foreign keys existing in the data, but not represented in the database. The value is a JSON array of JSON objects. See `example-config.json` for details.
+
+`dependency_breaks`: An array containing JSON objects with *"fk_table"* and *"target_table"* fields of table relationships to be ignored in order to break cycles
+
+`keep_disconnected_tables`: If `true` tables that the subset target(s) don't reach, when following foreign keys, will be copied 100% over. If it's `false` then their schema will be copied but the table contents will be empty. Put more mathematically, the tables and foreign keys create a graph (where tables are nodes and foreign keys are directed edges) disconnected tables are the tables in components that don't contain any targets. This setting decides how to import those tables.
+
+`max_rows_per_table`: This is interpreted as a limit on all of the tables to be copied. Useful if you have some very large tables that you want a sampling from. For an unlimited dataset (recommended) set this parameter to `ALL`.
+
+`pre_constraint_sql`: An array of SQL commands that will be issued on the destination database after subsetting is complete, but before the database constraints have been applied. Useful to perform tasks that will clean up any data that would otherwise violate the database constraints. `post_subset_sql` is the preferred option for any general purpose queries.
+
+`post_subset_sql`: An array of SQL commands that will be issued on the destination database after subsetting is complete, and after the database constraints have been applied. Useful to perform additional adhoc tasks after subsetting.
+
+# Running
+
+Almost all the configuration is in the `config.json` file, so running is as simple as
+
+```
+$ python direct_subset.py
+```
+
+Two commandline arguments are supported:
+
+`-v`: Verbose output. Useful for performance debugging. Lists almost every query made, and it's speed.
+
+`--no-constraints`: Do not add constraints found in the source database to the destination database.
+
+# Requirements
+
+Reference the `requirements.txt` file for a list of required Python packages. Python 3.8+ is required.

bp_condenser_postgresql-0.2.1/bp_condenser_postgresql.egg-info/PKG-INFO

@@ -0,0 +1,119 @@
+Metadata-Version: 2.4
+Name: bp-condenser-postgresql
+Version: 0.2.1
+Summary: Config-driven Postgres database subsetting tool.
+Author: Brightpick
+License-Expression: MIT
+Keywords: database,subset,subsetting,postgres,sampling,ETL
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Operating System :: OS Independent
+Classifier: Environment :: Console
+Classifier: Topic :: Database
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: toposort
+Requires-Dist: psycopg2-binary
+Dynamic: license-file
+
+# Condenser
+
+Condenser is a config-driven database subsetting tool for Postgres.
+
+Subsetting data is the process of taking a representative sample of your data in a manner that preserves the integrity of your database, e.g., give me 5% of my users. If you do this naively, e.g., just grab 5% of all the tables in your database, most likely, your database will break foreign key constraints. At best, you’ll end up with a statistically non-representative data sample.
+
+One common use-case is to scale down a production database to a more reasonable size so that it can be used in staging, test, and development environments. This can be done to save costs and, when used in tandem with PII removal, can be quite powerful as a productivity enhancer. Another example is copying specific rows from one database and placing them into another while maintaining referential integrity.
+
+You can find more details about how we built this [here](https://www.tonic.ai/blog/condenser-a-database-subsetting-tool) and [here](https://www.tonic.ai/blog/condenser-v2/).
+
+## Need to Subset a Large Database?
+
+Our open-source tool can subset databases up to 10GB, but it will struggle with larger databases. Our premium database subsetter can, among other things (graphical UI, job scheduling, fancy algorithms), subset multi-TB databases with ease. If you're interested find us at [hello@tonic.ai](mailto:hello@tonic.ai).
+
+# Installation
+
+Five steps to install, assuming Python 3.8+:
+
+1. Download the required Python modules. You can use [`pip`](https://pypi.org/project/pip/) for easy installation. The required modules are `toposort` and `psycopg2-binary`.
+```
+$ pip install toposort
+$ pip install psycopg2-binary
+```
+2. Install Postgres database tools. We need `pg_dump` and `psql`; they need to be on your `$PATH` or point to them with `$POSTGRES_PATH`.
+3. Download this repo. You can clone the repo or Download it as a zip. Scroll up, it's the green button that says "Clone or download".
+4. Setup your configuration and save it in `config.json`. The provided `config.json.example` has the skeleton of what you need to provide: source and destination database connection details, as well as subsetting goals in `initial_targets`. Here's an example that will collect 10% of a table named `public.target_table`.
+```
+"initial_targets": [
+    {
+        "table": "public.target_table",
+        "percent": 10
+    }
+]
+```
+There may be more required configuration depending on your database, but simple databases should be easy. See the Config section for more details, and `config.json.example_all` for all of the options in a single config file.
+
+5. Run! `$ python direct_subset.py`
+
+# Config
+
+Configuration must exist in `config.json`. There is an example configuration provided in `config.json.example`. Most of the configuration is straightforward: source and destination DB connection details and subsetting settings. There are three fields that deserve some additional attention.
+
+The first is `initial_targets`. This is where you tell the subsetter to begin the subset. You can specify any number of tables as an initial target, and provide either a percent goal (e.g. 5% of the `users` table) or a WHERE clause.
+
+Next is `dependency_breaks`. The best way to get a full understanding of this is to read our [blog post](https://www.tonic.ai/blog/condenser-a-database-subsetting-tool). But if you want a TLDR, it's this: The subsetting tool cannot operate on databases with cycles in their foreign key relationships. (Example: Table `events` references `users`, which references `company`, which references `events`, a cycle exists if you think about the foreign keys as a directed graph.) If your database has a foreign key cycle (any many do), have no fear! This field lets you tell the subsetter to ignore certain foreign keys, and essentially remove the cycle. You'll have to know a bit about your database to use this field effectively. The tool will warn you if you have a cycle that you haven't broken.
+
+The last is `fk_augmentation`. Databases frequently have foreign keys that are not codified as constraints on the database, these are implicit foreign keys. For a subsetter to create useful subsets if needs to know about this implicit constraints. This field lets you essentially add foreign keys to the subsetter that the DB doesn't have listed as a constraint.
+
+Below we describe the use of all configuration parameters, but the best place to start for the exact format is `config.json.example`.
+
+`db_type`: Required database type selector. The only supported value is `"postgres"`.
+
+`source_db_connection_info`: Source database connection details. These are recorded as a JSON object with the fields `user_name`, `host`, `db_name`, `ssl_mode`, `password` (optional), and `port`. If `password` is omitted, then you will be prompted for a password. See `config.json.example` for details.
+
+`destination_db_connection_info`: Destination database connection details. Same fields as `source_db_connection_info`.
+
+`initial_targets`: JSON array of JSON objects. The inner object must contain a `table` field, which is a target table, and either a `where` field or a `percent` field. The `where` field is used to specify a WHERE clause for the subsetting. The `percent` field indicates we want a specific percentage of the target table; it is equivalent to `"where": "random() < <percent>/100.0"`.
+
+`passthrough_tables`: Tables that will be copied to the destination database in whole. The value is a JSON array of strings in the form `"<schema>.<table>"`.
+
+`excluded_tables`: Tables that will be excluded from the subset. The table will exist in the output, but contain no rows. The value is a JSON array of strings in the form `"<schema>.<table>"`.
+
+`upstream_filters`: Additional filtering to be applied to tables during upstream subsetting. Upstream subsetting happens when a row is imported, and there are rows with foreign keys to that row. The subsetter then greedily grabs as many rows from the database as it can, based on the rows already imported. If you don't want such greedy behavior you can impose additional filters with this option. This is an advanced feature, you probably won't need for your first subsets. The value is a JSON array of JSON objects. See `example-config.json` for details.
+
+`fk_augmentation`: Additional foreign keys that, while not represented as constraints in the database, are logically present in the data. Foreign keys listed in `fk_augmentation` are unioned with the foreign keys provided by constraints in the database. `fk_augmentation` is useful when there are foreign keys existing in the data, but not represented in the database. The value is a JSON array of JSON objects. See `example-config.json` for details.
+
+`dependency_breaks`: An array containing JSON objects with *"fk_table"* and *"target_table"* fields of table relationships to be ignored in order to break cycles
+
+`keep_disconnected_tables`: If `true` tables that the subset target(s) don't reach, when following foreign keys, will be copied 100% over. If it's `false` then their schema will be copied but the table contents will be empty. Put more mathematically, the tables and foreign keys create a graph (where tables are nodes and foreign keys are directed edges) disconnected tables are the tables in components that don't contain any targets. This setting decides how to import those tables.
+
+`max_rows_per_table`: This is interpreted as a limit on all of the tables to be copied. Useful if you have some very large tables that you want a sampling from. For an unlimited dataset (recommended) set this parameter to `ALL`.
+
+`pre_constraint_sql`: An array of SQL commands that will be issued on the destination database after subsetting is complete, but before the database constraints have been applied. Useful to perform tasks that will clean up any data that would otherwise violate the database constraints. `post_subset_sql` is the preferred option for any general purpose queries.
+
+`post_subset_sql`: An array of SQL commands that will be issued on the destination database after subsetting is complete, and after the database constraints have been applied. Useful to perform additional adhoc tasks after subsetting.
+
+# Running
+
+Almost all the configuration is in the `config.json` file, so running is as simple as
+
+```
+$ python direct_subset.py
+```
+
+Two commandline arguments are supported:
+
+`-v`: Verbose output. Useful for performance debugging. Lists almost every query made, and it's speed.
+
+`--no-constraints`: Do not add constraints found in the source database to the destination database.
+
+# Requirements
+
+Reference the `requirements.txt` file for a list of required Python packages. Python 3.8+ is required.

bp_condenser_postgresql-0.2.1/bp_condenser_postgresql.egg-info/SOURCES.txt

@@ -0,0 +1,19 @@
+LICENSE
+README.md
+config_reader.py
+database_helper.py
+db_connect.py
+direct_subset.py
+psql_database_creator.py
+psql_database_helper.py
+pyproject.toml
+result_tabulator.py
+subset.py
+subset_utils.py
+topo_orderer.py
+bp_condenser_postgresql.egg-info/PKG-INFO
+bp_condenser_postgresql.egg-info/SOURCES.txt
+bp_condenser_postgresql.egg-info/dependency_links.txt
+bp_condenser_postgresql.egg-info/entry_points.txt
+bp_condenser_postgresql.egg-info/requires.txt
+bp_condenser_postgresql.egg-info/top_level.txt

bp_condenser_postgresql-0.2.1/bp_condenser_postgresql.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

bp_condenser_postgresql-0.2.1/bp_condenser_postgresql.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+condenser-direct-subset = direct_subset:main

bp_condenser_postgresql-0.2.1/bp_condenser_postgresql.egg-info/requires.txt

@@ -0,0 +1,2 @@
+toposort
+psycopg2-binary

bp_condenser_postgresql-0.2.1/bp_condenser_postgresql.egg-info/top_level.txt

@@ -0,0 +1,10 @@
+config_reader
+database_helper
+db_connect
+direct_subset
+psql_database_creator
+psql_database_helper
+result_tabulator
+subset
+subset_utils
+topo_orderer

bp_condenser_postgresql-0.2.1/config_reader.py

@@ -0,0 +1,93 @@
+import json, sys, collections
+
+_config = None
+
+def initialize(file_like = None):
+    global _config
+    if _config != None:
+        print('WARNING: Attempted to initialize configuration twice.', file=sys.stderr)
+
+    if not file_like:
+        with open('config.json', 'r') as fp:
+            _config = json.load(fp)
+    else:
+        _config = json.load(file_like)
+
+    if "desired_result" in _config:
+        raise ValueError("desired_result is a key in the old config spec. Check the README.md and config.json.example for the latest configuration parameters.")
+
+    _validate_db_type()
+
+def _validate_db_type():
+    if 'db_type' not in _config:
+        raise ValueError("Missing required config key 'db_type'. The only supported value is 'postgres'.")
+
+    db_type = _config['db_type']
+    if not isinstance(db_type, str):
+        raise ValueError("Invalid db_type {!r}. The only supported value is 'postgres'.".format(db_type))
+
+    normalized_db_type = db_type.lower()
+    if normalized_db_type != 'postgres':
+        raise ValueError("Unsupported db_type '{}'. Condenser supports only 'postgres'.".format(db_type))
+
+    _config['db_type'] = normalized_db_type
+
+DependencyBreak = collections.namedtuple('DependencyBreak', ['fk_table', 'target_table'])
+def get_dependency_breaks():
+    return set([DependencyBreak(b['fk_table'], b['target_table']) for b in _config['dependency_breaks']])
+
+def get_preserve_fk_opportunistically():
+    return set([DependencyBreak(b['fk_table'], b['target_table']) for b in _config['dependency_breaks'] if 'perserve_fk_opportunistically' in b and b['perserve_fk_opportunistically']])
+
+def get_initial_targets():
+    return _config['initial_targets']
+
+def get_initial_target_tables():
+    return [target["table"] for target in _config['initial_targets']]
+
+def keep_disconnected_tables():
+    return 'keep_disconnected_tables' in _config and bool(_config['keep_disconnected_tables'])
+
+def get_db_type():
+    return _config['db_type']
+
+def get_source_db_connection_info():
+    return _config['source_db_connection_info']
+
+def get_destination_db_connection_info():
+    return _config['destination_db_connection_info']
+
+def get_excluded_tables():
+    return list(_config['excluded_tables'])
+
+def get_passthrough_tables():
+    return list(_config['passthrough_tables'])
+
+def get_fk_augmentation():
+    return list(map(__convert_tonic_format, _config['fk_augmentation']))
+
+def get_upstream_filters():
+    return _config["upstream_filters"]
+
+def get_pre_constraint_sql():
+    return _config["pre_constraint_sql"] if "pre_constraint_sql" in _config else []
+
+def get_post_subset_sql():
+    return _config["post_subset_sql"] if "post_subset_sql" in _config else []
+
+def get_max_rows_per_table():
+    return _config["max_rows_per_table"] if "max_rows_per_table" in _config else None
+
+def __convert_tonic_format(obj):
+    if "fk_schema" in obj:
+        return {
+            "fk_table": obj["fk_schema"] + "." + obj["fk_table"],
+            "fk_columns": obj["fk_columns"],
+            "target_table": obj["target_schema"] + "." + obj["target_table"],
+            "target_columns": obj["target_columns"],
+        }
+    else:
+        return obj
+
+def verbose_logging():
+    return '-v' in sys.argv

bp_condenser_postgresql-0.2.1/database_helper.py

@@ -0,0 +1,8 @@
+import config_reader
+
+def get_specific_helper():
+    if config_reader.get_db_type() == 'postgres':
+        import psql_database_helper
+        return psql_database_helper
+    else:
+        raise ValueError('unsupported db_type ' + config_reader.get_db_type())

bp_condenser_postgresql-0.2.1/db_connect.py

@@ -0,0 +1,85 @@
+import config_reader
+import psycopg2
+import os, pathlib, re, urllib, subprocess, os.path, json, getpass, time, sys, datetime
+
+class DbConnect:
+
+    def __init__(self, db_type, connection_info):
+        requiredKeys = [
+            'user_name',
+            'host',
+            'db_name',
+            'port'
+        ]
+
+        for r in requiredKeys:
+            if r not in connection_info.keys():
+                raise Exception('Missing required key in database connection info: ' + r)
+        if 'password' not in connection_info.keys():
+            connection_info['password'] = getpass.getpass('Enter password for {0} on host {1}: '.format(connection_info['user_name'], connection_info['host']))
+
+        self.user = connection_info['user_name']
+        self.password = connection_info['password']
+        self.host = connection_info['host']
+        self.port = connection_info['port']
+        self.db_name = connection_info['db_name']
+        self.ssl_mode = connection_info['ssl_mode'] if 'ssl_mode' in connection_info else None
+        self.__db_type = db_type.lower()
+
+    def get_db_connection(self, read_repeatable=False):
+
+        if self.__db_type == 'postgres':
+            return PsqlConnection(self, read_repeatable)
+        else:
+            raise ValueError('unsupported db_type ' + self.__db_type)
+
+class DbConnection:
+    def __init__(self, connection):
+        self.connection = connection
+
+    def commit(self):
+        self.connection.commit()
+
+    def close(self):
+        self.connection.close()
+
+
+class LoggingCursor:
+    def __init__(self, cursor):
+        self.inner_cursor = cursor
+
+    def execute(self, query):
+        start_time = time.time()
+        if config_reader.verbose_logging():
+            print('Beginning query @ {}:\n\t{}'.format(str(datetime.datetime.now()), query))
+            sys.stdout.flush()
+        retval = self.inner_cursor.execute(query)
+        if config_reader.verbose_logging():
+            print('\tQuery completed in {}s'.format(time.time() - start_time))
+            sys.stdout.flush()
+        return retval
+
+    def __getattr__(self, name):
+        return self.inner_cursor.__getattribute__(name)
+
+    def __exit__(self, a, b, c):
+        return self.inner_cursor.__exit__(a, b, c)
+
+    def __enter__(self):
+        return LoggingCursor(self.inner_cursor.__enter__())
+
+# small wrapper to the connection class that gives us a common interface to the cursor()
+# method. This one is for Postgres.
+class PsqlConnection(DbConnection):
+    def __init__(self,  connect, read_repeatable):
+        connection_string = 'dbname=\'{0}\' user=\'{1}\' password=\'{2}\' host={3} port={4}'.format(connect.db_name, connect.user, connect.password, connect.host, connect.port)
+
+        if connect.ssl_mode :
+            connection_string = connection_string + ' sslmode={0}'.format(connect.ssl_mode)
+
+        DbConnection.__init__(self, psycopg2.connect(connection_string))
+        if read_repeatable:
+            self.connection.isolation_level = psycopg2.extensions.ISOLATION_LEVEL_REPEATABLE_READ
+
+    def cursor(self, name=None, withhold=False):
+        return LoggingCursor(self.connection.cursor(name=name, withhold=withhold))

bp_condenser_postgresql-0.2.1/direct_subset.py

@@ -0,0 +1,66 @@
+import uuid, sys
+import config_reader, result_tabulator
+import time
+from subset import Subset
+from psql_database_creator import PsqlDatabaseCreator
+from db_connect import DbConnect
+from subset_utils import print_progress
+import database_helper
+
+def db_creator(db_type, source, dest):
+    if db_type == 'postgres':
+        return PsqlDatabaseCreator(source, dest, False)
+    else:
+        raise ValueError('unsupported db_type ' + db_type)
+
+
+def main() -> None:
+    if "--stdin" in sys.argv:
+        config_reader.initialize(sys.stdin)
+    else:
+        config_reader.initialize()
+
+    db_type = config_reader.get_db_type()
+    source_dbc = DbConnect(db_type, config_reader.get_source_db_connection_info())
+    destination_dbc = DbConnect(db_type, config_reader.get_destination_db_connection_info())
+
+    database = db_creator(db_type, source_dbc, destination_dbc)
+    database.teardown()
+    database.create()
+
+    # Get list of tables to operate on
+    db_helper = database_helper.get_specific_helper()
+    all_tables = db_helper.list_all_tables(source_dbc)
+    all_tables = [x for x in all_tables if x not in config_reader.get_excluded_tables()]
+
+    subsetter = Subset(source_dbc, destination_dbc, all_tables)
+
+    try:
+        subsetter.prep_temp_dbs()
+        subsetter.run_middle_out()
+
+        print("Beginning pre constraint SQL calls")
+        start_time = time.time()
+        for idx, sql in enumerate(config_reader.get_pre_constraint_sql()):
+            print_progress(sql, idx+1, len(config_reader.get_pre_constraint_sql()))
+            db_helper.run_query(sql, destination_dbc.get_db_connection())
+        print("Completed pre constraint SQL calls in {}s".format(time.time()-start_time))
+
+        print("Adding database constraints")
+        if "--no-constraints" not in sys.argv:
+            database.add_constraints()
+
+        print("Beginning post subset SQL calls")
+        start_time = time.time()
+        for idx, sql in enumerate(config_reader.get_post_subset_sql()):
+            print_progress(sql, idx+1, len(config_reader.get_post_subset_sql()))
+            db_helper.run_query(sql, destination_dbc.get_db_connection())
+        print("Completed post subset SQL calls in {}s".format(time.time()-start_time))
+
+        result_tabulator.tabulate(source_dbc, destination_dbc, all_tables)
+    finally:
+        subsetter.unprep_temp_dbs()
+
+
+if __name__ == '__main__':
+    main()