compare-schema 0.1.1__py3-none-any.whl

compare_schema-0.1.1.dist-info/METADATA

@@ -0,0 +1,206 @@
+Metadata-Version: 2.4
+Name: compare-schema
+Version: 0.1.1
+Summary: Compare schema and data between databases
+Author-email: Gu Park <doctorgu@kakao.com>
+License-Expression: MIT
+Project-URL: homepage, https://github.com/doctorgu/compare_schema
+Project-URL: repository, https://github.com/doctorgu/compare_schema
+Project-URL: documentation, https://github.com/doctorgu/compare_schema
+Keywords: compare schema,database comparison,schema diff,mysql,sqlite
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: sqlite3-client>=1.2.0
+Requires-Dist: mysqlclient-client>=1.2.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pydantic-settings>=2.0
+Requires-Dist: xlsxwriter>=3.0
+Requires-Dist: tabulate>=0.9
+Requires-Dist: types-tabulate>=0.10
+Requires-Dist: requests>=2.28
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: SQLAlchemy>=2.0
+Requires-Dist: PyMySQL>=1.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: ruff
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+Requires-Dist: pytest-env; extra == "test"
+Requires-Dist: pytest-mock; extra == "test"
+Requires-Dist: pytest-asyncio; extra == "test"
+Requires-Dist: build; extra == "test"
+Dynamic: license-file
+
+# compare-schema — Database Schema and Data Comparison Tool
+
+[English](README.md) | [한국어](README_KO.md)
+
+A database schema and data comparison tool across different environments (e.g. dev, stg, prd).
+
+> [!NOTE]
+> Currently, only **MySQL** schema and data are supported. The comparison metadata and version history are stored locally in an embedded SQLite database.
+
+## Features
+
+- **Schema Comparison**: Compare tables, views, columns, indices, and foreign key references across databases.
+- **Data Comparison**: Compare key-based data records across environments.
+- **Database History Tracking**: Track changes to the same database over time by comparing versions.
+- **Multiple Output Formats**: Export comparison reports to Markdown (`.md`) or Excel (`.xlsx`).
+- **History & Versioning**: Tracks versions in SQLite to detect schema drift over time.
+- **Notifications**: Discord webhook notification support for schema and data changes.
+- **CLI & Module Support**: Run via `compare-schema` CLI command or `python -m src.main`.
+
+## Installation
+
+```bash
+pip install compare-schema
+```
+
+## Quick Start
+
+### 1. Configuration (`config/config.yaml`)
+
+Create a configuration YAML file:
+
+```yaml
+output_type: md # md or xlsx
+output_dir: ./compare_schema/result
+prev_version: ""
+
+data_tables:
+  - table: users
+    columns:
+      - name
+      - email
+      - nick_name
+    key_columns:
+      - id
+
+exclude_tables: []
+exclude_columns: []
+db_host: 127.0.0.1
+envs:
+  - name: dev
+    port: 3306
+    db_names: [shop, logs]
+  - name: stg
+    port: 3306
+    db_names: [shop]
+  - name: prd
+    port: 3306
+    db_names: [shop]
+
+compare_list:
+  # Compare between different databases or environments
+  - a: dev.shop
+    b: dev.logs
+  - a: dev.shop
+    b: stg.shop
+  - a: stg.shop
+    b: prd.shop
+  # If a and b is same, compares between recent version and new version (shows history of that DB)
+  - a: dev.shop
+    b: dev.shop
+  - a: stg.shop
+    b: stg.shop
+  - a: prd.shop
+    b: prd.shop
+```
+
+### 2. Version Comparison & History
+
+The tool supports two comparison modes based on the entries defined in `compare_list` (`{env}.{database}`):
+
+#### A. Cross-Database Comparison (`a != b`)
+
+Compares schemas and data between two distinct environments or databases (e.g., `dev.shop` vs `stg.shop`):
+
+- **Schema Differences**:
+  - `schema_not_exists`: Identifies missing tables, views, columns, indices, and foreign keys present in one database but absent in the other.
+  - `schema_diff`: Identifies structural discrepancies in matching objects:
+    - **Columns**: Data types, nullability (`is_nullable`), default values, character length, numeric precision/scale, and column comments.
+    - **Indices**: Index types (BTREE, etc.), uniqueness (`is_unique`), and indexed column combinations.
+    - **Foreign Keys**: Target referenced tables and columns.
+- **Data Differences**:
+  - Compares rows in tables specified in `data_tables`.
+  - Matches rows between databases using the primary/unique keys configured in `key_columns`.
+  - `data_not_exists`: Rows present in one database but missing in the other.
+  - `data_diff`: Rows that exist in both databases but have differing values in the monitored `columns`.
+- **Output**: Exported to `{version}_compare.md` (Markdown) or the `compare` sheets in Excel.
+
+#### B. Database History & Drift Tracking (`a == b`)
+
+When `a` and `b` reference the same database (e.g., `a: dev.shop, b: dev.shop`), the tool compares the new `version` (timestamped as `YYYYMMDDHHmm`) against `prev_version`:
+
+- **Change Categories**:
+  - `schema_added` / `data_added`: Newly created tables, views, columns, indices, foreign keys, or inserted rows.
+  - `schema_removed` / `data_removed`: Dropped schema objects or deleted rows.
+  - `schema_changed` / `data_changed`: Modified column types/properties or updated data row values over time.
+- **Output**: Exported to `{version}_log.md` (Markdown) or the `log` sheets in Excel.
+
+---
+
+### Configuration Reference (`config.yaml`)
+
+| Parameter | Type | Default | Description |
+| :--- | :--- | :--- | :--- |
+| `output_type` | `Literal["xlsx", "md"]` | Required | Output format. `"md"` generates Markdown files; `"xlsx"` generates an Excel workbook. |
+| `output_dir` | `str` | Required | Directory path where output reports are saved. |
+| `prev_version` | `str` | `""` | Previous version (`YYYYMMDDHHmm`) to compare against. If empty (`""`), automatically uses the latest version recorded in SQLite. |
+| `process_when_changed` | `bool` | Required | If `true`, stops execution without generating reports if no schema or data changes are detected against `prev_version`. If `false`, generates reports on every run. |
+| `data_tables` | `list[DataTableConfig]` | `[]` | List of tables and columns to compare data for. |
+| `data_tables[].table` | `str` | Required | Table name in MySQL. |
+| `data_tables[].columns` | `list[str]` | Required | Columns to compare for value differences. |
+| `data_tables[].key_columns` | `list[str]` | Required | Key columns (e.g., primary key) used to match corresponding rows across databases. |
+| `exclude_tables` | `list[str]` | Required | Table names to exclude from cross-environment schema comparison (`a != b`), such as migration tables (e.g., `["alembic_version"]`). *(Not excluded during same-database history tracking `a == b`)*. |
+| `exclude_columns` | `list[str]` | Required | Column names to ignore during cross-environment column comparisons. |
+| `db_host` | `str` | Required | Hostname or IP address for MySQL database connections. |
+| `envs` | `list[EnvConfig]` | Required | List of environments and databases to connect to and extract. |
+| `envs[].name` | `str` | Required | Environment identifier (e.g., `dev`, `stg`, `prd`). |
+| `envs[].port` | `int` | Required | MySQL port number for this environment. |
+| `envs[].db_names` | `list[str]` | Required | List of database names to extract from this environment. |
+| `compare_list` | `list[EnvCompareConfig]` | Required | Target database pairs to compare (e.g., `a: dev.shop`, `b: stg.shop`). |
+
+---
+
+### How It Works
+
+1. **Extraction & Snapshotting**:
+   - Connects to MySQL instances defined in `envs` using credentials from `.env`.
+   - Extracts schema definitions (`information_schema.TABLES`, `COLUMNS`, `STATISTICS`, `KEY_COLUMN_USAGE`) and records from `data_tables`.
+   - Saves a point-in-time snapshot into an embedded SQLite database (`db_client/schema_data.sqlite3`), tagged with the current version timestamp (`YYYYMMDDHHmm`).
+2. **Change Detection**:
+   - Queries SQLite to check whether any schema or data differences exist between the current version and `prev_version`.
+   - If `process_when_changed: true` and no changes are found, execution halts early.
+3. **Comparison Execution**:
+   - Compares all configured pairs in `compare_list` using parameterized queries with `#foreach` list filtering.
+4. **Report Export & Alerts**:
+   - Writes Markdown or Excel reports to `data/`.
+   - If schema or data changes are detected in log mode (`is_log: true`), an alert is automatically posted to Discord via `DISCORD_WEB_HOOK_URL`.
+   - Any runtime errors or exceptions are formatted and sent to Discord as alerts.
+
+### 3. Environment Variables (`.env`)
+
+Set database credentials in `.env`:
+
+```env
+DB_USERNAME=your_username
+DB_PASSWORD=your_password
+DISCORD_WEB_HOOK_URL=https://discord.com/api/webhooks/...
+```
+
+### 4. Run Comparison
+
+```bash
+# Run via CLI
+compare-schema --config_path config/config.yaml
+
+# Or run as Python module
+python -m src.main --config_path config/config.yaml
+```
+
+## License
+
+MIT License. See [LICENSE](LICENSE) for details.

compare_schema-0.1.1.dist-info/RECORD

@@ -0,0 +1,24 @@
+compare_schema-0.1.1.dist-info/licenses/LICENSE,sha256=SceEQXH2EqmhxRLvy-52rf03gwxH8LrGttgH-lannvU,1065
+db_client/schema.sql,sha256=L8c87TmnsCkpQp2uKW4x6nn1hTJ4bs5ONYuhiqa5yPY,1804
+db_client/schema_client.py,sha256=QI3w8pFuyYich_J4uy0YwHnj6h-E2idHjjNEb_IwKUs,285
+db_client/schema_settings.py,sha256=aGmfLtEVcAg3mrwHbB_GCvaJzL2JdC_FBvv0QKXMmto,1122
+db_client/source_client.py,sha256=y2EP4s_1qKCRPac_VcBJ5lzg8k0eSpOjELklIQ3gXiU,505
+db_client/source_settings.py,sha256=NR5XHBteLOUJFVahCs3fhNoldVVAJLOhen9chh_gwYo,1240
+db_client/queries/schema/data.yml,sha256=1izUdJpcC7kamR3Y9fpRM4WyXSe-ekqjpFLJqTozpwA,3848
+db_client/queries/schema/schema.yml,sha256=WtB__WOEZ--lLeVygrhME1Bmc9V5iyTDMr3sjXEmXNU,20524
+db_client/queries/source/import_data.yml,sha256=hFcwwZ0Fm_4VY9Lwcl9o-ewe3-SbriRXhpeJDEB6gOc,188
+db_client/queries/source/import_schema.yml,sha256=tKONtDwTx4Wpqt1oW93OJEr1J6-ErEqYky182f-99ew,2185
+src/config.py,sha256=_xfR1G8C9rzzvTwd8C0nszNL9Qx34j58dzmoCjbw6t8,1392
+src/env_config.py,sha256=-JeN-P_5X4J3r02_mrLCnbRlY5lcmYUEf8NUUue_Gio,385
+src/excel_helper.py,sha256=6zlzvUFggEznnWiIq0vA4P_nweBO7I7v2R1ZL_cGX2E,2452
+src/main.py,sha256=uO0fSNnW6c9dqksX3UqszTkyakcf77Wus7xS4JVU5iw,47555
+src/markdown_helper.py,sha256=cMyF0FSm0tu3PqfJ2joerrZPqouX_yVgUdbUOSppWXY,2624
+src/models.py,sha256=trzluwMu1gaCL_drEprGcDRiGVf0ftoeGNl5jGMgdi0,6774
+src/util_mysql.py,sha256=lknFYsM57d42-xY0LMG4UzV41bG1vFhRT8dkLdhWMxk,10781
+src/util_other.py,sha256=qMxTLqwpHLR5v90KZEYlvLh2qrsyxQbMklZjOmBHXJk,624
+src/util_path.py,sha256=6FBCRFjZPPFOxmh1SQXn4g8bGN-nQKvrrZua5-nry6U,3505
+compare_schema-0.1.1.dist-info/METADATA,sha256=fInsqSoxpttNg9UABnmWdsoU9G8FT8S1mCXJtFqV_00,8849
+compare_schema-0.1.1.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
+compare_schema-0.1.1.dist-info/entry_points.txt,sha256=A5fUzIs5Z1S51jWFufwMNoYLf1U8xPDiY-ShEdcZXUA,49
+compare_schema-0.1.1.dist-info/top_level.txt,sha256=nwhb5mKiVo0RuDBMfpvgFk3OC5T4z6FMIOq-Z2FD_is,14
+compare_schema-0.1.1.dist-info/RECORD,,

compare_schema-0.1.1.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.0.0)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

compare_schema-0.1.1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+compare-schema = src.main:main

compare_schema-0.1.1.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 doctorgu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

compare_schema-0.1.1.dist-info/top_level.txt

@@ -0,0 +1,2 @@
+db_client
+src

db_client/queries/schema/data.yml

@@ -0,0 +1,142 @@
+- name: delete_data_by_version
+  value: |
+    DELETE
+    FROM    ${table}
+    WHERE   version = :version
+
+- name: insert_data_from_mysql
+  value: |
+    INSERT INTO ${table}
+        (
+            version, env_name, db_name,
+            ${columns}
+        )
+    VALUES
+        (
+            :version, :env_name, :db_name,
+            ${values}
+        )
+
+- name: read_table_sql
+  value: |
+    SELECT  sql
+    FROM    sqlite_master
+    WHERE   type = 'table'
+            AND name = :table
+
+- name: read_table_info
+  value: |
+    PRAGMA table_info("${table}")
+
+- name: create_backup_table
+  value: |
+    DROP TABLE IF EXISTS "${backup_table}";
+
+    CREATE TABLE "${backup_table}" AS
+    SELECT  *
+    FROM    "${table}";
+
+    DROP TABLE "${table}";
+
+- name: create_table_index
+  value: |
+    CREATE TABLE "${table}" (
+    ${columns}
+    ) STRICT;
+
+    CREATE UNIQUE INDEX IF NOT EXISTS "${table}_unique_idx" ON "${table}" (${index_cols});
+
+- name: read_data_not_exists
+  value: |
+    SELECT  a.env_name, a.db_name, 
+            a.${key_columns}
+    FROM    ${table} a
+            LEFT JOIN ${table} b
+            ON
+    #if ${other_version}
+            b.version = :other_version
+    #else
+            a.version = b.version
+    #endif
+            AND b.env_name = :other_env_name
+            AND b.db_name = :other_db_name
+            AND ${key_columns_equal}
+    WHERE   a.version = :current_version
+            AND a.env_name = :current_env_name
+            AND a.db_name = :current_db_name
+            AND b.version IS NULL
+
+- name: read_data_different
+  value: |
+    SELECT  a.${key_columns},
+            ${other_columns_comma_a},
+            ${other_columns_comma_b},
+            a.env_name, a.db_name
+    FROM    ${table} a
+            INNER JOIN ${table} b
+            ON
+    #if ${other_version}
+            b.version = :other_version
+    #else
+            a.version = b.version
+    #endif
+            AND b.env_name = :other_env_name
+            AND b.db_name = :other_db_name
+            AND ${key_columns_equal}
+    WHERE   a.version = :current_version
+            AND a.env_name = :current_env_name
+            AND a.db_name = :current_db_name
+            AND
+            (
+                1 != 1
+                ${other_columns_inequal}
+            )
+
+- name: read_not_same_data
+  value: |
+    WITH
+    #foreach dt in ${data_tables} separator=','
+    n_${dt.table} AS (
+        SELECT  env_name, db_name,
+                #foreach col in ${dt.columns} separator=','
+                "${col}"
+                #endfor
+        FROM    "${dt.table}"
+        WHERE   version = :version
+    )
+    , o_${dt.table} AS (
+        SELECT  env_name, db_name,
+                #foreach col in ${dt.columns} separator=','
+                "${col}"
+                #endfor
+        FROM    "${dt.table}"
+        WHERE   version = :prev_version
+    )
+    #endfor
+    , cnt AS (
+        #foreach idx, dt in ${data_tables} separator=' UNION ALL '
+        SELECT  '${dt.table}' compare_type,
+                (
+                    SELECT  count(*) cnt_n
+                    FROM    n_${dt.table}
+                ) n,
+                (
+                    SELECT  count(*) cnt_o
+                    FROM    o_${dt.table}
+                ) o,
+                (
+                    SELECT  count(*) cnt_n_o
+                    FROM    n_${dt.table} n
+                            INNER JOIN o_${dt.table} o
+                            ON o.env_name = n.env_name
+                            AND o.db_name = n.db_name
+                            AND
+                            #foreach col in ${dt.columns} separator=' AND '
+                            COALESCE(o."${col}", '<NULL>') = COALESCE(n."${col}", '<NULL>')
+                            #endfor
+                ) n_o
+        #endfor
+    )
+    SELECT  compare_type
+    FROM    cnt
+    WHERE   n != o OR o != n_o