rowproof 0.1.0__tar.gz → 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {rowproof-0.1.0 → rowproof-0.1.2}/PKG-INFO +13 -1
- {rowproof-0.1.0 → rowproof-0.1.2}/README.md +12 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/pyproject.toml +1 -1
- rowproof-0.1.2/rowproof/__init__.py +1 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/cli/main.py +21 -12
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/core/errors.py +26 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/core/hashdiff.py +11 -1
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof.egg-info/PKG-INFO +13 -1
- rowproof-0.1.0/rowproof/__init__.py +0 -1
- {rowproof-0.1.0 → rowproof-0.1.2}/LICENSE +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/cli/__init__.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/cli/render.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/cli/spec.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/config/__init__.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/connectors/__init__.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/connectors/_chwire.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/connectors/_pgwire.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/connectors/_sfwire.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/connectors/base.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/connectors/clickhouse.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/connectors/postgres.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/connectors/snowflake.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/core/__init__.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/core/column_matching.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/core/joindiff.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/core/models.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/core/normalisation.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/core/segmentation.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/report/__init__.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof/report/html.py +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof.egg-info/SOURCES.txt +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof.egg-info/dependency_links.txt +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof.egg-info/entry_points.txt +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof.egg-info/requires.txt +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/rowproof.egg-info/top_level.txt +0 -0
- {rowproof-0.1.0 → rowproof-0.1.2}/setup.cfg +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: rowproof
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.2
|
|
4
4
|
Summary: Prove a database migration or replication didn't lose, add or change a single row.
|
|
5
5
|
License: Apache-2.0
|
|
6
6
|
Requires-Python: >=3.11
|
|
@@ -87,6 +87,18 @@ public.orders postgres:postgres/public.orders -> clickhouse:default/orders
|
|
|
87
87
|
| ClickHouse | `clickhouse-connect` | |
|
|
88
88
|
| Snowflake | `snowflake-connector-python` | Optional extra: `rowproof[snowflake]` |
|
|
89
89
|
|
|
90
|
+
### Snowflake: known limitations
|
|
91
|
+
|
|
92
|
+
Snowflake support works, but it is the least battle-tested of the three engines. It has been tested against a single trial account with small tables; Postgres and ClickHouse have had far more. Know these before you rely on it:
|
|
93
|
+
|
|
94
|
+
- **Login is username and password only.** The connection string has no place for a warehouse, role, SSO, MFA, or key-pair login. Accounts that require SSO or key-pair authentication won't work yet, and queries run on the user's *default* warehouse (a user with none set will fail).
|
|
95
|
+
- **Column names are case-sensitive.** Snowflake stores unquoted names in upper case, so `--key id` won't match a column stored as `ID`. Pass `--key ID`, or create the table with quoted lower-case names. The error message tells you when this is the problem.
|
|
96
|
+
- **Primary keys are often not declared** on Snowflake tables (Snowflake doesn't enforce them). If auto-detection finds none, pass `--key`.
|
|
97
|
+
- **JSON columns are compared as plain text.** `VARIANT`, `ARRAY` and `OBJECT` use rule UNK-1: two JSON values that mean the same thing but are formatted differently will show as different, and a JSON `null` is not distinguished from an SQL `NULL`.
|
|
98
|
+
- **Special characters in passwords must be percent-encoded** in the connection string (`@` becomes `%40`).
|
|
99
|
+
- **It uses your warehouse's credits.** Every query runs on your Snowflake warehouse. Start with a small table or `--sample`.
|
|
100
|
+
- **No Snowflake benchmark yet.** The 100M-row benchmark below is Postgres to ClickHouse only.
|
|
101
|
+
|
|
90
102
|
## Supported types (canonical comparison rules)
|
|
91
103
|
|
|
92
104
|
Every cross-engine type difference is governed by one of these published rules, and every difference rowproof reports cites the rule ID that applied — never a vague "values differ."
|
|
@@ -65,6 +65,18 @@ public.orders postgres:postgres/public.orders -> clickhouse:default/orders
|
|
|
65
65
|
| ClickHouse | `clickhouse-connect` | |
|
|
66
66
|
| Snowflake | `snowflake-connector-python` | Optional extra: `rowproof[snowflake]` |
|
|
67
67
|
|
|
68
|
+
### Snowflake: known limitations
|
|
69
|
+
|
|
70
|
+
Snowflake support works, but it is the least battle-tested of the three engines. It has been tested against a single trial account with small tables; Postgres and ClickHouse have had far more. Know these before you rely on it:
|
|
71
|
+
|
|
72
|
+
- **Login is username and password only.** The connection string has no place for a warehouse, role, SSO, MFA, or key-pair login. Accounts that require SSO or key-pair authentication won't work yet, and queries run on the user's *default* warehouse (a user with none set will fail).
|
|
73
|
+
- **Column names are case-sensitive.** Snowflake stores unquoted names in upper case, so `--key id` won't match a column stored as `ID`. Pass `--key ID`, or create the table with quoted lower-case names. The error message tells you when this is the problem.
|
|
74
|
+
- **Primary keys are often not declared** on Snowflake tables (Snowflake doesn't enforce them). If auto-detection finds none, pass `--key`.
|
|
75
|
+
- **JSON columns are compared as plain text.** `VARIANT`, `ARRAY` and `OBJECT` use rule UNK-1: two JSON values that mean the same thing but are formatted differently will show as different, and a JSON `null` is not distinguished from an SQL `NULL`.
|
|
76
|
+
- **Special characters in passwords must be percent-encoded** in the connection string (`@` becomes `%40`).
|
|
77
|
+
- **It uses your warehouse's credits.** Every query runs on your Snowflake warehouse. Start with a small table or `--sample`.
|
|
78
|
+
- **No Snowflake benchmark yet.** The 100M-row benchmark below is Postgres to ClickHouse only.
|
|
79
|
+
|
|
68
80
|
## Supported types (canonical comparison rules)
|
|
69
81
|
|
|
70
82
|
Every cross-engine type difference is governed by one of these published rules, and every difference rowproof reports cites the rule ID that applied — never a vague "values differ."
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
__version__ = "0.1.2"
|
|
@@ -135,23 +135,26 @@ def _parse_column_map(value: str | None) -> dict | None:
|
|
|
135
135
|
return result or None
|
|
136
136
|
|
|
137
137
|
|
|
138
|
+
def _hook_sql_log(connector, sql_log: list[str] | None) -> None:
|
|
139
|
+
"""spec §8.2: JSON output carries every generated SQL statement too —
|
|
140
|
+
reuse the exact same on_query hook --verbose already installs
|
|
141
|
+
(chaining onto it, so --verbose logging to stderr keeps working
|
|
142
|
+
unchanged) rather than a second SQL-capturing mechanism. `list.append`
|
|
143
|
+
is atomic in CPython, so pooled worker threads can share one log.
|
|
144
|
+
"""
|
|
145
|
+
if sql_log is None or not hasattr(connector, "on_query"):
|
|
146
|
+
return
|
|
147
|
+
previous = connector.on_query
|
|
148
|
+
connector.on_query = lambda sql, _prev=previous: (_prev(sql) if _prev else None, sql_log.append(sql))
|
|
149
|
+
|
|
150
|
+
|
|
138
151
|
def _connect_pair(args, sql_log: list[str] | None = None):
|
|
139
152
|
src_spec = parse_source_spec(args.source)
|
|
140
153
|
tgt_spec = parse_source_spec(args.target)
|
|
141
154
|
source = _make_connector(src_spec.engine, args.verbose)
|
|
142
155
|
target = _make_connector(tgt_spec.engine, args.verbose)
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
if hasattr(connector, "on_query"):
|
|
146
|
-
previous = connector.on_query
|
|
147
|
-
# spec §8.2: JSON output carries every generated SQL
|
|
148
|
-
# statement too — reuse the exact same on_query hook
|
|
149
|
-
# --verbose already installs (chaining onto it, so
|
|
150
|
-
# --verbose logging to stderr keeps working unchanged)
|
|
151
|
-
# rather than a second SQL-capturing mechanism.
|
|
152
|
-
connector.on_query = (
|
|
153
|
-
lambda sql, _prev=previous: (_prev(sql) if _prev else None, sql_log.append(sql))
|
|
154
|
-
)
|
|
156
|
+
_hook_sql_log(source, sql_log)
|
|
157
|
+
_hook_sql_log(target, sql_log)
|
|
155
158
|
_connect_with_retry(source, src_spec.connect_dsn)
|
|
156
159
|
_connect_with_retry(target, tgt_spec.connect_dsn)
|
|
157
160
|
|
|
@@ -168,6 +171,12 @@ def _connect_pair(args, sql_log: list[str] | None = None):
|
|
|
168
171
|
if threads > 1:
|
|
169
172
|
source_pool = [_make_connector(src_spec.engine, args.verbose) for _ in range(threads)]
|
|
170
173
|
target_pool = [_make_connector(tgt_spec.engine, args.verbose) for _ in range(threads)]
|
|
174
|
+
# The pooled connectors run the per-segment hash queries -- the
|
|
175
|
+
# actual proof of a match -- so they must feed the same SQL log as
|
|
176
|
+
# the main connectors, or the JSON/HTML "Reproduce" output would
|
|
177
|
+
# silently omit exactly the queries a reviewer wants to re-run.
|
|
178
|
+
for connector in (*source_pool, *target_pool):
|
|
179
|
+
_hook_sql_log(connector, sql_log)
|
|
171
180
|
for connector in source_pool:
|
|
172
181
|
_connect_with_retry(connector, src_spec.connect_dsn)
|
|
173
182
|
for connector in target_pool:
|
|
@@ -20,6 +20,32 @@ class NoPrimaryKeyError(RowProofError):
|
|
|
20
20
|
self.table_name = table_name
|
|
21
21
|
|
|
22
22
|
|
|
23
|
+
class TableNotFoundError(RowProofError):
|
|
24
|
+
def __init__(self, table_name: str):
|
|
25
|
+
super().__init__(
|
|
26
|
+
f"Table '{table_name}' not found, or it has no visible columns "
|
|
27
|
+
"(check the name, the schema, and that this login can read it)."
|
|
28
|
+
)
|
|
29
|
+
self.table_name = table_name
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
class KeyColumnNotFoundError(RowProofError):
|
|
33
|
+
def __init__(self, table_name: str, column: str, available: list[str]):
|
|
34
|
+
hint = ""
|
|
35
|
+
by_lower = {c.lower(): c for c in available}
|
|
36
|
+
if column.lower() in by_lower and by_lower[column.lower()] != column:
|
|
37
|
+
hint = (
|
|
38
|
+
f" Did you mean '{by_lower[column.lower()]}'? Column names are "
|
|
39
|
+
"case-sensitive here (Snowflake stores unquoted names in upper case)."
|
|
40
|
+
)
|
|
41
|
+
super().__init__(
|
|
42
|
+
f"Key column '{column}' not found in '{table_name}'.{hint} "
|
|
43
|
+
f"Available columns: {', '.join(available)}."
|
|
44
|
+
)
|
|
45
|
+
self.table_name = table_name
|
|
46
|
+
self.column = column
|
|
47
|
+
|
|
48
|
+
|
|
23
49
|
class NonUniqueKeyError(RowProofError):
|
|
24
50
|
def __init__(self, table_name: str, key_columns: list[str], duplicate_example: tuple):
|
|
25
51
|
example = ", ".join(f"{c}={v!r}" for c, v in zip(key_columns, duplicate_example))
|
|
@@ -19,7 +19,12 @@ from dataclasses import dataclass
|
|
|
19
19
|
|
|
20
20
|
from rowproof.connectors.base import Connector
|
|
21
21
|
from rowproof.core.column_matching import match_columns
|
|
22
|
-
from rowproof.core.errors import
|
|
22
|
+
from rowproof.core.errors import (
|
|
23
|
+
KeyColumnNotFoundError,
|
|
24
|
+
NonUniqueKeyError,
|
|
25
|
+
NoPrimaryKeyError,
|
|
26
|
+
TableNotFoundError,
|
|
27
|
+
)
|
|
23
28
|
from rowproof.core.models import (
|
|
24
29
|
Algorithm,
|
|
25
30
|
Column,
|
|
@@ -105,7 +110,12 @@ def _build_plan(
|
|
|
105
110
|
where: str | None = None,
|
|
106
111
|
) -> _Plan:
|
|
107
112
|
cols_by_name = _resolve_columns(connector, table)
|
|
113
|
+
if not cols_by_name:
|
|
114
|
+
raise TableNotFoundError(table.qualified_name)
|
|
108
115
|
resolved_key = _resolve_key_columns(connector, table, key_columns)
|
|
116
|
+
for key_col in resolved_key:
|
|
117
|
+
if key_col not in cols_by_name:
|
|
118
|
+
raise KeyColumnNotFoundError(table.qualified_name, key_col, list(cols_by_name))
|
|
109
119
|
|
|
110
120
|
if columns:
|
|
111
121
|
selected = list(columns)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: rowproof
|
|
3
|
-
Version: 0.1.
|
|
3
|
+
Version: 0.1.2
|
|
4
4
|
Summary: Prove a database migration or replication didn't lose, add or change a single row.
|
|
5
5
|
License: Apache-2.0
|
|
6
6
|
Requires-Python: >=3.11
|
|
@@ -87,6 +87,18 @@ public.orders postgres:postgres/public.orders -> clickhouse:default/orders
|
|
|
87
87
|
| ClickHouse | `clickhouse-connect` | |
|
|
88
88
|
| Snowflake | `snowflake-connector-python` | Optional extra: `rowproof[snowflake]` |
|
|
89
89
|
|
|
90
|
+
### Snowflake: known limitations
|
|
91
|
+
|
|
92
|
+
Snowflake support works, but it is the least battle-tested of the three engines. It has been tested against a single trial account with small tables; Postgres and ClickHouse have had far more. Know these before you rely on it:
|
|
93
|
+
|
|
94
|
+
- **Login is username and password only.** The connection string has no place for a warehouse, role, SSO, MFA, or key-pair login. Accounts that require SSO or key-pair authentication won't work yet, and queries run on the user's *default* warehouse (a user with none set will fail).
|
|
95
|
+
- **Column names are case-sensitive.** Snowflake stores unquoted names in upper case, so `--key id` won't match a column stored as `ID`. Pass `--key ID`, or create the table with quoted lower-case names. The error message tells you when this is the problem.
|
|
96
|
+
- **Primary keys are often not declared** on Snowflake tables (Snowflake doesn't enforce them). If auto-detection finds none, pass `--key`.
|
|
97
|
+
- **JSON columns are compared as plain text.** `VARIANT`, `ARRAY` and `OBJECT` use rule UNK-1: two JSON values that mean the same thing but are formatted differently will show as different, and a JSON `null` is not distinguished from an SQL `NULL`.
|
|
98
|
+
- **Special characters in passwords must be percent-encoded** in the connection string (`@` becomes `%40`).
|
|
99
|
+
- **It uses your warehouse's credits.** Every query runs on your Snowflake warehouse. Start with a small table or `--sample`.
|
|
100
|
+
- **No Snowflake benchmark yet.** The 100M-row benchmark below is Postgres to ClickHouse only.
|
|
101
|
+
|
|
90
102
|
## Supported types (canonical comparison rules)
|
|
91
103
|
|
|
92
104
|
Every cross-engine type difference is governed by one of these published rules, and every difference rowproof reports cites the rule ID that applied — never a vague "values differ."
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
__version__ = "0.1.0"
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|