compara 0.5.3a1__pha.1-py3-none-win_amd64.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
Binary file
|
|
@@ -0,0 +1,184 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: compara
|
|
3
|
+
Version: 0.5.3-alpha.1
|
|
4
|
+
Summary: Compare two tables (CSV, NDJSON, Parquet) on a composite key: counts, per-column statistics and a self-contained HTML report.
|
|
5
|
+
Home-page: https://github.com/andrey-usa/compara
|
|
6
|
+
Project-URL: Source, https://github.com/andrey-usa/compara
|
|
7
|
+
Project-URL: Changelog, https://github.com/andrey-usa/compara/blob/HEAD/CHANGELOG.md
|
|
8
|
+
License: MIT
|
|
9
|
+
Keywords: csv,parquet,diff,compare,data-quality
|
|
10
|
+
Classifier: Programming Language :: Rust
|
|
11
|
+
Classifier: Environment :: Console
|
|
12
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
13
|
+
Requires-Python: >=3.8
|
|
14
|
+
Description-Content-Type: text/markdown
|
|
15
|
+
|
|
16
|
+
# compara
|
|
17
|
+
|
|
18
|
+
[](https://opensource.org/licenses/MIT)
|
|
19
|
+
[](https://github.com/andrey-usa/compara/actions/workflows/ci.yml)
|
|
20
|
+
[](https://crates.io/crates/compara)
|
|
21
|
+
[](https://pypi.org/project/compara/)
|
|
22
|
+
[](https://www.npmjs.com/package/compara)
|
|
23
|
+
|
|
24
|
+
Extremely fast table comparison: two tables on a composite key, with counts,
|
|
25
|
+
per-column statistics and a self-contained HTML report. Key columns, compared
|
|
26
|
+
columns and normalisation rules are parameters, so one tool serves every
|
|
27
|
+
recurring comparison: a nightly export against yesterday's, a CSV against the
|
|
28
|
+
Parquet a warehouse emits, a migration's source against its target.
|
|
29
|
+
|
|
30
|
+
> Faster software for a better planet — less energy, less compute, more time for life.
|
|
31
|
+
|
|
32
|
+
- Reads **CSV (and TSV), newline-delimited JSON and Parquet**, and the two sides
|
|
33
|
+
need not be in the same format.
|
|
34
|
+
- Ten million rows × 20 columns in about a second on four cores
|
|
35
|
+
([numbers below](#speed)).
|
|
36
|
+
- One static binary; the report is a single HTML file with nothing to host.
|
|
37
|
+
- Exit codes made for pipelines: `0` identical, `1` differences, `2` error,
|
|
38
|
+
`3` duplicate keys (with `--fail-on-dups`).
|
|
39
|
+
|
|
40
|
+
## Install
|
|
41
|
+
|
|
42
|
+
Prebuilt binaries for Linux (x86_64, aarch64), macOS (arm64, x86_64) and
|
|
43
|
+
Windows (x64) are on the [releases page](https://github.com/andrey-usa/compara/releases),
|
|
44
|
+
and the same binaries through package managers:
|
|
45
|
+
|
|
46
|
+
| | |
|
|
47
|
+
|---|---|
|
|
48
|
+
| Cargo | `cargo install --locked compara` |
|
|
49
|
+
| npm | `npm install -g compara` (or `npx compara …`) |
|
|
50
|
+
| pip / uv | `pip install compara` · `uv tool install compara` |
|
|
51
|
+
| Homebrew | `brew install andrey-usa/tap/compara` |
|
|
52
|
+
| Scoop | `scoop bucket add andrey-usa https://github.com/andrey-usa/scoop-bucket` then `scoop install compara` |
|
|
53
|
+
|
|
54
|
+
## Use
|
|
55
|
+
|
|
56
|
+
```sh
|
|
57
|
+
compara compare a.csv b.csv -k account_id,txn_id -i updated_at
|
|
58
|
+
compara compare a.csv b.csv -k account_id,txn_id -i updated_at -o report.html
|
|
59
|
+
compara compare a.csv b.csv -k account_id,txn_id --trim --tolerance 0.005
|
|
60
|
+
compara compare export.csv warehouse.parquet -k id
|
|
61
|
+
compara compare a.csv b.csv --profile orders --json summary.json --export-dir out/
|
|
62
|
+
compara columns data.parquet # column names (reads only the footer)
|
|
63
|
+
compara head data.ndjson -n 5 # first rows, aligned
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
`compare` prints the counts and writes nothing else unless asked. `-o PATH`
|
|
67
|
+
writes the HTML report: counts, per-column statistics, and the changed,
|
|
68
|
+
added, removed and duplicate rows, each section capped at `--max-rows`
|
|
69
|
+
(counts are always exact). [`examples/report.html`](examples/report.html) is
|
|
70
|
+
one, made from the two files next to it:
|
|
71
|
+
|
|
72
|
+
```sh
|
|
73
|
+
compara compare examples/orders_2026-08.csv examples/orders_2026-09.csv -k order_id,line_no -i updated_at -o examples/report.html
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
| Option | Effect |
|
|
77
|
+
|---|---|
|
|
78
|
+
| `-k`, `--key` | composite key, required (or from a profile) |
|
|
79
|
+
| `-c`, `--compare` | columns to diff; default every column present in both files except the key |
|
|
80
|
+
| `-i`, `--ignore` | columns to skip (timestamps, run ids) |
|
|
81
|
+
| `--trim`, `--ignore-case`, `--empty-is-null` | normalisation before comparing (applies to key and values) |
|
|
82
|
+
| `--tolerance` | absolute numeric tolerance where both sides parse as numbers |
|
|
83
|
+
| `--delimiter`, `--encoding` | override auto-detection (CSV only) |
|
|
84
|
+
| `--threads` | how wide the engines run; the default is every core |
|
|
85
|
+
| `--memory` | memory to plan for (e.g. `8G`); decides when large text files are joined in one pass instead of two, and how many columns of two Parquet files are read at once (a Parquet pair that needs more than a stated budget is refused) |
|
|
86
|
+
| `--max-rows` | rows embedded per report section (default 50 000) |
|
|
87
|
+
| `-o`, `--out` | write the HTML report here; there is no report without it |
|
|
88
|
+
| `--export-dir` | full, uncapped changed/added/removed CSVs |
|
|
89
|
+
| `--json` | write a JSON summary (counts and column stats; schema in `docs/contract.schema.json`) |
|
|
90
|
+
| `--summary` | print the counts and refuse any flag that writes a file (the default without `-o`, `--json` or `--export-dir`) |
|
|
91
|
+
| `--engine` | `auto` (default), `turbo`, `sortmerge` or `native`; `sortmerge` and `native` read CSV only |
|
|
92
|
+
| `--no-compress` | plain JSON payload in the report, for pre-2023 browsers (with `-o`) |
|
|
93
|
+
| `--fail-on-dups` | exit 3 when either file has duplicate keys |
|
|
94
|
+
|
|
95
|
+
Duplicate keys are counted and listed per file; the first occurrence of each
|
|
96
|
+
key takes part in the join.
|
|
97
|
+
|
|
98
|
+
**Profiles.** A `compara.toml` in the working directory (or
|
|
99
|
+
`~/.config/compara/compara.toml`, or `--config PATH`) names recurring
|
|
100
|
+
comparisons: key, compared and ignored columns, normalisation, engine. See
|
|
101
|
+
[`compara.example.toml`](compara.example.toml). The command line overrides the
|
|
102
|
+
profile.
|
|
103
|
+
|
|
104
|
+
## Speed
|
|
105
|
+
|
|
106
|
+
Ten million rows × 20 columns, keyed on `(account_id, txn_id)`, `--ignore
|
|
107
|
+
updated_at`, one GitHub Actions runner (AMD EPYC 7763, 4 vCPU / 16 GB), best of
|
|
108
|
+
five. Each cell is wall time · CPU time · memory above the mapped input.
|
|
109
|
+
|
|
110
|
+
| Format | Input a side | compara |
|
|
111
|
+
|---|---:|---|
|
|
112
|
+
| CSV | 3,509 MB | 1.20 s · 4.0 s · 591 MB |
|
|
113
|
+
| ndjson | 8,487 MB | 2.79 s · 10.4 s · 592 MB |
|
|
114
|
+
| Parquet | 2,074 MB | 1.41 s · 4.7 s · 1,342 MB |
|
|
115
|
+
|
|
116
|
+
Contender benchmarks rerun weekly
|
|
117
|
+
([`bench-contenders.yml`](.github/workflows/bench-contenders.yml)) against the
|
|
118
|
+
latest DuckDB and pandas.
|
|
119
|
+
|
|
120
|
+
## Engines
|
|
121
|
+
|
|
122
|
+
Three backends, one result contract: every engine returns identical counts and
|
|
123
|
+
column statistics for the same input, and the test suite asserts it cell for
|
|
124
|
+
cell. `--engine auto` takes the first one that can load the input, in this
|
|
125
|
+
order.
|
|
126
|
+
|
|
127
|
+
| Engine | Implementation | Memory model | Use it for |
|
|
128
|
+
|---|---|---|---|
|
|
129
|
+
| `turbo` | mapped file, byte-level index; CSV, JSON and Parquet | off-heap bytes, index in memory | the default, and the fastest |
|
|
130
|
+
| `sortmerge` | external sort, merge join | bounded memory, spills to disk | files past what memory can index |
|
|
131
|
+
| `native` | over the `csv` crate | in-memory, row-oriented | the dependency-light baseline |
|
|
132
|
+
|
|
133
|
+
All three read values as text, with no type inference, so `1.0` and `1` stay
|
|
134
|
+
different unless a tolerance is set, and all three treat an empty field as
|
|
135
|
+
absent whether or not it is quoted. A Parquet column is the exception, on
|
|
136
|
+
either side: it stores typed values, so a column it holds as numbers
|
|
137
|
+
compares by value with the other file's text (`31.40` against the DOUBLE
|
|
138
|
+
31.4 is no change), and a date, time or timestamp column by instant (`T` or
|
|
139
|
+
a space, a zero fraction or none, `Z` or nothing).
|
|
140
|
+
|
|
141
|
+
### Input formats
|
|
142
|
+
|
|
143
|
+
`turbo` decides what a file is by its content rather than its name: a Parquet
|
|
144
|
+
magic number, a leading `{`, or a CSV header.
|
|
145
|
+
|
|
146
|
+
| Format | How it is read |
|
|
147
|
+
|---|---|
|
|
148
|
+
| CSV | mapped, scanned eight bytes at a time; a field is an offset and a length; a UTF-8 byte order mark is skipped; the separator (comma, semicolon, tab or pipe) is read from the header line unless `--delimiter` names it, so a TSV file is read the same way and gives the same answer and export as the same rows in CSV |
|
|
149
|
+
| newline-delimited JSON | the same, addressed by key rather than by column number; `\uXXXX` is decoded, so an escaped and a literal character compare equal |
|
|
150
|
+
| Parquet | pages decoded into an arena; dictionary-encoded columns point at the dictionary entry, so a repeated value costs eight bytes a row |
|
|
151
|
+
|
|
152
|
+
The Parquet reader is written here rather than taken from a crate, to keep that
|
|
153
|
+
representation instead of converting to Arrow arrays. It reads PLAIN,
|
|
154
|
+
dictionary (`PLAIN_DICTIONARY`, `RLE_DICTIONARY`), RLE booleans and the
|
|
155
|
+
version-2 delta encodings; uncompressed, snappy, gzip, zstd and LZ4-raw pages;
|
|
156
|
+
data pages v1 and v2; definition levels for optional columns. It refuses, by
|
|
157
|
+
name, nested or repeated schemas, `BYTE_STREAM_SPLIT`, LZO and Brotli.
|
|
158
|
+
|
|
159
|
+
A typed Parquet value is rendered to text by a fixed rule: byte arrays as their
|
|
160
|
+
bytes, booleans `true`/`false`, integers and decimals in full, floats in the
|
|
161
|
+
shortest round-trip form, dates `YYYY-MM-DD`, timestamps ISO 8601.
|
|
162
|
+
`tests/fixtures/formats/typed.csv` is that rule written down.
|
|
163
|
+
|
|
164
|
+
Two Parquet files take a columnar path instead, which never builds a row: it
|
|
165
|
+
joins on the key columns, then compares one column at a time, string columns
|
|
166
|
+
by dictionary id or bytes and booleans, integers, floats, dates, times,
|
|
167
|
+
timestamps and decimals of up to eight bytes as one word per value. A cell is
|
|
168
|
+
rendered by the rule above only when the report shows it or `--export-dir`
|
|
169
|
+
lists it. Added and removed rows are known from the join, so their cells are
|
|
170
|
+
written out during that pass; which rows changed is known only after the last
|
|
171
|
+
column, so an export reads the compared columns once more and writes out just
|
|
172
|
+
those rows. It costs that second pass and the text it writes, rather than
|
|
173
|
+
holding every column of both files at once. A column the two files store as
|
|
174
|
+
different types, or with an encoding or page outside PLAIN, dictionary and v1
|
|
175
|
+
pages (or LZO or Brotli compression), goes the row-by-row way instead, with
|
|
176
|
+
the same answer.
|
|
177
|
+
|
|
178
|
+
## Contributing
|
|
179
|
+
|
|
180
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
181
|
+
|
|
182
|
+
## License
|
|
183
|
+
|
|
184
|
+
[MIT](LICENSE)
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
compara-0.5.3_alpha.1.data/scripts/compara.exe,sha256=ewVZFRKd3c5CfsxpFY_uYUwrVPDgrWCYebSL2msS66s,2211840
|
|
2
|
+
compara-0.5.3_alpha.1.dist-info/METADATA,sha256=mSHg4zCof_1BRqZKTRfWpkT6DMW29h2G7eHhBUmKlFw,9851
|
|
3
|
+
compara-0.5.3_alpha.1.dist-info/WHEEL,sha256=g1aQ65onkFiWyh7QjiXnWXLnCV9xiZTddNDcljDxYOM,88
|
|
4
|
+
compara-0.5.3_alpha.1.dist-info/licenses/LICENSE,sha256=RSeVx-hjXtE643Nsk8gnOcomD9Wdf4PAXy7J7fSi4pQ,1067
|
|
5
|
+
compara-0.5.3_alpha.1.dist-info/RECORD,,
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 andrey-usa
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|