parq-cli 0.2.2__tar.gz → 0.2.4__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- parq_cli-0.2.4/CHANGELOG.md +53 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/CLAUDE.md +2 -1
- {parq_cli-0.2.2 → parq_cli-0.2.4}/PKG-INFO +15 -12
- {parq_cli-0.2.2 → parq_cli-0.2.4}/README.md +14 -11
- {parq_cli-0.2.2 → parq_cli-0.2.4}/README_CN.md +14 -11
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/cli.py +157 -117
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/formats/_chunk_writers.py +133 -52
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/formats/_common.py +83 -16
- parq_cli-0.2.4/parq/formats/_csv.py +182 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/formats/_parquet.py +46 -10
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/formats/_xlsx.py +11 -3
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/plain_output.py +13 -1
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/reader.py +374 -119
- {parq_cli-0.2.2 → parq_cli-0.2.4}/pyproject.toml +1 -1
- {parq_cli-0.2.2 → parq_cli-0.2.4}/scripts/check_version.py +0 -3
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/test_cli.py +169 -0
- parq_cli-0.2.4/tests/test_output_not_input.py +63 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/test_plain_output.py +18 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/test_reader.py +583 -1
- {parq_cli-0.2.2 → parq_cli-0.2.4}/uv.lock +1 -1
- parq_cli-0.2.2/CHANGELOG.md +0 -26
- parq_cli-0.2.2/parq/formats/_csv.py +0 -37
- parq_cli-0.2.2/tests/test_output_not_input.py +0 -26
- {parq_cli-0.2.2 → parq_cli-0.2.4}/.claude/settings.json +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/.github/workflows/publish.yml +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/.github/workflows/test.yml +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/.gitignore +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/.python-version +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/.tmp-bench/csv-large.csv +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/.tmp-bench/csv-small.csv +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/.tmp-bench/parquet-large.parquet +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/.tmp-bench/parquet-small.parquet +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/AGENTS.md +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/LICENSE +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/data/test.parquet +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/docs/QUICKSTART.md +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/docs/RELEASE.md +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/examples/README.md +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/examples/create_sample_data.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/__init__.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/__main__.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/formats/__init__.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/parq/output.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/scripts/README.md +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/scripts/bump_version.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/scripts/performance_benchmark.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/scripts/publish.sh +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/__init__.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/conftest.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/test_main_entry.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/test_output.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/test_package_meta.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/test_performance.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/test_release_scripts.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/test_workflows.py +0 -0
- {parq_cli-0.2.2 → parq_cli-0.2.4}/tests/test_xlsx_defused.py +0 -0
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## Unreleased
|
|
4
|
+
|
|
5
|
+
## 0.2.4
|
|
6
|
+
|
|
7
|
+
- CSV and TSV reads accept `--encoding` (default `utf8`). The flag works before a subcommand and on it, and the subcommand value wins. An unknown name fails with `Unknown encoding`. Writes stay UTF-8.
|
|
8
|
+
- `--encoding` is only validated for CSV and TSV input. A script that passes it globally no longer fails on a parquet or xlsx file.
|
|
9
|
+
- A CSV or TSV column that cannot be decoded with `--encoding` fails the read, including a header whose name is not valid in that encoding. A bad byte after the first block names that column. UTF-16 read as UTF-8 reports the NUL bytes instead of a column-count error. PyArrow would otherwise return an undecoded data column as raw bytes.
|
|
10
|
+
- `--encoding punycode` is rejected. PyArrow passes a buffer to that decoder and it raises AttributeError.
|
|
11
|
+
- `diff` reads `.xlsx` files. It parses the selected sheet in full so a value past the first 1000 rows can widen a column before types are compared. Rows are kept only up to 100,000 per side; a larger sheet is parsed again in batches instead of held in memory.
|
|
12
|
+
- A type difference stays in `schema_type_mismatches`, including with `--summary-only`. The text produced by `str(value)` when a number, date, or boolean is widened is not a cell edit, in any input format. Any other text is. Keys stay exact, so `1` and `"1"` do not match.
|
|
13
|
+
- `diff` can sample rows whose keys are null next to numbers or strings. A Decimal NaN key matches another Decimal NaN. A list or struct column used as a key is rejected with the column name.
|
|
14
|
+
- Parquet `head` stops after the requested rows, including inside one large row group. A preview no longer reads that whole group.
|
|
15
|
+
- Streamed `tail` keeps the trailing Arrow batches and slices them. CSV `tail` no longer converts every cell to Python.
|
|
16
|
+
|
|
17
|
+
## 0.2.3
|
|
18
|
+
|
|
19
|
+
- `merge --force` and `convert --force` replace an existing output file only after the new file is fully written. A schema mismatch or a write error leaves the previous file in place, and the temporary file is removed, including on Windows.
|
|
20
|
+
- `diff` treats NaN as equal to NaN, including when NaN is the key.
|
|
21
|
+
- `diff` and `merge` close Parquet file handles when they finish.
|
|
22
|
+
- JSON table output summarizes binary values, matching rich and plain output.
|
|
23
|
+
- `split --force` replaces existing part files only after every part is written. A later write error leaves the previous parts in place.
|
|
24
|
+
- Column stats ignore NaN when computing min, max, and mean. Nulls are still counted separately, and NaN stays in the non-null count.
|
|
25
|
+
- An empty, multi-character, or non-ASCII `--delimiter` is rejected with `delimiter must be a single ASCII character` before CSV parsing or writing.
|
|
26
|
+
- `stats --limit`, `stats --top-n`, and `diff --limit` reject negative values instead of slicing from the end. Zero still means an empty result.
|
|
27
|
+
- Selecting a duplicated column name reports that the name cannot be selected, instead of claiming the column does not exist. A unique column in the same file can still be selected.
|
|
28
|
+
- `diff` compares repeated column names by position. A diff that would compare those names by default asks for `--columns` instead of claiming the column does not exist.
|
|
29
|
+
|
|
30
|
+
## 0.2.2
|
|
31
|
+
|
|
32
|
+
- `merge --force` and `convert` now refuse an output path that is one of the inputs. Previously `--force` deleted that input before reading it.
|
|
33
|
+
- XLSX workbook XML is parsed with `defusedxml` (added to the `xlsx` extra).
|
|
34
|
+
- Table output escapes cell markup and summarizes binary values (#4).
|
|
35
|
+
- Errors include the file path (#5).
|
|
36
|
+
- Dependency security updates: certifi, urllib3, idna, pygments, cryptography, pyarrow.
|
|
37
|
+
|
|
38
|
+
## 0.2.1
|
|
39
|
+
|
|
40
|
+
- `diff` and `merge` honor the global `--delimiter` and `--sheet` options.
|
|
41
|
+
- `.tsv` is a supported output format for `convert`, `split`, and `merge`.
|
|
42
|
+
- XLSX `schema` still samples the first 1000 rows. Full reads widen a column when a later row does not fit that sample.
|
|
43
|
+
- String cardinality and top values stay exact past 10,000 distinct values. Nulls are not counted as empty strings.
|
|
44
|
+
- `diff` spills inputs larger than 100,000 rows per side into temporary files.
|
|
45
|
+
- `split --record-count` progress stays indeterminate while the total is still unknown.
|
|
46
|
+
- PyPI uploads run from a `v*` tag and use the repository `PYPI_API_TOKEN`.
|
|
47
|
+
- XLSX reads use `python-calamine`. A full read parses the workbook once and reuses those rows.
|
|
48
|
+
- CSV `convert` no longer counts every row before writing when a progress bar is shown.
|
|
49
|
+
- `--output`, `--delimiter`, and `--sheet` work before a subcommand and on the subcommand. The value on the subcommand wins when both are set.
|
|
50
|
+
|
|
51
|
+
## 0.2.0
|
|
52
|
+
|
|
53
|
+
- Inspect, split, convert, diff, and merge Parquet, CSV, TSV, and XLSX files.
|
|
@@ -20,5 +20,6 @@ pytest tests/test_performance.py -m performance -q -s
|
|
|
20
20
|
## Gotchas
|
|
21
21
|
|
|
22
22
|
- `parq schema` on XLSX samples the first 1000 rows. Full reads parse the sheet once with python-calamine and widen a column when a later value does not fit. Writing XLSX still uses openpyxl.
|
|
23
|
-
- `
|
|
23
|
+
- Parquet `head` stops inside a large row group. `tail` still reads the whole group when the last rows share one group. CSV type inference still uses PyArrow's 1 MiB block.
|
|
24
|
+
- `diff` and `merge` use the global `--delimiter`, `--sheet`, and `--encoding`. `diff` reads XLSX after a full sheet parse and keeps at most 100,000 rows per side; a larger sheet is parsed again in batches. Both sides spill to temporary files past that limit. CSV/TSV `--encoding` applies to reads; a column or header that does not decode fails the read. It is not validated for parquet or xlsx input. Writes stay UTF-8.
|
|
24
25
|
- `.tsv` is valid for both input and output. Output uses a tab delimiter.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.5
|
|
2
2
|
Name: parq-cli
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.4
|
|
4
4
|
Summary: A command-line tool for inspecting tabular files: Parquet, CSV, TSV, and XLSX
|
|
5
5
|
Project-URL: Homepage, https://github.com/Tendo33/parq-cli
|
|
6
6
|
Project-URL: Repository, https://github.com/Tendo33/parq-cli
|
|
@@ -131,10 +131,10 @@ parq merge chunks/*.parquet merged.parquet --force # overwrite if exists
|
|
|
131
131
|
| `split` | yes | yes | yes | yes |
|
|
132
132
|
| `stats` | yes | yes | yes | yes |
|
|
133
133
|
| `convert` | yes | yes | yes | yes |
|
|
134
|
-
| `diff` | yes | yes | yes |
|
|
134
|
+
| `diff` | yes | yes | yes | yes |
|
|
135
135
|
| `merge` | yes | yes | yes | yes |
|
|
136
136
|
|
|
137
|
-
`XLSX` reads use `python-calamine` and writes use `openpyxl`; both install with `parq-cli[xlsx]`. Formula cells are read from Excel's cached value, so a formula Excel has not calculated comes back empty. TSV files are auto-detected by the `.tsv` extension; a custom delimiter can be supplied with `--delimiter`.
|
|
137
|
+
`XLSX` reads use `python-calamine` and writes use `openpyxl`; both install with `parq-cli[xlsx]`. Formula cells are read from Excel's cached value, so a formula Excel has not calculated comes back empty. TSV files are auto-detected by the `.tsv` extension; a custom delimiter can be supplied with `--delimiter`. CSV and TSV reads default to UTF-8. Pass `--encoding gbk` (or another codec name) for a different input encoding. A column that does not decode with that encoding fails the read, including a header whose name is not valid in that encoding. UTF-16 read as UTF-8 fails the same way. Writes stay UTF-8.
|
|
138
138
|
|
|
139
139
|
## Command Reference
|
|
140
140
|
|
|
@@ -258,12 +258,14 @@ Compares two datasets by key and reports:
|
|
|
258
258
|
Notes:
|
|
259
259
|
|
|
260
260
|
- `--key` is required
|
|
261
|
-
-
|
|
262
|
-
-
|
|
263
|
-
- XLSX
|
|
261
|
+
- inputs can be Parquet, CSV, TSV, or XLSX
|
|
262
|
+
- `--delimiter`, `--encoding`, and `--sheet` apply to both inputs
|
|
263
|
+
- XLSX diff parses the selected sheet in full, so a later value can widen a column before types are compared
|
|
264
|
+
- a type difference is always reported. The text from `str(value)` is not a cell edit; any other text is. Keys stay exact, so `1` and `"1"` do not match
|
|
264
265
|
- duplicate keys on either side are treated as an error
|
|
266
|
+
- null keys are included in the sample. A list or struct column cannot be a key
|
|
265
267
|
- inputs larger than 100,000 rows on a side spill into temporary files instead of holding every key in memory
|
|
266
|
-
- `--summary-only` keeps the counts and omits sample payloads
|
|
268
|
+
- `--summary-only` keeps the counts and the type mismatches, and omits sample row payloads
|
|
267
269
|
|
|
268
270
|
### `merge`
|
|
269
271
|
|
|
@@ -280,7 +282,7 @@ Notes:
|
|
|
280
282
|
- schemas must be identical or safely unifiable by Arrow
|
|
281
283
|
- by default, existing output files raise an error; use `--force` / `-F` to overwrite
|
|
282
284
|
- output format is inferred from the output suffix, including `.tsv`
|
|
283
|
-
- the global `--delimiter
|
|
285
|
+
- the global `--delimiter`, `--encoding`, and `--sheet` apply to every input
|
|
284
286
|
- a live progress bar is shown during the merge
|
|
285
287
|
|
|
286
288
|
## Output Modes
|
|
@@ -290,10 +292,11 @@ Global options:
|
|
|
290
292
|
- `--version`, `-v`: show version information. Place this before the subcommand: `parq --version`
|
|
291
293
|
- `--output`, `-o`: select output format (`rich` | `plain` | `json`)
|
|
292
294
|
- `--delimiter`, `-d`: field delimiter for CSV/TSV input (default: `,`); `.tsv` files default to `\t` automatically
|
|
295
|
+
- `--encoding`: character encoding for CSV/TSV input (default: `utf8`). Writes stay UTF-8
|
|
293
296
|
- `--sheet`: XLSX sheet name or 0-based index to read (default: active sheet). A value made only of digits is an index, including a sheet whose name is `1` or `2`
|
|
294
297
|
- `--help`: show command help
|
|
295
298
|
|
|
296
|
-
`--output`, `--delimiter`, and `--
|
|
299
|
+
`--output`, `--delimiter`, `--sheet`, and `--encoding` work before the subcommand and on the subcommand. If you pass the same option in both places, the value on the subcommand wins. On `convert`, `--output` / `-o` still selects the display format; the destination file is the positional `OUTPUT` argument.
|
|
297
300
|
|
|
298
301
|
Available output modes:
|
|
299
302
|
|
|
@@ -315,11 +318,11 @@ On Windows terminals that cannot safely render emoji or extended characters, Ric
|
|
|
315
318
|
|
|
316
319
|
## Large File Notes
|
|
317
320
|
|
|
318
|
-
- Parquet metadata
|
|
319
|
-
- CSV `tail`
|
|
321
|
+
- Parquet metadata and row counts come from the file footer. `head` stops after the requested rows, including when they sit inside one large row group. `tail` reads the row groups that hold the last rows.
|
|
322
|
+
- CSV `tail` scans the file once, keeps the trailing Arrow batches, and slices the last rows.
|
|
320
323
|
- CSV `split --record-count` and CSV `convert` stream in one pass. Their progress bars stay indeterminate because the row count is not known ahead of time.
|
|
321
324
|
- `meta --fast` is the best option when you need quick metadata from large CSV/XLSX inputs.
|
|
322
|
-
- `parq schema` and a short `head` sample the first 1000 XLSX rows. `count`, `tail`, `stats`, `convert`, `split`, and `
|
|
325
|
+
- `parq schema` and a short `head` sample the first 1000 XLSX rows. `count`, `tail`, `stats`, `convert`, `split`, `merge`, and `diff` parse the selected sheet once, widen a column when a later value does not fit that sample, and keep those rows in memory.
|
|
323
326
|
- `split --record-count` keeps the progress bar indeterminate while it writes.
|
|
324
327
|
- `diff` indexes at most 100,000 rows per side in memory, then hash-partitions larger inputs into temporary Parquet files.
|
|
325
328
|
|
|
@@ -94,10 +94,10 @@ parq merge chunks/*.parquet merged.parquet --force # overwrite if exists
|
|
|
94
94
|
| `split` | yes | yes | yes | yes |
|
|
95
95
|
| `stats` | yes | yes | yes | yes |
|
|
96
96
|
| `convert` | yes | yes | yes | yes |
|
|
97
|
-
| `diff` | yes | yes | yes |
|
|
97
|
+
| `diff` | yes | yes | yes | yes |
|
|
98
98
|
| `merge` | yes | yes | yes | yes |
|
|
99
99
|
|
|
100
|
-
`XLSX` reads use `python-calamine` and writes use `openpyxl`; both install with `parq-cli[xlsx]`. Formula cells are read from Excel's cached value, so a formula Excel has not calculated comes back empty. TSV files are auto-detected by the `.tsv` extension; a custom delimiter can be supplied with `--delimiter`.
|
|
100
|
+
`XLSX` reads use `python-calamine` and writes use `openpyxl`; both install with `parq-cli[xlsx]`. Formula cells are read from Excel's cached value, so a formula Excel has not calculated comes back empty. TSV files are auto-detected by the `.tsv` extension; a custom delimiter can be supplied with `--delimiter`. CSV and TSV reads default to UTF-8. Pass `--encoding gbk` (or another codec name) for a different input encoding. A column that does not decode with that encoding fails the read, including a header whose name is not valid in that encoding. UTF-16 read as UTF-8 fails the same way. Writes stay UTF-8.
|
|
101
101
|
|
|
102
102
|
## Command Reference
|
|
103
103
|
|
|
@@ -221,12 +221,14 @@ Compares two datasets by key and reports:
|
|
|
221
221
|
Notes:
|
|
222
222
|
|
|
223
223
|
- `--key` is required
|
|
224
|
-
-
|
|
225
|
-
-
|
|
226
|
-
- XLSX
|
|
224
|
+
- inputs can be Parquet, CSV, TSV, or XLSX
|
|
225
|
+
- `--delimiter`, `--encoding`, and `--sheet` apply to both inputs
|
|
226
|
+
- XLSX diff parses the selected sheet in full, so a later value can widen a column before types are compared
|
|
227
|
+
- a type difference is always reported. The text from `str(value)` is not a cell edit; any other text is. Keys stay exact, so `1` and `"1"` do not match
|
|
227
228
|
- duplicate keys on either side are treated as an error
|
|
229
|
+
- null keys are included in the sample. A list or struct column cannot be a key
|
|
228
230
|
- inputs larger than 100,000 rows on a side spill into temporary files instead of holding every key in memory
|
|
229
|
-
- `--summary-only` keeps the counts and omits sample payloads
|
|
231
|
+
- `--summary-only` keeps the counts and the type mismatches, and omits sample row payloads
|
|
230
232
|
|
|
231
233
|
### `merge`
|
|
232
234
|
|
|
@@ -243,7 +245,7 @@ Notes:
|
|
|
243
245
|
- schemas must be identical or safely unifiable by Arrow
|
|
244
246
|
- by default, existing output files raise an error; use `--force` / `-F` to overwrite
|
|
245
247
|
- output format is inferred from the output suffix, including `.tsv`
|
|
246
|
-
- the global `--delimiter
|
|
248
|
+
- the global `--delimiter`, `--encoding`, and `--sheet` apply to every input
|
|
247
249
|
- a live progress bar is shown during the merge
|
|
248
250
|
|
|
249
251
|
## Output Modes
|
|
@@ -253,10 +255,11 @@ Global options:
|
|
|
253
255
|
- `--version`, `-v`: show version information. Place this before the subcommand: `parq --version`
|
|
254
256
|
- `--output`, `-o`: select output format (`rich` | `plain` | `json`)
|
|
255
257
|
- `--delimiter`, `-d`: field delimiter for CSV/TSV input (default: `,`); `.tsv` files default to `\t` automatically
|
|
258
|
+
- `--encoding`: character encoding for CSV/TSV input (default: `utf8`). Writes stay UTF-8
|
|
256
259
|
- `--sheet`: XLSX sheet name or 0-based index to read (default: active sheet). A value made only of digits is an index, including a sheet whose name is `1` or `2`
|
|
257
260
|
- `--help`: show command help
|
|
258
261
|
|
|
259
|
-
`--output`, `--delimiter`, and `--
|
|
262
|
+
`--output`, `--delimiter`, `--sheet`, and `--encoding` work before the subcommand and on the subcommand. If you pass the same option in both places, the value on the subcommand wins. On `convert`, `--output` / `-o` still selects the display format; the destination file is the positional `OUTPUT` argument.
|
|
260
263
|
|
|
261
264
|
Available output modes:
|
|
262
265
|
|
|
@@ -278,11 +281,11 @@ On Windows terminals that cannot safely render emoji or extended characters, Ric
|
|
|
278
281
|
|
|
279
282
|
## Large File Notes
|
|
280
283
|
|
|
281
|
-
- Parquet metadata
|
|
282
|
-
- CSV `tail`
|
|
284
|
+
- Parquet metadata and row counts come from the file footer. `head` stops after the requested rows, including when they sit inside one large row group. `tail` reads the row groups that hold the last rows.
|
|
285
|
+
- CSV `tail` scans the file once, keeps the trailing Arrow batches, and slices the last rows.
|
|
283
286
|
- CSV `split --record-count` and CSV `convert` stream in one pass. Their progress bars stay indeterminate because the row count is not known ahead of time.
|
|
284
287
|
- `meta --fast` is the best option when you need quick metadata from large CSV/XLSX inputs.
|
|
285
|
-
- `parq schema` and a short `head` sample the first 1000 XLSX rows. `count`, `tail`, `stats`, `convert`, `split`, and `
|
|
288
|
+
- `parq schema` and a short `head` sample the first 1000 XLSX rows. `count`, `tail`, `stats`, `convert`, `split`, `merge`, and `diff` parse the selected sheet once, widen a column when a later value does not fit that sample, and keep those rows in memory.
|
|
286
289
|
- `split --record-count` keeps the progress bar indeterminate while it writes.
|
|
287
290
|
- `diff` indexes at most 100,000 rows per side in memory, then hash-partitions larger inputs into temporary Parquet files.
|
|
288
291
|
|
|
@@ -82,10 +82,10 @@ parq merge part-001.parquet part-002.parquet merged.parquet
|
|
|
82
82
|
| `split` | 支持 | 支持 | 支持 | 支持 |
|
|
83
83
|
| `stats` | 支持 | 支持 | 支持 | 支持 |
|
|
84
84
|
| `convert` | 支持 | 支持 | 支持 | 支持 |
|
|
85
|
-
| `diff` | 支持 | 支持 | 支持 |
|
|
85
|
+
| `diff` | 支持 | 支持 | 支持 | 支持 |
|
|
86
86
|
| `merge` | 支持 | 支持 | 支持 | 支持 |
|
|
87
87
|
|
|
88
|
-
`XLSX` 的读取使用 `python-calamine`,写出仍使用 `openpyxl`,两者都随 `parq-cli[xlsx]` 安装。公式单元格读取的是 Excel 已经算出的缓存值,没有缓存的公式会是空值。`.tsv` 按扩展名自动使用制表符;其它分隔符用 `--delimiter`。`--sheet` 选择 XLSX 工作表,对 `merge` 同样生效。
|
|
88
|
+
`XLSX` 的读取使用 `python-calamine`,写出仍使用 `openpyxl`,两者都随 `parq-cli[xlsx]` 安装。公式单元格读取的是 Excel 已经算出的缓存值,没有缓存的公式会是空值。`.tsv` 按扩展名自动使用制表符;其它分隔符用 `--delimiter`。`--sheet` 选择 XLSX 工作表,对 `diff` 和 `merge` 同样生效。CSV/TSV 默认按 UTF-8 读取,其它编码用 `--encoding`(例如 `gbk`)。某一列无法按该编码解码时,读取会失败,表头对不上该编码时也一样。把 UTF-16 当成 UTF-8 读取同样会失败。写出仍然是 UTF-8。
|
|
89
89
|
|
|
90
90
|
## 命令说明
|
|
91
91
|
|
|
@@ -203,12 +203,14 @@ parq diff LEFT RIGHT --key id --summary-only
|
|
|
203
203
|
说明:
|
|
204
204
|
|
|
205
205
|
- `--key` 必填
|
|
206
|
-
-
|
|
207
|
-
-
|
|
208
|
-
- XLSX
|
|
206
|
+
- 输入可以是 Parquet、CSV、TSV 或 XLSX
|
|
207
|
+
- `--delimiter`、`--encoding` 和 `--sheet` 会同时用于左右两个输入
|
|
208
|
+
- XLSX 对比会完整解析选中的工作表,第 1000 行之后的值也能放宽列类型后再比较
|
|
209
|
+
- 类型不同一定会报告。`str(value)` 得到的文本不算单元格修改,其它文本算。主键仍按原值匹配,`1` 和 `"1"` 对不上
|
|
209
210
|
- 任一侧若出现重复 key,会直接报错
|
|
211
|
+
- 空主键会进入样本。列表或结构体列不能当主键
|
|
210
212
|
- 单侧超过 10 万行时,会把数据按主键散列到临时文件,而不是把全部 key 放进内存
|
|
211
|
-
- `--summary-only`
|
|
213
|
+
- `--summary-only` 保留计数和类型差异,不输出变更行样本
|
|
212
214
|
|
|
213
215
|
### `merge`
|
|
214
216
|
|
|
@@ -224,7 +226,7 @@ parq merge chunks/*.parquet merged.parquet
|
|
|
224
226
|
- schema 必须一致,或能被 Arrow 安全统一
|
|
225
227
|
- 默认遇到已存在的输出文件会报错;`--force` / `-F` 会覆盖
|
|
226
228
|
- 输出格式由输出文件后缀决定,包括 `.tsv`
|
|
227
|
-
- 全局 `--delimiter` 和 `--sheet` 会作用于每一个输入
|
|
229
|
+
- 全局 `--delimiter`、`--encoding` 和 `--sheet` 会作用于每一个输入
|
|
228
230
|
|
|
229
231
|
## 输出模式
|
|
230
232
|
|
|
@@ -233,10 +235,11 @@ parq merge chunks/*.parquet merged.parquet
|
|
|
233
235
|
- `--version`, `-v`:显示版本号。放在子命令前面:`parq --version`
|
|
234
236
|
- `--output`, `-o`:选择输出格式(`rich` | `plain` | `json`)
|
|
235
237
|
- `--delimiter`, `-d`:CSV/TSV 输入分隔符,默认 `,`;`.tsv` 会自动使用制表符
|
|
238
|
+
- `--encoding`:CSV/TSV 输入编码,默认 `utf8`。写出仍然是 UTF-8
|
|
236
239
|
- `--sheet`:XLSX 工作表名称或从 0 开始的序号,默认活动工作表。纯数字会按序号处理,名叫 `1` 或 `2` 的工作表也一样
|
|
237
240
|
- `--help`:显示帮助
|
|
238
241
|
|
|
239
|
-
`--output`、`--delimiter`、`--sheet` 可以写在子命令前面,也可以写在子命令上面。同一个选项两处都写时,子命令上的值生效。`convert` 的 `--output` / `-o` 仍然是显示格式,目标文件是位置参数 `OUTPUT`。
|
|
242
|
+
`--output`、`--delimiter`、`--sheet`、`--encoding` 可以写在子命令前面,也可以写在子命令上面。同一个选项两处都写时,子命令上的值生效。`convert` 的 `--output` / `-o` 仍然是显示格式,目标文件是位置参数 `OUTPUT`。
|
|
240
243
|
|
|
241
244
|
可选输出模式:
|
|
242
245
|
|
|
@@ -256,11 +259,11 @@ parq diff left.parquet right.parquet --key id --summary-only --output json
|
|
|
256
259
|
|
|
257
260
|
## 大文件说明
|
|
258
261
|
|
|
259
|
-
- Parquet
|
|
260
|
-
- CSV `tail`
|
|
262
|
+
- Parquet 的元数据和行数来自文件 footer。`head` 拿到请求的行数就停,包括这些行落在同一个大 row group 里的情况。`tail` 只读包含末尾行的 row group。
|
|
263
|
+
- CSV `tail` 扫一遍文件,保留末尾的 Arrow batch,再切出最后几行。
|
|
261
264
|
- CSV 的 `split --record-count` 和 `convert` 单遍流式处理,进度条保持不确定状态,因为行数要等这一遍结束才知道。
|
|
262
265
|
- 对大 CSV/XLSX,`meta --fast` 是最快的元数据探查入口。
|
|
263
|
-
- `parq schema` 和较短的 `head` 对 XLSX 只采样前 1000 行。`count`、`tail`、`stats`、`convert`、`split`、`merge` 把选中的工作表解析一次,后续值放不进样本类型时放宽该列,并把这些行留在内存里复用。
|
|
266
|
+
- `parq schema` 和较短的 `head` 对 XLSX 只采样前 1000 行。`count`、`tail`、`stats`、`convert`、`split`、`merge`、`diff` 把选中的工作表解析一次,后续值放不进样本类型时放宽该列,并把这些行留在内存里复用。
|
|
264
267
|
|
|
265
268
|
如果你会反复处理大文件,把 CSV/XLSX 先转成 Parquet 依然是吞吐量最好的路径。
|
|
266
269
|
|