jsonl-diff 0.1.2__tar.gz → 0.1.4__tar.gz

{jsonl_diff-0.1.2 → jsonl_diff-0.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: jsonl-diff
-Version: 0.1.2
+Version: 0.1.4
 Summary: A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory.
 Author-email: rmoralespp <rmoralespp@gmail.com>
 License: MIT License
@@ -49,8 +49,11 @@ Classifier: Topic :: File Formats :: JSON
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: py-jsonl==1.5.2
+Requires-Dist: ijson>=3.2
+Requires-Dist: py-jsonl==1.5.3
 Requires-Dist: jmespath>=1.0.0
+Provides-Extra: speedups
+Requires-Dist: msgspec>=0.18; python_version >= "3.10" and extra == "speedups"
 Dynamic: license-file
 
 # jsonl-diff
@@ -63,12 +66,12 @@ Dynamic: license-file
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-Compare large JSONL/NDJSON datasets **by record identity instead of line
-position**, without loading the complete inputs into memory.
+A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets and
+top-level JSON arrays **by record identity instead of line position**, without
+loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
-`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each
-source.
+`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
 
 It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
@@ -80,6 +83,19 @@ It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 pip install jsonl-diff
 ```
 
+### Optional speedups
+
+For large datasets, install the optional [`msgspec`](https://jcristharif.com/msgspec/)
+accelerator (Python 3.10+), which uses a C parser and encoder for JSON decoding
+and record canonicalization, speeding up large diffs by roughly **2x**:
+
+```bash
+pip install "jsonl-diff[speedups]"
+```
+
+It is used automatically when available — no configuration or code changes. When
+`msgspec` is absent, the pure-Python path runs instead and produces identical results.
+
 ## Quick start
 
 Given:
@@ -119,23 +135,27 @@ Record order does not matter.
 
 * Single or composite top-level identities.
 * Disk-backed comparison for large datasets ([architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md)).
-* Optional **JMESPath** filtering with `--where`.
+* Optional **[JMESPath](https://jmespath.org/)** filtering with `--where`.
 * Configurable duplicate handling with counts and diagnostics: `error`, `first`, or `last`.
-* Exact **RFC 6901** JSON Pointer ignores.
+* Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
 * Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
+* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
 * Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
 * Local, HTTP/HTTPS, file-like, and supported compressed sources.
+* Incremental top-level JSON array parsing with `--format json`.
 * CLI and Python API using the same comparison engine.
 
 ## CLI
 
 ```text
 jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
-           [--duplicates {error,first,last}] [--details FILE] [--quiet]
+           [--duplicates {error,first,last}] [--missing-key {error,null}]
+           [--details FILE] [--quiet]
            [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
+           [--format {jsonl,json}]
            old new
 ```
 
@@ -146,11 +166,13 @@ jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
 | `--ignore POINTER`    | RFC 6901 pointer to exclude from content comparison                            |
 | `--where EXPRESSION`  | JMESPath filter applied to each record                                         |
 | `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`          |
+| `--missing-key POLICY`| `error` (default), or `null` to treat an absent identity field as `null`        |
 | `--details FILE`      | Write deterministic machine-readable JSONL changes                             |
 | `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                   |
 | `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                      |
 | `--quiet`             | Suppress the normal summary                                                    |
 | `--max-temp BYTES`    | Best-effort budget for `jsonl-diff` workspace temporary storage                |
+| `--format FORMAT`     | Input format: `jsonl` (default) or a top-level JSON array with `json`       |
 
 Examples:
 
@@ -169,6 +191,11 @@ jsonl-diff old.jsonl new.jsonl \
   --key id \
   --where 'deleted_at == `null`'
 
+# Tolerate an identity field that is absent from some records
+jsonl-diff old.jsonl new.jsonl \
+  --key country,customerId \
+  --missing-key null
+
 # Detect observed schema drift
 jsonl-diff old.jsonl new.jsonl \
   --key id \
@@ -177,6 +204,9 @@ jsonl-diff old.jsonl new.jsonl \
 
 # Compressed input
 jsonl-diff old.jsonl.xz new.jsonl.gz --key id
+
+# Top-level JSON arrays (parsed incrementally)
+jsonl-diff old.json new.json --key id --format json
 ```
 
 ### Exit codes
@@ -237,8 +267,12 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 
 `jsonl-diff` is intentionally strict:
 
-* Every input line must contain exactly one valid top-level JSON object.
-* Blank lines, malformed JSON, duplicate properties, `NaN`, and infinities are rejected.
+* In the default `jsonl` format, every input line must contain exactly one
+  valid top-level JSON object.
+* With `--format json`, each input must be one top-level JSON array whose
+  elements are the records.
+* Blank lines, malformed JSON, `NaN`, and infinities are rejected.
+* Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
 * Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
 * Duplicate identities fail by default; `--duplicates first`/`last` select one
@@ -259,7 +293,9 @@ order, and canonical number formatting.
 ## Sources & compression
 
 Supported sources include local paths and HTTP/HTTPS URLs. The Python API
-also accepts file-like objects.
+also accepts file-like objects. JSON arrays use `py-jsonl.open_stream()` for
+the source and `ijson` for incremental parsing, so compression and remote
+source handling are shared with JSONL input.
 
 Supported compression:
 

{jsonl_diff-0.1.2 → jsonl_diff-0.1.4}/README.md

@@ -8,12 +8,12 @@
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-Compare large JSONL/NDJSON datasets **by record identity instead of line
-position**, without loading the complete inputs into memory.
+A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets and
+top-level JSON arrays **by record identity instead of line position**, without
+loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
-`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each
-source.
+`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
 
 It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
@@ -25,6 +25,19 @@ It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 pip install jsonl-diff
 ```
 
+### Optional speedups
+
+For large datasets, install the optional [`msgspec`](https://jcristharif.com/msgspec/)
+accelerator (Python 3.10+), which uses a C parser and encoder for JSON decoding
+and record canonicalization, speeding up large diffs by roughly **2x**:
+
+```bash
+pip install "jsonl-diff[speedups]"
+```
+
+It is used automatically when available — no configuration or code changes. When
+`msgspec` is absent, the pure-Python path runs instead and produces identical results.
+
 ## Quick start
 
 Given:
@@ -64,23 +77,27 @@ Record order does not matter.
 
 * Single or composite top-level identities.
 * Disk-backed comparison for large datasets ([architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md)).
-* Optional **JMESPath** filtering with `--where`.
+* Optional **[JMESPath](https://jmespath.org/)** filtering with `--where`.
 * Configurable duplicate handling with counts and diagnostics: `error`, `first`, or `last`.
-* Exact **RFC 6901** JSON Pointer ignores.
+* Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
 * Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
+* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
 * Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
 * Local, HTTP/HTTPS, file-like, and supported compressed sources.
+* Incremental top-level JSON array parsing with `--format json`.
 * CLI and Python API using the same comparison engine.
 
 ## CLI
 
 ```text
 jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
-           [--duplicates {error,first,last}] [--details FILE] [--quiet]
+           [--duplicates {error,first,last}] [--missing-key {error,null}]
+           [--details FILE] [--quiet]
            [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
+           [--format {jsonl,json}]
            old new
 ```
 
@@ -91,11 +108,13 @@ jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
 | `--ignore POINTER`    | RFC 6901 pointer to exclude from content comparison                            |
 | `--where EXPRESSION`  | JMESPath filter applied to each record                                         |
 | `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`          |
+| `--missing-key POLICY`| `error` (default), or `null` to treat an absent identity field as `null`        |
 | `--details FILE`      | Write deterministic machine-readable JSONL changes                             |
 | `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                   |
 | `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                      |
 | `--quiet`             | Suppress the normal summary                                                    |
 | `--max-temp BYTES`    | Best-effort budget for `jsonl-diff` workspace temporary storage                |
+| `--format FORMAT`     | Input format: `jsonl` (default) or a top-level JSON array with `json`       |
 
 Examples:
 
@@ -114,6 +133,11 @@ jsonl-diff old.jsonl new.jsonl \
   --key id \
   --where 'deleted_at == `null`'
 
+# Tolerate an identity field that is absent from some records
+jsonl-diff old.jsonl new.jsonl \
+  --key country,customerId \
+  --missing-key null
+
 # Detect observed schema drift
 jsonl-diff old.jsonl new.jsonl \
   --key id \
@@ -122,6 +146,9 @@ jsonl-diff old.jsonl new.jsonl \
 
 # Compressed input
 jsonl-diff old.jsonl.xz new.jsonl.gz --key id
+
+# Top-level JSON arrays (parsed incrementally)
+jsonl-diff old.json new.json --key id --format json
 ```
 
 ### Exit codes
@@ -182,8 +209,12 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 
 `jsonl-diff` is intentionally strict:
 
-* Every input line must contain exactly one valid top-level JSON object.
-* Blank lines, malformed JSON, duplicate properties, `NaN`, and infinities are rejected.
+* In the default `jsonl` format, every input line must contain exactly one
+  valid top-level JSON object.
+* With `--format json`, each input must be one top-level JSON array whose
+  elements are the records.
+* Blank lines, malformed JSON, `NaN`, and infinities are rejected.
+* Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
 * Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
 * Duplicate identities fail by default; `--duplicates first`/`last` select one
@@ -204,7 +235,9 @@ order, and canonical number formatting.
 ## Sources & compression
 
 Supported sources include local paths and HTTP/HTTPS URLs. The Python API
-also accepts file-like objects.
+also accepts file-like objects. JSON arrays use `py-jsonl.open_stream()` for
+the source and `ijson` for incremental parsing, so compression and remote
+source handling are shared with JSONL input.
 
 Supported compression:
 

{jsonl_diff-0.1.2 → jsonl_diff-0.1.4}/jsonl_diff.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: jsonl-diff
-Version: 0.1.2
+Version: 0.1.4
 Summary: A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory.
 Author-email: rmoralespp <rmoralespp@gmail.com>
 License: MIT License
@@ -49,8 +49,11 @@ Classifier: Topic :: File Formats :: JSON
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: py-jsonl==1.5.2
+Requires-Dist: ijson>=3.2
+Requires-Dist: py-jsonl==1.5.3
 Requires-Dist: jmespath>=1.0.0
+Provides-Extra: speedups
+Requires-Dist: msgspec>=0.18; python_version >= "3.10" and extra == "speedups"
 Dynamic: license-file
 
 # jsonl-diff
@@ -63,12 +66,12 @@ Dynamic: license-file
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-Compare large JSONL/NDJSON datasets **by record identity instead of line
-position**, without loading the complete inputs into memory.
+A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets and
+top-level JSON arrays **by record identity instead of line position**, without
+loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
-`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each
-source.
+`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
 
 It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
@@ -80,6 +83,19 @@ It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 pip install jsonl-diff
 ```
 
+### Optional speedups
+
+For large datasets, install the optional [`msgspec`](https://jcristharif.com/msgspec/)
+accelerator (Python 3.10+), which uses a C parser and encoder for JSON decoding
+and record canonicalization, speeding up large diffs by roughly **2x**:
+
+```bash
+pip install "jsonl-diff[speedups]"
+```
+
+It is used automatically when available — no configuration or code changes. When
+`msgspec` is absent, the pure-Python path runs instead and produces identical results.
+
 ## Quick start
 
 Given:
@@ -119,23 +135,27 @@ Record order does not matter.
 
 * Single or composite top-level identities.
 * Disk-backed comparison for large datasets ([architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md)).
-* Optional **JMESPath** filtering with `--where`.
+* Optional **[JMESPath](https://jmespath.org/)** filtering with `--where`.
 * Configurable duplicate handling with counts and diagnostics: `error`, `first`, or `last`.
-* Exact **RFC 6901** JSON Pointer ignores.
+* Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
 * Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
+* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
 * Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
 * Local, HTTP/HTTPS, file-like, and supported compressed sources.
+* Incremental top-level JSON array parsing with `--format json`.
 * CLI and Python API using the same comparison engine.
 
 ## CLI
 
 ```text
 jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
-           [--duplicates {error,first,last}] [--details FILE] [--quiet]
+           [--duplicates {error,first,last}] [--missing-key {error,null}]
+           [--details FILE] [--quiet]
            [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
+           [--format {jsonl,json}]
            old new
 ```
 
@@ -146,11 +166,13 @@ jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
 | `--ignore POINTER`    | RFC 6901 pointer to exclude from content comparison                            |
 | `--where EXPRESSION`  | JMESPath filter applied to each record                                         |
 | `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`          |
+| `--missing-key POLICY`| `error` (default), or `null` to treat an absent identity field as `null`        |
 | `--details FILE`      | Write deterministic machine-readable JSONL changes                             |
 | `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                   |
 | `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                      |
 | `--quiet`             | Suppress the normal summary                                                    |
 | `--max-temp BYTES`    | Best-effort budget for `jsonl-diff` workspace temporary storage                |
+| `--format FORMAT`     | Input format: `jsonl` (default) or a top-level JSON array with `json`       |
 
 Examples:
 
@@ -169,6 +191,11 @@ jsonl-diff old.jsonl new.jsonl \
   --key id \
   --where 'deleted_at == `null`'
 
+# Tolerate an identity field that is absent from some records
+jsonl-diff old.jsonl new.jsonl \
+  --key country,customerId \
+  --missing-key null
+
 # Detect observed schema drift
 jsonl-diff old.jsonl new.jsonl \
   --key id \
@@ -177,6 +204,9 @@ jsonl-diff old.jsonl new.jsonl \
 
 # Compressed input
 jsonl-diff old.jsonl.xz new.jsonl.gz --key id
+
+# Top-level JSON arrays (parsed incrementally)
+jsonl-diff old.json new.json --key id --format json
 ```
 
 ### Exit codes
@@ -237,8 +267,12 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 
 `jsonl-diff` is intentionally strict:
 
-* Every input line must contain exactly one valid top-level JSON object.
-* Blank lines, malformed JSON, duplicate properties, `NaN`, and infinities are rejected.
+* In the default `jsonl` format, every input line must contain exactly one
+  valid top-level JSON object.
+* With `--format json`, each input must be one top-level JSON array whose
+  elements are the records.
+* Blank lines, malformed JSON, `NaN`, and infinities are rejected.
+* Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
 * Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
 * Duplicate identities fail by default; `--duplicates first`/`last` select one
@@ -259,7 +293,9 @@ order, and canonical number formatting.
 ## Sources & compression
 
 Supported sources include local paths and HTTP/HTTPS URLs. The Python API
-also accepts file-like objects.
+also accepts file-like objects. JSON arrays use `py-jsonl.open_stream()` for
+the source and `ijson` for incremental parsing, so compression and remote
+source handling are shared with JSONL input.
 
 Supported compression:
 

jsonl_diff-0.1.4/jsonl_diff.egg-info/requires.txt

@@ -0,0 +1,8 @@
+ijson>=3.2
+py-jsonl==1.5.3
+jmespath>=1.0.0
+
+[speedups]
+
+[speedups:python_version >= "3.10"]
+msgspec>=0.18