jsonl-diff 0.1.3__tar.gz → 0.1.5__tar.gz

{jsonl_diff-0.1.3 → jsonl_diff-0.1.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: jsonl-diff
-Version: 0.1.3
+Version: 0.1.5
 Summary: A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory.
 Author-email: rmoralespp <rmoralespp@gmail.com>
 License: MIT License
@@ -49,7 +49,8 @@ Classifier: Topic :: File Formats :: JSON
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: py-jsonl==1.5.2
+Requires-Dist: ijson>=3.2
+Requires-Dist: py-jsonl==1.5.3
 Requires-Dist: jmespath>=1.0.0
 Provides-Extra: speedups
 Requires-Dist: msgspec>=0.18; python_version >= "3.10" and extra == "speedups"
@@ -65,8 +66,9 @@ Dynamic: license-file
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-A lightweight, pure-Python tool for comparing large [JSONL](https://jsonlines.org/)/[NDJSON](https://github.com/ndjson/ndjson-spec) datasets **by record identity instead of line
-position**, without loading the complete inputs into memory.
+A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets and
+top-level JSON arrays **by record identity instead of line position**, without
+loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
 `equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
@@ -138,19 +140,21 @@ Record order does not matter.
 * Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
 * Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
-* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
 * Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
 * Local, HTTP/HTTPS, file-like, and supported compressed sources.
+* Incremental parsing for JSONL/NDJSON by default, with support for top-level JSON arrays via `--format json`.
 * CLI and Python API using the same comparison engine.
 
 ## CLI
 
 ```text
 jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
-           [--duplicates {error,first,last}] [--details FILE] [--quiet]
+           [--duplicates {error,first,last}] [--missing-key {error,null}]
+           [--details FILE] [--quiet]
            [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
+           [--format {jsonl,json}]
            old new
 ```
 
@@ -160,12 +164,14 @@ jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
 | `--key KEY`           | Required top-level identity field; repeat or comma-separate for composite keys |
 | `--ignore POINTER`    | RFC 6901 pointer to exclude from content comparison                            |
 | `--where EXPRESSION`  | JMESPath filter applied to each record                                         |
-| `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`          |
+| `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`         |
+| `--missing-key POLICY`| `error` (default), or `null` to treat an absent identity field as `null`       |
 | `--details FILE`      | Write deterministic machine-readable JSONL changes                             |
-| `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                   |
-| `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                      |
+| `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                  |
+| `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                     |
 | `--quiet`             | Suppress the normal summary                                                    |
 | `--max-temp BYTES`    | Best-effort budget for `jsonl-diff` workspace temporary storage                |
+| `--format FORMAT`     | Input format: `jsonl` (default) or a top-level JSON array with `json`          |
 
 Examples:
 
@@ -184,6 +190,11 @@ jsonl-diff old.jsonl new.jsonl \
   --key id \
   --where 'deleted_at == `null`'
 
+# Tolerate an identity field that is absent from some records
+jsonl-diff old.jsonl new.jsonl \
+  --key country,customerId \
+  --missing-key null
+
 # Detect observed schema drift
 jsonl-diff old.jsonl new.jsonl \
   --key id \
@@ -192,6 +203,9 @@ jsonl-diff old.jsonl new.jsonl \
 
 # Compressed input
 jsonl-diff old.jsonl.xz new.jsonl.gz --key id
+
+# Top-level JSON arrays (parsed incrementally)
+jsonl-diff old.json new.json --key id --format json
 ```
 
 ### Exit codes
@@ -252,10 +266,15 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 
 `jsonl-diff` is intentionally strict:
 
-* Every input line must contain exactly one valid top-level JSON object.
+* In the default `jsonl` format, every input line must contain exactly one
+  valid top-level JSON object.
+* With `--format json`, each input must be one top-level JSON array whose
+  elements are the records.
 * Blank lines, malformed JSON, `NaN`, and infinities are rejected.
 * Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
-* Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
+* Identity fields must be top-level JSON values. Object member order is ignored
+  and array order is significant (`null` allowed only as one component of a
+  composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
 * Duplicate identities fail by default; `--duplicates first`/`last` select one
   occurrence, report every discarded occurrence, and return exit code `1`.
@@ -285,7 +304,7 @@ Supported compression:
 * Zstandard on Python 3.14
 
 See [architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) for the full source/compression
-support matrix and how sources are delegated to `py-jsonl`.
+support matrix and how sources are managed.
 
 ## Limitations
 

{jsonl_diff-0.1.3 → jsonl_diff-0.1.5}/README.md

@@ -8,8 +8,9 @@
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-A lightweight, pure-Python tool for comparing large [JSONL](https://jsonlines.org/)/[NDJSON](https://github.com/ndjson/ndjson-spec) datasets **by record identity instead of line
-position**, without loading the complete inputs into memory.
+A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets and
+top-level JSON arrays **by record identity instead of line position**, without
+loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
 `equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
@@ -81,19 +82,21 @@ Record order does not matter.
 * Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
 * Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
-* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
 * Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
 * Local, HTTP/HTTPS, file-like, and supported compressed sources.
+* Incremental parsing for JSONL/NDJSON by default, with support for top-level JSON arrays via `--format json`.
 * CLI and Python API using the same comparison engine.
 
 ## CLI
 
 ```text
 jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
-           [--duplicates {error,first,last}] [--details FILE] [--quiet]
+           [--duplicates {error,first,last}] [--missing-key {error,null}]
+           [--details FILE] [--quiet]
            [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
+           [--format {jsonl,json}]
            old new
 ```
 
@@ -103,12 +106,14 @@ jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
 | `--key KEY`           | Required top-level identity field; repeat or comma-separate for composite keys |
 | `--ignore POINTER`    | RFC 6901 pointer to exclude from content comparison                            |
 | `--where EXPRESSION`  | JMESPath filter applied to each record                                         |
-| `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`          |
+| `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`         |
+| `--missing-key POLICY`| `error` (default), or `null` to treat an absent identity field as `null`       |
 | `--details FILE`      | Write deterministic machine-readable JSONL changes                             |
-| `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                   |
-| `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                      |
+| `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                  |
+| `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                     |
 | `--quiet`             | Suppress the normal summary                                                    |
 | `--max-temp BYTES`    | Best-effort budget for `jsonl-diff` workspace temporary storage                |
+| `--format FORMAT`     | Input format: `jsonl` (default) or a top-level JSON array with `json`          |
 
 Examples:
 
@@ -127,6 +132,11 @@ jsonl-diff old.jsonl new.jsonl \
   --key id \
   --where 'deleted_at == `null`'
 
+# Tolerate an identity field that is absent from some records
+jsonl-diff old.jsonl new.jsonl \
+  --key country,customerId \
+  --missing-key null
+
 # Detect observed schema drift
 jsonl-diff old.jsonl new.jsonl \
   --key id \
@@ -135,6 +145,9 @@ jsonl-diff old.jsonl new.jsonl \
 
 # Compressed input
 jsonl-diff old.jsonl.xz new.jsonl.gz --key id
+
+# Top-level JSON arrays (parsed incrementally)
+jsonl-diff old.json new.json --key id --format json
 ```
 
 ### Exit codes
@@ -195,10 +208,15 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 
 `jsonl-diff` is intentionally strict:
 
-* Every input line must contain exactly one valid top-level JSON object.
+* In the default `jsonl` format, every input line must contain exactly one
+  valid top-level JSON object.
+* With `--format json`, each input must be one top-level JSON array whose
+  elements are the records.
 * Blank lines, malformed JSON, `NaN`, and infinities are rejected.
 * Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
-* Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
+* Identity fields must be top-level JSON values. Object member order is ignored
+  and array order is significant (`null` allowed only as one component of a
+  composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
 * Duplicate identities fail by default; `--duplicates first`/`last` select one
   occurrence, report every discarded occurrence, and return exit code `1`.
@@ -228,7 +246,7 @@ Supported compression:
 * Zstandard on Python 3.14
 
 See [architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) for the full source/compression
-support matrix and how sources are delegated to `py-jsonl`.
+support matrix and how sources are managed.
 
 ## Limitations
 

{jsonl_diff-0.1.3 → jsonl_diff-0.1.5}/jsonl_diff.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: jsonl-diff
-Version: 0.1.3
+Version: 0.1.5
 Summary: A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory.
 Author-email: rmoralespp <rmoralespp@gmail.com>
 License: MIT License
@@ -49,7 +49,8 @@ Classifier: Topic :: File Formats :: JSON
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: py-jsonl==1.5.2
+Requires-Dist: ijson>=3.2
+Requires-Dist: py-jsonl==1.5.3
 Requires-Dist: jmespath>=1.0.0
 Provides-Extra: speedups
 Requires-Dist: msgspec>=0.18; python_version >= "3.10" and extra == "speedups"
@@ -65,8 +66,9 @@ Dynamic: license-file
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-A lightweight, pure-Python tool for comparing large [JSONL](https://jsonlines.org/)/[NDJSON](https://github.com/ndjson/ndjson-spec) datasets **by record identity instead of line
-position**, without loading the complete inputs into memory.
+A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets and
+top-level JSON arrays **by record identity instead of line position**, without
+loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
 `equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
@@ -138,19 +140,21 @@ Record order does not matter.
 * Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
 * Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
-* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
 * Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
 * Local, HTTP/HTTPS, file-like, and supported compressed sources.
+* Incremental parsing for JSONL/NDJSON by default, with support for top-level JSON arrays via `--format json`.
 * CLI and Python API using the same comparison engine.
 
 ## CLI
 
 ```text
 jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
-           [--duplicates {error,first,last}] [--details FILE] [--quiet]
+           [--duplicates {error,first,last}] [--missing-key {error,null}]
+           [--details FILE] [--quiet]
            [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
+           [--format {jsonl,json}]
            old new
 ```
 
@@ -160,12 +164,14 @@ jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
 | `--key KEY`           | Required top-level identity field; repeat or comma-separate for composite keys |
 | `--ignore POINTER`    | RFC 6901 pointer to exclude from content comparison                            |
 | `--where EXPRESSION`  | JMESPath filter applied to each record                                         |
-| `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`          |
+| `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`         |
+| `--missing-key POLICY`| `error` (default), or `null` to treat an absent identity field as `null`       |
 | `--details FILE`      | Write deterministic machine-readable JSONL changes                             |
-| `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                   |
-| `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                      |
+| `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                  |
+| `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                     |
 | `--quiet`             | Suppress the normal summary                                                    |
 | `--max-temp BYTES`    | Best-effort budget for `jsonl-diff` workspace temporary storage                |
+| `--format FORMAT`     | Input format: `jsonl` (default) or a top-level JSON array with `json`          |
 
 Examples:
 
@@ -184,6 +190,11 @@ jsonl-diff old.jsonl new.jsonl \
   --key id \
   --where 'deleted_at == `null`'
 
+# Tolerate an identity field that is absent from some records
+jsonl-diff old.jsonl new.jsonl \
+  --key country,customerId \
+  --missing-key null
+
 # Detect observed schema drift
 jsonl-diff old.jsonl new.jsonl \
   --key id \
@@ -192,6 +203,9 @@ jsonl-diff old.jsonl new.jsonl \
 
 # Compressed input
 jsonl-diff old.jsonl.xz new.jsonl.gz --key id
+
+# Top-level JSON arrays (parsed incrementally)
+jsonl-diff old.json new.json --key id --format json
 ```
 
 ### Exit codes
@@ -252,10 +266,15 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 
 `jsonl-diff` is intentionally strict:
 
-* Every input line must contain exactly one valid top-level JSON object.
+* In the default `jsonl` format, every input line must contain exactly one
+  valid top-level JSON object.
+* With `--format json`, each input must be one top-level JSON array whose
+  elements are the records.
 * Blank lines, malformed JSON, `NaN`, and infinities are rejected.
 * Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
-* Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
+* Identity fields must be top-level JSON values. Object member order is ignored
+  and array order is significant (`null` allowed only as one component of a
+  composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
 * Duplicate identities fail by default; `--duplicates first`/`last` select one
   occurrence, report every discarded occurrence, and return exit code `1`.
@@ -285,7 +304,7 @@ Supported compression:
 * Zstandard on Python 3.14
 
 See [architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) for the full source/compression
-support matrix and how sources are delegated to `py-jsonl`.
+support matrix and how sources are managed.
 
 ## Limitations
 

{jsonl_diff-0.1.3 → jsonl_diff-0.1.5}/jsonl_diff.egg-info/requires.txt

@@ -1,4 +1,5 @@
-py-jsonl==1.5.2
+ijson>=3.2
+py-jsonl==1.5.3
 jmespath>=1.0.0
 
 [speedups]

{jsonl_diff-0.1.3 → jsonl_diff-0.1.5}/jsonl_diff.py

@@ -17,6 +17,7 @@ from json.encoder import encode_basestring_ascii as _encode_basestring_ascii
 from pathlib import Path
 from typing import Any, Callable, Dict, Iterable, Iterator, Optional, Sequence, Tuple, Union
 
+import ijson
 import jmespath
 import jsonl
 
@@ -25,9 +26,9 @@ try:  # Optional C-accelerated content canonicalization (Python 3.10+).
 except ImportError:
     _msgspec = None
 
-JsonScalar = Union[str, Decimal, bool, None]
+JsonIdentityValue = Union[str, Decimal, bool, int, None, list, dict]
 Number = Union[Decimal, int, float]
-IdentityKey = Tuple[JsonScalar, ...]
+IdentityKey = Tuple[JsonIdentityValue, ...]
 
 
 class _Num(Decimal):
@@ -118,6 +119,13 @@ class DuplicatePolicy(str, Enum):
     LAST = "last"
 
 
+class MissingKeyPolicy(str, Enum):
+    """How an absent identity field is handled."""
+
+    ERROR = "error"
+    NULL = "null"
+
+
 class SchemaChangeOperation(str, Enum):
     """An observed schema change classification."""
 
@@ -262,6 +270,8 @@ class DiffConfig:
     where_expression: Any = None
     schema_diff: bool = False
     schema_ignore: Tuple[str, ...] = ()
+    format: str = "jsonl"
+    missing_key: MissingKeyPolicy = MissingKeyPolicy.ERROR
 
 
 @dataclass(frozen=True)
@@ -574,13 +584,20 @@ def _remove_ignored(value: Any, tree: Dict[str, Any]) -> Any:
     return result
 
 
-def _identity(record: Dict[str, Any], keys: Sequence[str]) -> IdentityKey:
+def _identity(
+    record: Dict[str, Any],
+    keys: Sequence[str],
+    missing_key: MissingKeyPolicy = MissingKeyPolicy.ERROR,
+) -> IdentityKey:
     composite = len(keys) > 1
     values = []
     for name in keys:
         if name not in record:
-            raise ValueError("missing identity field {!r}".format(name))
-        value = record[name]
+            if missing_key != MissingKeyPolicy.NULL:
+                raise ValueError("missing identity field {!r}".format(name))
+            value = None
+        else:
+            value = record[name]
         if value is None:
             # Composite identities may contain null components; uniqueness is
             # still enforced on the complete identity tuple. A single-field
@@ -588,8 +605,8 @@ def _identity(record: Dict[str, Any], keys: Sequence[str]) -> IdentityKey:
             # record into one indistinguishable identity.
             if not composite:
                 raise ValueError("identity field {!r} must be a non-null scalar".format(name))
-        elif not isinstance(value, (str, Decimal, bool, int)):
-            raise ValueError("identity field {!r} must be a non-null scalar".format(name))
+        elif not isinstance(value, (str, Decimal, bool, int, list, dict)):
+            raise ValueError("identity field {!r} must be a JSON value".format(name))
         values.append(value)
     return tuple(values)
 
@@ -603,10 +620,36 @@ def _compile_where(expression: str) -> Any:
         ) from error
 
 
+def _normalize_array_numbers(value: Any) -> Any:
+    if isinstance(value, Decimal) and not isinstance(value, _Num):
+        return _Num(value)
+    if isinstance(value, list):
+        return [_normalize_array_numbers(item) for item in value]
+    if isinstance(value, dict):
+        return {name: _normalize_array_numbers(item) for name, item in value.items()}
+    return value
+
+
+def _array_records(source: Any) -> Iterator[Any]:
+    with jsonl.open_stream(source) as stream:
+        events = ijson.parse(stream, use_float=False)
+        try:
+            prefix, event, _value = next(events)
+        except StopIteration:
+            raise ValueError("empty JSON input") from None
+        if (prefix, event) != ("", "start_array"):
+            raise ValueError("expected a top-level JSON array")
+        for record in ijson.items(events, "item"):
+            yield _normalize_array_numbers(record)
+
+
 if _msgspec is not None:
     _INPUT_DECODER = _msgspec.json.Decoder(float_hook=_Num)
 
-    def _records(source: Any, on_error: Any) -> Iterator[Any]:
+    def _records(source: Any, on_error: Any, input_format: str) -> Iterator[Any]:
+        if input_format == "json":
+            yield from _array_records(source)
+            return
         # msgspec decodes input lines ~2.6x faster than the stdlib parser and
         # rejects NaN/Infinity natively. Floats become `_Num` (a Decimal subclass)
         # while integers stay native `int`; both are handled by the number
@@ -614,7 +657,10 @@ if _msgspec is not None:
         # matching the pure-Python fallback below.
         yield from jsonl.load(source, cls=_INPUT_DECODER.decode, _on_error=on_error)
 else:
-    def _records(source: Any, on_error: Any) -> Iterator[Any]:
+    def _records(source: Any, on_error: Any, input_format: str) -> Iterator[Any]:
+        if input_format == "json":
+            yield from _array_records(source)
+            return
         # No object_pairs_hook is installed, so duplicate object keys follow the
         # stdlib parser's last-wins semantics, matching the msgspec path above.
         yield from jsonl.load(
@@ -1053,7 +1099,7 @@ class DiffResult:
                             self._check_size()
                     continue
             try:
-                key = _identity(record, self.config.key)
+                key = _identity(record, self.config.key, self.config.missing_key)
                 normalized = _remove_ignored(record, self._ignore_tree)
                 canonical = _content_canonical(normalized)
                 identity = _canonical(list(key))
@@ -1176,10 +1222,10 @@ class DiffResult:
 
         try:
             with self._connection:
-                self._insert_records(_records(source, on_error), side, name)
+                self._insert_records(_records(source, on_error, self.config.format), side, name)
         except JsonlDiffError:
             raise
-        except (OSError, EOFError, ValueError, RuntimeError) as error:
+        except (OSError, EOFError, ValueError, RuntimeError, ijson.JSONError) as error:
             line = error_line[0]
             message = "invalid input ({})".format(type(error).__name__)
             raise InputError(message, name, line) from error
@@ -1264,6 +1310,8 @@ def _configuration(
     max_temp: Optional[int],
     schema_diff: bool,
     schema_ignore: Sequence[str],
+    input_format: str,
+    missing_key: Union[str, MissingKeyPolicy] = MissingKeyPolicy.ERROR,
 ) -> DiffConfig:
     keys = (key,) if isinstance(key, str) else tuple(key)
     if not keys or any(not isinstance(name, str) or not name for name in keys):
@@ -1282,20 +1330,30 @@ def _configuration(
         duplicate_policy = DuplicatePolicy(duplicates)
     except ValueError as error:
         raise ConfigurationError("invalid duplicate policy {!r}".format(duplicates)) from error
+    try:
+        missing_key_policy = MissingKeyPolicy(missing_key)
+    except ValueError as error:
+        raise ConfigurationError(
+            "invalid missing-key policy {!r}".format(missing_key),
+        ) from error
     if (
         max_temp is not None
         and (not isinstance(max_temp, int) or isinstance(max_temp, bool) or max_temp <= 0)
     ):
         raise ConfigurationError("max_temp must be a positive integer")
+    if input_format not in ("jsonl", "json"):
+        raise ConfigurationError("invalid format {!r}".format(input_format))
     return DiffConfig(
         key=keys,
         ignore=ignores,
         where=where,
         duplicates=duplicate_policy,
+        missing_key=missing_key_policy,
         max_temp=max_temp,
         where_expression=where_expression,
         schema_diff=schema_diff,
         schema_ignore=schema_ignores,
+        format=input_format,
     )
 
 
@@ -1307,9 +1365,11 @@ def diff(
     ignore: Sequence[str] = (),
     where: Optional[str] = None,
     duplicates: Union[str, DuplicatePolicy] = DuplicatePolicy.ERROR,
+    missing_key: Union[str, MissingKeyPolicy] = MissingKeyPolicy.ERROR,
     max_temp: Optional[int] = None,
     schema_diff: bool = False,
     schema_ignore: Sequence[str] = (),
+    format: str = "jsonl",  # noqa: A002
 ) -> DiffResult:
     """Create a context-managed, disk-backed comparison."""
     config = _configuration(
@@ -1320,6 +1380,8 @@ def diff(
         max_temp,
         schema_diff,
         schema_ignore,
+        format,
+        missing_key,
     )
     return DiffResult(old, new, config)
 
@@ -1408,6 +1470,7 @@ def _write_details(result: DiffResult, path: Union[str, os.PathLike]) -> None:
             "ignore": list(result.config.ignore),
             "where": result.config.where,
             "duplicates": result.config.duplicates.value,
+            "missing_key": result.config.missing_key.value,
         }
         if result.config.schema_diff:
             metadata["schema_diff"] = True
@@ -1454,11 +1517,17 @@ def _parser() -> argparse.ArgumentParser:
         choices=tuple(policy.value for policy in DuplicatePolicy),
         default=DuplicatePolicy.ERROR.value,
     )
+    parser.add_argument(
+        "--missing-key",
+        choices=tuple(policy.value for policy in MissingKeyPolicy),
+        default=MissingKeyPolicy.ERROR.value,
+    )
     parser.add_argument("--details", metavar="FILE")
     parser.add_argument("--schema-diff", action="store_true")
     parser.add_argument("--schema-ignore", action="append", default=[], metavar="POINTER")
     parser.add_argument("--quiet", action="store_true")
     parser.add_argument("--max-temp", type=int)
+    parser.add_argument("--format", choices=("jsonl", "json"), default="jsonl")
     return parser
 
 
@@ -1470,9 +1539,11 @@ def _run_comparison(arguments: argparse.Namespace, old: Any, new: Any, keys: Tup
         ignore=arguments.ignore,
         where=arguments.where,
         duplicates=arguments.duplicates,
+        missing_key=arguments.missing_key,
         max_temp=arguments.max_temp,
         schema_diff=arguments.schema_diff,
         schema_ignore=arguments.schema_ignore,
+        format=arguments.format,
     ) as result:
         if arguments.details:
             _write_details(result, arguments.details)

{jsonl_diff-0.1.3 → jsonl_diff-0.1.5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "jsonl-diff"
-version = "0.1.3"
+version = "0.1.5"
 description = "A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory."
 readme = "README.md"
 license = { file = "LICENSE" }
@@ -29,7 +29,8 @@ classifiers = [
     "Topic :: File Formats :: JSON",
 ]
 dependencies = [
-    "py-jsonl==1.5.2",
+    "ijson>=3.2",
+    "py-jsonl==1.5.3",
     "jmespath>=1.0.0",
 ]
 

{jsonl_diff-0.1.3 → jsonl_diff-0.1.5}/tests/test_cli.py

@@ -1,5 +1,6 @@
 import io
 import json
+import types
 from decimal import Decimal
 
 import pytest
@@ -26,6 +27,36 @@ def _different_sources(write_jsonl):
 
 
 class TestCliExitCodes:
+    def test_json_format_compares_array_inputs(self, tmp_path, capsys):
+        # Arrange
+        old = tmp_path / "old.json"
+        new = tmp_path / "new.json"
+        old.write_text('[{"id":1}]', encoding="utf-8")
+        new.write_text('[{"id":1},{"id":2}]', encoding="utf-8")
+
+        # Act
+        exit_code = main([str(old), str(new), "--key", "id", "--format", "json"])
+        captured = capsys.readouterr()
+
+        # Assert
+        assert exit_code == 1
+        assert "added:     1" in captured.out
+
+    def test_json_format_reads_stdin(self, tmp_path, monkeypatch, capsys):
+        # Arrange
+        new = tmp_path / "new.json"
+        new.write_text('[{"id":1}]', encoding="utf-8")
+        stdin = types.SimpleNamespace(buffer=io.BytesIO(b'[{"id":1},{"id":2}]'))
+        monkeypatch.setattr(jsonl_diff.sys, "stdin", stdin)
+
+        # Act
+        exit_code = main(["-", str(new), "--key", "id", "--format", "json"])
+        captured = capsys.readouterr()
+
+        # Assert
+        assert exit_code == 1
+        assert "deleted:   1" in captured.out
+
     def test_equal_inputs_return_zero(self, write_jsonl, capsys):
         # Arrange
         old = write_jsonl("old.jsonl", [{"id": 1}])
@@ -163,6 +194,44 @@ class TestCliExitCodes:
         assert exit_code == 1
         assert capsys.readouterr() == ("", "")
 
+    def test_missing_key_defaults_to_error(self, write_jsonl, capsys):
+        # Arrange
+        old = write_jsonl("old.jsonl", [{"value": "old"}])
+        new = write_jsonl("new.jsonl", [{"id": 1, "value": "new"}])
+
+        # Act
+        exit_code = main([str(old), str(new), "--key", "id", "--quiet"])
+
+        # Assert
+        assert exit_code == 2
+        assert "missing identity field" in capsys.readouterr().err
+
+    def test_missing_key_null_policy_tolerates_absent_composite_component(
+        self,
+        write_jsonl,
+        capsys,
+    ):
+        # Arrange
+        old = write_jsonl("old.jsonl", [{"a": 1, "value": "old"}])
+        new = write_jsonl("new.jsonl", [{"a": 1, "b": None, "value": "new"}])
+
+        # Act
+        exit_code = main(
+            [
+                str(old),
+                str(new),
+                "--key",
+                "a,b",
+                "--missing-key",
+                "null",
+                "--quiet",
+            ],
+        )
+
+        # Assert
+        assert exit_code == 1
+        assert capsys.readouterr() == ("", "")
+
     @pytest.mark.parametrize("stdin_side", ["old", "new"])
     def test_dash_reads_one_source_from_stdin(
         self,
@@ -459,6 +528,7 @@ class TestDetailsOutput:
             "ignore": ["/volatile"],
             "where": None,
             "duplicates": "error",
+            "missing_key": "error",
         }, {
             "type": "summary",
             "equal": 1,
@@ -527,6 +597,26 @@ class TestDetailsOutput:
         # Assert
         assert records[1]["key"] == [12345678901234567890]
 
+    def test_structured_key_is_written_to_details_as_json(
+        self,
+        write_jsonl,
+        tmp_path,
+    ):
+        old = write_jsonl(
+            "old.jsonl",
+            [{"id": {"region": "ES", "customer": 7}, "value": "before"}],
+        )
+        new = write_jsonl(
+            "new.jsonl",
+            [{"id": {"customer": 7, "region": "ES"}, "value": "after"}],
+        )
+        details = tmp_path / "changes.jsonl"
+
+        main([str(old), str(new), "--key", "id", "--details", str(details), "--quiet"])
+        records = [json.loads(line) for line in details.read_text(encoding="utf-8").splitlines()]
+
+        assert records[1]["key"] == [{"customer": 7, "region": "ES"}]
+
     def test_duplicate_events_follow_meta_and_include_policy_outcome(
         self,
         write_jsonl,

{jsonl_diff-0.1.3 → jsonl_diff-0.1.5}/tests/test_diff.py

@@ -11,6 +11,7 @@ from jsonl_diff import (
     DuplicateKeyError,
     DuplicatePolicy,
     InputError,
+    MissingKeyPolicy,
     SchemaChangeOperation,
     SchemaSummary,
     Summary,
@@ -19,6 +20,65 @@ from jsonl_diff import (
 
 
 class TestDiffSummary:
+    def test_json_array_format_compares_array_elements(self, tmp_path):
+        # Arrange
+        old = tmp_path / "old.json"
+        new = tmp_path / "new.json"
+        old.write_text('[{"id":1,"value":"before"}]', encoding="utf-8")
+        new.write_text('[{"id":1,"value":"after"}]', encoding="utf-8")
+
+        # Act
+        with diff(old, new, key="id", format="json") as result:
+            change = next(result.changes())
+
+        # Assert
+        assert change.operation == ChangeOperation.MODIFIED
+
+    def test_json_array_numbers_use_jsonl_numeric_semantics(self, tmp_path):
+        # Arrange
+        old = tmp_path / "old.json"
+        new = tmp_path / "new.json"
+        old.write_text('[{"id":1,"value":1}]', encoding="utf-8")
+        new.write_text('[{"id":1.0,"value":1e0}]', encoding="utf-8")
+
+        # Act
+        with diff(old, new, key="id", format="json") as result:
+            summary = result.summary
+
+        # Assert
+        assert summary == Summary(equal=1, added=0, deleted=0, modified=0)
+
+    def test_json_array_locations_are_one_based_element_ordinals(self, tmp_path):
+        # Arrange
+        old = tmp_path / "old.json"
+        new = tmp_path / "new.json"
+        old.write_text('[{"id":1},{"id":2}]', encoding="utf-8")
+        new.write_text('[{"id":1},{"id":3}]', encoding="utf-8")
+
+        # Act
+        with diff(old, new, key="id", format="json") as result:
+            changes = list(result.changes())
+
+        # Assert
+        assert [(change.operation, change.old_line, change.new_line) for change in changes] == [
+            (ChangeOperation.DELETED, 2, None),
+            (ChangeOperation.ADDED, None, 2),
+        ]
+
+    def test_json_array_duplicate_locations_are_element_ordinals(self, tmp_path):
+        # Arrange
+        old = tmp_path / "old.json"
+        new = tmp_path / "new.json"
+        old.write_text('[{"id":1},{"id":1}]', encoding="utf-8")
+        new.write_text('[{"id":1}]', encoding="utf-8")
+
+        # Act
+        with diff(old, new, key="id", format="json", duplicates="first") as result:
+            duplicate = next(result.duplicates())
+
+        # Assert
+        assert (duplicate.selected_line, duplicate.discarded_line) == (1, 2)
+
     def test_records_in_different_physical_order_are_classified_by_identity(self, write_jsonl):
         # Arrange
         old = write_jsonl(
@@ -71,6 +131,50 @@ class TestDiffSummary:
 
 
 class TestIdentityKeys:
+    @pytest.mark.parametrize(
+        "identity",
+        [
+            {"region": "ES", "customer": 7},
+            ["ES", 7, {"active": True}],
+        ],
+    )
+    def test_structured_identity_values_are_supported(self, write_jsonl, identity):
+        old = write_jsonl("old.jsonl", [{"id": identity, "value": "before"}])
+        new = write_jsonl("new.jsonl", [{"id": identity, "value": "after"}])
+
+        with diff(old, new, key="id") as result:
+            change = next(result.changes())
+
+        assert change.key == (identity,)
+        assert change.operation == ChangeOperation.MODIFIED
+
+    def test_identity_object_member_order_is_ignored(self, write_jsonl):
+        old = write_jsonl("old.jsonl", ['{"id":{"a":1,"b":{"x":2,"y":3}}}'])
+        new = write_jsonl("new.jsonl", ['{"id":{"b":{"y":3,"x":2},"a":1}}'])
+
+        with diff(old, new, key="id") as result:
+            summary = result.summary
+
+        assert summary == Summary(equal=1, added=0, deleted=0, modified=0)
+
+    def test_identity_array_order_is_significant(self, write_jsonl):
+        old = write_jsonl("old.jsonl", ['{"id":[1,2]}'])
+        new = write_jsonl("new.jsonl", ['{"id":[2,1]}'])
+
+        with diff(old, new, key="id") as result:
+            summary = result.summary
+
+        assert summary == Summary(equal=0, added=1, deleted=1, modified=0)
+
+    def test_identity_nested_numbers_use_json_numeric_semantics(self, write_jsonl):
+        old = write_jsonl("old.jsonl", ['{"id":{"version":1,"values":[2.0]}}'])
+        new = write_jsonl("new.jsonl", ['{"id":{"version":1.0,"values":[2e0]}}'])
+
+        with diff(old, new, key="id") as result:
+            summary = result.summary
+
+        assert summary == Summary(equal=1, added=0, deleted=0, modified=0)
+
     def test_composite_key_is_returned_as_typed_tuple(self, write_jsonl):
         # Arrange
         old = write_jsonl("old.jsonl", [{"country": "ES", "customer": 7, "value": "before"}])
@@ -521,8 +625,6 @@ class TestInputValidation:
         [
             ({}, "missing identity field"),
             ({"id": None}, "must be a non-null scalar"),
-            ({"id": []}, "must be a non-null scalar"),
-            ({"id": {}}, "must be a non-null scalar"),
         ],
     )
     def test_invalid_identity_component_is_rejected(self, write_jsonl, record, message):
@@ -566,6 +668,67 @@ class TestInputValidation:
         # Assert
         assert summary == Summary(added=0, deleted=0, equal=1, modified=1)
 
+    def test_missing_identity_field_defaults_to_error(self, write_jsonl):
+        # Arrange
+        old = write_jsonl("old.jsonl", [{"value": "old"}])
+        new = write_jsonl("new.jsonl", [{"id": 1, "value": "new"}])
+
+        # Act / Assert
+        with pytest.raises(InputError, match="missing identity field"):
+            with diff(old, new, key="id"):
+                pass
+
+    def test_missing_key_null_policy_treats_absent_field_as_null(self, write_jsonl):
+        # Arrange: "b" is absent from OLD and explicitly null in NEW; with the
+        # "null" policy both are treated as the same composite identity.
+        old = write_jsonl("old.jsonl", [{"a": 1, "value": "old"}])
+        new = write_jsonl("new.jsonl", [{"a": 1, "b": None, "value": "new"}])
+
+        # Act
+        with diff(old, new, key=("a", "b"), missing_key=MissingKeyPolicy.NULL) as result:
+            change = next(result.changes())
+
+        # Assert
+        assert change.key == (Decimal("1"), None)
+        assert change.operation == ChangeOperation.MODIFIED
+
+    def test_missing_key_null_policy_still_rejects_single_field_identity(self, write_jsonl):
+        # Arrange: a single-field identity may not collapse to null, so the
+        # "null" policy still raises for the missing field, same as an
+        # explicit null value would.
+        old = write_jsonl("old.jsonl", [{"value": "old"}])
+        new = write_jsonl("new.jsonl", [{"id": 1, "value": "new"}])
+
+        # Act / Assert
+        with pytest.raises(InputError, match="must be a non-null scalar"):
+            with diff(old, new, key="id", missing_key="null"):
+                pass
+
+    def test_missing_key_null_policy_can_collide_across_missing_records(self, write_jsonl):
+        # Arrange: two distinct OLD records both missing "b" collapse into one
+        # (1, None) identity under the "null" policy, matching the documented
+        # collision risk of explicit composite nulls.
+        old = write_jsonl(
+            "old.jsonl",
+            [{"a": 1, "value": "first"}, {"a": 1, "value": "second"}],
+        )
+        new = write_jsonl("new.jsonl", [])
+
+        # Act / Assert
+        with pytest.raises(DuplicateKeyError):
+            with diff(old, new, key=("a", "b"), missing_key="null"):
+                pass
+
+    def test_invalid_missing_key_policy_is_rejected(self, write_jsonl):
+        # Arrange
+        old = write_jsonl("old.jsonl", [{"id": 1}])
+        new = write_jsonl("new.jsonl", [{"id": 1}])
+
+        # Act / Assert
+        with pytest.raises(ConfigurationError, match="invalid missing-key policy"):
+            with diff(old, new, key="id", missing_key="bogus"):
+                pass
+
     def test_duplicate_identity_reports_source_and_physical_lines(self, write_jsonl):
         # Arrange
         old = write_jsonl("old.jsonl", [{"id": 1}, {"id": 2}, {"id": 1}])

{jsonl_diff-0.1.3 → jsonl_diff-0.1.5}/tests/test_sources.py

@@ -82,6 +82,83 @@ class TestStreamSources:
         # Assert
         assert summary == Summary(equal=1, added=0, deleted=0, modified=0)
 
+    def test_json_array_file_like_sources_are_supported(self):
+        # Arrange
+        old = io.BytesIO(b'[{"id":1,"value":"same"}]')
+        new = io.BytesIO(b'[{"value":"same","id":1}]')
+
+        # Act
+        with diff(old, new, key="id", format="json") as result:
+            summary = result.summary
+
+        # Assert
+        assert summary == Summary(equal=1, added=0, deleted=0, modified=0)
+        assert not old.closed
+        assert not new.closed
+
+
+class TestJsonArraySources:
+    def test_json_arrays_are_streamed_incrementally(self, tmp_path):
+        # Arrange
+        old = tmp_path / "old.json"
+        new = tmp_path / "new.json"
+        old.write_text('[{"id":1,"value":"same"},{"id":2}]', encoding="utf-8")
+        new.write_text('[{"id":2},{"id":3}]', encoding="utf-8")
+
+        # Act
+        with diff(old, new, key="id", format="json") as result:
+            summary = result.summary
+
+        # Assert
+        assert summary == Summary(equal=1, added=1, deleted=1, modified=0)
+
+    @pytest.mark.parametrize(
+        "suffix,writer",
+        [
+            (".json.gz", _write_gzip),
+            (".json.bz2", _write_bzip2),
+            (".json.xz", _write_xz),
+        ],
+    )
+    def test_compressed_json_arrays_are_supported(self, tmp_path, suffix, writer):
+        # Arrange
+        old = tmp_path / ("old" + suffix)
+        new = tmp_path / ("new" + suffix)
+        writer(old, b'[{"id":1,"value":"same"}]')
+        writer(new, b'[{"value":"same","id":1}]')
+
+        # Act
+        with diff(old, new, key="id", format="json") as result:
+            summary = result.summary
+
+        # Assert
+        assert summary == Summary(equal=1, added=0, deleted=0, modified=0)
+
+    def test_json_format_rejects_non_array_documents(self, tmp_path):
+        # Arrange
+        old = tmp_path / "old.json"
+        new = tmp_path / "new.json"
+        old.write_text('{"id":1}', encoding="utf-8")
+        new.write_text("[]", encoding="utf-8")
+
+        # Act / Assert
+        with pytest.raises(InputError, match="invalid input"):
+            with diff(old, new, key="id", format="json"):
+                pass
+
+    @pytest.mark.parametrize("content", ['[{"id":1}', '[{"id":1}] trailing'])
+    def test_json_format_rejects_malformed_documents(self, tmp_path, content):
+        # Arrange
+        old = tmp_path / "old.json"
+        new = tmp_path / "new.json"
+        old.write_text(content, encoding="utf-8")
+        new.write_text("[]", encoding="utf-8")
+
+        # Act / Assert
+        with pytest.raises(InputError, match="invalid input"):
+            with diff(old, new, key="id", format="json"):
+                pass
+
 
 class _SilentHandler(http.server.SimpleHTTPRequestHandler):
     def log_message(self, message, *args):
@@ -117,6 +194,35 @@ class TestHttpSources:
         # Assert
         assert summary == Summary(equal=1, added=0, deleted=0, modified=0)
 
+    def test_http_json_array_sources_are_supported(self, tmp_path):
+        # Arrange
+        old = tmp_path / "old.json"
+        old.write_text('[{"id":1,"value":"same"}]', encoding="utf-8")
+        new = tmp_path / "new.json"
+        new.write_text('[{"value":"same","id":1}]', encoding="utf-8")
+        handler = functools.partial(_SilentHandler, directory=str(tmp_path))
+        server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
+        thread = threading.Thread(target=server.serve_forever)
+        thread.start()
+
+        # Act
+        try:
+            port = server.server_address[1]
+            with diff(
+                "http://127.0.0.1:{}/old.json".format(port),
+                "http://127.0.0.1:{}/new.json".format(port),
+                key="id",
+                format="json",
+            ) as result:
+                summary = result.summary
+        finally:
+            server.shutdown()
+            server.server_close()
+            thread.join()
+
+        # Assert
+        assert summary == Summary(equal=1, added=0, deleted=0, modified=0)
+
 
 class TestTemporaryStorage:
     @pytest.mark.parametrize("limit", [0, -1, "invalid"])