jsonl-diff 0.1.2__tar.gz → 0.1.3__tar.gz

{jsonl_diff-0.1.2 → jsonl_diff-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: jsonl-diff
-Version: 0.1.2
+Version: 0.1.3
 Summary: A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory.
 Author-email: rmoralespp <rmoralespp@gmail.com>
 License: MIT License
@@ -51,6 +51,8 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: py-jsonl==1.5.2
 Requires-Dist: jmespath>=1.0.0
+Provides-Extra: speedups
+Requires-Dist: msgspec>=0.18; python_version >= "3.10" and extra == "speedups"
 Dynamic: license-file
 
 # jsonl-diff
@@ -63,12 +65,11 @@ Dynamic: license-file
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-Compare large JSONL/NDJSON datasets **by record identity instead of line
+A lightweight, pure-Python tool for comparing large [JSONL](https://jsonlines.org/)/[NDJSON](https://github.com/ndjson/ndjson-spec) datasets **by record identity instead of line
 position**, without loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
-`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each
-source.
+`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
 
 It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
@@ -80,6 +81,19 @@ It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 pip install jsonl-diff
 ```
 
+### Optional speedups
+
+For large datasets, install the optional [`msgspec`](https://jcristharif.com/msgspec/)
+accelerator (Python 3.10+), which uses a C parser and encoder for JSON decoding
+and record canonicalization, speeding up large diffs by roughly **2x**:
+
+```bash
+pip install "jsonl-diff[speedups]"
+```
+
+It is used automatically when available — no configuration or code changes. When
+`msgspec` is absent, the pure-Python path runs instead and produces identical results.
+
 ## Quick start
 
 Given:
@@ -119,11 +133,12 @@ Record order does not matter.
 
 * Single or composite top-level identities.
 * Disk-backed comparison for large datasets ([architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md)).
-* Optional **JMESPath** filtering with `--where`.
+* Optional **[JMESPath](https://jmespath.org/)** filtering with `--where`.
 * Configurable duplicate handling with counts and diagnostics: `error`, `first`, or `last`.
-* Exact **RFC 6901** JSON Pointer ignores.
+* Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
 * Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
+* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
 * Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
@@ -238,7 +253,8 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 `jsonl-diff` is intentionally strict:
 
 * Every input line must contain exactly one valid top-level JSON object.
-* Blank lines, malformed JSON, duplicate properties, `NaN`, and infinities are rejected.
+* Blank lines, malformed JSON, `NaN`, and infinities are rejected.
+* Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
 * Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
 * Duplicate identities fail by default; `--duplicates first`/`last` select one

{jsonl_diff-0.1.2 → jsonl_diff-0.1.3}/README.md

@@ -8,12 +8,11 @@
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-Compare large JSONL/NDJSON datasets **by record identity instead of line
+A lightweight, pure-Python tool for comparing large [JSONL](https://jsonlines.org/)/[NDJSON](https://github.com/ndjson/ndjson-spec) datasets **by record identity instead of line
 position**, without loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
-`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each
-source.
+`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
 
 It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
@@ -25,6 +24,19 @@ It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 pip install jsonl-diff
 ```
 
+### Optional speedups
+
+For large datasets, install the optional [`msgspec`](https://jcristharif.com/msgspec/)
+accelerator (Python 3.10+), which uses a C parser and encoder for JSON decoding
+and record canonicalization, speeding up large diffs by roughly **2x**:
+
+```bash
+pip install "jsonl-diff[speedups]"
+```
+
+It is used automatically when available — no configuration or code changes. When
+`msgspec` is absent, the pure-Python path runs instead and produces identical results.
+
 ## Quick start
 
 Given:
@@ -64,11 +76,12 @@ Record order does not matter.
 
 * Single or composite top-level identities.
 * Disk-backed comparison for large datasets ([architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md)).
-* Optional **JMESPath** filtering with `--where`.
+* Optional **[JMESPath](https://jmespath.org/)** filtering with `--where`.
 * Configurable duplicate handling with counts and diagnostics: `error`, `first`, or `last`.
-* Exact **RFC 6901** JSON Pointer ignores.
+* Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
 * Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
+* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
 * Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
@@ -183,7 +196,8 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 `jsonl-diff` is intentionally strict:
 
 * Every input line must contain exactly one valid top-level JSON object.
-* Blank lines, malformed JSON, duplicate properties, `NaN`, and infinities are rejected.
+* Blank lines, malformed JSON, `NaN`, and infinities are rejected.
+* Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
 * Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
 * Duplicate identities fail by default; `--duplicates first`/`last` select one

{jsonl_diff-0.1.2 → jsonl_diff-0.1.3}/jsonl_diff.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: jsonl-diff
-Version: 0.1.2
+Version: 0.1.3
 Summary: A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory.
 Author-email: rmoralespp <rmoralespp@gmail.com>
 License: MIT License
@@ -51,6 +51,8 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: py-jsonl==1.5.2
 Requires-Dist: jmespath>=1.0.0
+Provides-Extra: speedups
+Requires-Dist: msgspec>=0.18; python_version >= "3.10" and extra == "speedups"
 Dynamic: license-file
 
 # jsonl-diff
@@ -63,12 +65,11 @@ Dynamic: license-file
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-Compare large JSONL/NDJSON datasets **by record identity instead of line
+A lightweight, pure-Python tool for comparing large [JSONL](https://jsonlines.org/)/[NDJSON](https://github.com/ndjson/ndjson-spec) datasets **by record identity instead of line
 position**, without loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
-`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each
-source.
+`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
 
 It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
@@ -80,6 +81,19 @@ It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 pip install jsonl-diff
 ```
 
+### Optional speedups
+
+For large datasets, install the optional [`msgspec`](https://jcristharif.com/msgspec/)
+accelerator (Python 3.10+), which uses a C parser and encoder for JSON decoding
+and record canonicalization, speeding up large diffs by roughly **2x**:
+
+```bash
+pip install "jsonl-diff[speedups]"
+```
+
+It is used automatically when available — no configuration or code changes. When
+`msgspec` is absent, the pure-Python path runs instead and produces identical results.
+
 ## Quick start
 
 Given:
@@ -119,11 +133,12 @@ Record order does not matter.
 
 * Single or composite top-level identities.
 * Disk-backed comparison for large datasets ([architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md)).
-* Optional **JMESPath** filtering with `--where`.
+* Optional **[JMESPath](https://jmespath.org/)** filtering with `--where`.
 * Configurable duplicate handling with counts and diagnostics: `error`, `first`, or `last`.
-* Exact **RFC 6901** JSON Pointer ignores.
+* Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
 * Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
+* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
 * Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
@@ -238,7 +253,8 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 `jsonl-diff` is intentionally strict:
 
 * Every input line must contain exactly one valid top-level JSON object.
-* Blank lines, malformed JSON, duplicate properties, `NaN`, and infinities are rejected.
+* Blank lines, malformed JSON, `NaN`, and infinities are rejected.
+* Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
 * Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
 * Duplicate identities fail by default; `--duplicates first`/`last` select one

jsonl_diff-0.1.3/jsonl_diff.egg-info/requires.txt

@@ -0,0 +1,7 @@
+py-jsonl==1.5.2
+jmespath>=1.0.0
+
+[speedups]
+
+[speedups:python_version >= "3.10"]
+msgspec>=0.18

{jsonl_diff-0.1.2 → jsonl_diff-0.1.3}/jsonl_diff.py

@@ -12,21 +12,48 @@ import warnings
 from dataclasses import dataclass
 from decimal import Decimal
 from enum import Enum
+from json.encoder import encode_basestring as _encode_basestring
+from json.encoder import encode_basestring_ascii as _encode_basestring_ascii
 from pathlib import Path
 from typing import Any, Callable, Dict, Iterable, Iterator, Optional, Sequence, Tuple, Union
 
 import jmespath
 import jsonl
 
+try:  # Optional C-accelerated content canonicalization (Python 3.10+).
+    import msgspec as _msgspec
+except ImportError:
+    _msgspec = None
+
 JsonScalar = Union[str, Decimal, bool, None]
 Number = Union[Decimal, int, float]
 IdentityKey = Tuple[JsonScalar, ...]
 
+
+class _Num(Decimal):
+    """Decimal subclass parsed numbers use so msgspec routes them through enc_hook."""
+
+    __slots__ = ()
+
+
 _MISSING = object()
 
+_DIGITS = ("0", "1", "2", "3", "4", "5", "6", "7", "8", "9")
+
+# Integral digit limit for the msgspec content encoder only. Because msgspec's
+# parser emits integer literals as native `int` and its encoder always writes
+# them out in full, `_Num` float integrals must use the fast plain-integer form
+# over the same range to stay byte-consistent. The bound stays within CPython's
+# default integer-to-string limit so huge float exponents (e.g. 1e50000000) still
+# fall back to scientific form and never materialise as gigantic integers. The
+# pure-Python fallback keeps the lower default limit in `_digest_number`.
+_DIGEST_INTEGRAL_LIMIT = 4300
+
 # Avoid per-record filesystem scans: SQLite already enforces the main size limit.
 # Check periodically and once after each side finishes to catch extra temp/journal growth.
 _SIZE_CHECK_INTERVAL = 1024
+# Records are inserted in batches to amortize per-statement overhead.
+_INSERT_BATCH = 2048
 _SCHEMA_TYPE_INDEXES = {
     "boolean": 3,
     "integer": 4,
@@ -286,22 +313,12 @@ def _reject_constant(value: str) -> None:
     raise ValueError("invalid JSON number {!r}".format(value))
 
 
-def _object(pairs: Iterable[Tuple[str, Any]]) -> Dict[str, Any]:
-    result = {}
-    for name, value in pairs:
-        if name in result:
-            raise ValueError("duplicate property {!r}".format(name))
-        result[name] = value
-    return result
-
-
 def _decode_json(value: str) -> Any:
     return json.loads(
         value,
         parse_int=Decimal,
         parse_float=Decimal,
         parse_constant=_reject_constant,
-        object_pairs_hook=_object,
     )
 
 
@@ -318,17 +335,22 @@ def _number(value: Number) -> str:
         return "0"
 
     sign, digits, exponent = value.as_tuple()
+    count = len(digits)
     trailing_zeros = 0
-    while len(digits) - trailing_zeros > 1 and digits[-trailing_zeros - 1] == 0:
+    while count - trailing_zeros > 1 and digits[count - trailing_zeros - 1] == 0:
         trailing_zeros += 1
-
-    digits = digits[:-trailing_zeros] if trailing_zeros else digits
-    exponent += trailing_zeros
-    mantissa = str(digits[0])
-    if len(digits) > 1:
-        mantissa += "." + "".join(str(digit) for digit in digits[1:])
-
-    scientific_exponent = exponent + len(digits) - 1
+    if trailing_zeros:
+        digits = digits[:count - trailing_zeros]
+        exponent += trailing_zeros
+        count -= trailing_zeros
+
+    if count == 1:
+        mantissa = _DIGITS[digits[0]]
+    else:
+        text = "".join([_DIGITS[digit] for digit in digits])
+        mantissa = text[0] + "." + text[1:]
+
+    scientific_exponent = exponent + count - 1
     return "{}{}e{:+d}".format("-" if sign else "", mantissa, scientific_exponent)
 
 
@@ -340,11 +362,50 @@ def _details_number(value: Number) -> str:
     return str(value).lower()
 
 
+def _digest_number(value: Number, integral_limit: int = 18) -> str:
+    # Formats numbers for the content fingerprint only (hashed, never decoded or
+    # displayed), so it just needs value-consistent bytes. Integrals with fewer
+    # than `integral_limit` digits use the fast plain-integer form; classifying by
+    # value keeps `1`, `1.0` and `1e0` identical. Fractions and larger integrals
+    # fall back to the exact scientific form and are never materialised in full.
+    # The default limit matches the pure-Python fallback; the msgspec encoder
+    # raises it (see `_digest_raw`) so `_Num` integrals agree with the full-digit
+    # bytes msgspec writes for native ints.
+    if (
+        isinstance(value, Decimal)
+        and value.adjusted() < integral_limit
+        and value == value.to_integral_value()
+    ):
+        return str(int(value))
+    return _number(value)
+
+
+def _require_str_key(name: Any) -> bool:
+    if isinstance(name, str):
+        return True
+    raise ValueError("JSON object property names must be strings")
+
+
 def _json_text(
     value: Any,
-    ensure_ascii: bool,
+    escape: Callable[[str], str],
     number: Callable[[Number], str],
 ) -> str:
+    # Checks are ordered by frequency for object-heavy payloads. String escaping
+    # is delegated to CPython's C-accelerated encoder (`escape`) rather than a
+    # per-value `json.dumps`.
+    kind = type(value)
+    if kind is str:
+        return escape(value)
+    if kind is dict:
+        members = [
+            escape(name) + ":" + _json_text(value[name], escape, number)
+            for name in sorted(value)
+            if type(name) is str or _require_str_key(name)
+        ]
+        return "{" + ",".join(members) + "}"
+    if kind is list or kind is tuple:
+        return "[" + ",".join([_json_text(item, escape, number) for item in value]) + "]"
     if value is None:
         return "null"
     if value is True:
@@ -354,35 +415,56 @@ def _json_text(
     if isinstance(value, (Decimal, int, float)):
         return number(value)
     if isinstance(value, str):
-        return json.dumps(value, ensure_ascii=ensure_ascii)
+        return escape(value)
     if isinstance(value, (list, tuple)):
-        return "[{}]".format(",".join(_json_text(item, ensure_ascii, number) for item in value))
+        return "[" + ",".join([_json_text(item, escape, number) for item in value]) + "]"
     if isinstance(value, dict):
-        if any(not isinstance(name, str) for name in value):
-            raise ValueError("JSON object property names must be strings")
-        members = (
-            "{}:{}".format(
-                json.dumps(name, ensure_ascii=ensure_ascii),
-                _json_text(value[name], ensure_ascii, number),
-            )
+        members = [
+            escape(name) + ":" + _json_text(value[name], escape, number)
             for name in sorted(value)
-        )
-        return "{{{}}}".format(",".join(members))
+            if type(name) is str or _require_str_key(name)
+        ]
+        return "{" + ",".join(members) + "}"
     raise ValueError("unsupported JSON value type: {}".format(type(value).__name__))
 
 
 def _canonical_text(value: Any, ensure_ascii: bool = False) -> str:
-    return _json_text(value, ensure_ascii, _number)
+    escape = _encode_basestring_ascii if ensure_ascii else _encode_basestring
+    return _json_text(value, escape, _number)
 
 
 def _details_text(value: Any, ensure_ascii: bool = False) -> str:
-    return _json_text(value, ensure_ascii, _details_number)
+    escape = _encode_basestring_ascii if ensure_ascii else _encode_basestring
+    return _json_text(value, escape, _details_number)
 
 
 def _canonical(value: Any) -> bytes:
     return _canonical_text(value).encode("utf-8")
 
 
+def _content_canonical(value: Any) -> bytes:
+    # Bytes hashed for the content fingerprint. Uses the fast digest-only number
+    # formatter; identity encoding keeps `_canonical` so key notation round-trips.
+    return _json_text(value, _encode_basestring, _digest_number).encode("utf-8")
+
+
+if _msgspec is not None:
+    def _digest_raw(value: Number) -> Any:
+        # enc_hook only fires for `_Num`, msgspec's parser writes integer literals
+        # as native ints and encodes them in full, so `_Num` integrals use the
+        # raised digit limit to produce the same bytes. Tokens are emitted bare
+        # (no quotes) so they never collide with real strings.
+        return _msgspec.Raw(_digest_number(value, _DIGEST_INTEGRAL_LIMIT).encode("utf-8"))
+
+    _CONTENT_ENCODER = _msgspec.json.Encoder(order="sorted", enc_hook=_digest_raw)
+
+    def _content_canonical(value: Any) -> bytes:
+        # C-accelerated equivalent of the pure-Python serializer above. Every
+        # number is a _Num subclass, so enc_hook fires and yields identical
+        # value-consistent bytes; the digest is hashed only, never decoded.
+        return _CONTENT_ENCODER.encode(value)
+
+
 def _fingerprint(canonical: bytes) -> Tuple[int, bytes]:
     return len(canonical), hashlib.sha256(canonical).digest()
 
@@ -506,7 +588,7 @@ def _identity(record: Dict[str, Any], keys: Sequence[str]) -> IdentityKey:
             # record into one indistinguishable identity.
             if not composite:
                 raise ValueError("identity field {!r} must be a non-null scalar".format(name))
-        elif not isinstance(value, (str, Decimal, bool)):
+        elif not isinstance(value, (str, Decimal, bool, int)):
             raise ValueError("identity field {!r} must be a non-null scalar".format(name))
         values.append(value)
     return tuple(values)
@@ -521,15 +603,27 @@ def _compile_where(expression: str) -> Any:
         ) from error
 
 
-def _records(source: Any, on_error: Any) -> Iterator[Any]:
-    yield from jsonl.load(
-        source,
-        parse_int=Decimal,
-        parse_float=Decimal,
-        parse_constant=_reject_constant,
-        object_pairs_hook=_object,
-        _on_error=on_error,
-    )
+if _msgspec is not None:
+    _INPUT_DECODER = _msgspec.json.Decoder(float_hook=_Num)
+
+    def _records(source: Any, on_error: Any) -> Iterator[Any]:
+        # msgspec decodes input lines ~2.6x faster than the stdlib parser and
+        # rejects NaN/Infinity natively. Floats become `_Num` (a Decimal subclass)
+        # while integers stay native `int`; both are handled by the number
+        # canonicalizers. Duplicate object keys follow JSON's last-wins semantics,
+        # matching the pure-Python fallback below.
+        yield from jsonl.load(source, cls=_INPUT_DECODER.decode, _on_error=on_error)
+else:
+    def _records(source: Any, on_error: Any) -> Iterator[Any]:
+        # No object_pairs_hook is installed, so duplicate object keys follow the
+        # stdlib parser's last-wins semantics, matching the msgspec path above.
+        yield from jsonl.load(
+            source,
+            parse_int=Decimal,
+            parse_float=Decimal,
+            parse_constant=_reject_constant,
+            _on_error=on_error,
+        )
 
 
 class DiffResult:
@@ -561,11 +655,17 @@ class DiffResult:
 
     def _configure_database(self) -> None:
         self._connection.execute("PRAGMA journal_mode = OFF")
+        self._connection.execute("PRAGMA synchronous = OFF")
         self._connection.execute("PRAGMA temp_store = FILE")
         if self.config.max_temp is not None:
             pages = max(1, self.config.max_temp // 4096)
             self._connection.execute("PRAGMA page_size = 4096")
             self._connection.execute("PRAGMA max_page_count = {}".format(pages))
+        else:
+            # A larger page cache speeds bulk index writes and the summary read.
+            # Skipped under max_temp so on-disk growth stays observable and the
+            # workspace memory footprint remains bounded.
+            self._connection.execute("PRAGMA cache_size = -65536")
 
     def _create_schema(self) -> None:
         self._connection.execute(
@@ -927,6 +1027,7 @@ class DiffResult:
         cursor = self._connection.cursor()
         schema_fields = {}
         schema_objects = {}
+        pending = []
         for line, record in enumerate(records, start=1):
             if not isinstance(record, dict):
                 raise InputError("each record must be a JSON object", source, line)
@@ -939,6 +1040,8 @@ class DiffResult:
                     if (
                         self.config.schema_diff or self.config.max_temp is not None
                     ) and line % _SIZE_CHECK_INTERVAL == 0:
+                        if pending:
+                            self._flush_record_batch(cursor, pending, source)
                         if self.config.schema_diff:
                             self._flush_schema_profile(
                                 cursor,
@@ -952,7 +1055,7 @@ class DiffResult:
             try:
                 key = _identity(record, self.config.key)
                 normalized = _remove_ignored(record, self._ignore_tree)
-                canonical = _canonical(normalized)
+                canonical = _content_canonical(normalized)
                 identity = _canonical(list(key))
                 length, digest = _fingerprint(canonical)
             except (TypeError, ValueError) as error:
@@ -969,16 +1072,9 @@ class DiffResult:
                     raise InputError(str(error), source, line) from error
             values = (side, identity, line, length, digest)
             if self.config.duplicates == DuplicatePolicy.ERROR:
-                try:
-                    cursor.execute("INSERT INTO records VALUES (?, ?, ?, ?, ?)", values)
-                except sqlite3.IntegrityError as error:
-                    first = cursor.execute(
-                        "SELECT line FROM records WHERE side = ? AND identity = ?",
-                        (side, identity),
-                    ).fetchone()[0]
-                    raise DuplicateKeyError(key, source, (first, line)) from error
-                except sqlite3.DatabaseError as error:
-                    raise ResourceError("could not write the temporary index") from error
+                pending.append(values)
+                if len(pending) >= _INSERT_BATCH:
+                    self._flush_record_batch(cursor, pending, source)
             else:
                 try:
                     self._insert_tolerated_record(cursor, values)
@@ -987,14 +1083,52 @@ class DiffResult:
             if (
                 self.config.schema_diff or self.config.max_temp is not None
             ) and line % _SIZE_CHECK_INTERVAL == 0:
+                if pending:
+                    self._flush_record_batch(cursor, pending, source)
                 if self.config.schema_diff:
                     self._flush_schema_profile(cursor, side, schema_fields, schema_objects)
                 if self.config.max_temp is not None:
                     self._check_size()
+        if pending:
+            self._flush_record_batch(cursor, pending, source)
         if self.config.schema_diff:
             self._flush_schema_profile(cursor, side, schema_fields, schema_objects)
         self._check_size()
 
+    @staticmethod
+    def _flush_record_batch(
+        cursor: sqlite3.Cursor,
+        pending: list,
+        source: str,
+    ) -> None:
+        try:
+            cursor.executemany("INSERT INTO records VALUES (?, ?, ?, ?, ?)", pending)
+        except sqlite3.IntegrityError:
+            # A duplicate identity exists within this batch or against a
+            # previously inserted row. Undo this batch's partial inserts (each
+            # row has a unique physical line, so this can never remove an earlier
+            # row) and replay row by row to report the exact colliding identity
+            # and its first physical line, matching the per-record behavior.
+            cursor.executemany(
+                "DELETE FROM records WHERE side = ? AND identity = ? AND line = ?",
+                [(row[0], row[1], row[2]) for row in pending],
+            )
+            for row in pending:
+                try:
+                    cursor.execute("INSERT INTO records VALUES (?, ?, ?, ?, ?)", row)
+                except sqlite3.IntegrityError as error:
+                    first = cursor.execute(
+                        "SELECT line FROM records WHERE side = ? AND identity = ?",
+                        (row[0], row[1]),
+                    ).fetchone()[0]
+                    key = tuple(_decode_json(row[1].decode("utf-8")))
+                    raise DuplicateKeyError(key, source, (first, row[2])) from error
+            pending.clear()
+        except sqlite3.DatabaseError as error:
+            raise ResourceError("could not write the temporary index") from error
+        else:
+            pending.clear()
+
     @staticmethod
     def _flush_schema_profile(
         cursor: sqlite3.Cursor,
@@ -1066,43 +1200,48 @@ class DiffResult:
             )
 
     def _calculate_summary(self) -> Summary:
+        # Derive the four record totals from a single identity join (matched
+        # pairs, plus how many of them are content-equal) and two per-side range
+        # counts, instead of running the join twice and two anti-joins. Every
+        # access uses the (side, identity) primary key, and the arithmetic below
+        # recovers added/deleted/modified without extra scans.
         row = self._connection.execute(
             """
             SELECT
-                (SELECT COUNT(*)
-                 FROM records AS o JOIN records AS n USING (identity)
-                 WHERE o.side = 0 AND n.side = 1
-                   AND o.length = n.length
-                   AND o.digest = n.digest),
-                (SELECT COUNT(*)
-                 FROM records AS n
-                 WHERE n.side = 1 AND NOT EXISTS (
-                     SELECT 1 FROM records AS o
-                     WHERE o.side = 0 AND o.identity = n.identity
-                 )),
-                (SELECT COUNT(*)
-                 FROM records AS o
-                 WHERE o.side = 0 AND NOT EXISTS (
-                     SELECT 1 FROM records AS n
-                     WHERE n.side = 1 AND n.identity = o.identity
-                 )),
-                (SELECT COUNT(*)
-                 FROM records AS o JOIN records AS n USING (identity)
-                 WHERE o.side = 0 AND n.side = 1
-                   AND (
-                       o.length != n.length
-                       OR o.digest != n.digest
-                   )),
-                (SELECT COUNT(*)
-                 FROM duplicate_records
-                 WHERE side = 0),
-                (SELECT COUNT(*)
-                 FROM duplicate_records
-                 WHERE side = 1)
+                (SELECT COUNT(*) FROM records WHERE side = 0),
+                (SELECT COUNT(*) FROM records WHERE side = 1),
+                matches.matched,
+                matches.equal,
+                (SELECT COUNT(*) FROM duplicate_records WHERE side = 0),
+                (SELECT COUNT(*) FROM duplicate_records WHERE side = 1)
+            FROM (
+                SELECT
+                    COUNT(*) AS matched,
+                    COALESCE(
+                        SUM(o.length = n.length AND o.digest = n.digest), 0
+                    ) AS equal
+                FROM records AS o
+                JOIN records AS n
+                  ON o.side = 0 AND n.side = 1 AND o.identity = n.identity
+            ) AS matches
             """,
         ).fetchone()
+        total_old, total_new, matched, equal, old_dups, new_dups = (
+            int(value or 0) for value in row
+        )
+        added = total_new - matched
+        deleted = total_old - matched
+        modified = matched - equal
         schema = self._calculate_schema_summary() if self.config.schema_diff else None
-        return Summary(*(int(value or 0) for value in row), schema=schema)
+        return Summary(
+            equal,
+            added,
+            deleted,
+            modified,
+            old_dups,
+            new_dups,
+            schema=schema,
+        )
 
     def _calculate_schema_summary(self) -> SchemaSummary:
         counts = dict.fromkeys(SchemaChangeOperation, 0)

{jsonl_diff-0.1.2 → jsonl_diff-0.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "jsonl-diff"
-version = "0.1.2"
+version = "0.1.3"
 description = "A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory."
 readme = "README.md"
 license = { file = "LICENSE" }
@@ -33,6 +33,11 @@ dependencies = [
     "jmespath>=1.0.0",
 ]
 
+[project.optional-dependencies]
+speedups = [
+    "msgspec>=0.18; python_version >= '3.10'",
+]
+
 [project.scripts]
 jsonl-diff = "jsonl_diff:main"
 
@@ -48,6 +53,7 @@ test = [
     "coverage",
     "pytest-cov",
     "setuptools; python_version >= '3.12'",
+    "msgspec>=0.18; python_version >= '3.10'",
 ]
 lint = [
     "ruff",

{jsonl_diff-0.1.2 → jsonl_diff-0.1.3}/tests/test_diff.py

@@ -480,7 +480,6 @@ class TestInputValidation:
             '{"id":NaN}',
             '{"id":Infinity}',
             '{"id":-Infinity}',
-            '{"id":1,"id":2}',
         ],
     )
     def test_invalid_json_is_rejected(self, write_jsonl, record):
@@ -493,6 +492,19 @@ class TestInputValidation:
             with diff(old, new, key="id"):
                 pass
 
+    def test_duplicate_object_keys_follow_last_wins(self, write_jsonl):
+        # Arrange: a repeated property keeps the last occurrence, so both records
+        # collapse to {"id": 2} and compare equal.
+        old = write_jsonl("old.jsonl", ['{"id":1,"id":2}'])
+        new = write_jsonl("new.jsonl", ['{"id":2}'])
+
+        # Act
+        with diff(old, new, key="id") as result:
+            summary = result.summary
+
+        # Assert
+        assert summary == Summary(equal=1, added=0, deleted=0, modified=0)
+
     @pytest.mark.parametrize("record", ["[]", '"record"', "1", "true", "null"])
     def test_non_object_record_is_rejected(self, write_jsonl, record):
         # Arrange
@@ -568,6 +580,28 @@ class TestInputValidation:
         assert captured.value.lines == (1, 3)
         assert captured.value.source == "OLD"
 
+    def test_duplicate_identity_is_detected_across_insert_batches(self, write_jsonl):
+        # A duplicate whose two occurrences are separated by more than one
+        # insert batch must still report the exact first and duplicate physical
+        # lines, exercising the batched-insert cleanup-and-replay path.
+        import jsonl_diff
+
+        span = jsonl_diff._INSERT_BATCH + 500
+        records = [{"id": index + 100_000} for index in range(span)]
+        records[-1] = {"id": 100_000}
+        old = write_jsonl("old.jsonl", records)
+        new = write_jsonl("new.jsonl", [])
+
+        # Act
+        with pytest.raises(DuplicateKeyError) as captured:
+            with diff(old, new, key="id"):
+                pass
+
+        # Assert
+        assert captured.value.key == (Decimal(100_000),)
+        assert captured.value.lines == (1, span)
+        assert captured.value.source == "OLD"
+
     @pytest.mark.parametrize(
         "policy,expected",
         [

jsonl_diff-0.1.2/jsonl_diff.egg-info/requires.txt

@@ -1,2 +0,0 @@
-py-jsonl==1.5.2
-jmespath>=1.0.0