jsonl-diff 0.1.1__tar.gz → 0.1.3__tar.gz

{jsonl_diff-0.1.1 → jsonl_diff-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: jsonl-diff
-Version: 0.1.1
+Version: 0.1.3
 Summary: A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory.
 Author-email: rmoralespp <rmoralespp@gmail.com>
 License: MIT License
@@ -51,6 +51,8 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: py-jsonl==1.5.2
 Requires-Dist: jmespath>=1.0.0
+Provides-Extra: speedups
+Requires-Dist: msgspec>=0.18; python_version >= "3.10" and extra == "speedups"
 Dynamic: license-file
 
 # jsonl-diff
@@ -63,15 +65,11 @@ Dynamic: license-file
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-Compare large JSONL/NDJSON datasets **by record identity instead of line
+A lightweight, pure-Python tool for comparing large [JSONL](https://jsonlines.org/)/[NDJSON](https://github.com/ndjson/ndjson-spec) datasets **by record identity instead of line
 position**, without loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
-
-* `equal`
-* `added`
-* `deleted`
-* `modified`
+`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
 
 It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
@@ -79,12 +77,23 @@ It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
 ## Installation
 
-Python 3.8–3.14.
+```bash
+pip install jsonl-diff
+```
+
+### Optional speedups
+
+For large datasets, install the optional [`msgspec`](https://jcristharif.com/msgspec/)
+accelerator (Python 3.10+), which uses a C parser and encoder for JSON decoding
+and record canonicalization, speeding up large diffs by roughly **2x**:
 
 ```bash
-python -m pip install jsonl-diff
+pip install "jsonl-diff[speedups]"
 ```
 
+It is used automatically when available — no configuration or code changes. When
+`msgspec` is absent, the pure-Python path runs instead and produces identical results.
+
 ## Quick start
 
 Given:
@@ -114,6 +123,8 @@ Records:
   added:     1
   deleted:   0
   modified:  1
+  OLD duplicates:  0
+  NEW duplicates:  0
 ```
 
 Record order does not matter.
@@ -121,14 +132,16 @@ Record order does not matter.
 ## Features
 
 * Single or composite top-level identities.
-* Disk-backed comparison for large datasets ([architecture](docs/architecture.md)).
-* Optional JMESPath filtering with `--where`.
-* Configurable duplicate handling: `error`, `first`, or `last`.
-* Exact RFC 6901 JSON Pointer ignores.
+* Disk-backed comparison for large datasets ([architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md)).
+* Optional **[JMESPath](https://jmespath.org/)** filtering with `--where`.
+* Configurable duplicate handling with counts and diagnostics: `error`, `first`, or `last`.
+* Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
+* Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
+* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
-* Machine-readable JSONL change log with `--details` ([format](docs/details-format.md)).
+* Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
 * Local, HTTP/HTTPS, file-like, and supported compressed sources.
 * CLI and Python API using the same comparison engine.
 
@@ -137,7 +150,7 @@ Record order does not matter.
 ```text
 jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
            [--duplicates {error,first,last}] [--details FILE] [--quiet]
-           [--max-temp MAX_TEMP]
+           [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
            old new
 ```
 
@@ -147,10 +160,12 @@ jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
 | `--key KEY`           | Required top-level identity field; repeat or comma-separate for composite keys |
 | `--ignore POINTER`    | RFC 6901 pointer to exclude from content comparison                            |
 | `--where EXPRESSION`  | JMESPath filter applied to each record                                         |
-| `--duplicates POLICY` | `error` (default), `first`, or `last`                                          |
+| `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`          |
 | `--details FILE`      | Write deterministic machine-readable JSONL changes                             |
+| `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                   |
+| `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                      |
 | `--quiet`             | Suppress the normal summary                                                    |
-| `--max-temp BYTES`    | Limit `jsonl-diff` temporary storage                                           |
+| `--max-temp BYTES`    | Best-effort budget for `jsonl-diff` workspace temporary storage                |
 
 Examples:
 
@@ -169,18 +184,24 @@ jsonl-diff old.jsonl new.jsonl \
   --key id \
   --where 'deleted_at == `null`'
 
+# Detect observed schema drift
+jsonl-diff old.jsonl new.jsonl \
+  --key id \
+  --schema-diff \
+  --schema-ignore /metadata
+
 # Compressed input
 jsonl-diff old.jsonl.xz new.jsonl.gz --key id
 ```
 
 ### Exit codes
 
-| Code | Meaning                                   |
-| ---: | ----------------------------------------- |
-|  `0` | Inputs are equal                          |
-|  `1` | Added, deleted, or modified records found |
-|  `2` | Input, resource, output, or runtime error |
-|  `3` | Invalid CLI configuration or usage        |
+| Code | Meaning                                                    |
+| ---: | ---------------------------------------------------------- |
+|  `0` | No record, duplicate, or requested observed-schema issues   |
+|  `1` | Record differences, tolerated duplicates, or schema changes |
+|  `2` | Input, resource, output, or runtime error                  |
+|  `3` | Invalid CLI configuration or usage                         |
 
 ## Python API
 
@@ -193,11 +214,15 @@ with diff(
     key=("country", "customerId"),
     ignore=("/updated_at",),
     where='country == `"ES"`',
+    schema_diff=True,
 ) as result:
     print(result.summary)
 
     for change in result.changes(ChangeOperation.MODIFIED):
         print(change.key, change.old_line, change.new_line)
+
+    for change in result.schema_changes():
+        print(change.operation, change.path)
 ```
 
 `diff()` returns a disk-backed `DiffResult`, used as a context manager.
@@ -220,7 +245,7 @@ The public exception hierarchy is rooted at `JsonlDiffError`:
 * `DuplicateKeyError`
 * `ResourceError`
 
-See the [Python API reference](docs/python-api.md) for the full callable
+See the [Python API reference](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/python-api.md) for the full callable
 signature, result models, `Decimal` key semantics, and error hierarchy.
 
 ## Comparison semantics
@@ -228,18 +253,22 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 `jsonl-diff` is intentionally strict:
 
 * Every input line must contain exactly one valid top-level JSON object.
-* Blank lines, malformed JSON, duplicate properties, `NaN`, and infinities are rejected.
+* Blank lines, malformed JSON, `NaN`, and infinities are rejected.
+* Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
 * Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
-* Duplicate identities fail by default; `--duplicates first`/`last` tolerate them.
+* Duplicate identities fail by default; `--duplicates first`/`last` select one
+  occurrence, report every discarded occurrence, and return exit code `1`.
 * Object property order is ignored; array order is significant.
 * Numbers are compared by mathematical value: `1`, `1.0`, and `1e0` are equal.
 * Unicode strings are compared without normalization.
 * Changes are reported in deterministic identity order.
 
 `--where` selects which records participate; `--key` defines identity;
-`--ignore` removes fields from content comparison. See
-[Comparison semantics](docs/comparison-semantics.md) for the full rules,
+`--ignore` removes fields from content comparison. With `--schema-diff`,
+`--schema-ignore` independently removes fields from observed-schema profiling.
+See
+[Comparison semantics](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/comparison-semantics.md) for the full rules,
 including duplicate handling, ignore-pointer edge cases, `--where` evaluation
 order, and canonical number formatting.
 
@@ -255,7 +284,7 @@ Supported compression:
 * xz
 * Zstandard on Python 3.14
 
-See [architecture](docs/architecture.md) for the full source/compression
+See [architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) for the full source/compression
 support matrix and how sources are delegated to `py-jsonl`.
 
 ## Limitations
@@ -268,7 +297,7 @@ support matrix and how sources are delegated to `py-jsonl`.
 * nested identities or automatic key detection
 * fuzzy matching or numeric tolerances
 * unordered-array comparison
-* schema validation or input repair
+* validation against a declared schema or input repair
 * ZIP, database, or cloud-provider inputs
 * GUI or HTML reports
 
@@ -276,10 +305,10 @@ It is a **dataset reconciliation tool**, not a general-purpose visual JSON diff.
 
 ## Further reading
 
-* [Comparison semantics](docs/comparison-semantics.md) — identity, duplicates, ignores, `--where`, canonical numbers, determinism.
-* [Disk-backed architecture](docs/architecture.md) — SQLite index, `max_temp`, cleanup, sources and compression.
-* [Details JSONL format](docs/details-format.md) — machine-readable `--details` output schema.
-* [Python API reference](docs/python-api.md) — full signature, result models, and error hierarchy.
+* [Comparison semantics](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/comparison-semantics.md) — identity, duplicates, ignores, `--where`, canonical numbers, determinism.
+* [Disk-backed architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) — SQLite index, `max_temp`, cleanup, sources and compression.
+* [Details JSONL format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md) — machine-readable `--details` output schema.
+* [Python API reference](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/python-api.md) — full signature, result models, and error hierarchy.
 
 ## Development
 

{jsonl_diff-0.1.1 → jsonl_diff-0.1.3}/README.md

@@ -8,15 +8,11 @@
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-Compare large JSONL/NDJSON datasets **by record identity instead of line
+A lightweight, pure-Python tool for comparing large [JSONL](https://jsonlines.org/)/[NDJSON](https://github.com/ndjson/ndjson-spec) datasets **by record identity instead of line
 position**, without loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
-
-* `equal`
-* `added`
-* `deleted`
-* `modified`
+`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
 
 It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
@@ -24,12 +20,23 @@ It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
 ## Installation
 
-Python 3.8–3.14.
+```bash
+pip install jsonl-diff
+```
+
+### Optional speedups
+
+For large datasets, install the optional [`msgspec`](https://jcristharif.com/msgspec/)
+accelerator (Python 3.10+), which uses a C parser and encoder for JSON decoding
+and record canonicalization, speeding up large diffs by roughly **2x**:
 
 ```bash
-python -m pip install jsonl-diff
+pip install "jsonl-diff[speedups]"
 ```
 
+It is used automatically when available — no configuration or code changes. When
+`msgspec` is absent, the pure-Python path runs instead and produces identical results.
+
 ## Quick start
 
 Given:
@@ -59,6 +66,8 @@ Records:
   added:     1
   deleted:   0
   modified:  1
+  OLD duplicates:  0
+  NEW duplicates:  0
 ```
 
 Record order does not matter.
@@ -66,14 +75,16 @@ Record order does not matter.
 ## Features
 
 * Single or composite top-level identities.
-* Disk-backed comparison for large datasets ([architecture](docs/architecture.md)).
-* Optional JMESPath filtering with `--where`.
-* Configurable duplicate handling: `error`, `first`, or `last`.
-* Exact RFC 6901 JSON Pointer ignores.
+* Disk-backed comparison for large datasets ([architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md)).
+* Optional **[JMESPath](https://jmespath.org/)** filtering with `--where`.
+* Configurable duplicate handling with counts and diagnostics: `error`, `first`, or `last`.
+* Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
+* Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
+* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
-* Machine-readable JSONL change log with `--details` ([format](docs/details-format.md)).
+* Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
 * Local, HTTP/HTTPS, file-like, and supported compressed sources.
 * CLI and Python API using the same comparison engine.
 
@@ -82,7 +93,7 @@ Record order does not matter.
 ```text
 jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
            [--duplicates {error,first,last}] [--details FILE] [--quiet]
-           [--max-temp MAX_TEMP]
+           [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
            old new
 ```
 
@@ -92,10 +103,12 @@ jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
 | `--key KEY`           | Required top-level identity field; repeat or comma-separate for composite keys |
 | `--ignore POINTER`    | RFC 6901 pointer to exclude from content comparison                            |
 | `--where EXPRESSION`  | JMESPath filter applied to each record                                         |
-| `--duplicates POLICY` | `error` (default), `first`, or `last`                                          |
+| `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`          |
 | `--details FILE`      | Write deterministic machine-readable JSONL changes                             |
+| `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                   |
+| `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                      |
 | `--quiet`             | Suppress the normal summary                                                    |
-| `--max-temp BYTES`    | Limit `jsonl-diff` temporary storage                                           |
+| `--max-temp BYTES`    | Best-effort budget for `jsonl-diff` workspace temporary storage                |
 
 Examples:
 
@@ -114,18 +127,24 @@ jsonl-diff old.jsonl new.jsonl \
   --key id \
   --where 'deleted_at == `null`'
 
+# Detect observed schema drift
+jsonl-diff old.jsonl new.jsonl \
+  --key id \
+  --schema-diff \
+  --schema-ignore /metadata
+
 # Compressed input
 jsonl-diff old.jsonl.xz new.jsonl.gz --key id
 ```
 
 ### Exit codes
 
-| Code | Meaning                                   |
-| ---: | ----------------------------------------- |
-|  `0` | Inputs are equal                          |
-|  `1` | Added, deleted, or modified records found |
-|  `2` | Input, resource, output, or runtime error |
-|  `3` | Invalid CLI configuration or usage        |
+| Code | Meaning                                                    |
+| ---: | ---------------------------------------------------------- |
+|  `0` | No record, duplicate, or requested observed-schema issues   |
+|  `1` | Record differences, tolerated duplicates, or schema changes |
+|  `2` | Input, resource, output, or runtime error                  |
+|  `3` | Invalid CLI configuration or usage                         |
 
 ## Python API
 
@@ -138,11 +157,15 @@ with diff(
     key=("country", "customerId"),
     ignore=("/updated_at",),
     where='country == `"ES"`',
+    schema_diff=True,
 ) as result:
     print(result.summary)
 
     for change in result.changes(ChangeOperation.MODIFIED):
         print(change.key, change.old_line, change.new_line)
+
+    for change in result.schema_changes():
+        print(change.operation, change.path)
 ```
 
 `diff()` returns a disk-backed `DiffResult`, used as a context manager.
@@ -165,7 +188,7 @@ The public exception hierarchy is rooted at `JsonlDiffError`:
 * `DuplicateKeyError`
 * `ResourceError`
 
-See the [Python API reference](docs/python-api.md) for the full callable
+See the [Python API reference](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/python-api.md) for the full callable
 signature, result models, `Decimal` key semantics, and error hierarchy.
 
 ## Comparison semantics
@@ -173,18 +196,22 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 `jsonl-diff` is intentionally strict:
 
 * Every input line must contain exactly one valid top-level JSON object.
-* Blank lines, malformed JSON, duplicate properties, `NaN`, and infinities are rejected.
+* Blank lines, malformed JSON, `NaN`, and infinities are rejected.
+* Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
 * Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
-* Duplicate identities fail by default; `--duplicates first`/`last` tolerate them.
+* Duplicate identities fail by default; `--duplicates first`/`last` select one
+  occurrence, report every discarded occurrence, and return exit code `1`.
 * Object property order is ignored; array order is significant.
 * Numbers are compared by mathematical value: `1`, `1.0`, and `1e0` are equal.
 * Unicode strings are compared without normalization.
 * Changes are reported in deterministic identity order.
 
 `--where` selects which records participate; `--key` defines identity;
-`--ignore` removes fields from content comparison. See
-[Comparison semantics](docs/comparison-semantics.md) for the full rules,
+`--ignore` removes fields from content comparison. With `--schema-diff`,
+`--schema-ignore` independently removes fields from observed-schema profiling.
+See
+[Comparison semantics](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/comparison-semantics.md) for the full rules,
 including duplicate handling, ignore-pointer edge cases, `--where` evaluation
 order, and canonical number formatting.
 
@@ -200,7 +227,7 @@ Supported compression:
 * xz
 * Zstandard on Python 3.14
 
-See [architecture](docs/architecture.md) for the full source/compression
+See [architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) for the full source/compression
 support matrix and how sources are delegated to `py-jsonl`.
 
 ## Limitations
@@ -213,7 +240,7 @@ support matrix and how sources are delegated to `py-jsonl`.
 * nested identities or automatic key detection
 * fuzzy matching or numeric tolerances
 * unordered-array comparison
-* schema validation or input repair
+* validation against a declared schema or input repair
 * ZIP, database, or cloud-provider inputs
 * GUI or HTML reports
 
@@ -221,10 +248,10 @@ It is a **dataset reconciliation tool**, not a general-purpose visual JSON diff.
 
 ## Further reading
 
-* [Comparison semantics](docs/comparison-semantics.md) — identity, duplicates, ignores, `--where`, canonical numbers, determinism.
-* [Disk-backed architecture](docs/architecture.md) — SQLite index, `max_temp`, cleanup, sources and compression.
-* [Details JSONL format](docs/details-format.md) — machine-readable `--details` output schema.
-* [Python API reference](docs/python-api.md) — full signature, result models, and error hierarchy.
+* [Comparison semantics](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/comparison-semantics.md) — identity, duplicates, ignores, `--where`, canonical numbers, determinism.
+* [Disk-backed architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) — SQLite index, `max_temp`, cleanup, sources and compression.
+* [Details JSONL format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md) — machine-readable `--details` output schema.
+* [Python API reference](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/python-api.md) — full signature, result models, and error hierarchy.
 
 ## Development
 

{jsonl_diff-0.1.1 → jsonl_diff-0.1.3}/jsonl_diff.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: jsonl-diff
-Version: 0.1.1
+Version: 0.1.3
 Summary: A lightweight, pure-Python tool for comparing large JSONL/NDJSON datasets without loading them entirely into memory.
 Author-email: rmoralespp <rmoralespp@gmail.com>
 License: MIT License
@@ -51,6 +51,8 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: py-jsonl==1.5.2
 Requires-Dist: jmespath>=1.0.0
+Provides-Extra: speedups
+Requires-Dist: msgspec>=0.18; python_version >= "3.10" and extra == "speedups"
 Dynamic: license-file
 
 # jsonl-diff
@@ -63,15 +65,11 @@ Dynamic: license-file
   <img src="https://img.shields.io/github/license/rmoralespp/jsonl-diff.svg" alt="License">
 </p>
 
-Compare large JSONL/NDJSON datasets **by record identity instead of line
+A lightweight, pure-Python tool for comparing large [JSONL](https://jsonlines.org/)/[NDJSON](https://github.com/ndjson/ndjson-spec) datasets **by record identity instead of line
 position**, without loading the complete inputs into memory.
 
 `jsonl-diff` matches records using one or more top-level fields and reports:
-
-* `equal`
-* `added`
-* `deleted`
-* `modified`
+`equal`, `added`, `deleted`, `modified`, and tolerated duplicates in each source.
 
 It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
@@ -79,12 +77,23 @@ It is useful for snapshots, ETL validation, migrations, exports, and CI checks.
 
 ## Installation
 
-Python 3.8–3.14.
+```bash
+pip install jsonl-diff
+```
+
+### Optional speedups
+
+For large datasets, install the optional [`msgspec`](https://jcristharif.com/msgspec/)
+accelerator (Python 3.10+), which uses a C parser and encoder for JSON decoding
+and record canonicalization, speeding up large diffs by roughly **2x**:
 
 ```bash
-python -m pip install jsonl-diff
+pip install "jsonl-diff[speedups]"
 ```
 
+It is used automatically when available — no configuration or code changes. When
+`msgspec` is absent, the pure-Python path runs instead and produces identical results.
+
 ## Quick start
 
 Given:
@@ -114,6 +123,8 @@ Records:
   added:     1
   deleted:   0
   modified:  1
+  OLD duplicates:  0
+  NEW duplicates:  0
 ```
 
 Record order does not matter.
@@ -121,14 +132,16 @@ Record order does not matter.
 ## Features
 
 * Single or composite top-level identities.
-* Disk-backed comparison for large datasets ([architecture](docs/architecture.md)).
-* Optional JMESPath filtering with `--where`.
-* Configurable duplicate handling: `error`, `first`, or `last`.
-* Exact RFC 6901 JSON Pointer ignores.
+* Disk-backed comparison for large datasets ([architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md)).
+* Optional **[JMESPath](https://jmespath.org/)** filtering with `--where`.
+* Configurable duplicate handling with counts and diagnostics: `error`, `first`, or `last`.
+* Exact **[RFC 6901](https://www.rfc-editor.org/info/rfc6901/)** JSON Pointer ignores.
+* Optional disk-backed observed-schema diff for fields, types, nullability, and requiredness.
 * Semantic number comparison using `Decimal`.
+* Optional `msgspec` accelerator for ~2x faster large diffs (`pip install "jsonl-diff[speedups]"`).
 * Deterministic summaries and change iteration.
 * Original OLD/NEW physical line numbers.
-* Machine-readable JSONL change log with `--details` ([format](docs/details-format.md)).
+* Machine-readable JSONL change log with `--details` ([format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md)).
 * Local, HTTP/HTTPS, file-like, and supported compressed sources.
 * CLI and Python API using the same comparison engine.
 
@@ -137,7 +150,7 @@ Record order does not matter.
 ```text
 jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
            [--duplicates {error,first,last}] [--details FILE] [--quiet]
-           [--max-temp MAX_TEMP]
+           [--schema-diff] [--schema-ignore POINTER] [--max-temp MAX_TEMP]
            old new
 ```
 
@@ -147,10 +160,12 @@ jsonl-diff [-h] --key KEY [--ignore IGNORE] [--where EXPRESSION]
 | `--key KEY`           | Required top-level identity field; repeat or comma-separate for composite keys |
 | `--ignore POINTER`    | RFC 6901 pointer to exclude from content comparison                            |
 | `--where EXPRESSION`  | JMESPath filter applied to each record                                         |
-| `--duplicates POLICY` | `error` (default), `first`, or `last`                                          |
+| `--duplicates POLICY` | `error` (default), or select and report duplicates with `first`/`last`          |
 | `--details FILE`      | Write deterministic machine-readable JSONL changes                             |
+| `--schema-diff`       | Compare observed fields, types, nullability, and requiredness                   |
+| `--schema-ignore`     | RFC 6901 pointer to exclude from observed-schema profiling                      |
 | `--quiet`             | Suppress the normal summary                                                    |
-| `--max-temp BYTES`    | Limit `jsonl-diff` temporary storage                                           |
+| `--max-temp BYTES`    | Best-effort budget for `jsonl-diff` workspace temporary storage                |
 
 Examples:
 
@@ -169,18 +184,24 @@ jsonl-diff old.jsonl new.jsonl \
   --key id \
   --where 'deleted_at == `null`'
 
+# Detect observed schema drift
+jsonl-diff old.jsonl new.jsonl \
+  --key id \
+  --schema-diff \
+  --schema-ignore /metadata
+
 # Compressed input
 jsonl-diff old.jsonl.xz new.jsonl.gz --key id
 ```
 
 ### Exit codes
 
-| Code | Meaning                                   |
-| ---: | ----------------------------------------- |
-|  `0` | Inputs are equal                          |
-|  `1` | Added, deleted, or modified records found |
-|  `2` | Input, resource, output, or runtime error |
-|  `3` | Invalid CLI configuration or usage        |
+| Code | Meaning                                                    |
+| ---: | ---------------------------------------------------------- |
+|  `0` | No record, duplicate, or requested observed-schema issues   |
+|  `1` | Record differences, tolerated duplicates, or schema changes |
+|  `2` | Input, resource, output, or runtime error                  |
+|  `3` | Invalid CLI configuration or usage                         |
 
 ## Python API
 
@@ -193,11 +214,15 @@ with diff(
     key=("country", "customerId"),
     ignore=("/updated_at",),
     where='country == `"ES"`',
+    schema_diff=True,
 ) as result:
     print(result.summary)
 
     for change in result.changes(ChangeOperation.MODIFIED):
         print(change.key, change.old_line, change.new_line)
+
+    for change in result.schema_changes():
+        print(change.operation, change.path)
 ```
 
 `diff()` returns a disk-backed `DiffResult`, used as a context manager.
@@ -220,7 +245,7 @@ The public exception hierarchy is rooted at `JsonlDiffError`:
 * `DuplicateKeyError`
 * `ResourceError`
 
-See the [Python API reference](docs/python-api.md) for the full callable
+See the [Python API reference](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/python-api.md) for the full callable
 signature, result models, `Decimal` key semantics, and error hierarchy.
 
 ## Comparison semantics
@@ -228,18 +253,22 @@ signature, result models, `Decimal` key semantics, and error hierarchy.
 `jsonl-diff` is intentionally strict:
 
 * Every input line must contain exactly one valid top-level JSON object.
-* Blank lines, malformed JSON, duplicate properties, `NaN`, and infinities are rejected.
+* Blank lines, malformed JSON, `NaN`, and infinities are rejected.
+* Duplicate object property names within a record follow JSON's last-wins semantics: the last occurrence is kept.
 * Identity fields must be top-level scalar values (`null` allowed only as one component of a composite key).
 * Identity types remain significant: `"1"` ≠ `1`, `true` ≠ `1`.
-* Duplicate identities fail by default; `--duplicates first`/`last` tolerate them.
+* Duplicate identities fail by default; `--duplicates first`/`last` select one
+  occurrence, report every discarded occurrence, and return exit code `1`.
 * Object property order is ignored; array order is significant.
 * Numbers are compared by mathematical value: `1`, `1.0`, and `1e0` are equal.
 * Unicode strings are compared without normalization.
 * Changes are reported in deterministic identity order.
 
 `--where` selects which records participate; `--key` defines identity;
-`--ignore` removes fields from content comparison. See
-[Comparison semantics](docs/comparison-semantics.md) for the full rules,
+`--ignore` removes fields from content comparison. With `--schema-diff`,
+`--schema-ignore` independently removes fields from observed-schema profiling.
+See
+[Comparison semantics](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/comparison-semantics.md) for the full rules,
 including duplicate handling, ignore-pointer edge cases, `--where` evaluation
 order, and canonical number formatting.
 
@@ -255,7 +284,7 @@ Supported compression:
 * xz
 * Zstandard on Python 3.14
 
-See [architecture](docs/architecture.md) for the full source/compression
+See [architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) for the full source/compression
 support matrix and how sources are delegated to `py-jsonl`.
 
 ## Limitations
@@ -268,7 +297,7 @@ support matrix and how sources are delegated to `py-jsonl`.
 * nested identities or automatic key detection
 * fuzzy matching or numeric tolerances
 * unordered-array comparison
-* schema validation or input repair
+* validation against a declared schema or input repair
 * ZIP, database, or cloud-provider inputs
 * GUI or HTML reports
 
@@ -276,10 +305,10 @@ It is a **dataset reconciliation tool**, not a general-purpose visual JSON diff.
 
 ## Further reading
 
-* [Comparison semantics](docs/comparison-semantics.md) — identity, duplicates, ignores, `--where`, canonical numbers, determinism.
-* [Disk-backed architecture](docs/architecture.md) — SQLite index, `max_temp`, cleanup, sources and compression.
-* [Details JSONL format](docs/details-format.md) — machine-readable `--details` output schema.
-* [Python API reference](docs/python-api.md) — full signature, result models, and error hierarchy.
+* [Comparison semantics](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/comparison-semantics.md) — identity, duplicates, ignores, `--where`, canonical numbers, determinism.
+* [Disk-backed architecture](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/architecture.md) — SQLite index, `max_temp`, cleanup, sources and compression.
+* [Details JSONL format](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/details-format.md) — machine-readable `--details` output schema.
+* [Python API reference](https://github.com/rmoralespp/jsonl-diff/blob/main/docs/python-api.md) — full signature, result models, and error hierarchy.
 
 ## Development