bytesense 1.1.0__tar.gz → 1.2.0__tar.gz

{bytesense-1.1.0 → bytesense-1.2.0}/CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## 1.2.0
+
+### UTF-8 validation
+- Validate complete UTF-8 and UTF-8-SIG inputs in `from_bytes()` with SIMD in native builds, without allocating decoded Unicode. Pin `simdutf8` 0.1.5 and preserve strict rejection and first-error offsets with its compatibility validator.
+- Retain runtime x86 CPU dispatch and scalar fallbacks; no target-specific CPU flags or Python runtime dependencies are required.
+- Decode pure-Python UTF-8 inputs larger than 64 KiB in bounded blocks. Preserve the existing threshold for other codecs, including custom decoders.
+- Reuse the ASCII check within each detection call. Public results, scoring and stream behavior are unchanged.
+
+### Verification
+- Add strict-decoder parity, malformed-scalar, SIMD/block boundary, allocation and custom-codec regressions.
+- Extend rotating-input benchmarks with 64 KiB and 8 MiB UTF-8 workloads and isolated before/after comparisons.
+- Preserve existing corpus accuracy and native/Python predictions; include the native dependency's license in distributions.
+
+See the benchmark documentation for measured gains and their scope.
+
 ## 1.1.0
 
 ### Faster legacy detection

{bytesense-1.1.0 → bytesense-1.2.0}/MANIFEST.in

@@ -1,4 +1,4 @@
-include LICENSE README.md CHANGELOG.md
+include LICENSE THIRD_PARTY_LICENSES README.md CHANGELOG.md
 recursive-include rust *.toml *.lock *.rs
 recursive-include src/bytesense *.py py.typed
 prune rust/target

{bytesense-1.1.0/src/bytesense.egg-info → bytesense-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bytesense
-Version: 1.1.0
+Version: 1.2.0
 Summary: Encoding detection with full-input validation, bounded-memory streams, and optional Rust acceleration.
 Author-email: Oğuzhan Kır <oguzhankir17@gmail.com>
 Maintainer-email: Oğuzhan Kır <oguzhankir17@gmail.com>
@@ -33,6 +33,7 @@ Classifier: Typing :: Typed
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
+License-File: THIRD_PARTY_LICENSES
 Provides-Extra: fast
 Provides-Extra: docs
 Requires-Dist: mkdocs>=1.6; extra == "docs"
@@ -131,13 +132,21 @@ cat report.csv | bytesense --minimal -
 
 The CLI exits **0** when every input has an encoding, **1** for read failures or unknown/binary inputs, and **2** for invalid arguments. JSON goes to stdout; errors go to stderr.
 
+## Used by
+
+[ingest-sentry](https://github.com/oguzhankir/ingest-sentry) uses bytesense for encoding
+detection and full-snapshot validation before CSV/TSV imports. It adds import contracts,
+safe UTF-8 normalization, string-preserving pandas/Polars adapters, and a GitHub Action.
+Built by the same maintainer; an independent project, not affiliated with Sentry.
+
 ## Measured accuracy and performance
 
 The v1 engine replaces the previous collection of special-case ranking rules with a small, reproducibly generated statistical model. The evaluation separates development documents from held-out documents, grouping identical decoded text across encodings. Accuracy means **exact Unicode equality**, not just a compatible codec name.
 
 | Detector | Exact Unicode matches | Accuracy |
 |---|---:|---:|
-| **bytesense 1.1.0** | **1907 / 2078** | **91.77%** |
+| **bytesense 1.2.0** | **1907 / 2078** | **91.77%** |
+| bytesense 1.1.0 | 1907 / 2078 | 91.77% |
 | bytesense 1.0.0 | 1906 / 2078 | 91.72% |
 | bytesense 0.1.2 | 883 / 2078 | 42.49% |
 | chardet 7.6.0 | 2060 / 2078 | 99.13% |
@@ -146,9 +155,9 @@ The v1 engine replaces the previous collection of special-case ranking rules wit
 
 See [benchmarks and methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for measured comparisons with chardet, charset-normalizer and chardetng, including the cases where another detector wins. Timing reports distinguish default behavior from an additional full-decode validation step.
 
-**New in 1.1:** faster legacy scoring, canonical-Unicode evidence, and deeper full-input validation. On a new 432-case transfer check built from 28 UDHR translations, exact recovery increased from **374 to 394 cases**; charset-normalizer recovered 381 and chardet 422. These are correlated variants of one translated document, not a universal accuracy ranking. Chardet remains more accurate on both reported corpora.
+**New in 1.2:** `from_bytes()` validates UTF-8 using native SIMD without allocating decoded Unicode; its pure-Python path uses 64 KiB blocks above 64 KiB. On the measured 1 MiB UTF-8 workload, native `from_bytes()` median latency decreased from **1.564 ms to 0.294 ms (5.3×)** versus 1.1, with every byte still validated. Peak temporary Python allocations decreased from about **6 MiB to 6 KiB**; this is not total process memory.
 
-The measured Turkish CP1254 and Japanese EUC-JP workloads run approximately **2.8× faster than bytesense 1.0** with the native backend. See the benchmark table for absolute timings and where competitors remain faster.
+The existing 2,078-file and 432-case transfer results are unchanged. This release improves UTF-8 validation cost, not general encoding accuracy; chardet remains more accurate on both corpora. See the [benchmark methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for paired measurements, competitors, backend scope and reproduction commands.
 
 ## Upgrading from 0.x
 

{bytesense-1.1.0 → bytesense-1.2.0}/README.md

@@ -74,13 +74,21 @@ cat report.csv | bytesense --minimal -
 
 The CLI exits **0** when every input has an encoding, **1** for read failures or unknown/binary inputs, and **2** for invalid arguments. JSON goes to stdout; errors go to stderr.
 
+## Used by
+
+[ingest-sentry](https://github.com/oguzhankir/ingest-sentry) uses bytesense for encoding
+detection and full-snapshot validation before CSV/TSV imports. It adds import contracts,
+safe UTF-8 normalization, string-preserving pandas/Polars adapters, and a GitHub Action.
+Built by the same maintainer; an independent project, not affiliated with Sentry.
+
 ## Measured accuracy and performance
 
 The v1 engine replaces the previous collection of special-case ranking rules with a small, reproducibly generated statistical model. The evaluation separates development documents from held-out documents, grouping identical decoded text across encodings. Accuracy means **exact Unicode equality**, not just a compatible codec name.
 
 | Detector | Exact Unicode matches | Accuracy |
 |---|---:|---:|
-| **bytesense 1.1.0** | **1907 / 2078** | **91.77%** |
+| **bytesense 1.2.0** | **1907 / 2078** | **91.77%** |
+| bytesense 1.1.0 | 1907 / 2078 | 91.77% |
 | bytesense 1.0.0 | 1906 / 2078 | 91.72% |
 | bytesense 0.1.2 | 883 / 2078 | 42.49% |
 | chardet 7.6.0 | 2060 / 2078 | 99.13% |
@@ -89,9 +97,9 @@ The v1 engine replaces the previous collection of special-case ranking rules wit
 
 See [benchmarks and methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for measured comparisons with chardet, charset-normalizer and chardetng, including the cases where another detector wins. Timing reports distinguish default behavior from an additional full-decode validation step.
 
-**New in 1.1:** faster legacy scoring, canonical-Unicode evidence, and deeper full-input validation. On a new 432-case transfer check built from 28 UDHR translations, exact recovery increased from **374 to 394 cases**; charset-normalizer recovered 381 and chardet 422. These are correlated variants of one translated document, not a universal accuracy ranking. Chardet remains more accurate on both reported corpora.
+**New in 1.2:** `from_bytes()` validates UTF-8 using native SIMD without allocating decoded Unicode; its pure-Python path uses 64 KiB blocks above 64 KiB. On the measured 1 MiB UTF-8 workload, native `from_bytes()` median latency decreased from **1.564 ms to 0.294 ms (5.3×)** versus 1.1, with every byte still validated. Peak temporary Python allocations decreased from about **6 MiB to 6 KiB**; this is not total process memory.
 
-The measured Turkish CP1254 and Japanese EUC-JP workloads run approximately **2.8× faster than bytesense 1.0** with the native backend. See the benchmark table for absolute timings and where competitors remain faster.
+The existing 2,078-file and 432-case transfer results are unchanged. This release improves UTF-8 validation cost, not general encoding accuracy; chardet remains more accurate on both corpora. See the [benchmark methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for paired measurements, competitors, backend scope and reproduction commands.
 
 ## Upgrading from 0.x
 

bytesense-1.2.0/THIRD_PARTY_LICENSES

@@ -0,0 +1,23 @@
+The optional native extension includes simdutf8 0.1.5.
+Source: https://github.com/rusticstuff/simdutf8/tree/87ee8d9d20b849eae1974a82b248ee6fe7491bcc
+Upstream license: MIT OR Apache-2.0; distributed under the MIT option below.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

{bytesense-1.1.0 → bytesense-1.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "bytesense"
-version = "1.1.0"
+version = "1.2.0"
 description = "Encoding detection with full-input validation, bounded-memory streams, and optional Rust acceleration."
 readme = "README.md"
 license = { text = "MIT" }
@@ -82,6 +82,9 @@ Issues      = "https://github.com/oguzhankir/bytesense/issues"
 Changelog     = "https://github.com/oguzhankir/bytesense/blob/main/CHANGELOG.md"
 Documentation = "https://oguzhankir.github.io/bytesense/"
 
+[tool.setuptools]
+license-files = ["LICENSE", "THIRD_PARTY_LICENSES"]
+
 [tool.setuptools.packages.find]
 where = ["src"]
 

{bytesense-1.1.0 → bytesense-1.2.0}/rust/Cargo.lock

@@ -4,9 +4,10 @@ version = 4
 
 [[package]]
 name = "bytesense-core"
-version = "1.1.0"
+version = "1.2.0"
 dependencies = [
  "pyo3",
+ "simdutf8",
 ]
 
 [[package]]
@@ -109,6 +110,12 @@ dependencies = [
  "proc-macro2",
 ]
 
+[[package]]
+name = "simdutf8"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
+
 [[package]]
 name = "syn"
 version = "2.0.117"

{bytesense-1.1.0 → bytesense-1.2.0}/rust/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name    = "bytesense-core"
-version = "1.1.0"
+version = "1.2.0"
 edition = "2021"
 rust-version = "1.88"
 
@@ -10,6 +10,7 @@ crate-type = ["cdylib"]
 
 [dependencies]
 pyo3 = { version = "0.28.2", features = ["abi3-py39"] }
+simdutf8 = "=0.1.5"
 
 [profile.release]
 opt-level     = 3

{bytesense-1.1.0 → bytesense-1.2.0}/rust/src/utf8.rs

@@ -1,6 +1,8 @@
 /// Returns (is_valid, confidence_0_to_1).
 pub fn utf8_check(data: &[u8]) -> (bool, f64) {
-    match std::str::from_utf8(data) {
+    // The compat validator preserves the first-error offset and early exit.
+    // x86 CPU features are detected at runtime; unsupported targets use scalar validation.
+    match simdutf8::compat::from_utf8(data) {
         Ok(_) => (true, 1.0),
         Err(e) => {
             let conf = if data.is_empty() {

bytesense-1.2.0/scripts/benchmark_compare.py

@@ -0,0 +1,234 @@
+"""Compare installed baseline/candidate packages in paired, isolated processes."""
+
+from __future__ import annotations
+
+import argparse
+import contextlib
+import hashlib
+import importlib.metadata
+import json
+import os
+import platform
+import random
+import statistics
+import subprocess
+import sys
+import time
+import tracemalloc
+from pathlib import Path
+from typing import Any
+
+from benchmark_v1 import samples
+from evaluate_corpus import engines
+
+EXPECTED = {
+    "ascii_1mib": "ascii",
+    "utf8_1mib": "utf_8",
+    "utf8_8kib": "utf_8",
+    "utf8_64kib": "utf_8",
+    "utf8_8mib": "utf_8",
+    "turkish_cp1254_2100b": "cp1254",
+    "japanese_eucjp_4kib": "euc_jp",
+}
+COMPETITORS = ["chardet", "charset-normalizer", "chardetng-py"]
+
+
+def worker(variant: str, competitors: list[str]) -> None:
+    import bytesense.api
+    from bytesense._rust import is_rust_available
+
+    names = ["bytesense"] + (competitors if variant == "candidate" else [])
+    detectors = engines(names)
+    payloads = samples()
+    for inputs in payloads.values():
+        for data in inputs[:4]:
+            for detector in detectors.values():
+                detector(data)
+    metadata = {
+        "ready": True,
+        "native": is_rust_available(),
+        "python": platform.python_version(),
+        "versions": {name: importlib.metadata.version(name) for name in names},
+        "api_sha256": hashlib.sha256(Path(bytesense.api.__file__).read_bytes()).hexdigest(),
+    }
+    if is_rust_available():
+        import bytesense._rust_core
+
+        metadata["native_extension_sha256"] = hashlib.sha256(
+            Path(bytesense._rust_core.__file__).read_bytes()
+        ).hexdigest()
+    print(json.dumps(metadata), flush=True)
+    for line in sys.stdin:
+        request = json.loads(line)
+        case, engine = request["case"], request["engine"]
+        data = payloads[case][request["index"]]
+
+        def call(
+            payload: bytes = data,
+            detect=detectors[engine],
+            validate: bool = request["mode"] == "full_validation",
+        ) -> str | None:
+            encoding = detect(payload)
+            if validate and encoding:
+                payload.decode(encoding, errors="strict")
+            return encoding
+
+        if request.get("allocation"):
+            tracemalloc.start()
+            try:
+                call()
+                _, peak = tracemalloc.get_traced_memory()
+            finally:
+                tracemalloc.stop()
+            result = {"peak_bytes": peak}
+        else:
+            start = time.perf_counter_ns()
+            encoding = call()
+            elapsed = (time.perf_counter_ns() - start) / 1e6
+            try:
+                correct = bool(encoding and data.decode(encoding) == data.decode(EXPECTED[case]))
+            except UnicodeError:
+                correct = False
+            result = {"ms": elapsed, "correct": correct}
+        print(json.dumps(result), flush=True)
+
+
+class Worker:
+    def __init__(self, python: Path, variant: str, competitors: list[str]):
+        self.variant = variant
+        environment = dict(os.environ)
+        # Each supplied interpreter must resolve its own installed package.
+        environment.pop("PYTHONPATH", None)
+        # Keep virtualenv interpreter symlinks intact: resolving one loses its
+        # environment and can import a different installed package.
+        arguments = [str(python.absolute()), "-u", str(Path(__file__).resolve()), "--worker", variant]
+        for competitor in competitors:
+            arguments.extend(["--engine", competitor])
+        self.process = subprocess.Popen(
+            arguments,
+            stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE,
+            text=True,
+            env=environment,
+        )
+
+    def read(self) -> dict[str, Any]:
+        assert self.process.stdout is not None
+        line = self.process.stdout.readline()
+        if not line:
+            raise RuntimeError(
+                f"{self.variant} worker closed its output; inspect its stderr above "
+                f"(exit status: {self.process.poll()})."
+            )
+        return json.loads(line)
+
+    def request(self, **payload: Any) -> dict[str, Any]:
+        assert self.process.stdin is not None
+        self.process.stdin.write(json.dumps(payload) + "\n")
+        self.process.stdin.flush()
+        return self.read()
+
+    def close(self) -> None:
+        if self.process.stdin is not None:
+            with contextlib.suppress(BrokenPipeError):
+                self.process.stdin.close()
+        try:
+            self.process.wait(timeout=5)
+        except subprocess.TimeoutExpired:
+            self.process.terminate()
+            try:
+                self.process.wait(timeout=5)
+            except subprocess.TimeoutExpired:
+                self.process.kill()
+                self.process.wait()
+        if self.process.stdout is not None:
+            self.process.stdout.close()
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--baseline-python", type=Path)
+    parser.add_argument("--candidate-python", type=Path)
+    parser.add_argument("--output", type=Path)
+    parser.add_argument("--rounds", type=int, default=128)
+    parser.add_argument("--engine", action="append", choices=COMPETITORS)
+    parser.add_argument("--worker", choices=["baseline", "candidate"], help=argparse.SUPPRESS)
+    args = parser.parse_args()
+    competitors = list(dict.fromkeys(args.engine or COMPETITORS))
+    if args.worker:
+        worker(args.worker, competitors)
+        return
+    if not args.baseline_python or not args.candidate_python or not args.output:
+        parser.error("--baseline-python, --candidate-python and --output are required")
+    if args.rounds < 1:
+        parser.error("--rounds must be positive")
+    processes: dict[str, Worker] = {}
+    metadata = {}
+    try:
+        for variant, python in [
+            ("baseline", args.baseline_python), ("candidate", args.candidate_python)
+        ]:
+            processes[variant] = Worker(python, variant, competitors)
+            metadata[variant] = processes[variant].read()
+            if not metadata[variant].get("ready"):
+                raise RuntimeError(f"{variant} worker did not initialize")
+        print(json.dumps(metadata), flush=True)
+        detectors = [("baseline", "bytesense"), ("candidate", "bytesense")]
+        detectors += [("candidate", name) for name in competitors]
+        rng = random.Random(20260919)
+        results = []
+        for case, payloads in samples().items():
+            for mode in ["defaults", "full_validation"]:
+                times: dict[tuple[str, str], list[float]] = {key: [] for key in detectors}
+                correct = dict.fromkeys(detectors, 0)
+                for i in range(args.rounds):
+                    order = list(detectors)
+                    rng.shuffle(order)
+                    for variant, engine in order:
+                        result = processes[variant].request(
+                            case=case, engine=engine, index=i % len(payloads), mode=mode
+                        )
+                        times[(variant, engine)].append(result["ms"])
+                        correct[(variant, engine)] += result["correct"]
+                for variant, engine in detectors:
+                    durations = times[(variant, engine)]
+                    allocation = processes[variant].request(
+                        case=case, engine=engine, index=0, mode=mode, allocation=True
+                    )
+                    result = {
+                        "case": case,
+                        "mode": mode,
+                        "variant": variant,
+                        "engine": engine,
+                        "bytes": len(payloads[0]),
+                        "sha256": hashlib.sha256(b"".join(payloads)).hexdigest(),
+                        "median_ms": statistics.median(durations),
+                        "p95_ms": sorted(durations)[int(0.95 * (args.rounds - 1))],
+                        "peak_bytes": allocation["peak_bytes"],
+                        "correct": correct[(variant, engine)],
+                        "rounds": args.rounds,
+                    }
+                    results.append(result)
+                    print(json.dumps(result), flush=True)
+                args.output.write_text(
+                    json.dumps(
+                        {
+                            "python": platform.python_version(),
+                            "platform": platform.platform(),
+                            "metadata": metadata,
+                            "notes": "32 rotating payloads; four warmups per case; persistent isolated "
+                            "baseline/candidate processes; seed 20260919 randomized engine order each "
+                            "round; IPC excluded; tracemalloc excludes native allocations.",
+                            "results": results,
+                        },
+                        indent=2,
+                    ),
+                    encoding="utf-8",
+                )
+    finally:
+        for process in processes.values():
+            process.close()
+
+
+if __name__ == "__main__":
+    main()

{bytesense-1.1.0 → bytesense-1.2.0}/scripts/benchmark_v1.py

@@ -21,6 +21,8 @@ def samples() -> dict[str, list[bytes]]:
         "ascii_1mib": (b"The quick brown fox jumps over the lazy dog. ", 1_048_576, "ascii"),
         "utf8_1mib": ("Merhaba dünya! 日本語 Ελληνικά 🙂 ".encode(), 1_048_576, "utf_8"),
         "utf8_8kib": ("Café dünyası: 中文测试 🎉 ".encode(), 8192, "utf_8"),
+        "utf8_64kib": ("Merhaba dünya! 日本語 Ελληνικά 🙂 ".encode(), 65536, "utf_8"),
+        "utf8_8mib": ("Merhaba dünya! 日本語 Ελληνικά 🙂 ".encode(), 8_388_608, "utf_8"),
         "turkish_cp1254_2100b": (
             ("İstanbul'da çalışan mühendisler için doğru metin çözümleme. " * 35).encode("cp1254"),
             2100,
@@ -65,6 +67,8 @@ def main() -> None:
         "ascii_1mib": "ascii",
         "utf8_1mib": "utf_8",
         "utf8_8kib": "utf_8",
+        "utf8_64kib": "utf_8",
+        "utf8_8mib": "utf_8",
         "turkish_cp1254_2100b": "cp1254",
         "japanese_eucjp_4kib": "euc_jp",
     }

{bytesense-1.1.0 → bytesense-1.2.0}/scripts/check_dist.py

@@ -13,12 +13,14 @@ for path in sorted(Path(sys.argv[1]).iterdir()):
                 assert not any(n.endswith((".so", ".pyd", ".dll")) for n in names)
             assert "bytesense/py.typed" in names
             assert "bytesense/data/language.json.gz" in names
+            assert any(n.endswith("/THIRD_PARTY_LICENSES") for n in names)
             assert not any("/target/" in n for n in names)
     elif path.name.endswith(".tar.gz"):
         with tarfile.open(path) as archive:
             names = archive.getnames()
             for required in (
                 "setup.py",
+                "THIRD_PARTY_LICENSES",
                 "scripts/evaluate_corpus.py",
                 "scripts/evaluate_udhr.py",
                 "scripts/check_evaluation.py",

{bytesense-1.1.0 → bytesense-1.2.0}/src/bytesense/api.py

@@ -10,6 +10,7 @@ from functools import lru_cache
 from os import PathLike
 from typing import Any, BinaryIO, List, Optional
 
+from ._rust import is_rust_available, rust_utf8_check
 from .candidate import CandidateSelector
 from .coherence import detect_language
 from .constant import ALL_ENCODINGS
@@ -236,8 +237,13 @@ def _unicode_fallbacks(
 
 
 def _valid(data: bytes, encoding: str) -> bool:
+    utf8 = encoding in ("utf_8", "utf_8_sig")
+    if utf8 and is_rust_available():
+        # A UTF-8 BOM changes decoded text, not strict byte validity. Native
+        # validation checks every byte without allocating decoded Unicode.
+        return rust_utf8_check(data)[0]
     try:
-        if len(data) <= 1_048_576:
+        if len(data) <= (65536 if utf8 else 1_048_576):
             data.decode(encoding, errors="strict")
         else:
             decoder = codecs.getincrementaldecoder(encoding)(errors="strict")
@@ -375,7 +381,8 @@ def from_bytes(
             )
         r = _result(bom, size, f"BOM declares {bom}; all bytes validated.", 1.0)
         return replace(r, bom_detected=True)
-    transport = _transport_encoding(data) if data.isascii() else None
+    ascii_data = data.isascii()
+    transport = _transport_encoding(data) if ascii_data else None
     if transport and _allowed(transport, include, exclude):
         if min_confidence <= 0.85:
             return _result(
@@ -412,7 +419,7 @@ def from_bytes(
         )
     shape = detect_null_pattern(data)
     if not shape and not _looks_like_iso2022(data):
-        if data.isascii() and _allowed("ascii", include, exclude):
+        if ascii_data and _allowed("ascii", include, exclude):
             return _result("ascii", size, "All bytes are valid ASCII text.", 1.0)
         if _allowed("utf_8", include, exclude) and _valid(data, "utf_8"):
             r = _result("utf_8", size, "All bytes validated as UTF-8.", 0.99)

bytesense-1.2.0/src/bytesense/version.py

@@ -0,0 +1,4 @@
+from __future__ import annotations
+
+__version__: str = "1.2.0"
+VERSION: tuple[int, int, int] = (1, 2, 0)

{bytesense-1.1.0 → bytesense-1.2.0/src/bytesense.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bytesense
-Version: 1.1.0
+Version: 1.2.0
 Summary: Encoding detection with full-input validation, bounded-memory streams, and optional Rust acceleration.
 Author-email: Oğuzhan Kır <oguzhankir17@gmail.com>
 Maintainer-email: Oğuzhan Kır <oguzhankir17@gmail.com>
@@ -33,6 +33,7 @@ Classifier: Typing :: Typed
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
+License-File: THIRD_PARTY_LICENSES
 Provides-Extra: fast
 Provides-Extra: docs
 Requires-Dist: mkdocs>=1.6; extra == "docs"
@@ -131,13 +132,21 @@ cat report.csv | bytesense --minimal -
 
 The CLI exits **0** when every input has an encoding, **1** for read failures or unknown/binary inputs, and **2** for invalid arguments. JSON goes to stdout; errors go to stderr.
 
+## Used by
+
+[ingest-sentry](https://github.com/oguzhankir/ingest-sentry) uses bytesense for encoding
+detection and full-snapshot validation before CSV/TSV imports. It adds import contracts,
+safe UTF-8 normalization, string-preserving pandas/Polars adapters, and a GitHub Action.
+Built by the same maintainer; an independent project, not affiliated with Sentry.
+
 ## Measured accuracy and performance
 
 The v1 engine replaces the previous collection of special-case ranking rules with a small, reproducibly generated statistical model. The evaluation separates development documents from held-out documents, grouping identical decoded text across encodings. Accuracy means **exact Unicode equality**, not just a compatible codec name.
 
 | Detector | Exact Unicode matches | Accuracy |
 |---|---:|---:|
-| **bytesense 1.1.0** | **1907 / 2078** | **91.77%** |
+| **bytesense 1.2.0** | **1907 / 2078** | **91.77%** |
+| bytesense 1.1.0 | 1907 / 2078 | 91.77% |
 | bytesense 1.0.0 | 1906 / 2078 | 91.72% |
 | bytesense 0.1.2 | 883 / 2078 | 42.49% |
 | chardet 7.6.0 | 2060 / 2078 | 99.13% |
@@ -146,9 +155,9 @@ The v1 engine replaces the previous collection of special-case ranking rules wit
 
 See [benchmarks and methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for measured comparisons with chardet, charset-normalizer and chardetng, including the cases where another detector wins. Timing reports distinguish default behavior from an additional full-decode validation step.
 
-**New in 1.1:** faster legacy scoring, canonical-Unicode evidence, and deeper full-input validation. On a new 432-case transfer check built from 28 UDHR translations, exact recovery increased from **374 to 394 cases**; charset-normalizer recovered 381 and chardet 422. These are correlated variants of one translated document, not a universal accuracy ranking. Chardet remains more accurate on both reported corpora.
+**New in 1.2:** `from_bytes()` validates UTF-8 using native SIMD without allocating decoded Unicode; its pure-Python path uses 64 KiB blocks above 64 KiB. On the measured 1 MiB UTF-8 workload, native `from_bytes()` median latency decreased from **1.564 ms to 0.294 ms (5.3×)** versus 1.1, with every byte still validated. Peak temporary Python allocations decreased from about **6 MiB to 6 KiB**; this is not total process memory.
 
-The measured Turkish CP1254 and Japanese EUC-JP workloads run approximately **2.8× faster than bytesense 1.0** with the native backend. See the benchmark table for absolute timings and where competitors remain faster.
+The existing 2,078-file and 432-case transfer results are unchanged. This release improves UTF-8 validation cost, not general encoding accuracy; chardet remains more accurate on both corpora. See the [benchmark methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for paired measurements, competitors, backend scope and reproduction commands.
 
 ## Upgrading from 0.x
 

{bytesense-1.1.0 → bytesense-1.2.0}/src/bytesense.egg-info/SOURCES.txt

@@ -4,6 +4,7 @@ CONTRIBUTING.md
 LICENSE
 MANIFEST.in
 README.md
+THIRD_PARTY_LICENSES
 pyproject.toml
 setup.py
 benchmarks/__init__.py
@@ -18,6 +19,7 @@ rust/src/histogram.rs
 rust/src/language.rs
 rust/src/lib.rs
 rust/src/utf8.rs
+scripts/benchmark_compare.py
 scripts/benchmark_v1.py
 scripts/build_fingerprints.py
 scripts/check_dist.py
@@ -75,4 +77,5 @@ tests/test_rust.py
 tests/test_rust_layer.py
 tests/test_streaming.py
 tests/test_streaming_v2.py
+tests/test_utf8_validation.py
 tests/test_v1_contracts.py

bytesense-1.2.0/tests/test_utf8_validation.py

@@ -0,0 +1,109 @@
+"""Strict UTF-8 validation across SIMD, BOM, and incremental decoder boundaries."""
+
+from __future__ import annotations
+
+import codecs
+import platform
+
+import pytest
+from hypothesis import given, settings
+from hypothesis import strategies as st
+
+from bytesense import api, from_bytes
+from bytesense._rust import is_rust_available, rust_utf8_check
+
+
+def _strictly_valid(data: bytes, encoding: str) -> bool:
+    try:
+        data.decode(encoding, errors="strict")
+    except UnicodeError:
+        return False
+    return True
+
+
+@pytest.mark.parametrize("encoding", ["utf_8", "utf_8_sig"])
+@given(data=st.one_of(st.binary(max_size=4096), st.text(max_size=4096).map(str.encode)))
+@settings(max_examples=300, deadline=None)
+def test_utf8_validation_matches_strict_python(data: bytes, encoding: str) -> None:
+    assert api._valid(data, encoding) == _strictly_valid(data, encoding)
+
+
+@pytest.mark.parametrize("encoding", ["utf_8", "utf_8_sig"])
+@pytest.mark.parametrize("offset", [31, 32, 63, 64, 65, 65535, 65536, 65537, 1_048_575])
+@pytest.mark.parametrize(
+    "suffix",
+    [
+        "é中🙂".encode(),
+        b"\xc0\xaf",  # Overlong sequence.
+        b"\xed\xa0\x80",  # Surrogate scalar.
+        b"\xf4\x90\x80\x80",  # Above U+10FFFF.
+        b"\xe2\x82",  # Truncated code point.
+        b"\x80",  # Unpaired continuation byte.
+    ],
+)
+def test_utf8_boundaries_preserve_full_input_validation(
+    encoding: str, offset: int, suffix: bytes
+) -> None:
+    prefix = codecs.BOM_UTF8 if encoding == "utf_8_sig" else b""
+    data = prefix + b"a" * offset + suffix
+    expected = _strictly_valid(data, encoding)
+    assert api._valid(data, encoding) == expected
+    result = from_bytes(data, cp_isolation=[encoding], enable_fallback=False)
+    assert (result.encoding is not None) == expected
+    if expected:
+        assert result.bytes_validated == len(data)
+
+
+@pytest.mark.skipif(not is_rust_available(), reason="Native extension not compiled")
+@given(data=st.binary(max_size=8192))
+@settings(max_examples=500, deadline=None)
+def test_native_error_offset_matches_python(data: bytes) -> None:
+    try:
+        data.decode("utf_8", errors="strict")
+        expected = (True, 1.0)
+    except UnicodeDecodeError as error:
+        expected = (False, error.start / len(data))
+    assert rust_utf8_check(data) == expected
+
+
+@pytest.mark.skipif(platform.python_implementation() != "CPython", reason="CPython allocation probe")
+@pytest.mark.parametrize("native", [False, True])
+def test_utf8_validation_does_not_allocate_full_decoded_input(monkeypatch, native: bool) -> None:
+    import tracemalloc
+
+    if native and not is_rust_available():
+        pytest.skip("Native extension not compiled")
+    if not native:
+        monkeypatch.setattr(api, "is_rust_available", lambda: False)
+    data = ("Merhaba dünya! 日本語 Ελληνικά 🙂 ".encode() * 30_000)[:1_048_576]
+    data = data.decode("utf_8", errors="ignore").encode("utf_8")
+    # Warm the decoder and feature dispatch before measuring call allocations.
+    assert api._valid(data, "utf_8")
+    tracemalloc.start()
+    try:
+        assert api._valid(data, "utf_8")
+        _, peak = tracemalloc.get_traced_memory()
+    finally:
+        tracemalloc.stop()
+    assert peak < 600_000
+
+
+def test_non_utf8_custom_codec_keeps_existing_one_shot_boundary() -> None:
+    class BytesOnlyDecoder(codecs.IncrementalDecoder):
+        def decode(self, data: bytes, final: bool = False) -> str:
+            if not isinstance(data, bytes):
+                raise TypeError("decoder requires bytes")
+            return data.decode("ascii")
+
+    def search(name: str) -> codecs.CodecInfo | None:
+        if name != "bytesense_test_bytes_only":
+            return None
+        return codecs.CodecInfo(
+            name=name,
+            encode=codecs.ascii_encode,
+            decode=codecs.ascii_decode,
+            incrementaldecoder=BytesOnlyDecoder,
+        )
+
+    codecs.register(search)
+    assert api._valid(b"a" * 65537, "bytesense_test_bytes_only")

bytesense-1.1.0/src/bytesense/version.py

@@ -1,4 +0,0 @@
-from __future__ import annotations
-
-__version__: str = "1.1.0"
-VERSION: tuple[int, int, int] = (1, 1, 0)