bytesense 1.0.0__tar.gz → 1.2.0__tar.gz

{bytesense-1.0.0 → bytesense-1.2.0}/CHANGELOG.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## 1.2.0
+
+### UTF-8 validation
+- Validate complete UTF-8 and UTF-8-SIG inputs in `from_bytes()` with SIMD in native builds, without allocating decoded Unicode. Pin `simdutf8` 0.1.5 and preserve strict rejection and first-error offsets with its compatibility validator.
+- Retain runtime x86 CPU dispatch and scalar fallbacks; no target-specific CPU flags or Python runtime dependencies are required.
+- Decode pure-Python UTF-8 inputs larger than 64 KiB in bounded blocks. Preserve the existing threshold for other codecs, including custom decoders.
+- Reuse the ASCII check within each detection call. Public results, scoring and stream behavior are unchanged.
+
+### Verification
+- Add strict-decoder parity, malformed-scalar, SIMD/block boundary, allocation and custom-codec regressions.
+- Extend rotating-input benchmarks with 64 KiB and 8 MiB UTF-8 workloads and isolated before/after comparisons.
+- Preserve existing corpus accuracy and native/Python predictions; include the native dependency's license in distributions.
+
+See the benchmark documentation for measured gains and their scope.
+
+## 1.1.0
+
+### Faster legacy detection
+- Cache Unicode scalar properties in a fixed 64 KiB atomic table; remove per-character property hash lookups and read locks.
+- Use a bounded 512 KiB native lookup table for Latin/ASCII model pairs, with sparse lookup for other scripts.
+- Prune pair scoring only when an optimistic evidence bound proves that a candidate cannot qualify. Retain matching Python/native decisions.
+- Normalize and deduplicate the built-in codec catalog lazily once; caller-supplied codecs are still validated on each call.
+
+### Correctness
+- Normalize scoring input to NFC and stop penalizing combining marks as symbol noise. Original bytes and decoded text are never normalized or rewritten.
+- Reach all statistically eligible candidates during full-input validation instead of stopping at the six display hypotheses. Public alternatives remain limited to five.
+- Recover plausible BOM-less UTF-16 with NULs in both lanes before declaring binary content, subject to strict full-input validation and caller filters.
+- Recognize ISO-2022 shift controls and UTF-7 Unicode spacing/format characters as text syntax.
+
+### Verification
+- Add cache-boundary, concurrent cold-cache, dense/sparse pair parity, pruning-bound, canonical-equivalence and validation-depth regressions.
+- Add a pinned 28-document UDHR transfer evaluation, with exact Unicode comparisons, recorded exclusions and separate legacy/Unicode counts.
+- Add single-engine benchmark runs for before/after comparisons. Model weights and public API signatures are unchanged.
+
+See the benchmark documentation for measured gains and remaining limitations.
+
 
 ## 1.0.0
 

{bytesense-1.0.0 → bytesense-1.2.0}/MANIFEST.in

@@ -1,4 +1,4 @@
-include LICENSE README.md CHANGELOG.md
+include LICENSE THIRD_PARTY_LICENSES README.md CHANGELOG.md
 recursive-include rust *.toml *.lock *.rs
 recursive-include src/bytesense *.py py.typed
 prune rust/target

{bytesense-1.0.0 → bytesense-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bytesense
-Version: 1.0.0
+Version: 1.2.0
 Summary: Encoding detection with full-input validation, bounded-memory streams, and optional Rust acceleration.
 Author-email: Oğuzhan Kır <oguzhankir17@gmail.com>
 Maintainer-email: Oğuzhan Kır <oguzhankir17@gmail.com>
@@ -33,6 +33,7 @@ Classifier: Typing :: Typed
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
+License-File: THIRD_PARTY_LICENSES
 Provides-Extra: fast
 Provides-Extra: docs
 Requires-Dist: mkdocs>=1.6; extra == "docs"
@@ -67,7 +68,7 @@ Dynamic: license-file
 
 **bytesense** is a Python encoding detector for file imports, multilingual text pipelines and legacy data. It combines fast Unicode checks, compact character-pair statistics and optional Rust acceleration—with **zero runtime dependencies**.
 
-Every returned encoding strictly decodes the complete supplied input. Linguistic scoring uses a bounded sample; stream inputs spill to a temporary file after a configurable memory limit. Results tell you what was examined, what was validated, and whether the input ended.
+Final detection results strictly validate the selected encoding against the complete supplied input. Linguistic scoring uses a bounded sample; stream inputs spill to a temporary file after a configurable memory limit. Results tell you what was examined, what was validated, and whether the input ended.
 
 ```python
 from bytesense import from_bytes
@@ -131,13 +132,22 @@ cat report.csv | bytesense --minimal -
 
 The CLI exits **0** when every input has an encoding, **1** for read failures or unknown/binary inputs, and **2** for invalid arguments. JSON goes to stdout; errors go to stderr.
 
+## Used by
+
+[ingest-sentry](https://github.com/oguzhankir/ingest-sentry) uses bytesense for encoding
+detection and full-snapshot validation before CSV/TSV imports. It adds import contracts,
+safe UTF-8 normalization, string-preserving pandas/Polars adapters, and a GitHub Action.
+Built by the same maintainer; an independent project, not affiliated with Sentry.
+
 ## Measured accuracy and performance
 
 The v1 engine replaces the previous collection of special-case ranking rules with a small, reproducibly generated statistical model. The evaluation separates development documents from held-out documents, grouping identical decoded text across encodings. Accuracy means **exact Unicode equality**, not just a compatible codec name.
 
 | Detector | Exact Unicode matches | Accuracy |
 |---|---:|---:|
-| **bytesense 1.0.0** | **1906 / 2078** | **91.72%** |
+| **bytesense 1.2.0** | **1907 / 2078** | **91.77%** |
+| bytesense 1.1.0 | 1907 / 2078 | 91.77% |
+| bytesense 1.0.0 | 1906 / 2078 | 91.72% |
 | bytesense 0.1.2 | 883 / 2078 | 42.49% |
 | chardet 7.6.0 | 2060 / 2078 | 99.13% |
 | charset-normalizer 3.5.1 | 1679 / 2078 | 80.80% |
@@ -145,6 +155,10 @@ The v1 engine replaces the previous collection of special-case ranking rules wit
 
 See [benchmarks and methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for measured comparisons with chardet, charset-normalizer and chardetng, including the cases where another detector wins. Timing reports distinguish default behavior from an additional full-decode validation step.
 
+**New in 1.2:** `from_bytes()` validates UTF-8 using native SIMD without allocating decoded Unicode; its pure-Python path uses 64 KiB blocks above 64 KiB. On the measured 1 MiB UTF-8 workload, native `from_bytes()` median latency decreased from **1.564 ms to 0.294 ms (5.3×)** versus 1.1, with every byte still validated. Peak temporary Python allocations decreased from about **6 MiB to 6 KiB**; this is not total process memory.
+
+The existing 2,078-file and 432-case transfer results are unchanged. This release improves UTF-8 validation cost, not general encoding accuracy; chardet remains more accurate on both corpora. See the [benchmark methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for paired measurements, competitors, backend scope and reproduction commands.
+
 ## Upgrading from 0.x
 
 - Language reporting is opt-in: `include_language=True`.

{bytesense-1.0.0 → bytesense-1.2.0}/README.md

@@ -10,7 +10,7 @@
 
 **bytesense** is a Python encoding detector for file imports, multilingual text pipelines and legacy data. It combines fast Unicode checks, compact character-pair statistics and optional Rust acceleration—with **zero runtime dependencies**.
 
-Every returned encoding strictly decodes the complete supplied input. Linguistic scoring uses a bounded sample; stream inputs spill to a temporary file after a configurable memory limit. Results tell you what was examined, what was validated, and whether the input ended.
+Final detection results strictly validate the selected encoding against the complete supplied input. Linguistic scoring uses a bounded sample; stream inputs spill to a temporary file after a configurable memory limit. Results tell you what was examined, what was validated, and whether the input ended.
 
 ```python
 from bytesense import from_bytes
@@ -74,13 +74,22 @@ cat report.csv | bytesense --minimal -
 
 The CLI exits **0** when every input has an encoding, **1** for read failures or unknown/binary inputs, and **2** for invalid arguments. JSON goes to stdout; errors go to stderr.
 
+## Used by
+
+[ingest-sentry](https://github.com/oguzhankir/ingest-sentry) uses bytesense for encoding
+detection and full-snapshot validation before CSV/TSV imports. It adds import contracts,
+safe UTF-8 normalization, string-preserving pandas/Polars adapters, and a GitHub Action.
+Built by the same maintainer; an independent project, not affiliated with Sentry.
+
 ## Measured accuracy and performance
 
 The v1 engine replaces the previous collection of special-case ranking rules with a small, reproducibly generated statistical model. The evaluation separates development documents from held-out documents, grouping identical decoded text across encodings. Accuracy means **exact Unicode equality**, not just a compatible codec name.
 
 | Detector | Exact Unicode matches | Accuracy |
 |---|---:|---:|
-| **bytesense 1.0.0** | **1906 / 2078** | **91.72%** |
+| **bytesense 1.2.0** | **1907 / 2078** | **91.77%** |
+| bytesense 1.1.0 | 1907 / 2078 | 91.77% |
+| bytesense 1.0.0 | 1906 / 2078 | 91.72% |
 | bytesense 0.1.2 | 883 / 2078 | 42.49% |
 | chardet 7.6.0 | 2060 / 2078 | 99.13% |
 | charset-normalizer 3.5.1 | 1679 / 2078 | 80.80% |
@@ -88,6 +97,10 @@ The v1 engine replaces the previous collection of special-case ranking rules wit
 
 See [benchmarks and methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for measured comparisons with chardet, charset-normalizer and chardetng, including the cases where another detector wins. Timing reports distinguish default behavior from an additional full-decode validation step.
 
+**New in 1.2:** `from_bytes()` validates UTF-8 using native SIMD without allocating decoded Unicode; its pure-Python path uses 64 KiB blocks above 64 KiB. On the measured 1 MiB UTF-8 workload, native `from_bytes()` median latency decreased from **1.564 ms to 0.294 ms (5.3×)** versus 1.1, with every byte still validated. Peak temporary Python allocations decreased from about **6 MiB to 6 KiB**; this is not total process memory.
+
+The existing 2,078-file and 432-case transfer results are unchanged. This release improves UTF-8 validation cost, not general encoding accuracy; chardet remains more accurate on both corpora. See the [benchmark methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for paired measurements, competitors, backend scope and reproduction commands.
+
 ## Upgrading from 0.x
 
 - Language reporting is opt-in: `include_language=True`.

bytesense-1.2.0/THIRD_PARTY_LICENSES

@@ -0,0 +1,23 @@
+The optional native extension includes simdutf8 0.1.5.
+Source: https://github.com/rusticstuff/simdutf8/tree/87ee8d9d20b849eae1974a82b248ee6fe7491bcc
+Upstream license: MIT OR Apache-2.0; distributed under the MIT option below.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

{bytesense-1.0.0 → bytesense-1.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "bytesense"
-version = "1.0.0"
+version = "1.2.0"
 description = "Encoding detection with full-input validation, bounded-memory streams, and optional Rust acceleration."
 readme = "README.md"
 license = { text = "MIT" }
@@ -82,6 +82,9 @@ Issues      = "https://github.com/oguzhankir/bytesense/issues"
 Changelog     = "https://github.com/oguzhankir/bytesense/blob/main/CHANGELOG.md"
 Documentation = "https://oguzhankir.github.io/bytesense/"
 
+[tool.setuptools]
+license-files = ["LICENSE", "THIRD_PARTY_LICENSES"]
+
 [tool.setuptools.packages.find]
 where = ["src"]
 

{bytesense-1.0.0 → bytesense-1.2.0}/rust/Cargo.lock

@@ -4,9 +4,10 @@ version = 4
 
 [[package]]
 name = "bytesense-core"
-version = "1.0.0"
+version = "1.2.0"
 dependencies = [
  "pyo3",
+ "simdutf8",
 ]
 
 [[package]]
@@ -109,6 +110,12 @@ dependencies = [
  "proc-macro2",
 ]
 
+[[package]]
+name = "simdutf8"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
+
 [[package]]
 name = "syn"
 version = "2.0.117"

{bytesense-1.0.0 → bytesense-1.2.0}/rust/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name    = "bytesense-core"
-version = "1.0.0"
+version = "1.2.0"
 edition = "2021"
 rust-version = "1.88"
 
@@ -10,6 +10,7 @@ crate-type = ["cdylib"]
 
 [dependencies]
 pyo3 = { version = "0.28.2", features = ["abi3-py39"] }
+simdutf8 = "=0.1.5"
 
 [profile.release]
 opt-level     = 3

{bytesense-1.0.0 → bytesense-1.2.0}/rust/src/language.rs

@@ -1,47 +1,93 @@
 use pyo3::prelude::*;
 use std::collections::{HashMap, HashSet};
-use std::sync::RwLock;
+use std::sync::atomic::{AtomicU8, Ordering};
+
+const BMP_LEN: usize = 1 << 16;
+const INITIALIZED: u8 = 1;
+const ALPHABETIC: u8 = 1 << 1;
+const UNPRINTABLE: u8 = 1 << 2;
+const SYMBOL: u8 = 1 << 3;
+const KNOWN_LETTER: u8 = 1 << 4;
+
+fn pair_key(a: char, b: char) -> u64 {
+    ((a as u64) << 32) | b as u64
+}
 
 /// Immutable corpus statistics only. No input text is retained between calls.
 #[pyclass(frozen)]
 pub struct NgramModel {
     letters: HashSet<char>,
-    pairs: HashMap<(char, char), f64>,
-    properties: RwLock<HashMap<char, (bool, bool, bool)>>,
+    pairs: HashMap<u64, f64>,
+    latin_pairs: Box<[f64]>,
+    max_pair_support: f64,
+    // One byte per BMP scalar. Each atomic contains the complete property
+    // record, so relaxed loads/stores need not publish any additional state.
+    properties: Box<[AtomicU8]>,
+}
+
+impl NgramModel {
+    #[inline]
+    fn pair_support(&self, before: char, current: char) -> f64 {
+        if before <= '\u{ff}' && current <= '\u{ff}' {
+            self.latin_pairs[((before as usize) << 8) | current as usize]
+        } else {
+            self.pairs
+                .get(&pair_key(before, current))
+                .copied()
+                .unwrap_or(0.0)
+        }
+    }
 }
 
 #[pymethods]
 impl NgramModel {
     #[new]
     fn new(letters: &str, pairs: Vec<(String, f64)>) -> Self {
+        // A bounded 512 KiB table avoids hashing common Latin/ASCII pairs.
+        // It contains model weights only, independent of observed documents.
+        let mut latin_pairs = vec![0.0; 256 * 256].into_boxed_slice();
+        for (pair, weight) in &pairs {
+            let mut chars = pair.chars();
+            if let (Some(a), Some(b)) = (chars.next(), chars.next()) {
+                if a <= '\u{ff}' && b <= '\u{ff}' {
+                    latin_pairs[((a as usize) << 8) | b as usize] = *weight;
+                }
+            }
+        }
         Self {
-            properties: RwLock::new(HashMap::new()),
+            latin_pairs,
+            max_pair_support: pairs.iter().fold(1.0_f64, |m, (_, weight)| m.max(*weight)),
+            properties: (0..BMP_LEN).map(|_| AtomicU8::new(0)).collect(),
             letters: letters.chars().collect(),
             pairs: pairs
                 .iter()
                 .filter_map(|(s, weight)| {
                     let mut chars = s.chars();
-                    Some(((chars.next()?, chars.next()?), *weight))
+                    Some((pair_key(chars.next()?, chars.next()?), *weight))
                 })
                 .collect(),
         }
     }
 
     /// Cache bounded Unicode scalar metadata, never input strings or sequences.
+    #[pyo3(signature = (raw, lower, classify, minimum=None))]
     fn quality(
         &self,
         py: Python<'_>,
         raw: &str,
         lower: &str,
         classify: &Bound<'_, PyAny>,
-    ) -> PyResult<(f64, f64)> {
-        let unknown: HashSet<char> = {
-            let properties = self.properties.read().unwrap();
-            raw.chars()
-                .chain(lower.chars())
-                .filter(|c| !properties.contains_key(c))
-                .collect()
-        };
+        minimum: Option<f64>,
+    ) -> PyResult<Option<(f64, f64)>> {
+        let unknown: HashSet<char> = raw
+            .chars()
+            .chain(lower.chars())
+            .filter(|&c| {
+                self.properties
+                    .get(c as usize)
+                    .is_none_or(|flags| flags.load(Ordering::Relaxed) == 0)
+            })
+            .collect();
         let chars: Vec<char> = unknown.into_iter().collect();
         let mut local = HashMap::new();
         if !chars.is_empty() {
@@ -53,27 +99,46 @@ impl NgramModel {
                     "one property record per scalar is required",
                 ));
             }
-            let mut properties = self.properties.write().unwrap();
-            for (c, flags) in chars.into_iter().zip(values) {
-                // BMP cache has at most 65,536 entries. Other scalars are local.
-                if c <= '\u{ffff}' {
-                    properties.insert(c, flags);
+            for (c, (alpha, bad, symbol)) in chars.into_iter().zip(values) {
+                let flags = INITIALIZED
+                    | if alpha { ALPHABETIC } else { 0 }
+                    | if bad { UNPRINTABLE } else { 0 }
+                    | if symbol { SYMBOL } else { 0 }
+                    | if self.letters.contains(&c) {
+                        KNOWN_LETTER
+                    } else {
+                        0
+                    };
+                if let Some(entry) = self.properties.get(c as usize) {
+                    entry.store(flags, Ordering::Relaxed);
                 } else {
                     local.insert(c, flags);
                 }
             }
         }
         Ok(py.detach(|| {
-            let properties = self.properties.read().unwrap();
-            let flags = |c: &char| properties.get(c).or_else(|| local.get(c)).unwrap();
+            let flags = |c: char| {
+                self.properties
+                    .get(c as usize)
+                    .map_or_else(|| local[&c], |entry| entry.load(Ordering::Relaxed))
+            };
             let mut n = 0usize;
             let mut bad = 0usize;
             let mut symbols = 0usize;
             for c in raw.chars() {
                 n += 1;
-                let (_, unprintable, symbol) = flags(&c);
-                bad += usize::from(*unprintable);
-                symbols += usize::from(*symbol);
+                let properties = flags(c);
+                bad += usize::from(properties & UNPRINTABLE != 0);
+                symbols += usize::from(properties & SYMBOL != 0);
+            }
+            let bad_ratio = bad as f64 / n.max(1) as f64;
+            let penalty = 5.0 * bad_ratio + 3.0 * symbols as f64 / n.max(1) as f64;
+            // An optimistic bound: even perfect letter/pair coverage cannot
+            // recover this candidate. Keep slack for Python's score rounding.
+            if minimum
+                .is_some_and(|floor| 0.25 + 0.75 * self.max_pair_support - penalty + 1e-9 < floor)
+            {
+                return None;
             }
             let mut letter_count = 0usize;
             let mut known_letters = 0usize;
@@ -81,10 +146,11 @@ impl NgramModel {
             let mut known_pair_weight = 0.0f64;
             let mut previous = None;
             for c in lower.chars() {
-                let current = if flags(&c).0 { c } else { ' ' };
+                let properties = flags(c);
+                let current = if properties & ALPHABETIC != 0 { c } else { ' ' };
                 if current != ' ' {
                     letter_count += 1;
-                    known_letters += usize::from(self.letters.contains(&current));
+                    known_letters += usize::from(properties & KNOWN_LETTER != 0);
                 }
                 if let Some(before) = previous {
                     if before != ' ' || current != ' ' {
@@ -94,21 +160,18 @@ impl NgramModel {
                             1
                         };
                         pair_weight += weight;
-                        if let Some(support) = self.pairs.get(&(before, current)) {
-                            known_pair_weight += weight as f64 * support;
-                        }
+                        known_pair_weight += weight as f64 * self.pair_support(before, current);
                     }
                 }
                 previous = Some(current);
             }
-            let bad_ratio = bad as f64 / n.max(1) as f64;
-            (
+            Some((
                 0.25 * known_letters as f64 / letter_count.max(1) as f64
                     + 0.75 * known_pair_weight / pair_weight.max(1) as f64
                     - 5.0 * bad_ratio
                     - 3.0 * symbols as f64 / n.max(1) as f64,
                 bad_ratio,
-            )
+            ))
         }))
     }
 
@@ -135,9 +198,7 @@ impl NgramModel {
                             1
                         };
                         pair_weight += weight;
-                        if let Some(support) = self.pairs.get(&(before, current)) {
-                            known_pair_weight += weight as f64 * support;
-                        }
+                        known_pair_weight += weight as f64 * self.pair_support(before, current);
                     }
                 }
                 previous = Some(current);

{bytesense-1.0.0 → bytesense-1.2.0}/rust/src/utf8.rs

@@ -1,6 +1,8 @@
 /// Returns (is_valid, confidence_0_to_1).
 pub fn utf8_check(data: &[u8]) -> (bool, f64) {
-    match std::str::from_utf8(data) {
+    // The compat validator preserves the first-error offset and early exit.
+    // x86 CPU features are detected at runtime; unsupported targets use scalar validation.
+    match simdutf8::compat::from_utf8(data) {
         Ok(_) => (true, 1.0),
         Err(e) => {
             let conf = if data.is_empty() {