bytesense 0.1.2__tar.gz → 1.0.0__tar.gz

bytesense-1.0.0/CHANGELOG.md

@@ -0,0 +1,83 @@
+# Changelog
+
+
+## 1.0.0
+
+### Detection and correctness
+- Replace the bespoke ranking pipeline with a reproducibly generated character-pair model and optional native scoring.
+- Strictly validate every returned codec against the complete input, with explicit sample/validation counts and completion status.
+- Normalize filters on all paths; an empty isolation list permits no codecs.
+- Use BOM-consuming UTF-16/32 codecs, improve non-Latin Unicode lane detection, and recognize 7-bit shift syntax.
+- Remove fabricated confidence intervals; scores are explicitly uncalibrated. Language reporting is opt-in.
+
+### Streams and data handling
+- Consume streams to EOF by default with bounded memory and temporary-file spooling.
+- Preserve late informative samples across chunk boundaries; reject feeding after finalization.
+- Retain every mixed-document byte, including short tails, and preserve Unicode boundaries.
+- Use strict decoding for repair and segments; do not silently replace undecodable bytes.
+- Add CLI stdin support, configuration flags and meaningful exit codes.
+
+### Packaging and verification
+- Python 3.9+, portable compiler-free wheel/sdist installs, typed package marker, and CPython abi3 native wheels.
+- Distinct pure/native CI, installed-wheel tests, version checks, pinned/hash-verified corpus inputs and reproducible comparison tools.
+- Consolidate documentation; retain historical details in this changelog.
+
+See the README migration notes for intentional 0.x behavior changes.
+
+## [0.1.2] — 2025-03-26
+
+### Changed
+
+- **BOM fast path:** UTF-8 with BOM now reports `encoding="utf_8_sig"` (aligned with `codecs` / `CandidateSelector`), not `utf_8`.
+- **Streaming:** In-band HTML/XML hints are probed on every `feed()` until a hint is found (scan limited to the first 4KB inside `_probe_inband_hint`); shared regex patterns imported from `hints.py`; `codecs` imported at module level.
+- **`detect_multi`:** Adjacent same-encoding merges no longer re-run `from_bytes` on the full merged span; `byte_count` on the shared `DetectionResult` is updated with `dataclasses.replace`.
+- **`repair_bytes`:** Explicit `max_iterations` and `chains` parameters instead of `**kwargs: object`.
+- **Coverage:** `fail_under` raised from 50 to **75** (current suite ~75%; 85% remains a stretch goal with more `api` / fingerprint tests).
+
+### Fixed
+
+- `DocumentSegment` / `MultiEncodingResult` now use `slots` on Python 3.10+ like other dataclasses in the package.
+
+## [0.1.1] — 2025-03-26
+
+### Added
+
+- `examples/` scripts (basic detection, `detect()`, streaming, repair, HTTP hints, multi-encoding)
+- MkDocs documentation site and GitHub Actions workflow **Docs Pages** → [GitHub Pages](https://oguzhankir.github.io/bytesense/)
+- `[project.optional-dependencies]` group `docs` (`mkdocs`, `mkdocs-material`, `pymdown-extensions`)
+- `project.urls.Documentation` in `pyproject.toml`
+
+### Fixed
+
+- README logo on PyPI: use `raw.githubusercontent.com` URL (sdist has no `assets/` for the image)
+- Source distribution: include `LICENSE` in maturin `include` so PyPI accepts `License-File` metadata
+
+## [0.1.0] — 2025-03-26
+
+First published release.
+
+### Added
+
+- `from_bytes()`, `from_path()`, `from_fp()`, `is_binary()` API
+- `detect()` chardet / charset-normalizer drop-in compatibility
+- `detect_stream()` for iterator-based streaming detection
+- `StreamDetector` for incremental detection; `snapshot()`, `is_stable`, auto-stop when encoding is stable (configurable)
+- In-band hints from HTML meta tags and XML declarations in `StreamDetector`
+- Byte-distribution fingerprinting with pre-computed lookup table
+- `repair()` and `repair_bytes()` for mojibake detection and repair; `is_mojibake()`; `RepairResult`
+- `hint_from_http_headers()`, `hint_from_content()`, `best_hint()` for standalone hint extraction
+- `detect_multi()` for multi-encoding documents; `MultiEncodingResult` and `DocumentSegment`
+- Optional Rust core extension (`pip install "bytesense[fast]"`)
+- CLI: `bytesense <file>`
+- `py.typed` marker for PEP 561 compliance
+- Full type annotations, mypy strict mode
+- GitHub Actions CI across 3 OS × multiple Python versions
+
+### Changed
+
+- `LANGUAGE_ENCODINGS` aligned with languages in `CHAR_FREQUENCIES`
+- Candidate shortlist tuned for speed while preserving accuracy targets
+
+### Fixed
+
+- Benchmark and packaging fixes (e.g. UTF-8 BOM test fixture, build backend configuration)

bytesense-1.0.0/CODE_OF_CONDUCT.md

@@ -0,0 +1,83 @@
+# Contributor Covenant Code of Conduct
+
+## Our pledge
+
+We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
+
+We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
+
+## Our standards
+
+Examples of behavior that contributes to a positive environment for our community include:
+
+- Demonstrating empathy and kindness toward other people
+- Being respectful of differing opinions, viewpoints, and experiences
+- Giving and gracefully accepting constructive feedback
+- Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
+- Focusing on what is best not just for us as individuals, but for the overall community
+
+Examples of unacceptable behavior include:
+
+- The use of sexualized language or imagery, and sexual attention or advances of any kind
+- Trolling, insulting or derogatory comments, and personal or political attacks
+- Public or private harassment
+- Publishing others’ private information, such as a physical or email address, without their explicit permission
+- Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Enforcement responsibilities
+
+Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
+
+Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
+
+## Scope
+
+This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at **oguzhankir17@gmail.com**. All complaints will be reviewed and investigated promptly and fairly.
+
+All community leaders are obligated to respect the privacy and security of the reporter of any incident.
+
+## Enforcement guidelines
+
+Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
+
+### 1. Correction
+
+**Community Impact**: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
+
+**Consequence**: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
+
+### 2. Warning
+
+**Community Impact**: A violation through a single incident or series of actions.
+
+**Consequence**: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
+
+### 3. Temporary ban
+
+**Community Impact**: A serious violation of community standards, including sustained inappropriate behavior.
+
+**Consequence**: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
+
+### 4. Permanent ban
+
+**Community Impact**: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
+
+**Consequence**: A permanent ban from any sort of public interaction within the community.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1, available at [https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
+
+Community Impact Guidelines were inspired by [Mozilla’s code of conduct enforcement ladder][Mozilla CoC].
+
+For answers to common questions about this code of conduct, see the FAQ at [https://www.contributor-covenant.org/faq][FAQ]. Translations are available at [https://www.contributor-covenant.org/translations][translations].
+
+[homepage]: https://www.contributor-covenant.org
+[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
+[Mozilla CoC]: https://github.com/mozilla/diversity
+[FAQ]: https://www.contributor-covenant.org/faq
+[translations]: https://www.contributor-covenant.org/translations

bytesense-1.0.0/CONTRIBUTING.md

@@ -0,0 +1,47 @@
+# Contributing
+
+Follow the [Code of Conduct](CODE_OF_CONDUCT.md). Use a focused `oguzhankir/<topic>` branch and sign commits with `git commit -s` under the project's DCO.
+
+## Development
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+BYTESENSE_BUILD_RUST=0 pip install -e '.[dev,docs]'
+python scripts/fetch_cn_benchmark_samples.py
+ruff check src tests benchmarks scripts setup.py
+mypy src/bytesense
+BYTESENSE_PURE_PYTHON=1 pytest tests --cov=bytesense --cov-branch
+pytest benchmarks/test_bench_detection.py benchmarks/test_hard_scenarios.py --benchmark-disable
+mkdocs build --strict
+```
+
+On Windows, set environment variables using your shell's syntax. Native development requires Rust 1.88+:
+
+```bash
+BYTESENSE_BUILD_RUST=1 pip install -e . --no-build-isolation
+BYTESENSE_EXPECT_RUST=1 pytest tests
+cargo test --manifest-path rust/Cargo.toml --locked
+```
+
+CI tests pure and native installations separately on Linux, macOS and Windows. Packaging jobs install and test actual wheels, check abi3 compatibility, and test the sdist without compiling Rust. The coverage floor remains 75%; timing measurements are reports, not noisy speed assertions.
+
+## Model and benchmarks
+
+The [benchmark documentation](docs/benchmarks.md) describes corpus pins, Unicode-based grouping and reproduction commands. Keep development and holdout documents separate. Do not tune the model on the holdout, silently omit missing inputs, compare differently sized payloads, or advertise sample-only latency as full-input validation latency.
+
+`language.json.gz` contains aggregate character-pair statistics; no test documents are bundled. The core retains static model metadata, never document text in a global cache. Native and Python paths must agree within numerical tolerance and produce the same rounded detection scores.
+
+## Release
+
+Keep `pyproject.toml`, `rust/Cargo.toml`, and `src/bytesense/version.py` aligned; `python scripts/check_version.py` checks them. Update the existing changelog and API/migration documentation when behavior changes.
+
+```bash
+BYTESENSE_BUILD_RUST=0 python -m build
+python scripts/check_dist.py dist
+python -m twine check --strict dist/*
+```
+
+Publishing a GitHub Release triggers the release workflow. It checks the tag, builds and tests portable/native distributions, then uses the configured PyPI trusted publisher environment. A PR does not publish a release.
+
+Keep documentation focused: README, quick start, API reference, benchmarks, and the shared changelog. Put rationale and validation detail in the PR instead of creating additional planning documents.

bytesense-1.0.0/MANIFEST.in

@@ -0,0 +1,8 @@
+include LICENSE README.md CHANGELOG.md
+recursive-include rust *.toml *.lock *.rs
+recursive-include src/bytesense *.py py.typed
+prune rust/target
+
+include CONTRIBUTING.md CODE_OF_CONDUCT.md
+recursive-include scripts *.py *.sh
+recursive-include benchmarks *.py cn_official_manifest.json

bytesense-1.0.0/PKG-INFO

@@ -0,0 +1,160 @@
+Metadata-Version: 2.4
+Name: bytesense
+Version: 1.0.0
+Summary: Encoding detection with full-input validation, bounded-memory streams, and optional Rust acceleration.
+Author-email: Oğuzhan Kır <oguzhankir17@gmail.com>
+Maintainer-email: Oğuzhan Kır <oguzhankir17@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/oguzhankir/bytesense
+Project-URL: Repository, https://github.com/oguzhankir/bytesense
+Project-URL: Issues, https://github.com/oguzhankir/bytesense/issues
+Project-URL: Changelog, https://github.com/oguzhankir/bytesense/blob/main/CHANGELOG.md
+Project-URL: Documentation, https://oguzhankir.github.io/bytesense/
+Keywords: encoding,charset,detection,unicode,bytes,chardet,charset-normalizer
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Classifier: Programming Language :: Rust
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Topic :: Utilities
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: fast
+Provides-Extra: docs
+Requires-Dist: mkdocs>=1.6; extra == "docs"
+Requires-Dist: mkdocs-material>=9.5; extra == "docs"
+Requires-Dist: pymdown-extensions>=10.0; extra == "docs"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
+Requires-Dist: pytest-benchmark>=4.0; extra == "dev"
+Requires-Dist: mypy>=1.8; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: hypothesis>=6.100; (platform_python_implementation != "PyPy" or python_version >= "3.11") and extra == "dev"
+Requires-Dist: hypothesis==6.100.0; (platform_python_implementation == "PyPy" and python_version < "3.11") and extra == "dev"
+Requires-Dist: setuptools-rust>=1.12; extra == "dev"
+Requires-Dist: charset-normalizer>=3.3; extra == "dev"
+Requires-Dist: chardet>=5.0; extra == "dev"
+Requires-Dist: requests>=2.31; extra == "dev"
+Requires-Dist: certifi>=2023.0; extra == "dev"
+Requires-Dist: twine>=5.0; extra == "dev"
+Dynamic: license-file
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/oguzhankir/bytesense/main/assets/bytesense_logo.svg" alt="bytesense" width="480" />
+</p>
+<p align="center"><strong>Detect the encoding. Validate every byte. Keep your data intact.</strong></p>
+<p align="center">
+  <a href="https://pypi.org/project/bytesense/"><img src="https://img.shields.io/pypi/v/bytesense.svg" alt="PyPI" /></a>
+  <a href="https://github.com/oguzhankir/bytesense/actions/workflows/ci.yml"><img src="https://github.com/oguzhankir/bytesense/actions/workflows/ci.yml/badge.svg" alt="CI" /></a>
+  <a href="https://github.com/oguzhankir/bytesense/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT license" /></a>
+</p>
+
+**bytesense** is a Python encoding detector for file imports, multilingual text pipelines and legacy data. It combines fast Unicode checks, compact character-pair statistics and optional Rust acceleration—with **zero runtime dependencies**.
+
+Every returned encoding strictly decodes the complete supplied input. Linguistic scoring uses a bounded sample; stream inputs spill to a temporary file after a configurable memory limit. Results tell you what was examined, what was validated, and whether the input ended.
+
+```python
+from bytesense import from_bytes
+
+data = "İstanbul'da çalışan mühendisler için doğru metin çözümleme. ".encode("cp1254")
+result = from_bytes(data)
+if result.encoding:
+    text = data.decode(result.encoding)  # strict decoding; no replacement characters
+    print(result.encoding, result.bytes_validated, result.complete)
+```
+
+## Install
+
+```bash
+pip install bytesense
+```
+
+Python **3.9+**. Platform wheels include Rust acceleration; the portable wheel works without a compiler. Source installations automatically use Rust when a toolchain is available. To explicitly choose a source build:
+
+```bash
+BYTESENSE_BUILD_RUST=0 pip install --no-binary=bytesense bytesense  # pure Python
+BYTESENSE_BUILD_RUST=1 pip install --no-binary=bytesense bytesense  # require Rust
+```
+
+The old `[fast]` extra is a compatibility alias. It does not install an accelerator separately. `BYTESENSE_PURE_PYTHON=1` selects the Python implementation at runtime.
+
+## Built for dependable imports
+
+- **Complete validation:** a valid prefix cannot hide an undecodable suffix. Unknown and binary inputs have explicit outcomes.
+- **Bounded streams:** feed small chunks or large ones; finalization accounts for every accepted byte. Preview stability never silently ends a stream.
+- **Useful controls:** codec aliases, inclusion/exclusion filters, encoding hints, optional language estimates and a minimum evidence score.
+- **Transparent results:** `why`, alternatives, sample/validation counts and completion status. Confidence is a heuristic score; no fabricated statistical intervals.
+- **Portable acceleration:** native character-pair scoring and byte histograms, with matching Python behavior and typed public APIs.
+- **Data repair tools:** opt-in mojibake repair and byte-preserving mixed-document segmentation, with strict decoding.
+
+Decodability alone cannot prove the original encoding. Several legacy codecs can decode identical bytes into different, plausible text. Use a known encoding or an explicit hint when you have one, and evaluate representative data before switching detectors.
+
+## Files, streams and existing integrations
+
+```python
+from bytesense import detect, detect_stream, from_path
+
+result = from_path("export.csv", include_language=True)
+print(result.to_dict())
+
+with open("archive.txt", "rb") as source:
+    result = detect_stream(iter(lambda: source.read(64 * 1024), b""))
+assert result.complete
+
+# Familiar dictionary interface for code that uses chardet.detect().
+metadata = detect(b"hello world")
+```
+
+`detect_stream()` consumes to EOF by default. `max_bytes=...` or `early_stop=True` is an explicit sampling choice and returns `complete=False` when it stops early. Streaming uses temporary disk space beyond `memory_limit` (1 MiB by default); close an unfinished `StreamDetector` or use it as a context manager.
+
+```bash
+bytesense report.csv
+bytesense --language --verbose report.csv
+cat report.csv | bytesense --minimal -
+```
+
+The CLI exits **0** when every input has an encoding, **1** for read failures or unknown/binary inputs, and **2** for invalid arguments. JSON goes to stdout; errors go to stderr.
+
+## Measured accuracy and performance
+
+The v1 engine replaces the previous collection of special-case ranking rules with a small, reproducibly generated statistical model. The evaluation separates development documents from held-out documents, grouping identical decoded text across encodings. Accuracy means **exact Unicode equality**, not just a compatible codec name.
+
+| Detector | Exact Unicode matches | Accuracy |
+|---|---:|---:|
+| **bytesense 1.0.0** | **1906 / 2078** | **91.72%** |
+| bytesense 0.1.2 | 883 / 2078 | 42.49% |
+| chardet 7.6.0 | 2060 / 2078 | 99.13% |
+| charset-normalizer 3.5.1 | 1679 / 2078 | 80.80% |
+| chardetng-py 0.3.5 | 776 / 2078 | 37.34% |
+
+See [benchmarks and methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for measured comparisons with chardet, charset-normalizer and chardetng, including the cases where another detector wins. Timing reports distinguish default behavior from an additional full-decode validation step.
+
+## Upgrading from 0.x
+
+- Language reporting is opt-in: `include_language=True`.
+- `confidence_interval` remains available and is `None`; scores are not calibrated probabilities.
+- An empty `cp_isolation=[]` allows no codecs. Filters apply to every detection path.
+- BOM-bearing UTF-16/32 selects `utf_16`/`utf_32`, which consume the signature when decoding.
+- No unvalidated UTF-8 fallback; stream finalization and repair decoding reject silent data loss.
+- `StreamDetector.feed()` after finalization raises. `reset()` starts a new stream.
+- Python 3.8 is no longer supported. `steps` and `chunk_size` remain accepted compatibility arguments; use `sample_size` to control statistical work.
+
+[Quick start](https://oguzhankir.github.io/bytesense/quickstart/) · [API](https://oguzhankir.github.io/bytesense/api/) · [Changes](https://github.com/oguzhankir/bytesense/blob/main/CHANGELOG.md) · [Contributing](https://github.com/oguzhankir/bytesense/blob/main/CONTRIBUTING.md)
+
+Created by [Oğuzhan Kır](https://github.com/oguzhankir). MIT-licensed code. The aggregate model's data provenance and reproduction steps are documented with the benchmarks; upstream test documents remain copyrighted by their respective publishers.

bytesense-1.0.0/README.md

@@ -0,0 +1,103 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/oguzhankir/bytesense/main/assets/bytesense_logo.svg" alt="bytesense" width="480" />
+</p>
+<p align="center"><strong>Detect the encoding. Validate every byte. Keep your data intact.</strong></p>
+<p align="center">
+  <a href="https://pypi.org/project/bytesense/"><img src="https://img.shields.io/pypi/v/bytesense.svg" alt="PyPI" /></a>
+  <a href="https://github.com/oguzhankir/bytesense/actions/workflows/ci.yml"><img src="https://github.com/oguzhankir/bytesense/actions/workflows/ci.yml/badge.svg" alt="CI" /></a>
+  <a href="https://github.com/oguzhankir/bytesense/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT license" /></a>
+</p>
+
+**bytesense** is a Python encoding detector for file imports, multilingual text pipelines and legacy data. It combines fast Unicode checks, compact character-pair statistics and optional Rust acceleration—with **zero runtime dependencies**.
+
+Every returned encoding strictly decodes the complete supplied input. Linguistic scoring uses a bounded sample; stream inputs spill to a temporary file after a configurable memory limit. Results tell you what was examined, what was validated, and whether the input ended.
+
+```python
+from bytesense import from_bytes
+
+data = "İstanbul'da çalışan mühendisler için doğru metin çözümleme. ".encode("cp1254")
+result = from_bytes(data)
+if result.encoding:
+    text = data.decode(result.encoding)  # strict decoding; no replacement characters
+    print(result.encoding, result.bytes_validated, result.complete)
+```
+
+## Install
+
+```bash
+pip install bytesense
+```
+
+Python **3.9+**. Platform wheels include Rust acceleration; the portable wheel works without a compiler. Source installations automatically use Rust when a toolchain is available. To explicitly choose a source build:
+
+```bash
+BYTESENSE_BUILD_RUST=0 pip install --no-binary=bytesense bytesense  # pure Python
+BYTESENSE_BUILD_RUST=1 pip install --no-binary=bytesense bytesense  # require Rust
+```
+
+The old `[fast]` extra is a compatibility alias. It does not install an accelerator separately. `BYTESENSE_PURE_PYTHON=1` selects the Python implementation at runtime.
+
+## Built for dependable imports
+
+- **Complete validation:** a valid prefix cannot hide an undecodable suffix. Unknown and binary inputs have explicit outcomes.
+- **Bounded streams:** feed small chunks or large ones; finalization accounts for every accepted byte. Preview stability never silently ends a stream.
+- **Useful controls:** codec aliases, inclusion/exclusion filters, encoding hints, optional language estimates and a minimum evidence score.
+- **Transparent results:** `why`, alternatives, sample/validation counts and completion status. Confidence is a heuristic score; no fabricated statistical intervals.
+- **Portable acceleration:** native character-pair scoring and byte histograms, with matching Python behavior and typed public APIs.
+- **Data repair tools:** opt-in mojibake repair and byte-preserving mixed-document segmentation, with strict decoding.
+
+Decodability alone cannot prove the original encoding. Several legacy codecs can decode identical bytes into different, plausible text. Use a known encoding or an explicit hint when you have one, and evaluate representative data before switching detectors.
+
+## Files, streams and existing integrations
+
+```python
+from bytesense import detect, detect_stream, from_path
+
+result = from_path("export.csv", include_language=True)
+print(result.to_dict())
+
+with open("archive.txt", "rb") as source:
+    result = detect_stream(iter(lambda: source.read(64 * 1024), b""))
+assert result.complete
+
+# Familiar dictionary interface for code that uses chardet.detect().
+metadata = detect(b"hello world")
+```
+
+`detect_stream()` consumes to EOF by default. `max_bytes=...` or `early_stop=True` is an explicit sampling choice and returns `complete=False` when it stops early. Streaming uses temporary disk space beyond `memory_limit` (1 MiB by default); close an unfinished `StreamDetector` or use it as a context manager.
+
+```bash
+bytesense report.csv
+bytesense --language --verbose report.csv
+cat report.csv | bytesense --minimal -
+```
+
+The CLI exits **0** when every input has an encoding, **1** for read failures or unknown/binary inputs, and **2** for invalid arguments. JSON goes to stdout; errors go to stderr.
+
+## Measured accuracy and performance
+
+The v1 engine replaces the previous collection of special-case ranking rules with a small, reproducibly generated statistical model. The evaluation separates development documents from held-out documents, grouping identical decoded text across encodings. Accuracy means **exact Unicode equality**, not just a compatible codec name.
+
+| Detector | Exact Unicode matches | Accuracy |
+|---|---:|---:|
+| **bytesense 1.0.0** | **1906 / 2078** | **91.72%** |
+| bytesense 0.1.2 | 883 / 2078 | 42.49% |
+| chardet 7.6.0 | 2060 / 2078 | 99.13% |
+| charset-normalizer 3.5.1 | 1679 / 2078 | 80.80% |
+| chardetng-py 0.3.5 | 776 / 2078 | 37.34% |
+
+See [benchmarks and methodology](https://oguzhankir.github.io/bytesense/benchmarks/) for measured comparisons with chardet, charset-normalizer and chardetng, including the cases where another detector wins. Timing reports distinguish default behavior from an additional full-decode validation step.
+
+## Upgrading from 0.x
+
+- Language reporting is opt-in: `include_language=True`.
+- `confidence_interval` remains available and is `None`; scores are not calibrated probabilities.
+- An empty `cp_isolation=[]` allows no codecs. Filters apply to every detection path.
+- BOM-bearing UTF-16/32 selects `utf_16`/`utf_32`, which consume the signature when decoding.
+- No unvalidated UTF-8 fallback; stream finalization and repair decoding reject silent data loss.
+- `StreamDetector.feed()` after finalization raises. `reset()` starts a new stream.
+- Python 3.8 is no longer supported. `steps` and `chunk_size` remain accepted compatibility arguments; use `sample_size` to control statistical work.
+
+[Quick start](https://oguzhankir.github.io/bytesense/quickstart/) · [API](https://oguzhankir.github.io/bytesense/api/) · [Changes](https://github.com/oguzhankir/bytesense/blob/main/CHANGELOG.md) · [Contributing](https://github.com/oguzhankir/bytesense/blob/main/CONTRIBUTING.md)
+
+Created by [Oğuzhan Kır](https://github.com/oguzhankir). MIT-licensed code. The aggregate model's data provenance and reproduction steps are documented with the benchmarks; upstream test documents remain copyrighted by their respective publishers.

bytesense-1.0.0/benchmarks/cn_official_manifest.json

@@ -0,0 +1,118 @@
+{
+  "source_repo": "https://github.com/jawah/charset_normalizer",
+  "source_path": "data/",
+  "reference_test": "tests/test_full_detection.py",
+  "notice": "Sample files originate from charset-normalizer data/ (see upstream data/NOTICE.md). Used only for benchmark comparison.",
+  "base_url": "https://raw.githubusercontent.com/jawah/charset_normalizer/b130b7dae36658c7ba67381fe06a62c7c7c03894/data/",
+  "files": [
+    {
+      "file": "sample-arabic-1.txt",
+      "encoding": "cp1256",
+      "sha256": "00c83c71aaf34f6c17ee5ed4e6ad868fe7f281c56d96d5f88f21fc5d39203648",
+      "size": 906
+    },
+    {
+      "file": "sample-french-1.txt",
+      "encoding": "cp1252",
+      "sha256": "6b88988aa8cfd689df08f91a25ae0ea8032cc28b4557092432849a2712fce716",
+      "size": 3251
+    },
+    {
+      "file": "sample-arabic.txt",
+      "encoding": "utf_8",
+      "sha256": "3606d50f8dbd20d80aa4470bd89e74696ceda8ec9138f9524330869eb9aafa46",
+      "size": 1637
+    },
+    {
+      "file": "sample-russian-3.txt",
+      "encoding": "utf_8",
+      "sha256": "81c49a881b7175f68201edc41966016b5c0f16ddb4ac4a30407af0e4ac8b84e6",
+      "size": 3070
+    },
+    {
+      "file": "sample-french.txt",
+      "encoding": "utf_8",
+      "sha256": "ab1b0ebf22b7bd85d2a45600844c0a2c89ba6217b862a6d96b9fa46ce1e132bb",
+      "size": 3375
+    },
+    {
+      "file": "sample-chinese.txt",
+      "encoding": "big5",
+      "sha256": "5f4aa09479ff9366936b242697680f4f0a50dcb3829fb10d855644bc035a7b7a",
+      "size": 743
+    },
+    {
+      "file": "sample-greek.txt",
+      "encoding": "cp1253",
+      "sha256": "cf9afae3a58f28a422fa8edecf255744a8cffb559262d3b3f7dfa3e1d6f59cb4",
+      "size": 570
+    },
+    {
+      "file": "sample-greek-2.txt",
+      "encoding": "cp1253",
+      "sha256": "cf9afae3a58f28a422fa8edecf255744a8cffb559262d3b3f7dfa3e1d6f59cb4",
+      "size": 570
+    },
+    {
+      "file": "sample-hebrew-2.txt",
+      "encoding": "cp1255",
+      "sha256": "eb3a0aa5646487ee6b06c7ef8a9223a89cc2c7b920c787f982ba712d8fedf5f4",
+      "size": 340
+    },
+    {
+      "file": "sample-hebrew-3.txt",
+      "encoding": "cp1255",
+      "sha256": "eb3a0aa5646487ee6b06c7ef8a9223a89cc2c7b920c787f982ba712d8fedf5f4",
+      "size": 340
+    },
+    {
+      "file": "sample-bulgarian.txt",
+      "encoding": "utf_8",
+      "sha256": "d6f347a930569baa7150ab528cfd64420fd9335d89d6617cbed4addec7c4ad30",
+      "size": 2198
+    },
+    {
+      "file": "sample-english.bom.txt",
+      "encoding": "utf_8",
+      "sha256": "4a5850a424c075e25e86fbee489561d5869efdb42297ed08ae074238f312e818",
+      "size": 859
+    },
+    {
+      "file": "sample-spanish.txt",
+      "encoding": "utf_8",
+      "sha256": "b43d16b5398e26565153b1c1118a7f040fa2350493eac5cea51721ffe5367edd",
+      "size": 7281
+    },
+    {
+      "file": "sample-korean.txt",
+      "encoding": "cp949",
+      "sha256": "281246aaaefceb47f6bc0120b2646ebe1832d57101cc0d7bfaeedfbfed979329",
+      "size": 387
+    },
+    {
+      "file": "sample-turkish.txt",
+      "encoding": "cp1254",
+      "sha256": "df265e595aac51e92faafd41afefadbd21b820d1be3eec2e35dab4213fb16fba",
+      "size": 1840
+    },
+    {
+      "file": "sample-russian-2.txt",
+      "encoding": "utf_8",
+      "sha256": "2492ff4b9b15c174a998457ff02233cd1367bdfa5d7c066145f15616aaaa941a",
+      "size": 2209
+    },
+    {
+      "file": "sample-russian.txt",
+      "encoding": "mac_cyrillic",
+      "sha256": "becee0937e700266497ef4cbe84352be0449476a9299a7fc7c18647344e3ff8c",
+      "size": 1211
+    },
+    {
+      "file": "sample-polish.txt",
+      "encoding": "utf_8",
+      "sha256": "fe130e75df06b484e1a00cfa6c7679f2ab2b2c44f9a69780b89e729c651e5fcf",
+      "size": 5815
+    }
+  ],
+  "source_commit": "b130b7dae36658c7ba67381fe06a62c7c7c03894"
+}