hashcodecs 1.4.0__tar.gz → 1.4.2__tar.gz

hashcodecs-1.4.2/BENCHMARK.md

@@ -0,0 +1,165 @@
+# Benchmarks
+
+Use this reference to compare `hashcodecs` APIs and reproduce the measurements for a change. For the implementation
+behind the results, read [Architecture](docs/ARCHITECTURE.md).
+
+## Measurement conditions
+
+| Setting | Recorded comparison |
+| --- | --- |
+| Host | Windows 10 x64, Intel Core Ultra 7 265K |
+| Execution | One thread, pinned to one logical CPU |
+| Rust | Criterion, 50 samples per case, release optimization |
+| Python timing | Median of 15 samples, calibrated to at least 0.2 seconds per sample |
+| Python | Free-threaded CPython 3.14.6 (`3.14t`), GIL disabled |
+| Python baselines | pybase64 1.5.0, mmh3 5.3.0, xxhash 4.0.1 |
+| Allocation | `mimalloc` for Rust benchmark allocations and Rust allocations in the Python extension |
+| XXH3 C baseline | xxHash 0.8.3 through `xxhash-c-sys`, compiled with AVX2 to match this host's selected backend |
+
+Charts report GiB/s (2³⁰ bytes per second). Higher values mean greater throughput. Base64 uses the original binary
+payload size for both encoding and decoding. Batch throughput counts the total payload across items. Default
+Python Base64 comparisons use immutable `bytes` and strict decoding (`validate=True`).
+
+Cases that return output include result allocation. Cases with reusable output allocate destinations before timing.
+The harnesses reuse inputs across iterations, so cache residency affects the results. Compare like input types,
+sizes, and output models. These measurements describe this host and workload.
+
+The repository's [results.csv](docs/benchmarks/results.csv) supplies the chart values. Focused updates replace the
+affected series and retain the other measurements, so the charts combine results from multiple runs.
+
+## Base64 results
+
+| API and workload | Charts |
+| --- | --- |
+| Rust, standard Base64 and Base64 for URLs | [Rust Base64](docs/benchmarks/base64-rust.svg) |
+| Python, standard Base64 and Base64 for URLs | [Python Base64](docs/benchmarks/base64-python.svg) |
+| Python, reusable `bytearray` output | [Reusable buffers](docs/benchmarks/base64-python-reusable.svg) |
+| Python, MIME whitespace and ignored characters outside the alphabet | [Lenient decoding](docs/benchmarks/base64-python-lenient.svg) |
+| Python, full and sliced immutable views | [Memoryview inputs](docs/benchmarks/base64-python-memoryview.svg) |
+| Python, ASCII `str` decoding | [String inputs](docs/benchmarks/base64-python-str.svg) |
+| Python, mutable `bytearray` input | [Mutable inputs](docs/benchmarks/base64-python-mutable.svg) |
+| Python, batches | [Returned bytes](docs/benchmarks/base64-python-batch.svg), [reusable outputs](docs/benchmarks/base64-python-batch-reusable.svg) |
+| Python, batches of memoryviews | [Memoryview batches](docs/benchmarks/base64-python-batch-memoryview.svg) |
+| Python, batches of 1 MiB items | [Large batches](docs/benchmarks/base64-python-batch-large.svg) |
+
+Batch charts use item count on the horizontal axis. Reusable Base64 batches provide one destination per item.
+Lenient cases insert CRLF or `!` after each line of 76 characters.
+
+## MurmurHash3 results
+
+| API and workload | Charts |
+| --- | --- |
+| Rust, x86-32, x86-128, and x64-128 | [Rust MurmurHash3](docs/benchmarks/murmur3-rust.svg) |
+| Python, single calls and incremental hashing | [Python MurmurHash3](docs/benchmarks/murmur3-python.svg) |
+| Python, mutable `bytearray` input | [Mutable inputs](docs/benchmarks/murmur3-python-mutable.svg) |
+
+Incremental cases include construction, `update`, and digest creation.
+
+## XXH3 results
+
+| API and workload | Charts |
+| --- | --- |
+| Rust, XXH3-64 and XXH3-128, single calls and batches of 32 items | [Rust XXH3](docs/benchmarks/xxh3-rust.svg) |
+| Rust, batches of two and three items | [Batch remainders](docs/benchmarks/xxh3-rust-batch-remainders.svg) |
+| Python, single calls, list results, and packed output | [Python XXH3](docs/benchmarks/xxh3-python.svg) |
+
+Rust allocating batches include allocation of the result vector. Python list batches create one integer per digest.
+Packed batches write digests in little endian order into one reusable `bytearray`. Python batch comparisons use
+32 inputs of equal size by default and compare against the upstream `xxhash` extension.
+
+The Python small-input panels cover 16, 17, 33, 65, 97, and 240 bytes. These sizes use scalar formulas; runtime SIMD
+dispatch starts at 241 bytes.
+
+## Reproduce a benchmark
+
+Run commands from the repository root. Install Rust 1.89 or newer, a C/C++ compiler and linker for your platform,
+and `uv`. Use free-threaded CPython 3.14 for the Python charts. The Python setup below builds and installs the
+current checkout as a CPython wheel through Hatchling.
+
+Run the group affected by your change. Reserve a complete run for changes that can affect all groups. Keep
+benchmarks out of CI. The standard harnesses set CPU affinity on Windows and Linux. On other platforms, arrange
+equivalent CPU pinning before collecting comparison results.
+
+### Rust
+
+Choose the relevant harness:
+
+```sh
+cargo bench --manifest-path benches/Cargo.toml --bench base64
+cargo bench --manifest-path benches/Cargo.toml --bench murmur3
+cargo bench --manifest-path benches/Cargo.toml --bench xxhash
+```
+
+To reproduce the Windows XXH3 C baseline, set the compiler flags and rebuild its package before running the harness:
+
+```powershell
+$env:CFLAGS = '/O2 /arch:AVX2'
+cargo clean -p xxhash-c-sys
+cargo bench --manifest-path benches/Cargo.toml --bench xxhash
+```
+
+Use the equivalent AVX2 compiler flag on other platforms. Match the baseline instruction set to the backend under
+comparison. Pass a Criterion name filter after `--` to narrow a run:
+
+```sh
+cargo bench --manifest-path benches/Cargo.toml --bench murmur3 -- x64_128/hashcodecs
+```
+
+### Python
+
+Prepare the free-threaded interpreter (`3.14t`) and pinned benchmark dependencies. Repeat the wheel installation
+after changing native code or switching interpreters:
+
+```sh
+uv sync --python 3.14t --frozen --group benchmark --no-install-project
+uv run --python 3.14t --frozen --no-sync python tools/install_local_wheel.py
+```
+
+Choose the relevant script:
+
+```sh
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_base64.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_base64_batch.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_murmur3.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_xxhash.py
+```
+
+The scripts check outputs before timing and print GiB/s. Append a mode from the table to select a workload. Use
+`--help` for the full option list.
+
+| Script | Focused modes |
+| --- | --- |
+| `python_base64.py` | `--into`, `--lenient`, `--custom-lenient`, `--bytearray-input`, `--memoryview-input`, `--sliced-memoryview-input`, `--str-input` |
+| `python_base64_batch.py` | `--large`, `--memoryview-input`, `--decode-only`. Select sizes with `--item-sizes` and `--batch-sizes`. |
+| `python_murmur3.py` | `--incremental`, `--bytearray-input` |
+| `python_xxhash.py` | `--one-shot-only`, `--sizes 17 33 65 97 240`, `--batches-only`, `--batch-counts 2 3` |
+
+`python_base64.py` accepts one mode per run. Its `--configured` and `--wrapped` modes require CPython 3.15 or newer.
+Use that interpreter for both setup commands and the benchmark. These modes cover configured decoding and
+encoding with newlines after 76 output characters.
+
+The four scripts accept `--hashcodecs-only` to skip competitor timing. The Base64 script treats that flag as a
+mode, so run it apart from `--into`, `--lenient`, and the other Base64 modes. All four accept `--samples` and
+`--minimum-sample-seconds`. Keep the defaults for published comparisons. Lower them for local exploration.
+
+For call overhead, run `benchmarks/python_calls.py` with the same `uv run` prefix. It reports nanoseconds per call
+and supports `--keywords`, `--thresholds`, and `--buffer-inputs`. Its `--thread-scaling` mode measures aggregate
+throughput without pinning to one CPU. Keep those results separate from the charts above.
+
+## Update the charts
+
+1. Copy the measured GiB/s values into [results.csv](docs/benchmarks/results.csv). Update the series you measured.
+   Retain unmeasured values and leave `gib_per_second` empty for unavailable results.
+2. Keep matching input categories across implementations in each panel. Each row identifies a chart, panel,
+   input category, and implementation. Row order controls chart, panel, category, and legend order.
+3. Render the SVG files:
+
+   ```sh
+   uv run --python 3.14t --no-project python benchmarks/render_charts.py
+   ```
+
+4. Review the CSV and SVG diff. A focused update should change the corresponding charts, including any README
+   overview that uses those values.
+
+The renderer reads the CSV without running benchmarks or changing measurements. Keep temporary tuning sweeps,
+branch comparisons, and profiler output out of this reference.

{hashcodecs-1.4.0 → hashcodecs-1.4.2}/CHANGELOG.md

@@ -4,6 +4,38 @@ This file records notable user-facing changes to `hashcodecs`. Version 1.0.0 sta
 
 ## [Unreleased]
 
+## [1.4.2] - 2026-09-29
+
+### What's Changed
+* perf: Improve AVX2 Base64 decoding throughput by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/125
+* perf: Reduce SSE Base64 decode stores by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/126
+* perf: Reduce Python Base64 ASCII decoding overhead by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/127
+* docs: Clarify benchmarks, architecture, and safety guidance by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/128
+* chore: Refresh all Python benchmarks on free-threaded 3.14 by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/129
+* perf: Reduce Python XXH3 seed conversion overhead by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/130
+* perf: Speed up short XXH3 inputs by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/131
+
+
+**Full Changelog**: https://github.com/kozistr/hashcodecs-rs/compare/v1.4.1...v1.4.2
+
+## [1.4.1] - 2026-09-13
+
+### What's Changed
+* fix: match CPython Base64 and XXH3 buffer semantics by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/114
+* fix: preserve Base64 callback buffer semantics by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/115
+* fix: preserve CPython Base64 observable behavior by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/116
+* fix: resolve repository type diagnostics by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/117
+* fix: release Base64 input buffers before writing output by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/118
+* refactor: centralize buffer callback policy and clarify finalizers by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/119
+* fix: accelerate custom-alphabet lenient Base64 decoding by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/120
+* fix: reduce XXH3 packed batch detachment overhead by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/121
+* fix: delay MurmurHash3 x64 SIMD dispatch until 512 bytes by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/122
+* fix: restore Python buffer and batch throughput by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/123
+* fix: typo in the performance viz image by @kozistr in https://github.com/kozistr/hashcodecs-rs/pull/124
+
+
+**Full Changelog**: https://github.com/kozistr/hashcodecs-rs/compare/v1.4.0...v1.4.1
+
 ## [1.4.0] - 2026-09-10
 
 ### What's Changed
@@ -193,7 +225,9 @@ This file records notable user-facing changes to `hashcodecs`. Version 1.0.0 sta
 - Initial Python and Rust APIs for Base64 and MurmurHash3.
 - Runtime SIMD dispatch and platform-specific CPython wheels.
 
-[Unreleased]: https://github.com/kozistr/hashcodecs-rs/compare/v1.4.0...HEAD
+[Unreleased]: https://github.com/kozistr/hashcodecs-rs/compare/v1.4.2...HEAD
+[1.4.2]: https://github.com/kozistr/hashcodecs-rs/compare/v1.4.1...v1.4.2
+[1.4.1]: https://github.com/kozistr/hashcodecs-rs/compare/v1.4.0...v1.4.1
 [1.4.0]: https://github.com/kozistr/hashcodecs-rs/compare/v1.3.0...v1.4.0
 [1.3.0]: https://github.com/kozistr/hashcodecs-rs/compare/v1.2.1...v1.3.0
 [1.2.1]: https://github.com/kozistr/hashcodecs-rs/compare/v1.2.0...v1.2.1

{hashcodecs-1.4.0 → hashcodecs-1.4.2}/CITATION.cff

@@ -5,8 +5,8 @@ authors:
     given-names: Hyeongchan
     orcid: https://orcid.org/0000-0002-1729-0580
 title: "hashcodecs: SIMD-accelerated Base64, MurmurHash3, and XXH3 for Python and Rust"
-version: 1.4.0
-date-released: 2026-09-10
+version: 1.4.2
+date-released: 2026-09-29
 license: "MIT OR Apache-2.0"
 repository-code: "https://github.com/kozistr/hashcodecs-rs"
 url: "https://github.com/kozistr/hashcodecs-rs"

{hashcodecs-1.4.0 → hashcodecs-1.4.2}/Cargo.lock

@@ -194,7 +194,7 @@ dependencies = [
 
 [[package]]
 name = "hashcodecs"
-version = "1.4.0"
+version = "1.4.2"
 dependencies = [
  "base64",
  "memchr",

{hashcodecs-1.4.0 → hashcodecs-1.4.2}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "hashcodecs"
-version = "1.4.0"
+version = "1.4.2"
 edition = "2024"
 rust-version = "1.89"
 autobenches = false

{hashcodecs-1.4.0 → hashcodecs-1.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hashcodecs
-Version: 1.4.0
+Version: 1.4.2
 Summary: SIMD-accelerated Base64, MurmurHash3, and xxHash codecs
 Project-URL: Documentation, https://hashcodecs-rs.readthedocs.io/
 Project-URL: Repository, https://github.com/kozistr/hashcodecs-rs
@@ -43,7 +43,7 @@ Description-Content-Type: text/markdown
 
 <p align="center">
   <a href="BENCHMARK.md">
-    <img src="docs/benchmarks/performance-at-a-glance.svg" alt="CPython 3.12 standard Base64 encoding and decoding benchmark">
+    <img src="docs/benchmarks/performance-at-a-glance.svg" alt="Free-threaded CPython 3.14.6 standard Base64 encoding and decoding benchmark">
   </a>
 </p>
 
@@ -184,6 +184,7 @@ CPython boundary, and safety invariants.
 
 Run the suite on Windows 10 x64 with an Intel Core Ultra 7 265K. Pin one logical CPU and run each case in one thread.
 Collect 50 Rust samples and 15 Python samples. Higher throughput wins.
+Use free-threaded CPython 3.14.6 with the GIL disabled for all Python charts.
 
 ### Base64: Rust
 
@@ -230,12 +231,13 @@ cargo bench --manifest-path benches/Cargo.toml --bench murmur3
 cargo bench --manifest-path benches/Cargo.toml --bench xxhash
 cargo bench --manifest-path benches/Cargo.toml --bench crossover
 
-uv sync --group benchmark --no-install-project
-uv run --no-project --with . python benchmarks/python_base64.py
-uv run --no-project --with . python benchmarks/python_base64_batch.py
-uv run --no-project --with . python benchmarks/python_calls.py
-uv run --no-project --with . python benchmarks/python_murmur3.py
-uv run --no-project --with . python benchmarks/python_xxhash.py
+uv sync --python 3.14t --frozen --group benchmark --no-install-project
+uv run --python 3.14t --frozen --no-sync python tools/install_local_wheel.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_base64.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_base64_batch.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_calls.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_murmur3.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_xxhash.py
 ```
 
 The Python benchmarks expose focused modes such as `--into`, `--lenient`, `--bytearray-input`, `--memoryview-input`,
@@ -253,11 +255,10 @@ cargo bench --manifest-path benches/Cargo.toml --bench xxhash
 
 ## Performance snapshot
 
-In the focused 2026-09-09 hashcodecs-only run on the benchmark host, `hashcodecs.xxh3_64` processes a 1 MiB input
-at 91.27 GiB/s. In the full 2026-09-02 run, the Base64 batch API reaches 11.84 GiB/s for encode and 6.23 GiB/s for
-decode with 256 B items in batches of 64. Each run pins one logical CPU and uses 15 samples with a 0.2-second
-minimum per sample. Read the
-[benchmark details](BENCHMARK.md) and [raw comparison results](docs/benchmarks/results.csv).
+On the benchmark host with CPython 3.14.6t, `hashcodecs.xxh3_64` processes a 1 MiB input at 80.87 GiB/s. The Base64
+batch API reaches 5.10 GiB/s for encode and 5.03 GiB/s for decode with 256 B items in batches of 64. Each run pins
+one logical CPU and uses 15 samples with a 0.2-second minimum per sample. Read the [benchmark details](BENCHMARK.md)
+and [raw comparison results](docs/benchmarks/results.csv).
 
 ## Development
 

{hashcodecs-1.4.0 → hashcodecs-1.4.2}/README.md

@@ -12,7 +12,7 @@
 
 <p align="center">
   <a href="BENCHMARK.md">
-    <img src="docs/benchmarks/performance-at-a-glance.svg" alt="CPython 3.12 standard Base64 encoding and decoding benchmark">
+    <img src="docs/benchmarks/performance-at-a-glance.svg" alt="Free-threaded CPython 3.14.6 standard Base64 encoding and decoding benchmark">
   </a>
 </p>
 
@@ -153,6 +153,7 @@ CPython boundary, and safety invariants.
 
 Run the suite on Windows 10 x64 with an Intel Core Ultra 7 265K. Pin one logical CPU and run each case in one thread.
 Collect 50 Rust samples and 15 Python samples. Higher throughput wins.
+Use free-threaded CPython 3.14.6 with the GIL disabled for all Python charts.
 
 ### Base64: Rust
 
@@ -199,12 +200,13 @@ cargo bench --manifest-path benches/Cargo.toml --bench murmur3
 cargo bench --manifest-path benches/Cargo.toml --bench xxhash
 cargo bench --manifest-path benches/Cargo.toml --bench crossover
 
-uv sync --group benchmark --no-install-project
-uv run --no-project --with . python benchmarks/python_base64.py
-uv run --no-project --with . python benchmarks/python_base64_batch.py
-uv run --no-project --with . python benchmarks/python_calls.py
-uv run --no-project --with . python benchmarks/python_murmur3.py
-uv run --no-project --with . python benchmarks/python_xxhash.py
+uv sync --python 3.14t --frozen --group benchmark --no-install-project
+uv run --python 3.14t --frozen --no-sync python tools/install_local_wheel.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_base64.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_base64_batch.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_calls.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_murmur3.py
+uv run --python 3.14t --frozen --no-sync python benchmarks/python_xxhash.py
 ```
 
 The Python benchmarks expose focused modes such as `--into`, `--lenient`, `--bytearray-input`, `--memoryview-input`,
@@ -222,11 +224,10 @@ cargo bench --manifest-path benches/Cargo.toml --bench xxhash
 
 ## Performance snapshot
 
-In the focused 2026-09-09 hashcodecs-only run on the benchmark host, `hashcodecs.xxh3_64` processes a 1 MiB input
-at 91.27 GiB/s. In the full 2026-09-02 run, the Base64 batch API reaches 11.84 GiB/s for encode and 6.23 GiB/s for
-decode with 256 B items in batches of 64. Each run pins one logical CPU and uses 15 samples with a 0.2-second
-minimum per sample. Read the
-[benchmark details](BENCHMARK.md) and [raw comparison results](docs/benchmarks/results.csv).
+On the benchmark host with CPython 3.14.6t, `hashcodecs.xxh3_64` processes a 1 MiB input at 80.87 GiB/s. The Base64
+batch API reaches 5.10 GiB/s for encode and 5.03 GiB/s for decode with 256 B items in batches of 64. Each run pins
+one logical CPU and uses 15 samples with a 0.2-second minimum per sample. Read the [benchmark details](BENCHMARK.md)
+and [raw comparison results](docs/benchmarks/results.csv).
 
 ## Development
 

hashcodecs-1.4.2/SAFETY.md

@@ -0,0 +1,120 @@
+# Memory safety verification
+
+Use this reference when reviewing changes to unsafe Rust or CPython buffer handling. It identifies the memory
+contracts, the checks that exercise them, and the commands to reproduce those checks. For performance mechanisms,
+see [Architecture](docs/ARCHITECTURE.md).
+
+## Required invariants
+
+| Boundary | Required contract | Source |
+| --- | --- | --- |
+| CPU dispatch | Check the features required by a kernel and its fallback paths before executing SIMD instructions. | [CPU detection](src/backend.rs), [Base64 dispatch](src/base64/backend.rs), [MurmurHash3 dispatch](src/murmur3/dispatch.rs), [XXH3 engine](src/xxhash/long_inputs.rs) |
+| Input loads | Keep scalar word loads, vector loads, and final overlapping stripes within the input allocation. | [MurmurHash3 loads](src/murmur3/primitives.rs), [XXH3 loads](src/xxhash/primitives.rs), [XXH3 schedule](src/xxhash/long_inputs.rs) |
+| Output stores | Check capacity before writing. Overlapping vector stores need space for their full width. Exact outputs must preserve bytes beyond the returned length. | [Base64 store policies](src/base64/decode/x86_contracts.rs), [output initialization](src/base64/output_buffer.rs) |
+| Result initialization | Initialize the returned prefix before exposing uninitialized storage as a result. | [Base64 allocation](src/base64/output_buffer.rs), [XXH3 batch results](src/bindings/xxhash/batch.rs) |
+| Python ownership | Keep owners alive and stabilize inputs before callbacks, overlapping writes, or interpreter detachment can invalidate a borrow. | [Buffer policy](src/bindings/buffer.rs), [Base64 batches](src/bindings/base64/batch.rs) |
+
+Exact output bounds do not imply rollback on errors. Reusable Base64 batches retain prior destination writes.
+Packed XXH3 batches validate and stabilize inputs before mutating the destination.
+
+## CPython ownership and callbacks
+
+The global interpreter lock (GIL) does not prevent callbacks from running on the same thread. Python allocation
+can trigger garbage collection and finalizers that clear an input list or resize a `bytearray`. Argument
+conversion and buffer release can also invoke user code. Bindings must preserve input ownership across these
+operations and copy mutable or overlapping data where the buffer policy requires it.
+
+XXH3 list batches finish input reads before allocating Python result containers. They store up to 32 native
+results on the stack and use a vector with fallible allocation for larger batches. Base64 batches retain input
+owners across result allocation. Subprocess tests on CPython 3.10 and 3.11 trigger finalizers during allocation
+and reuse freed storage to check these lifetime rules.
+
+Detached packed XXH3 batches retain immutable owners and stage digests in native memory. The path for exact
+`bytes` borrows 64 retained inputs at a time in a stack array. After reattaching, the binding rechecks destination
+capacity under synchronization before writing. The private `PackedDigest` contract requires initialized bytes
+without padding and a representation matching the packed output on hosts that use little endian order. Those
+hosts can copy the staged results in one operation. Other hosts serialize each word.
+
+Tests cover source list mutation, output resizing, overlapping buffers, and access to mutable data on CPython
+builds without the GIL. See the [XXH3 binding tests](src/bindings/xxhash/batch.rs),
+[XXH3 Python tests](tests/test_xxhash.py), [Base64 batch tests](tests/test_base64_batch.py), and
+[buffer tests](tests/test_base64_buffers.py).
+
+## Verification scope
+
+| Check | What it verifies | Limits |
+| --- | --- | --- |
+| Kani | Base64 scalar output bounds, MurmurHash3 and XXH3 word loads, MurmurHash3 scalar block loops, and XXH3 scheduling bounds. | Proofs apply within each harness's assumptions and unwind limits. The XXH3 schedule proof covers 241 through 3,072 bytes. |
+| Miri | Scalar allocation, pointer provenance, exact Base64 outputs, MurmurHash3 incremental state, and XXH3 length classes and batches. | The Miri configuration uses scalar dispatch and does not execute hardware intrinsics. |
+| AddressSanitizer | Invalid memory access in the sanitizer executable, including runtime SIMD paths, exact Base64 outputs, and hash batches. | Exercises inputs in the harness and backends available on the host. |
+| MemorySanitizer | Reads of uninitialized memory in the same executable. | Exercises the instrumented Rust core, including a rebuilt standard library. |
+| libFuzzer | Differential output checks against `base64`, `murmur3`, and `xxhash-rust` under sanitizers. | A timed run samples inputs. XXH3 batches cover one through nine independent buffers with equal or mixed lengths. |
+| Python and Rust binding tests | Callback order, retained owners, alias snapshots, output publication, and thread coordination. | Some cases require a specific CPython version or a build without the GIL. |
+
+The Rust Kani, Miri, sanitizer, and fuzz jobs exclude the optional CPython bindings. Binding tests cover that
+boundary. Passing a check establishes its stated coverage, not a proof of all unsafe code.
+
+Proof harnesses live in [Base64](src/base64/proofs.rs), [MurmurHash3](src/murmur3/proofs.rs), and
+[XXH3](src/xxhash/proofs.rs). Miri cases live in the corresponding `miri_tests.rs` files.
+The [sanitizer executable](tests/sanitizers.rs), [fuzz targets](fuzz/fuzz_targets), and
+[CI workflow](.github/workflows/ci.yml) define the executed checks.
+[Fuzz dependencies](fuzz/Cargo.toml) record the reference implementations.
+
+## Run the Rust checks on Linux
+
+Run these commands from the repository root on Linux x86_64 with a C/C++ compiler and linker installed.
+Install nightly Rust with the `rust-src` and `miri` components, plus `cargo-fuzz`. Install
+[Kani](https://model-checking.github.io/kani/install-guide.html) and complete `cargo kani setup` before running proofs.
+
+```sh
+rustup toolchain install nightly --component rust-src --component miri
+cargo install cargo-fuzz --locked
+```
+
+Run the proofs and scalar interpreter checks:
+
+```sh
+cargo kani
+MIRIFLAGS=-Zmiri-strict-provenance \
+  cargo +nightly miri test --lib miri_tests
+```
+
+Run the sanitizer executable with each instrumentation mode:
+
+```sh
+RUSTFLAGS="-Zsanitizer=address" \
+RUSTDOCFLAGS="-Zsanitizer=address" \
+  cargo +nightly test -Zbuild-std --target x86_64-unknown-linux-gnu \
+  --test sanitizers
+
+RUSTFLAGS="-Zsanitizer=memory -Zsanitizer-memory-track-origins" \
+RUSTDOCFLAGS="-Zsanitizer=memory -Zsanitizer-memory-track-origins" \
+  cargo +nightly test -Zbuild-std --target x86_64-unknown-linux-gnu \
+  --test sanitizers
+```
+
+Run the bounded fuzz checks used in CI:
+
+```sh
+cargo +nightly fuzz run base64 -- -max_total_time=30 -rss_limit_mb=2048
+cargo +nightly fuzz run murmur3 -- -max_total_time=30 -rss_limit_mb=2048
+cargo +nightly fuzz run xxhash -- -max_total_time=30 -rss_limit_mb=2048
+```
+
+Successful runs exit with status zero and no failed proof, assertion, or sanitizer diagnostic. Preserve any
+failing input or counterexample and reproduce the failure before changing the affected code.
+
+## Run the CPython checks
+
+Build the current wheel before testing the installed package:
+
+```sh
+uv sync --python 3.12 --frozen --no-install-project
+uv run --python 3.12 --frozen --no-sync python tools/install_local_wheel.py
+uv run --python 3.12 --frozen --no-sync pytest tests
+uv run --python 3.12 --frozen --no-sync cargo test --features python
+```
+
+Repeat the setup and test commands with `3.10` or `3.11` for the finalizer subprocess cases and with `3.15t` for
+cases that require CPython without the GIL. Those tests skip on incompatible interpreters. Check the selected
+interpreter and skipped cases before treating a run as coverage of those behaviors.

{hashcodecs-1.4.0 → hashcodecs-1.4.2}/benches/crossover.rs

@@ -8,6 +8,7 @@ mod support;
 const BASE64_ENCODE_SIZES: [usize; 10] = [15, 16, 31, 32, 47, 48, 51, 52, 103, 104];
 const BASE64_DECODE_SIZES: [usize; 8] = [12, 16, 28, 32, 60, 64, 124, 128];
 const MURMUR_SIZES: [usize; 6] = [15, 16, 31, 32, 255, 256];
+const MURMUR_X64_SIZES: [usize; 12] = [15, 16, 31, 32, 64, 255, 256, 384, 511, 512, 513, 1024];
 
 fn data(size: usize) -> Vec<u8> {
     (0..size)
@@ -41,9 +42,9 @@ fn base64_decode(c: &mut Criterion) {
 }
 
 macro_rules! murmur_group {
-    ($criterion:expr, $name:literal, $function:path) => {{
+    ($criterion:expr, $name:literal, $function:path, $sizes:expr) => {{
         let mut group = $criterion.benchmark_group($name);
-        for size in MURMUR_SIZES {
+        for size in $sizes {
             let input = data(size);
             group.throughput(Throughput::Bytes(size as u64));
             group.bench_with_input(BenchmarkId::from_parameter(size), &input, |bench, input| {
@@ -58,17 +59,20 @@ fn murmur3(c: &mut Criterion) {
     murmur_group!(
         c,
         "murmur_x86_32_crossover",
-        hashcodecs::murmur3::murmur3_x86_32
+        hashcodecs::murmur3::murmur3_x86_32,
+        MURMUR_SIZES
     );
     murmur_group!(
         c,
         "murmur_x86_128_crossover",
-        hashcodecs::murmur3::murmur3_x86_128
+        hashcodecs::murmur3::murmur3_x86_128,
+        MURMUR_SIZES
     );
     murmur_group!(
         c,
         "murmur_x64_128_crossover",
-        hashcodecs::murmur3::murmur3_x64_128
+        hashcodecs::murmur3::murmur3_x64_128,
+        MURMUR_X64_SIZES
     );
 }
 

{hashcodecs-1.4.0 → hashcodecs-1.4.2}/benches/xxhash.rs

@@ -6,13 +6,19 @@ use criterion::{BenchmarkId, Criterion, Throughput, criterion_group, criterion_m
 
 mod support;
 
-const SIZES: [usize; 16] = [
+const SIZES: [usize; 22] = [
     16,
     17,
     32,
+    33,
     64,
+    65,
+    97,
     128,
     129,
+    160,
+    192,
+    224,
     240,
     241,
     512,