hashcodecs 0.3.0__tar.gz → 0.4.0__tar.gz

hashcodecs-0.4.0/BENCHMARK.md

@@ -0,0 +1,71 @@
+# Benchmark Details
+
+Environment: Windows 10 x64 and Intel Core Ultra 7 265K.
+
+Conditions: clean builds, one pinned logical CPU, single-threaded execution, and 15 Python samples. Higher is better.
+
+## Reusable Python Buffers
+
+| Alphabet | Input | Operation | hashcodecs |
+| --- | --- | --- | ---: |
+| Standard | 4 KiB | encode | **15.09 GiB/s** |
+|  | 4 KiB | decode | **18.43 GiB/s** |
+|  | 1 MiB | encode | **39.66 GiB/s** |
+|  | 1 MiB | decode | **29.03 GiB/s** |
+|  | 32 MiB | encode | **10.89 GiB/s** |
+|  | 32 MiB | decode | **10.36 GiB/s** |
+| URL-safe | 4 KiB | encode | **14.83 GiB/s** |
+|  | 4 KiB | decode | **12.26 GiB/s** |
+|  | 1 MiB | encode | **39.71 GiB/s** |
+|  | 1 MiB | decode | **20.72 GiB/s** |
+|  | 32 MiB | encode | **10.84 GiB/s** |
+|  | 32 MiB | decode | **10.44 GiB/s** |
+
+## Python Base64 Batches
+
+Each value is total input throughput; the parenthesized figure is items per second. Higher is better.
+
+| Item | Batch | Operation | hashcodecs | hashcodecs loop | pybase64 loop | CPython loop |
+| ---: | ---: | --- | ---: | ---: | ---: | ---: |
+| 16 B | 8 | encode | **0.41 GiB/s (27.42M)** | 0.27 GiB/s (18.11M) | 0.13 GiB/s (8.45M) | 0.18 GiB/s (12.37M) |
+|  |  | decode | **0.36 GiB/s (24.24M)** | 0.21 GiB/s (14.13M) | 0.08 GiB/s (5.44M) | 0.12 GiB/s (7.85M) |
+| 16 B | 64 | encode | **0.55 GiB/s (36.69M)** | 0.31 GiB/s (20.81M) | 0.14 GiB/s (9.17M) | 0.21 GiB/s (13.81M) |
+|  |  | decode | **0.49 GiB/s (32.86M)** | 0.24 GiB/s (16.13M) | 0.09 GiB/s (5.87M) | 0.13 GiB/s (8.63M) |
+| 16 B | 1,024 | encode | **0.56 GiB/s (37.26M)** | 0.31 GiB/s (20.64M) | 0.14 GiB/s (9.07M) | 0.20 GiB/s (13.73M) |
+|  |  | decode | **0.49 GiB/s (32.70M)** | 0.25 GiB/s (16.60M) | 0.09 GiB/s (5.95M) | 0.13 GiB/s (8.71M) |
+| 256 B | 8 | encode | 1.77 GiB/s (7.43M) | 1.52 GiB/s (6.38M) | **1.93 GiB/s (8.08M)** | 0.40 GiB/s (1.67M) |
+|  |  | decode | **5.03 GiB/s (21.12M)** | 3.02 GiB/s (12.66M) | 1.22 GiB/s (5.12M) | 0.76 GiB/s (3.19M) |
+| 256 B | 64 | encode | 1.92 GiB/s (8.05M) | 1.62 GiB/s (6.82M) | **2.02 GiB/s (8.48M)** | 0.41 GiB/s (1.71M) |
+|  |  | decode | **6.09 GiB/s (25.56M)** | 3.36 GiB/s (14.07M) | 1.29 GiB/s (5.40M) | 0.76 GiB/s (3.18M) |
+| 256 B | 1,024 | encode | 1.92 GiB/s (8.05M) | 1.63 GiB/s (6.82M) | **1.99 GiB/s (8.35M)** | 0.40 GiB/s (1.69M) |
+|  |  | decode | **5.87 GiB/s (24.61M)** | 3.30 GiB/s (13.83M) | 1.29 GiB/s (5.43M) | 0.76 GiB/s (3.17M) |
+| 4 KiB | 8 | encode | **15.12 GiB/s (3.96M)** | 13.72 GiB/s (3.60M) | 12.69 GiB/s (3.33M) | 0.48 GiB/s (0.13M) |
+|  |  | decode | **19.20 GiB/s (5.03M)** | 16.52 GiB/s (4.33M) | 8.29 GiB/s (2.17M) | 1.13 GiB/s (0.30M) |
+| 4 KiB | 64 | encode | **12.59 GiB/s (3.30M)** | 11.55 GiB/s (3.03M) | 9.54 GiB/s (2.50M) | 0.48 GiB/s (0.12M) |
+|  |  | decode | **18.95 GiB/s (4.97M)** | 16.12 GiB/s (4.23M) | 8.12 GiB/s (2.13M) | 1.12 GiB/s (0.29M) |
+| 4 KiB | 1,024 | encode | 5.84 GiB/s (1.53M) | **8.81 GiB/s (2.31M)** | 7.83 GiB/s (2.05M) | 0.47 GiB/s (0.12M) |
+|  |  | decode | 3.52 GiB/s (0.92M) | **3.81 GiB/s (1.00M)** | 2.99 GiB/s (0.79M) | 0.92 GiB/s (0.24M) |
+
+## Mutable Python Inputs
+
+### Base64
+
+| Input | Operation | Returned `bytes` | Reusable `bytearray` |
+| --- | --- | ---: | ---: |
+| 4 KiB | encode | 14.12 GiB/s | **15.16 GiB/s** |
+|  | decode | 16.76 GiB/s | **18.27 GiB/s** |
+| 1 MiB | encode | 2.88 GiB/s | **38.96 GiB/s** |
+|  | decode | 3.59 GiB/s | **28.99 GiB/s** |
+| 32 MiB | encode | 2.92 GiB/s | **10.83 GiB/s** |
+|  | decode | 3.44 GiB/s | **10.35 GiB/s** |
+
+### MurmurHash3
+
+| Variant | API | 4 KiB | 1 MiB | 32 MiB |
+| --- | --- | ---: | ---: | ---: |
+| x86 32-bit | one-shot | **3.79 GiB/s** | **3.98 GiB/s** | **3.97 GiB/s** |
+|  | incremental | **3.58 GiB/s** | **3.96 GiB/s** | **3.92 GiB/s** |
+| x86 128-bit | one-shot | **8.09 GiB/s** | **9.22 GiB/s** | **9.12 GiB/s** |
+|  | incremental | **7.02 GiB/s** | **9.17 GiB/s** | **9.01 GiB/s** |
+| x64 128-bit | one-shot | **8.68 GiB/s** | **10.10 GiB/s** | **9.54 GiB/s** |
+|  | incremental | **7.70 GiB/s** | **10.01 GiB/s** | **9.41 GiB/s** |

{hashcodecs-0.3.0 → hashcodecs-0.4.0}/Cargo.lock

@@ -194,7 +194,7 @@ dependencies = [
 
 [[package]]
 name = "hashcodecs"
-version = "0.3.0"
+version = "0.4.0"
 dependencies = [
  "base64",
  "base64-turbo",

{hashcodecs-0.3.0 → hashcodecs-0.4.0}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "hashcodecs"
-version = "0.3.0"
+version = "0.4.0"
 edition = "2024"
 rust-version = "1.89"
 description = "SIMD-accelerated Base64 codecs and fast MurmurHash3 implementations"

hashcodecs-0.4.0/PKG-INFO

@@ -0,0 +1,190 @@
+Metadata-Version: 2.4
+Name: hashcodecs
+Version: 0.4.0
+Summary: SIMD-accelerated Base64 and MurmurHash3 codecs
+License-Expression: MIT OR Apache-2.0
+License-File: LICENSE
+License-File: LICENSE-MIT
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Programming Language :: Rust
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# hashcodecs
+
+[![CI](https://img.shields.io/github/actions/workflow/status/kozistr/hashcodecs-rs/ci.yml?branch=main&style=for-the-badge&logo=github)](https://github.com/kozistr/hashcodecs-rs/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/hashcodecs?style=for-the-badge&logo=pypi)](https://pypi.org/project/hashcodecs/)
+[![Python](https://img.shields.io/pypi/pyversions/hashcodecs?style=for-the-badge&logo=python)](https://pypi.org/project/hashcodecs/)
+[![License](https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-brightgreen?style=for-the-badge)](https://github.com/kozistr/hashcodecs-rs#license)
+[![Downloads](https://img.shields.io/pypi/dm/hashcodecs?style=for-the-badge&label=downloads)](https://pypi.org/project/hashcodecs/)
+
+`hashcodecs` provides runtime-dispatched SIMD Base64 codecs and fast, reference-compatible MurmurHash3 functions for Rust and Python.
+
+## Design
+
+- Runtime-dispatched SIMD Base64 with portable scalar fallbacks.
+- Reference-compatible MurmurHash3 with SIMD acceleration.
+- Rust and Python `*_into` APIs for caller-managed output buffers.
+- A familiar Python Base64 API, including native batch encode and decode operations.
+
+## Install
+
+```
+pip3 install hashcodecs
+```
+
+## Usage
+
+### Rust
+
+```rust
+let encoded = hashcodecs::b64encode(b"hello");
+assert_eq!(encoded, "aGVsbG8=");
+
+let mut output = [0_u8; 8];
+let written = hashcodecs::b64encode_into(b"hello", &mut output).unwrap();
+assert_eq!(&output[..written], b"aGVsbG8=");
+assert_eq!(hashcodecs::murmur3_x86_32(b"hello", 0), 0x248b_fa47);
+```
+
+### Python
+
+```python
+import hashcodecs.base64 as base64
+from hashcodecs import murmur3_32, murmur3_x64_128
+
+assert base64.b64encode(b'hello') == b'aGVsbG8='
+assert base64.b64decode(b'aGVsbG8=') == b'hello'
+assert base64.b64encode_batch([b'hello', b'world']) == [b'aGVsbG8=', b'd29ybGQ=']
+assert base64.b64decode_batch([b'aGVsbG8=', 'd29ybGQ=']) == [b'hello', b'world']
+assert base64.b64encode(b'hello', padded=False) == b'aGVsbG8'
+assert base64.b64decode(b'aGVsbG8', padded=False, canonical=True) == b'hello'
+assert murmur3_32(b'hello') == 0x248BFA47
+
+payload = b'hello'
+encoded = bytearray(4 * ((len(payload) + 2) // 3))
+encoded_len = base64.b64encode_into(payload, encoded)
+decoded = bytearray(len(encoded))
+decoded_len = base64.b64decode_into(encoded, decoded, validate=True)
+assert encoded[:encoded_len] == b'aGVsbG8='
+assert decoded[:decoded_len] == payload
+
+hasher = murmur3_x64_128(seed=42)
+hasher.update(b'hello')
+assert hasher.hexdigest() == hasher.digest().hex()
+```
+
+Build an installable wheel and source distribution with:
+
+```sh
+uv build
+```
+
+# Benchmark
+
+Environment: Windows 10 x64 and Intel Core Ultra 7 265K.
+
+Conditions: clean builds, one pinned logical CPU, single-threaded execution, 50 Rust samples, and 15 Python samples. Returned-output allocation is included except in the reusable-buffer table. Higher is better.
+
+## Run Locally
+
+Comparison crates are development-only dependencies and are not included in consumer builds.
+
+```sh
+cargo bench --bench base64
+cargo bench --bench murmur3
+uv sync --group benchmark --no-install-project
+uv run --no-project --with . python benchmarks/python_base64.py
+uv run --no-project --with . python benchmarks/python_base64.py --into
+uv run --no-project --with . python benchmarks/python_base64.py --bytearray-input
+uv run --no-project --with . python benchmarks/python_base64_batch.py
+uv run --no-project --with . python benchmarks/python_murmur3.py
+uv run --no-project --with . python benchmarks/python_murmur3.py --incremental
+```
+
+## Base64: Rust
+
+| Alphabet | Input | Operation | hashcodecs | `base64` | `base64-turbo` |
+| --- | --- | --- | ---: | ---: | ---: |
+| Standard | 4 KiB | encode | **20.41 GiB/s** | 5.81 GiB/s | 18.29 GiB/s |
+|  | 4 KiB | decode | **27.11 GiB/s** | 4.33 GiB/s | 16.95 GiB/s |
+|  | 1 MiB | encode | **42.38 GiB/s** | 5.16 GiB/s | 19.60 GiB/s |
+|  | 1 MiB | decode | **31.07 GiB/s** | 4.10 GiB/s | 18.17 GiB/s |
+|  | 32 MiB | encode | **11.97 GiB/s** | 3.28 GiB/s | 10.94 GiB/s |
+|  | 32 MiB | decode | **11.51 GiB/s** | 3.35 GiB/s | 10.45 GiB/s |
+| URL-safe | 4 KiB | encode | **20.46 GiB/s** | 5.81 GiB/s | 18.31 GiB/s |
+|  | 4 KiB | decode | **25.65 GiB/s** | 4.35 GiB/s | 16.96 GiB/s |
+|  | 1 MiB | encode | **42.56 GiB/s** | 5.18 GiB/s | 19.66 GiB/s |
+|  | 1 MiB | decode | **29.42 GiB/s** | 4.08 GiB/s | 18.16 GiB/s |
+|  | 32 MiB | encode | **11.86 GiB/s** | 3.29 GiB/s | 11.09 GiB/s |
+|  | 32 MiB | decode | **11.67 GiB/s** | 3.36 GiB/s | 10.62 GiB/s |
+
+## MurmurHash3: Rust
+
+| Variant | Input | hashcodecs | `murmur3` | `murmurs` | `fastmurmur3` | `mm3h` |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| x86 32-bit | 4 KiB | **4.30 GiB/s** | 2.60 GiB/s | 4.09 GiB/s | n/a | **4.30 GiB/s** |
+|  | 1 MiB | **4.26 GiB/s** | 2.59 GiB/s | 4.04 GiB/s | n/a | 4.25 GiB/s |
+|  | 32 MiB | **4.21 GiB/s** | 2.54 GiB/s | 3.87 GiB/s | n/a | 4.03 GiB/s |
+| x86 128-bit | 4 KiB | **9.54 GiB/s** | 4.97 GiB/s | 8.64 GiB/s | n/a | n/a |
+|  | 1 MiB | **9.84 GiB/s** | 5.06 GiB/s | 8.77 GiB/s | n/a | n/a |
+|  | 32 MiB | **9.63 GiB/s** | 4.84 GiB/s | 6.05 GiB/s | n/a | n/a |
+| x64 128-bit | 4 KiB | **10.76 GiB/s** | 6.83 GiB/s | 9.45 GiB/s | 10.06 GiB/s | 9.54 GiB/s |
+|  | 1 MiB | **10.84 GiB/s** | 6.89 GiB/s | 9.51 GiB/s | 10.04 GiB/s | 9.51 GiB/s |
+|  | 32 MiB | **10.12 GiB/s** | 6.11 GiB/s | 6.64 GiB/s | 7.25 GiB/s | 6.70 GiB/s |
+
+## Base64: Python
+
+Python decoding uses `validate=True`, and `hashcodecs` passes `bytes` directly into Rust without an input copy.
+
+| Alphabet | Input | Operation | hashcodecs | CPython `base64` | `pybase64` |
+| --- | --- | --- | ---: | ---: | ---: |
+| Standard | 4 KiB | encode | 13.57 GiB/s | 0.48 GiB/s | **14.06 GiB/s** |
+|  | 4 KiB | decode | **16.70 GiB/s** | 1.13 GiB/s | 8.47 GiB/s |
+|  | 1 MiB | encode | **3.28 GiB/s** | 0.44 GiB/s | 3.02 GiB/s |
+|  | 1 MiB | decode | **4.02 GiB/s** | 0.95 GiB/s | 3.68 GiB/s |
+|  | 32 MiB | encode | **2.92 GiB/s** | 0.43 GiB/s | 2.84 GiB/s |
+|  | 32 MiB | decode | 3.40 GiB/s | 0.94 GiB/s | **3.69 GiB/s** |
+| URL-safe | 4 KiB | encode | **12.63 GiB/s** | 0.41 GiB/s | 1.19 GiB/s |
+|  | 4 KiB | decode | **11.34 GiB/s** | 0.76 GiB/s | 1.56 GiB/s |
+|  | 1 MiB | encode | **3.24 GiB/s** | 0.36 GiB/s | 0.91 GiB/s |
+|  | 1 MiB | decode | **4.15 GiB/s** | 0.63 GiB/s | 1.45 GiB/s |
+|  | 32 MiB | encode | **3.03 GiB/s** | 0.36 GiB/s | 0.90 GiB/s |
+|  | 32 MiB | decode | **3.40 GiB/s** | 0.61 GiB/s | 1.48 GiB/s |
+
+## MurmurHash3: Python
+
+| Variant | API | Input | hashcodecs | `mmh3` |
+| --- | --- | --- | ---: | ---: |
+| x86 32-bit | one-shot | 4 KiB | **3.79 GiB/s** | 3.71 GiB/s |
+|  |  | 1 MiB | **3.98 GiB/s** | 3.83 GiB/s |
+|  |  | 32 MiB | **3.97 GiB/s** | 3.66 GiB/s |
+|  | incremental | 4 KiB | **3.58 GiB/s** | **3.58 GiB/s** |
+|  |  | 1 MiB | **3.96 GiB/s** | 3.81 GiB/s |
+|  |  | 32 MiB | **3.92 GiB/s** | 3.74 GiB/s |
+| x86 128-bit | one-shot | 4 KiB | 8.09 GiB/s | **8.21 GiB/s** |
+|  |  | 1 MiB | **9.22 GiB/s** | 8.87 GiB/s |
+|  |  | 32 MiB | **9.12 GiB/s** | 6.15 GiB/s |
+|  | incremental | 4 KiB | **7.02 GiB/s** | 0.77 GiB/s |
+|  |  | 1 MiB | **9.17 GiB/s** | 0.80 GiB/s |
+|  |  | 32 MiB | **9.01 GiB/s** | 0.79 GiB/s |
+| x64 128-bit | one-shot | 4 KiB | 8.68 GiB/s | **9.46 GiB/s** |
+|  |  | 1 MiB | 10.10 GiB/s | **10.24 GiB/s** |
+|  |  | 32 MiB | **9.54 GiB/s** | 6.61 GiB/s |
+|  | incremental | 4 KiB | 7.70 GiB/s | **7.99 GiB/s** |
+|  |  | 1 MiB | **10.01 GiB/s** | 9.28 GiB/s |
+|  |  | 32 MiB | **9.41 GiB/s** | 7.34 GiB/s |
+
+Reusable-buffer and mutable-input results are available in [BENCHMARK.md](https://github.com/kozistr/hashcodecs-rs/blob/main/BENCHMARK.md).
+
+## SIMD References
+
+The SIMD implementation follows the approach described in [Faster Base64 Encoding and Decoding using AVX2 Instructions](https://arxiv.org/abs/1704.00605), with AVX-512 VBMI and AArch64 NEON backends selected automatically when available.

hashcodecs-0.4.0/README.md

@@ -0,0 +1,170 @@
+# hashcodecs
+
+[![CI](https://img.shields.io/github/actions/workflow/status/kozistr/hashcodecs-rs/ci.yml?branch=main&style=for-the-badge&logo=github)](https://github.com/kozistr/hashcodecs-rs/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/hashcodecs?style=for-the-badge&logo=pypi)](https://pypi.org/project/hashcodecs/)
+[![Python](https://img.shields.io/pypi/pyversions/hashcodecs?style=for-the-badge&logo=python)](https://pypi.org/project/hashcodecs/)
+[![License](https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-brightgreen?style=for-the-badge)](https://github.com/kozistr/hashcodecs-rs#license)
+[![Downloads](https://img.shields.io/pypi/dm/hashcodecs?style=for-the-badge&label=downloads)](https://pypi.org/project/hashcodecs/)
+
+`hashcodecs` provides runtime-dispatched SIMD Base64 codecs and fast, reference-compatible MurmurHash3 functions for Rust and Python.
+
+## Design
+
+- Runtime-dispatched SIMD Base64 with portable scalar fallbacks.
+- Reference-compatible MurmurHash3 with SIMD acceleration.
+- Rust and Python `*_into` APIs for caller-managed output buffers.
+- A familiar Python Base64 API, including native batch encode and decode operations.
+
+## Install
+
+```
+pip3 install hashcodecs
+```
+
+## Usage
+
+### Rust
+
+```rust
+let encoded = hashcodecs::b64encode(b"hello");
+assert_eq!(encoded, "aGVsbG8=");
+
+let mut output = [0_u8; 8];
+let written = hashcodecs::b64encode_into(b"hello", &mut output).unwrap();
+assert_eq!(&output[..written], b"aGVsbG8=");
+assert_eq!(hashcodecs::murmur3_x86_32(b"hello", 0), 0x248b_fa47);
+```
+
+### Python
+
+```python
+import hashcodecs.base64 as base64
+from hashcodecs import murmur3_32, murmur3_x64_128
+
+assert base64.b64encode(b'hello') == b'aGVsbG8='
+assert base64.b64decode(b'aGVsbG8=') == b'hello'
+assert base64.b64encode_batch([b'hello', b'world']) == [b'aGVsbG8=', b'd29ybGQ=']
+assert base64.b64decode_batch([b'aGVsbG8=', 'd29ybGQ=']) == [b'hello', b'world']
+assert base64.b64encode(b'hello', padded=False) == b'aGVsbG8'
+assert base64.b64decode(b'aGVsbG8', padded=False, canonical=True) == b'hello'
+assert murmur3_32(b'hello') == 0x248BFA47
+
+payload = b'hello'
+encoded = bytearray(4 * ((len(payload) + 2) // 3))
+encoded_len = base64.b64encode_into(payload, encoded)
+decoded = bytearray(len(encoded))
+decoded_len = base64.b64decode_into(encoded, decoded, validate=True)
+assert encoded[:encoded_len] == b'aGVsbG8='
+assert decoded[:decoded_len] == payload
+
+hasher = murmur3_x64_128(seed=42)
+hasher.update(b'hello')
+assert hasher.hexdigest() == hasher.digest().hex()
+```
+
+Build an installable wheel and source distribution with:
+
+```sh
+uv build
+```
+
+# Benchmark
+
+Environment: Windows 10 x64 and Intel Core Ultra 7 265K.
+
+Conditions: clean builds, one pinned logical CPU, single-threaded execution, 50 Rust samples, and 15 Python samples. Returned-output allocation is included except in the reusable-buffer table. Higher is better.
+
+## Run Locally
+
+Comparison crates are development-only dependencies and are not included in consumer builds.
+
+```sh
+cargo bench --bench base64
+cargo bench --bench murmur3
+uv sync --group benchmark --no-install-project
+uv run --no-project --with . python benchmarks/python_base64.py
+uv run --no-project --with . python benchmarks/python_base64.py --into
+uv run --no-project --with . python benchmarks/python_base64.py --bytearray-input
+uv run --no-project --with . python benchmarks/python_base64_batch.py
+uv run --no-project --with . python benchmarks/python_murmur3.py
+uv run --no-project --with . python benchmarks/python_murmur3.py --incremental
+```
+
+## Base64: Rust
+
+| Alphabet | Input | Operation | hashcodecs | `base64` | `base64-turbo` |
+| --- | --- | --- | ---: | ---: | ---: |
+| Standard | 4 KiB | encode | **20.41 GiB/s** | 5.81 GiB/s | 18.29 GiB/s |
+|  | 4 KiB | decode | **27.11 GiB/s** | 4.33 GiB/s | 16.95 GiB/s |
+|  | 1 MiB | encode | **42.38 GiB/s** | 5.16 GiB/s | 19.60 GiB/s |
+|  | 1 MiB | decode | **31.07 GiB/s** | 4.10 GiB/s | 18.17 GiB/s |
+|  | 32 MiB | encode | **11.97 GiB/s** | 3.28 GiB/s | 10.94 GiB/s |
+|  | 32 MiB | decode | **11.51 GiB/s** | 3.35 GiB/s | 10.45 GiB/s |
+| URL-safe | 4 KiB | encode | **20.46 GiB/s** | 5.81 GiB/s | 18.31 GiB/s |
+|  | 4 KiB | decode | **25.65 GiB/s** | 4.35 GiB/s | 16.96 GiB/s |
+|  | 1 MiB | encode | **42.56 GiB/s** | 5.18 GiB/s | 19.66 GiB/s |
+|  | 1 MiB | decode | **29.42 GiB/s** | 4.08 GiB/s | 18.16 GiB/s |
+|  | 32 MiB | encode | **11.86 GiB/s** | 3.29 GiB/s | 11.09 GiB/s |
+|  | 32 MiB | decode | **11.67 GiB/s** | 3.36 GiB/s | 10.62 GiB/s |
+
+## MurmurHash3: Rust
+
+| Variant | Input | hashcodecs | `murmur3` | `murmurs` | `fastmurmur3` | `mm3h` |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| x86 32-bit | 4 KiB | **4.30 GiB/s** | 2.60 GiB/s | 4.09 GiB/s | n/a | **4.30 GiB/s** |
+|  | 1 MiB | **4.26 GiB/s** | 2.59 GiB/s | 4.04 GiB/s | n/a | 4.25 GiB/s |
+|  | 32 MiB | **4.21 GiB/s** | 2.54 GiB/s | 3.87 GiB/s | n/a | 4.03 GiB/s |
+| x86 128-bit | 4 KiB | **9.54 GiB/s** | 4.97 GiB/s | 8.64 GiB/s | n/a | n/a |
+|  | 1 MiB | **9.84 GiB/s** | 5.06 GiB/s | 8.77 GiB/s | n/a | n/a |
+|  | 32 MiB | **9.63 GiB/s** | 4.84 GiB/s | 6.05 GiB/s | n/a | n/a |
+| x64 128-bit | 4 KiB | **10.76 GiB/s** | 6.83 GiB/s | 9.45 GiB/s | 10.06 GiB/s | 9.54 GiB/s |
+|  | 1 MiB | **10.84 GiB/s** | 6.89 GiB/s | 9.51 GiB/s | 10.04 GiB/s | 9.51 GiB/s |
+|  | 32 MiB | **10.12 GiB/s** | 6.11 GiB/s | 6.64 GiB/s | 7.25 GiB/s | 6.70 GiB/s |
+
+## Base64: Python
+
+Python decoding uses `validate=True`, and `hashcodecs` passes `bytes` directly into Rust without an input copy.
+
+| Alphabet | Input | Operation | hashcodecs | CPython `base64` | `pybase64` |
+| --- | --- | --- | ---: | ---: | ---: |
+| Standard | 4 KiB | encode | 13.57 GiB/s | 0.48 GiB/s | **14.06 GiB/s** |
+|  | 4 KiB | decode | **16.70 GiB/s** | 1.13 GiB/s | 8.47 GiB/s |
+|  | 1 MiB | encode | **3.28 GiB/s** | 0.44 GiB/s | 3.02 GiB/s |
+|  | 1 MiB | decode | **4.02 GiB/s** | 0.95 GiB/s | 3.68 GiB/s |
+|  | 32 MiB | encode | **2.92 GiB/s** | 0.43 GiB/s | 2.84 GiB/s |
+|  | 32 MiB | decode | 3.40 GiB/s | 0.94 GiB/s | **3.69 GiB/s** |
+| URL-safe | 4 KiB | encode | **12.63 GiB/s** | 0.41 GiB/s | 1.19 GiB/s |
+|  | 4 KiB | decode | **11.34 GiB/s** | 0.76 GiB/s | 1.56 GiB/s |
+|  | 1 MiB | encode | **3.24 GiB/s** | 0.36 GiB/s | 0.91 GiB/s |
+|  | 1 MiB | decode | **4.15 GiB/s** | 0.63 GiB/s | 1.45 GiB/s |
+|  | 32 MiB | encode | **3.03 GiB/s** | 0.36 GiB/s | 0.90 GiB/s |
+|  | 32 MiB | decode | **3.40 GiB/s** | 0.61 GiB/s | 1.48 GiB/s |
+
+## MurmurHash3: Python
+
+| Variant | API | Input | hashcodecs | `mmh3` |
+| --- | --- | --- | ---: | ---: |
+| x86 32-bit | one-shot | 4 KiB | **3.79 GiB/s** | 3.71 GiB/s |
+|  |  | 1 MiB | **3.98 GiB/s** | 3.83 GiB/s |
+|  |  | 32 MiB | **3.97 GiB/s** | 3.66 GiB/s |
+|  | incremental | 4 KiB | **3.58 GiB/s** | **3.58 GiB/s** |
+|  |  | 1 MiB | **3.96 GiB/s** | 3.81 GiB/s |
+|  |  | 32 MiB | **3.92 GiB/s** | 3.74 GiB/s |
+| x86 128-bit | one-shot | 4 KiB | 8.09 GiB/s | **8.21 GiB/s** |
+|  |  | 1 MiB | **9.22 GiB/s** | 8.87 GiB/s |
+|  |  | 32 MiB | **9.12 GiB/s** | 6.15 GiB/s |
+|  | incremental | 4 KiB | **7.02 GiB/s** | 0.77 GiB/s |
+|  |  | 1 MiB | **9.17 GiB/s** | 0.80 GiB/s |
+|  |  | 32 MiB | **9.01 GiB/s** | 0.79 GiB/s |
+| x64 128-bit | one-shot | 4 KiB | 8.68 GiB/s | **9.46 GiB/s** |
+|  |  | 1 MiB | 10.10 GiB/s | **10.24 GiB/s** |
+|  |  | 32 MiB | **9.54 GiB/s** | 6.61 GiB/s |
+|  | incremental | 4 KiB | 7.70 GiB/s | **7.99 GiB/s** |
+|  |  | 1 MiB | **10.01 GiB/s** | 9.28 GiB/s |
+|  |  | 32 MiB | **9.41 GiB/s** | 7.34 GiB/s |
+
+Reusable-buffer and mutable-input results are available in [BENCHMARK.md](https://github.com/kozistr/hashcodecs-rs/blob/main/BENCHMARK.md).
+
+## SIMD References
+
+The SIMD implementation follows the approach described in [Faster Base64 Encoding and Decoding using AVX2 Instructions](https://arxiv.org/abs/1704.00605), with AVX-512 VBMI and AArch64 NEON backends selected automatically when available.

{hashcodecs-0.3.0 → hashcodecs-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hashcodecs"
-version = "0.3.0"
+version = "0.4.0"
 description = "SIMD-accelerated Base64 and MurmurHash3 codecs"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -12,6 +12,12 @@ license = "MIT OR Apache-2.0"
 license-files = ["LICENSE", "LICENSE-MIT"]
 classifiers = [
     "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Programming Language :: Python :: 3.15",
     "Programming Language :: Rust",
     "Operating System :: OS Independent",
     "Topic :: Software Development :: Libraries",
@@ -33,6 +39,7 @@ packages = ["python/hashcodecs"]
 
 [tool.hatch.build.targets.sdist]
 include = [
+    "/BENCHMARK.md",
     "/Cargo.lock",
     "/Cargo.toml",
     "/build.rs",

{hashcodecs-0.3.0 → hashcodecs-0.4.0}/python/hashcodecs/__init__.py

@@ -2,16 +2,22 @@
 
 from .base64 import (
     b64decode,
+    b64decode_batch,
     b64decode_into,
     b64encode,
+    b64encode_batch,
     b64encode_into,
     standard_b64decode,
+    standard_b64decode_batch,
     standard_b64decode_into,
     standard_b64encode,
+    standard_b64encode_batch,
     standard_b64encode_into,
     urlsafe_b64decode,
+    urlsafe_b64decode_batch,
     urlsafe_b64decode_into,
     urlsafe_b64encode,
+    urlsafe_b64encode_batch,
     urlsafe_b64encode_into,
 )
 from .murmur3 import (
@@ -25,8 +31,10 @@ from .murmur3 import (
 
 __all__ = [
     'b64decode',
+    'b64decode_batch',
     'b64decode_into',
     'b64encode',
+    'b64encode_batch',
     'b64encode_into',
     'murmur3_32',
     'murmur3_x64_128',
@@ -35,11 +43,15 @@ __all__ = [
     'murmur3_x86_128',
     'murmur3_x86_128_digest',
     'standard_b64decode',
+    'standard_b64decode_batch',
     'standard_b64decode_into',
     'standard_b64encode',
+    'standard_b64encode_batch',
     'standard_b64encode_into',
     'urlsafe_b64decode',
+    'urlsafe_b64decode_batch',
     'urlsafe_b64decode_into',
     'urlsafe_b64encode',
+    'urlsafe_b64encode_batch',
     'urlsafe_b64encode_into',
 ]

{hashcodecs-0.3.0 → hashcodecs-0.4.0}/python/hashcodecs/_hashcodecs.pyi

@@ -1,17 +1,44 @@
 from _typeshed import ReadableBuffer
 
-def b64encode(s: ReadableBuffer, altchars: ReadableBuffer | None = None) -> bytes: ...
-def b64encode_into(s: ReadableBuffer, output: bytearray, altchars: ReadableBuffer | None = None) -> int: ...
+def b64encode(
+    s: ReadableBuffer,
+    altchars: ReadableBuffer | None = None,
+    *,
+    padded: bool = True,
+    wrapcol: int = 0,
+) -> bytes: ...
+def b64encode_batch(items: list[ReadableBuffer], altchars: ReadableBuffer | None = None) -> list[bytes]: ...
+def b64encode_into(
+    s: ReadableBuffer,
+    output: bytearray,
+    altchars: ReadableBuffer | None = None,
+    *,
+    padded: bool = True,
+    wrapcol: int = 0,
+) -> int: ...
 def b64decode(
     s: str | ReadableBuffer,
     altchars: str | ReadableBuffer | None = None,
-    validate: bool = False,
+    validate: bool = ...,
+    *,
+    padded: bool = True,
+    ignorechars: ReadableBuffer = ...,
+    canonical: bool = False,
 ) -> bytes: ...
+def b64decode_batch(
+    items: list[str | ReadableBuffer],
+    altchars: str | ReadableBuffer | None = None,
+    validate: bool = False,
+) -> list[bytes]: ...
 def b64decode_into(
     s: str | ReadableBuffer,
     output: bytearray,
     altchars: str | ReadableBuffer | None = None,
-    validate: bool = False,
+    validate: bool = ...,
+    *,
+    padded: bool = True,
+    ignorechars: ReadableBuffer = ...,
+    canonical: bool = False,
 ) -> int: ...
 def murmur3_32(s: ReadableBuffer, seed: int = 0) -> int: ...
 def murmur3_x86_128_digest(s: ReadableBuffer, seed: int = 0) -> bytes: ...

hashcodecs-0.4.0/python/hashcodecs/base64.py

@@ -0,0 +1,89 @@
+"""A fast, API-compatible subset of Python's :mod:`base64` module."""
+
+import sys
+
+from ._hashcodecs import b64decode, b64decode_batch, b64decode_into, b64encode, b64encode_batch, b64encode_into
+
+_URLSAFE_PADDED_DEFAULT = sys.version_info < (3, 15)
+
+
+def standard_b64encode(s) -> bytes:
+    """Encode *s* with the standard Base64 alphabet."""
+    return b64encode(s)
+
+
+def standard_b64encode_batch(items) -> list[bytes]:
+    """Encode a list of inputs with the standard Base64 alphabet."""
+    return b64encode_batch(items)
+
+
+def standard_b64encode_into(s, output: bytearray) -> int:
+    """Encode *s* with the standard Base64 alphabet into *output*."""
+    return b64encode_into(s, output)
+
+
+def standard_b64decode(s) -> bytes:
+    """Decode *s* with the standard Base64 alphabet."""
+    return b64decode(s)
+
+
+def standard_b64decode_batch(items) -> list[bytes]:
+    """Decode a list of inputs with the standard Base64 alphabet."""
+    return b64decode_batch(items)
+
+
+def standard_b64decode_into(s, output: bytearray) -> int:
+    """Decode standard Base64 *s* into *output*."""
+    return b64decode_into(s, output)
+
+
+def urlsafe_b64encode(s, *, padded: bool = True) -> bytes:
+    """Encode *s* with the URL-safe Base64 alphabet."""
+    return b64encode(s, b'-_', padded=padded)
+
+
+def urlsafe_b64encode_batch(items) -> list[bytes]:
+    """Encode a list of inputs with the URL-safe Base64 alphabet."""
+    return b64encode_batch(items, b'-_')
+
+
+def urlsafe_b64encode_into(s, output: bytearray, *, padded: bool = True) -> int:
+    """Encode *s* with the URL-safe Base64 alphabet into *output*."""
+    return b64encode_into(s, output, b'-_', padded=padded)
+
+
+def urlsafe_b64decode(s, *, padded: bool = _URLSAFE_PADDED_DEFAULT) -> bytes:
+    """Decode *s* with the URL-safe Base64 alphabet."""
+    return b64decode(s, b'-_', padded=padded)
+
+
+def urlsafe_b64decode_batch(items) -> list[bytes]:
+    """Decode a list of inputs with the URL-safe Base64 alphabet."""
+    return b64decode_batch(items, b'-_')
+
+
+def urlsafe_b64decode_into(s, output: bytearray, *, padded: bool = _URLSAFE_PADDED_DEFAULT) -> int:
+    """Decode URL-safe Base64 *s* into *output*."""
+    return b64decode_into(s, output, b'-_', padded=padded)
+
+
+__all__ = [
+    'b64decode',
+    'b64decode_batch',
+    'b64decode_into',
+    'b64encode',
+    'b64encode_batch',
+    'b64encode_into',
+    'standard_b64decode',
+    'standard_b64decode_batch',
+    'standard_b64decode_into',
+    'standard_b64encode',
+    'standard_b64encode_batch',
+    'standard_b64encode_into',
+    'urlsafe_b64decode',
+    'urlsafe_b64decode_batch',
+    'urlsafe_b64decode_into',
+    'urlsafe_b64encode',
+    'urlsafe_b64encode_batch',
+    'urlsafe_b64encode_into',
+]