BloomFilter-py 1.3.0__tar.gz → 2.0.0__tar.gz

bloomfilter_py-2.0.0/CHANGELOG.md

@@ -0,0 +1,22 @@
+# Changelog
+
+## 2.0.0
+
+- Retain master security checks that bound hexadecimal and Base64 decoding
+  before allocating the complete serialized payload.
+- Remove all mandatory runtime dependencies; ship packed bit storage and a
+  seed-zero MurmurHash3 x64_128 implementation supporting Python 3.8+.
+- Add explicit `backend="python"` (default) and `backend="native"` modes.
+  Native acceleration is available through the `native` installation extra.
+- Preserve both hashing strategies, existing funnels and the Guava compact
+  serialization format. Support cross-backend comparison and merging.
+- Replace per-word serialization objects with bounded chunk transformations;
+  chunk large counts and merges as well.
+- Export a stable `BitArray` facade. `setup` copies external third-party
+  bitarray inputs; third-party-only data methods and aliasing need migration.
+- Reject non-finite error rates, nonintegral sizing and misbehaving streams.
+- Add fixed and random hash differential tests, packed-bit/backend tests,
+  memory checks, packaging validation, benchmarks and a dual-backend CI matrix.
+
+Public bit storage and external-array aliasing changes require a major version.
+The version is prepared locally; publishing requires a separate release action.

bloomfilter_py-2.0.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include LICENSE README.md CHANGELOG.md
+recursive-include tests *.py *.json *.java
+recursive-include benchmarks *.py
+recursive-include scripts *.py

{bloomfilter_py-1.3.0 → bloomfilter_py-2.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: bloomfilter-py
-Version: 1.3.0
+Version: 2.0.0
 Summary: Yet another bloomfilter implementation in Python
 Author-email: OldPanda <me@old-panda.com>
 License: MIT License
@@ -47,10 +47,9 @@ Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: bitarray==3.11.0
-Requires-Dist: mmh3==5.0.1; python_version < "3.9"
-Requires-Dist: mmh3==5.2.0; python_version >= "3.9" and python_version < "3.10"
-Requires-Dist: mmh3==5.3.1; python_version >= "3.10"
+Provides-Extra: native
+Requires-Dist: bitarray<4,>=3.0; extra == "native"
+Requires-Dist: mmh3<6,>=5.0; extra == "native"
 Dynamic: license-file
 
 # bloomfilter-py
@@ -81,6 +80,68 @@ Here's a brief [introduction](https://en.wikipedia.org/wiki/Bloom_filter) to Blo
 pip install bloomfilter-py
 ```
 
+## Backends and migration from 1.3
+
+Version 2.0 runs on the Python standard library alone. `bitarray` and `mmh3`
+are no longer required to install or import this package. The default is always
+`backend="python"`, even if those packages are already installed. For explicit
+native acceleration:
+
+```bash
+pip install 'bloomfilter-py[native]'
+```
+
+```python
+from bloomfilter import BloomFilter
+
+fast = BloomFilter(500, 0.01, backend="native")
+fast.put("example")
+loaded = BloomFilter.loads(fast.dumps(), backend="native")
+assert "example" in loaded
+assert loaded.backend == "native"
+```
+
+The constructor and all loading methods accept the keyword-only `backend`
+argument (`"python"` or `"native"`). An invalid name raises `ValueError`;
+requesting native mode without **both** optional packages raises `ImportError`
+with installation instructions. There is no automatic fallback. `copy()`
+preserves the backend. Filters using different backends can be compared and
+merged when their strategies, funnels, sizes, and hash counts are compatible.
+Backend choice is not serialized: all old dumps remain readable, and new dumps
+use exactly the same Guava compact format.
+
+Pure Python hashing is slower than the C extension, especially on long keys.
+Native mode is available for throughput-sensitive workloads. Serialization,
+counting, and merging process at most 64 KiB of payload per chunk; copies and
+returned dumps necessarily allocate their complete result. `dumps()` and
+`loads()` also retain full input/output bytes, so use stream methods for large
+filters. Backing storage remains packed (roughly one byte per eight bits).
+
+### Bit storage and custom strategies
+
+`filter.data` is now a `bloomfilter.BitArray` facade with consistent semantics
+on either backend. It supports `len`, iteration, negative/integer indexing,
+slice copies, scalar or equal-length iterable slice assignment, `setall`,
+`count(0/1)`, `copy`, equality, `|=`, `+=`, `frombytes`, and `tobytes`.
+Slices preserve the backend; slice assignment cannot resize. Appending via
+`frombytes` or `+=` works even at non-byte-aligned lengths. `tobytes()` uses
+most-significant-bit-first bytes with zero padding at the end. The buffer
+protocol and the complete third-party `bitarray` API are **not** provided.
+
+Existing `setup(hash_count, data, strategy, funnel)` calls accept this facade.
+For migration, a third-party `bitarray.bitarray` is also accepted when installed;
+it is copied into Python-backed storage and normalized to logical bit order,
+including little-endian inputs. It is no longer aliased to `filter.data`.
+Callers relying on that alias or on third-party-only methods must update.
+Passing a `BitArray` to `setup` retains it, as before, so external mutations
+remain visible. `backend` reflects the current data object.
+
+Custom strategies still receive the same positional `put` / `might_contain`
+arguments, now with the facade as their data argument. Overrides of built-in
+strategies' `hash_key` are honored on both backends. Strategies must not depend
+on private storage fields or a specific third-party data type. Direct calls to built-in strategies with legacy third-party bit arrays remain
+supported. Direct calls to `Strategy.hash_key` always use the standard-library MurmurHash3 implementation.
+
 ## Usage Examples
 
 ### Basic Usage
@@ -291,10 +352,59 @@ install the additional tools and execute:
 
 ```bash
 python -m pip install black mypy bandit
-python -m black --check bloomfilter tests
+python -m black --check bloomfilter tests benchmarks scripts
 python -m mypy bloomfilter --strict
 python -m bandit -r bloomfilter
 ```
 
 On Windows PowerShell, activate the virtual environment with
 `.venv\Scripts\Activate.ps1` instead of `source .venv/bin/activate`.
+
+### Reproducible backend benchmarks
+
+```bash
+python benchmarks/benchmark_backends.py --backend python --output python.json
+python benchmarks/benchmark_backends.py --backend native --output native.json
+# Against a separate checkout of 1.3.0, with its dependencies installed:
+python benchmarks/benchmark_backends.py --backend baseline \
+  --package-path /path/to/1.3.0 --output baseline.json
+```
+
+The script reports median timings, raw repeats, workload sizes, and interpreter
+metadata. It covers fresh and repeated insertions, hits and misses, short and
+long byte keys, serialization, statistics, and merging. Results describe the
+specified workload rather than a universal throughput guarantee.
+
+For the zero-dependency source suite run `python -S -m unittest discover`.
+Install `.[native]` to enable optional differential tests against the C packages.
+The checked-in hash vectors were generated with mmh3 5.3.1 and cover every
+length from 0 through 512 plus larger block boundaries.
+Run Java integration tests for both backends with `BLOOMFILTER_TEST_BACKEND`
+set to `python` and `native` respectively. CI covers Python 3.8 through 3.14.
+
+Build and verify release artifacts with `python -m build` followed by
+`python scripts/verify_distribution.py dist`. The latter installs the wheel
+offline into a clean temporary environment and verifies it outside the source
+path, including the absence of mandatory runtime dependencies.
+
+### Auditing compatibility against an earlier checkout
+
+Use an environment containing the baseline's dependencies to compare actual
+behavior in separate processes:
+
+```bash
+python scripts/check_api_compatibility.py --baseline /path/to/baseline \
+  --candidate . --output compatibility.json
+```
+
+The audit checks twelve strategy/funnel combinations, serialized bytes, query
+results, copies/merges/statistics, streams, custom hashes and existing call
+signatures. It separately reports changes to bit-storage type, third-party
+methods, buffer support and `setup` input aliasing. A successful audit command
+means the comparison completed; inspect `fully_backward_compatible` for the
+compatibility verdict. Version 2.0 preserves the tested core operations but
+changes these public storage behaviors, so it is a major release.
+
+Hex and Base64 loaders include the master allocation-bound security update:
+encoded sizes and headers are checked before full Base64 decoding, and hex
+inputs are decoded in bounded chunks even with substantial whitespace.

{bloomfilter_py-1.3.0 → bloomfilter_py-2.0.0}/README.md

@@ -26,6 +26,68 @@ Here's a brief [introduction](https://en.wikipedia.org/wiki/Bloom_filter) to Blo
 pip install bloomfilter-py
 ```
 
+## Backends and migration from 1.3
+
+Version 2.0 runs on the Python standard library alone. `bitarray` and `mmh3`
+are no longer required to install or import this package. The default is always
+`backend="python"`, even if those packages are already installed. For explicit
+native acceleration:
+
+```bash
+pip install 'bloomfilter-py[native]'
+```
+
+```python
+from bloomfilter import BloomFilter
+
+fast = BloomFilter(500, 0.01, backend="native")
+fast.put("example")
+loaded = BloomFilter.loads(fast.dumps(), backend="native")
+assert "example" in loaded
+assert loaded.backend == "native"
+```
+
+The constructor and all loading methods accept the keyword-only `backend`
+argument (`"python"` or `"native"`). An invalid name raises `ValueError`;
+requesting native mode without **both** optional packages raises `ImportError`
+with installation instructions. There is no automatic fallback. `copy()`
+preserves the backend. Filters using different backends can be compared and
+merged when their strategies, funnels, sizes, and hash counts are compatible.
+Backend choice is not serialized: all old dumps remain readable, and new dumps
+use exactly the same Guava compact format.
+
+Pure Python hashing is slower than the C extension, especially on long keys.
+Native mode is available for throughput-sensitive workloads. Serialization,
+counting, and merging process at most 64 KiB of payload per chunk; copies and
+returned dumps necessarily allocate their complete result. `dumps()` and
+`loads()` also retain full input/output bytes, so use stream methods for large
+filters. Backing storage remains packed (roughly one byte per eight bits).
+
+### Bit storage and custom strategies
+
+`filter.data` is now a `bloomfilter.BitArray` facade with consistent semantics
+on either backend. It supports `len`, iteration, negative/integer indexing,
+slice copies, scalar or equal-length iterable slice assignment, `setall`,
+`count(0/1)`, `copy`, equality, `|=`, `+=`, `frombytes`, and `tobytes`.
+Slices preserve the backend; slice assignment cannot resize. Appending via
+`frombytes` or `+=` works even at non-byte-aligned lengths. `tobytes()` uses
+most-significant-bit-first bytes with zero padding at the end. The buffer
+protocol and the complete third-party `bitarray` API are **not** provided.
+
+Existing `setup(hash_count, data, strategy, funnel)` calls accept this facade.
+For migration, a third-party `bitarray.bitarray` is also accepted when installed;
+it is copied into Python-backed storage and normalized to logical bit order,
+including little-endian inputs. It is no longer aliased to `filter.data`.
+Callers relying on that alias or on third-party-only methods must update.
+Passing a `BitArray` to `setup` retains it, as before, so external mutations
+remain visible. `backend` reflects the current data object.
+
+Custom strategies still receive the same positional `put` / `might_contain`
+arguments, now with the facade as their data argument. Overrides of built-in
+strategies' `hash_key` are honored on both backends. Strategies must not depend
+on private storage fields or a specific third-party data type. Direct calls to built-in strategies with legacy third-party bit arrays remain
+supported. Direct calls to `Strategy.hash_key` always use the standard-library MurmurHash3 implementation.
+
 ## Usage Examples
 
 ### Basic Usage
@@ -236,10 +298,59 @@ install the additional tools and execute:
 
 ```bash
 python -m pip install black mypy bandit
-python -m black --check bloomfilter tests
+python -m black --check bloomfilter tests benchmarks scripts
 python -m mypy bloomfilter --strict
 python -m bandit -r bloomfilter
 ```
 
 On Windows PowerShell, activate the virtual environment with
 `.venv\Scripts\Activate.ps1` instead of `source .venv/bin/activate`.
+
+### Reproducible backend benchmarks
+
+```bash
+python benchmarks/benchmark_backends.py --backend python --output python.json
+python benchmarks/benchmark_backends.py --backend native --output native.json
+# Against a separate checkout of 1.3.0, with its dependencies installed:
+python benchmarks/benchmark_backends.py --backend baseline \
+  --package-path /path/to/1.3.0 --output baseline.json
+```
+
+The script reports median timings, raw repeats, workload sizes, and interpreter
+metadata. It covers fresh and repeated insertions, hits and misses, short and
+long byte keys, serialization, statistics, and merging. Results describe the
+specified workload rather than a universal throughput guarantee.
+
+For the zero-dependency source suite run `python -S -m unittest discover`.
+Install `.[native]` to enable optional differential tests against the C packages.
+The checked-in hash vectors were generated with mmh3 5.3.1 and cover every
+length from 0 through 512 plus larger block boundaries.
+Run Java integration tests for both backends with `BLOOMFILTER_TEST_BACKEND`
+set to `python` and `native` respectively. CI covers Python 3.8 through 3.14.
+
+Build and verify release artifacts with `python -m build` followed by
+`python scripts/verify_distribution.py dist`. The latter installs the wheel
+offline into a clean temporary environment and verifies it outside the source
+path, including the absence of mandatory runtime dependencies.
+
+### Auditing compatibility against an earlier checkout
+
+Use an environment containing the baseline's dependencies to compare actual
+behavior in separate processes:
+
+```bash
+python scripts/check_api_compatibility.py --baseline /path/to/baseline \
+  --candidate . --output compatibility.json
+```
+
+The audit checks twelve strategy/funnel combinations, serialized bytes, query
+results, copies/merges/statistics, streams, custom hashes and existing call
+signatures. It separately reports changes to bit-storage type, third-party
+methods, buffer support and `setup` input aliasing. A successful audit command
+means the comparison completed; inspect `fully_backward_compatible` for the
+compatibility verdict. Version 2.0 preserves the tested core operations but
+changes these public storage behaviors, so it is a major release.
+
+Hex and Base64 loaders include the master allocation-bound security update:
+encoded sizes and headers are checked before full Base64 decoding, and hex
+inputs are decoded in bounded chunks even with substantial whitespace.

bloomfilter_py-2.0.0/benchmarks/benchmark_backends.py

@@ -0,0 +1,134 @@
+"""Reproducible stdlib-only driver; baseline/native modes need their packages."""
+
+import argparse
+import hashlib
+import json
+from pathlib import Path
+import platform
+import statistics
+import sys
+import timeit
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--backend", choices=("python", "native", "baseline"), default="python"
+    )
+    parser.add_argument(
+        "--package-path", type=Path, default=Path(__file__).resolve().parents[1]
+    )
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--repeat", type=int, default=5)
+    parser.add_argument("--number", type=int, default=3)
+    args = parser.parse_args()
+    if args.repeat < 1 or args.number < 1:
+        parser.error("repeat and number must be positive")
+    sys.path.insert(0, str(args.package_path.resolve()))
+    from bloomfilter import BloomFilter, BYTE_ARRAY_FUNNEL
+
+    def make(expected=5000):
+        if args.backend == "baseline":
+            return BloomFilter(expected, 0.01, funnel=BYTE_ARRAY_FUNNEL)
+        return BloomFilter(
+            expected, 0.01, funnel=BYTE_ARRAY_FUNNEL, backend=args.backend
+        )
+
+    measurements = {}
+    fingerprints = {}
+    for key_size in (16, 1024):
+        keys = [
+            i.to_bytes(4, "little") + bytes([i % 256]) * (key_size - 4)
+            for i in range(500)
+        ]
+        misses = [
+            (i + 500).to_bytes(4, "little") + bytes([(i + 500) % 256]) * (key_size - 4)
+            for i in range(500)
+        ]
+        bf = make()
+        for key in keys:
+            bf.put(key)
+        fingerprints[str(key_size)] = hashlib.sha256(bf.dumps()).hexdigest()
+
+        def insert_new():
+            fresh = make()
+            for key in keys:
+                fresh.put(key)
+
+        tasks = {
+            "insert_new": insert_new,
+            "insert_repeat": lambda: [bf.put(key) for key in keys],
+            "query_hit": lambda: [bf.might_contain(key) for key in keys],
+            "query_miss": lambda: [bf.might_contain(key) for key in misses],
+        }
+        for name, task in tasks.items():
+            raw = [
+                value / args.number
+                for value in timeit.repeat(task, number=args.number, repeat=args.repeat)
+            ]
+            measurements[f"{name}_{key_size}B_500keys"] = {
+                "median_seconds": statistics.median(raw),
+                "raw_seconds": raw,
+            }
+
+    # Approximately 1 MiB, enough to exercise bounded chunks repeatedly.
+    large = make(875000)
+    for i in range(500):
+        large.put(i.to_bytes(4, "little"))
+    serialized = large.dumps()
+    second = large.copy()
+    second.put(b"second-filter")
+    if args.backend == "baseline":
+        load = lambda: BloomFilter.loads(serialized, BYTE_ARRAY_FUNNEL)
+    else:
+        load = lambda: BloomFilter.loads(
+            serialized, BYTE_ARRAY_FUNNEL, backend=args.backend
+        )
+    for name, task in {
+        "dump_1MiB": large.dumps,
+        "load_1MiB": load,
+        "statistics_1MiB": large.expected_fpp,
+        "merge_repeat_1MiB": lambda: large.put_all(second),
+    }.items():
+        raw = [
+            value / args.number
+            for value in timeit.repeat(task, number=args.number, repeat=args.repeat)
+        ]
+        measurements[name] = {
+            "median_seconds": statistics.median(raw),
+            "raw_seconds": raw,
+        }
+    report = {
+        "backend": args.backend,
+        "python": sys.version,
+        "platform": platform.platform(),
+        "package_path": str(args.package_path.resolve()),
+        "repeat": args.repeat,
+        "number": args.number,
+        "short_filter_capacity": 5000,
+        "fpp": 0.01,
+        "keys_per_batch": 500,
+        "large_filter_bytes": len(serialized),
+        "fingerprints": fingerprints,
+        "notes": [
+            "Fresh insertion includes constructing an empty filter.",
+            "Repeated insertion and merge operate on already populated filters.",
+            "Query misses are absent keys and may include false positives.",
+            "Operations return full serialized bytes for dump/load.",
+        ],
+        "measurements": measurements,
+    }
+    args.output.write_text(json.dumps(report, indent=2) + "\n")
+    print(
+        json.dumps(
+            {
+                name: round(value["median_seconds"] * 1000, 3)
+                for name, value in measurements.items()
+            },
+            indent=2,
+        )
+    )
+
+
+if __name__ == "__main__":
+    main()

{bloomfilter_py-1.3.0 → bloomfilter_py-2.0.0}/bloomfilter/__init__.py

@@ -1,3 +1,4 @@
+from bloomfilter._bitarray import BitArray
 from bloomfilter.bloomfilter import BloomFilter
 from bloomfilter.funnel import (
     BYTE_ARRAY_FUNNEL,
@@ -11,6 +12,7 @@ from bloomfilter.funnel import (
 
 __all__ = [
     "BloomFilter",
+    "BitArray",
     "Funnel",
     "StringFunnel",
     "INTEGER_FUNNEL",