pyfastcdc 0.2.0b1__tar.gz → 0.3.0__tar.gz

{pyfastcdc-0.2.0b1/pyfastcdc.egg-info → pyfastcdc-0.3.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: pyfastcdc
-Version: 0.2.0b1
-Summary: FastCDC 2020 implementation in Python, with Cython acceleration
+Version: 0.3.0
+Summary: A high-performance FastCDC 2020 implementation written in Python + Cython
 Author: Fallen_Breath
 License: MIT
 Project-URL: Homepage, https://github.com/Fallen-Breath/pyfastcdc
@@ -42,7 +42,7 @@ Dynamic: summary
 [![Issues](https://img.shields.io/github/issues/Fallen-Breath/pyfastcdc.svg)](https://github.com/Fallen-Breath/pyfastcdc/issues)
 [![PyPI Version](https://img.shields.io/pypi/v/pyfastcdc.svg?label=PyPI)](https://pypi.org/project/pyfastcdc)
 
-A FastCDC 2020 implementation written in Python, with [Cython](https://github.com/cython/cython) acceleration
+A high-performance FastCDC 2020 implementation written in Python + [Cython](https://github.com/cython/cython)
 
 Supports Python 3.6+. Provides prebuilt wheels for Python 3.8+
 
@@ -63,14 +63,46 @@ For platforms without prebuilt wheels, a suitable build environment capable of c
 For example, on Debian, you might need to install `gcc` and `python3-dev` via `apt`
 
 If the Cython extension fails to compile, the installation will fall back to a pure Python implementation,
-which is significantly slower (around 0.01× or less in memory chunking speed)
+which is significantly slower (about 0.5% or less in memory chunking speed)
+
+<details>
+
+<summary>I only want to use the Cython implemetion, not the slow pure-Python one</summary>
+
+You can set the environment variable `PYFASTCDC_REQUIRE_CYTHON=true` or `PYFASTCDC_REQUIRE_CYTHON=1` for the `pip` installation command
+to disable the pure-Python fallback on extension compilation error and make the installation fail hard.
+Thus, after a successful installation, you will always have a working Cython extension
+
+Example bash command using `pip`:
+
+```
+$ PYFASTCDC_REQUIRE_CYTHON=true pip install pyfastcdc
+... some pip output ...
+Building wheels for collected packages: pyfastcdc
+  Building wheel for pyfastcdc (pyproject.toml) ... error
+  ... some pip output ...
+  ###########################################################################################################
+  Failed to compile pyfastcdc Cython extension, fail hard since PYFASTCDC_REQUIRE_CYTHON is set to true
+  Unset PYFASTCDC_REQUIRE_CYTHON to allow pure-Python fallback if that's acceptable for your use case.
+  <class 'distutils.compilers.C.errors.CompileError'> command 'gcc' failed: No such file or directory
+  ###########################################################################################################
+  error: command 'gcc' failed: No such file or directory
+  ----------------------------------------
+  ERROR: Failed building wheel for pyfastcdc
+  ... some pip output ...
+```
+
+</details>
 
 ## Usage
 
-The basic usage is simple:
+The usage of PyFastCDC is simple:
 
 1. Construct a `FastCDC` instance with desired parameters
 2. Call `FastCDC.cut_xxx()` function to chunk your data
+    - Call `cut_buf()` to chunk in-memory data buffers
+    - Call `cut_file()` to chunk a regular file using mmap
+    - Call `cut_stream()` to chunk a custom file-like streaming object
 
 Example:
 
@@ -88,8 +120,8 @@ Please only import members from `pyfastcdc` in your application code and avoid i
 Only public APIs inside the `pyfastcdc` module are guaranteed to be stable across releases
 
 ```python
-from pyfastcdc import Chunk         # GOOD
-from pyfastcdc.common import Chunk  # BAD, no API stability guarantee
+from pyfastcdc import NormalizedChunking         # GOOD
+from pyfastcdc.common import NormalizedChunking  # BAD, no API stability guarantee
 ```
 
 ## Performance
@@ -114,7 +146,7 @@ FastCDC parameters:
 
 Test environment:
 
-- PyFastCDC 0.1.1b, precompiled wheel from Test PyPI, Cython 3.2.4
+- PyFastCDC 0.2.0b1, precompiled wheel from Test PyPI, Cython 3.2.4
 - Python 3.11.14 using docker image `python:3.11`
 - Ryzen 7 6800H @ 4.55GHz, NVMe SSD, Debian 13.2
 

{pyfastcdc-0.2.0b1 → pyfastcdc-0.3.0}/README.md

@@ -4,7 +4,7 @@
 [![Issues](https://img.shields.io/github/issues/Fallen-Breath/pyfastcdc.svg)](https://github.com/Fallen-Breath/pyfastcdc/issues)
 [![PyPI Version](https://img.shields.io/pypi/v/pyfastcdc.svg?label=PyPI)](https://pypi.org/project/pyfastcdc)
 
-A FastCDC 2020 implementation written in Python, with [Cython](https://github.com/cython/cython) acceleration
+A high-performance FastCDC 2020 implementation written in Python + [Cython](https://github.com/cython/cython)
 
 Supports Python 3.6+. Provides prebuilt wheels for Python 3.8+
 
@@ -25,14 +25,46 @@ For platforms without prebuilt wheels, a suitable build environment capable of c
 For example, on Debian, you might need to install `gcc` and `python3-dev` via `apt`
 
 If the Cython extension fails to compile, the installation will fall back to a pure Python implementation,
-which is significantly slower (around 0.01× or less in memory chunking speed)
+which is significantly slower (about 0.5% or less in memory chunking speed)
+
+<details>
+
+<summary>I only want to use the Cython implemetion, not the slow pure-Python one</summary>
+
+You can set the environment variable `PYFASTCDC_REQUIRE_CYTHON=true` or `PYFASTCDC_REQUIRE_CYTHON=1` for the `pip` installation command
+to disable the pure-Python fallback on extension compilation error and make the installation fail hard.
+Thus, after a successful installation, you will always have a working Cython extension
+
+Example bash command using `pip`:
+
+```
+$ PYFASTCDC_REQUIRE_CYTHON=true pip install pyfastcdc
+... some pip output ...
+Building wheels for collected packages: pyfastcdc
+  Building wheel for pyfastcdc (pyproject.toml) ... error
+  ... some pip output ...
+  ###########################################################################################################
+  Failed to compile pyfastcdc Cython extension, fail hard since PYFASTCDC_REQUIRE_CYTHON is set to true
+  Unset PYFASTCDC_REQUIRE_CYTHON to allow pure-Python fallback if that's acceptable for your use case.
+  <class 'distutils.compilers.C.errors.CompileError'> command 'gcc' failed: No such file or directory
+  ###########################################################################################################
+  error: command 'gcc' failed: No such file or directory
+  ----------------------------------------
+  ERROR: Failed building wheel for pyfastcdc
+  ... some pip output ...
+```
+
+</details>
 
 ## Usage
 
-The basic usage is simple:
+The usage of PyFastCDC is simple:
 
 1. Construct a `FastCDC` instance with desired parameters
 2. Call `FastCDC.cut_xxx()` function to chunk your data
+    - Call `cut_buf()` to chunk in-memory data buffers
+    - Call `cut_file()` to chunk a regular file using mmap
+    - Call `cut_stream()` to chunk a custom file-like streaming object
 
 Example:
 
@@ -50,8 +82,8 @@ Please only import members from `pyfastcdc` in your application code and avoid i
 Only public APIs inside the `pyfastcdc` module are guaranteed to be stable across releases
 
 ```python
-from pyfastcdc import Chunk         # GOOD
-from pyfastcdc.common import Chunk  # BAD, no API stability guarantee
+from pyfastcdc import NormalizedChunking         # GOOD
+from pyfastcdc.common import NormalizedChunking  # BAD, no API stability guarantee
 ```
 
 ## Performance
@@ -76,7 +108,7 @@ FastCDC parameters:
 
 Test environment:
 
-- PyFastCDC 0.1.1b, precompiled wheel from Test PyPI, Cython 3.2.4
+- PyFastCDC 0.2.0b1, precompiled wheel from Test PyPI, Cython 3.2.4
 - Python 3.11.14 using docker image `python:3.11`
 - Ryzen 7 6800H @ 4.55GHz, NVMe SSD, Debian 13.2
 

{pyfastcdc-0.2.0b1 → pyfastcdc-0.3.0}/pyfastcdc/__init__.py

@@ -1,4 +1,4 @@
-__version__ = '0.2.0b1'
+__version__ = '0.3.0'
 
 __all__ = [
 	'BinaryStreamReader',

{pyfastcdc-0.2.0b1 → pyfastcdc-0.3.0}/pyfastcdc/__init__.pyi

@@ -3,6 +3,8 @@ from typing import Optional, Union, Iterator
 
 from typing_extensions import Protocol, Literal
 
+__version__: str
+
 
 class _BinaryStreamReaderWithRead(Protocol):
 	def read(self, n: int) -> bytes: ...

{pyfastcdc-0.2.0b1 → pyfastcdc-0.3.0}/pyfastcdc/cy/fastcdc.pyx

@@ -32,7 +32,7 @@ cdef uint64_t AVG_SIZE_UPPER_BOUND = 4 * 1048576
 cdef uint64_t MAX_SIZE_UPPER_BOUND = 16 * 1048576
 
 
-# see the .pyi file for docstrings
+# docstrings are in pyfastcdc/__init__.pyi
 cdef class FastCDC:
 	cdef _Config config
 	cdef uint64_t* gear_holder
@@ -66,7 +66,7 @@ cdef class FastCDC:
 		self.config.min_size = min_size
 		self.config.max_size = max_size
 
-		bits = avg_size.bit_length() - 1
+		bits = utils.logarithm2(avg_size)
 		self.config.mask_s = MASKS[bits + normalized_chunking]
 		self.config.mask_l = MASKS[bits - normalized_chunking]
 		self.config.mask_s_ls = self.config.mask_s << 1

{pyfastcdc-0.2.0b1 → pyfastcdc-0.3.0}/pyfastcdc/py/fastcdc.py

@@ -48,6 +48,7 @@ class _Config:
 		self.gear_ls = gear_ls
 
 
+# docstrings are in pyfastcdc/__init__.pyi
 class FastCDC:
 	MIN_SIZE_LOWER_BOUND: ClassVar[int] = 64
 	AVG_SIZE_LOWER_BOUND: ClassVar[int] = 256
@@ -80,7 +81,7 @@ class FastCDC:
 		if not (0 <= normalized_chunking <= 3):
 			raise ValueError(f'normalized_chunking {normalized_chunking} is out of range [0, 3]')
 
-		bits = avg_size.bit_length() - 1
+		bits = utils.logarithm2(avg_size)
 		mask_s = MASKS[bits + normalized_chunking]
 		mask_l = MASKS[bits - normalized_chunking]
 		mask_s_ls = (mask_s << 1) & _UINT64_MASK

{pyfastcdc-0.2.0b1 → pyfastcdc-0.3.0}/pyfastcdc/utils.py

@@ -1,3 +1,4 @@
+import math
 import mmap
 import os
 from pathlib import Path
@@ -8,6 +9,15 @@ from pyfastcdc.common import BinaryStreamReader
 ReadintoFunc = Callable[[memoryview], int]
 
 
+def logarithm2(value: int) -> int:
+	# The same as the nlfiedler/fastcdc-rs v4.0.1 Rust implementation: (value as f64).log2().round() as u32
+	if value < 0:
+		raise ValueError(f'value must be non-negative, got {value}')
+	if value == 0:
+		return 0
+	return math.floor(math.log2(value) + 0.5)
+
+
 def create_memoryview_from_buffer(buf: Union[bytes, bytearray, memoryview]) -> memoryview:
 	if isinstance(buf, (bytes, bytearray, memoryview)):
 		return memoryview(buf)

{pyfastcdc-0.2.0b1 → pyfastcdc-0.3.0/pyfastcdc.egg-info}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: pyfastcdc
-Version: 0.2.0b1
-Summary: FastCDC 2020 implementation in Python, with Cython acceleration
+Version: 0.3.0
+Summary: A high-performance FastCDC 2020 implementation written in Python + Cython
 Author: Fallen_Breath
 License: MIT
 Project-URL: Homepage, https://github.com/Fallen-Breath/pyfastcdc
@@ -42,7 +42,7 @@ Dynamic: summary
 [![Issues](https://img.shields.io/github/issues/Fallen-Breath/pyfastcdc.svg)](https://github.com/Fallen-Breath/pyfastcdc/issues)
 [![PyPI Version](https://img.shields.io/pypi/v/pyfastcdc.svg?label=PyPI)](https://pypi.org/project/pyfastcdc)
 
-A FastCDC 2020 implementation written in Python, with [Cython](https://github.com/cython/cython) acceleration
+A high-performance FastCDC 2020 implementation written in Python + [Cython](https://github.com/cython/cython)
 
 Supports Python 3.6+. Provides prebuilt wheels for Python 3.8+
 
@@ -63,14 +63,46 @@ For platforms without prebuilt wheels, a suitable build environment capable of c
 For example, on Debian, you might need to install `gcc` and `python3-dev` via `apt`
 
 If the Cython extension fails to compile, the installation will fall back to a pure Python implementation,
-which is significantly slower (around 0.01× or less in memory chunking speed)
+which is significantly slower (about 0.5% or less in memory chunking speed)
+
+<details>
+
+<summary>I only want to use the Cython implemetion, not the slow pure-Python one</summary>
+
+You can set the environment variable `PYFASTCDC_REQUIRE_CYTHON=true` or `PYFASTCDC_REQUIRE_CYTHON=1` for the `pip` installation command
+to disable the pure-Python fallback on extension compilation error and make the installation fail hard.
+Thus, after a successful installation, you will always have a working Cython extension
+
+Example bash command using `pip`:
+
+```
+$ PYFASTCDC_REQUIRE_CYTHON=true pip install pyfastcdc
+... some pip output ...
+Building wheels for collected packages: pyfastcdc
+  Building wheel for pyfastcdc (pyproject.toml) ... error
+  ... some pip output ...
+  ###########################################################################################################
+  Failed to compile pyfastcdc Cython extension, fail hard since PYFASTCDC_REQUIRE_CYTHON is set to true
+  Unset PYFASTCDC_REQUIRE_CYTHON to allow pure-Python fallback if that's acceptable for your use case.
+  <class 'distutils.compilers.C.errors.CompileError'> command 'gcc' failed: No such file or directory
+  ###########################################################################################################
+  error: command 'gcc' failed: No such file or directory
+  ----------------------------------------
+  ERROR: Failed building wheel for pyfastcdc
+  ... some pip output ...
+```
+
+</details>
 
 ## Usage
 
-The basic usage is simple:
+The usage of PyFastCDC is simple:
 
 1. Construct a `FastCDC` instance with desired parameters
 2. Call `FastCDC.cut_xxx()` function to chunk your data
+    - Call `cut_buf()` to chunk in-memory data buffers
+    - Call `cut_file()` to chunk a regular file using mmap
+    - Call `cut_stream()` to chunk a custom file-like streaming object
 
 Example:
 
@@ -88,8 +120,8 @@ Please only import members from `pyfastcdc` in your application code and avoid i
 Only public APIs inside the `pyfastcdc` module are guaranteed to be stable across releases
 
 ```python
-from pyfastcdc import Chunk         # GOOD
-from pyfastcdc.common import Chunk  # BAD, no API stability guarantee
+from pyfastcdc import NormalizedChunking         # GOOD
+from pyfastcdc.common import NormalizedChunking  # BAD, no API stability guarantee
 ```
 
 ## Performance
@@ -114,7 +146,7 @@ FastCDC parameters:
 
 Test environment:
 
-- PyFastCDC 0.1.1b, precompiled wheel from Test PyPI, Cython 3.2.4
+- PyFastCDC 0.2.0b1, precompiled wheel from Test PyPI, Cython 3.2.4
 - Python 3.11.14 using docker image `python:3.11`
 - Ryzen 7 6800H @ 4.55GHz, NVMe SSD, Debian 13.2
 

{pyfastcdc-0.2.0b1 → pyfastcdc-0.3.0}/pyfastcdc.egg-info/SOURCES.txt

@@ -26,4 +26,5 @@ pyfastcdc/py/chunk.py
 pyfastcdc/py/constants.py
 pyfastcdc/py/fastcdc.py
 tests/test_arguments.py
-tests/test_correctness.py
+tests/test_correctness.py
+tests/test_utils.py

{pyfastcdc-0.2.0b1 → pyfastcdc-0.3.0}/setup.py

@@ -1,6 +1,7 @@
 import ast
 import contextlib
 import functools
+import os
 import sys
 from pathlib import Path
 from typing import List
@@ -56,21 +57,23 @@ class BuildExt(build_ext):
 	def __init__(self, *args, **kwargs):
 		super().__init__(*args, **kwargs)
 
-		try:
-			from Cython import __version__ as cython_version
-		except ImportError:
-			cython_version = 'N/A'
-		print(f'Cython version: {cython_version}')
-
 	@classmethod
 	@contextlib.contextmanager
 	def __wrap_ext_err(cls):
+		require_cython = os.environ.get('PYFASTCDC_REQUIRE_CYTHON', '').lower() in ['true', '1']
 		try:
 			yield
 		except Exception as e:
+			if require_cython:
+				print("###########################################################################################################", file=sys.stderr)
+				print("Failed to compile pyfastcdc Cython extension, fail hard since PYFASTCDC_REQUIRE_CYTHON is set to true", file=sys.stderr)
+				print("Unset PYFASTCDC_REQUIRE_CYTHON to allow pure-Python fallback if that's acceptable for your use case.", file=sys.stderr)
+				print(type(e), e, file=sys.stderr)
+				print("###########################################################################################################", file=sys.stderr)
+				raise e
 			print("###########################################################################################################", file=sys.stderr)
-			print("Failed to compile pyfastcdc cython extension, fallback to pure python implementation with is a lot slower", file=sys.stderr)
-			print(e)
+			print("Failed to compile pyfastcdc Cython extension, fallback to pure python implementation with is a lot slower", file=sys.stderr)
+			print(type(e), e, file=sys.stderr)
 			print("###########################################################################################################", file=sys.stderr)
 
 	def run(self):
@@ -94,10 +97,16 @@ else:
 
 print(f'setuptools_version: {setuptools_version}')
 use_license_expression = parse_major(setuptools_version) >= 77
+try:
+	from Cython import __version__ as cython_version
+except ImportError:
+	cython_version = 'N/A'
+print(f'Cython version: {cython_version}')
+
 setup(
 	name='pyfastcdc',
 	version=get_version(),
-	description='FastCDC 2020 implementation in Python, with Cython acceleration',
+	description='A high-performance FastCDC 2020 implementation written in Python + Cython',
 	long_description=read_file('README.md'),
 	long_description_content_type='text/markdown',
 	author='Fallen_Breath',

{pyfastcdc-0.2.0b1 → pyfastcdc-0.3.0}/tests/test_arguments.py

@@ -102,4 +102,4 @@ class TestAvgSize:
 
 		# larger avg_size means fewer chunks
 		for i in range(1, len(chunk_counts)):
-			assert chunk_counts[i] <= chunk_counts[i - 1]
+			assert chunk_counts[i] <= chunk_counts[i - 1]