blosc2 3.3.3__tar.gz → 3.4.0__tar.gz

{blosc2-3.3.3 → blosc2-3.4.0}/.pre-commit-config.yaml

@@ -16,7 +16,7 @@ repos:
       - id: trailing-whitespace
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.11.8
+    rev: v0.11.12
     hooks:
       - id: ruff
         args: ["--fix", "--show-fixes"]

{blosc2-3.3.3 → blosc2-3.4.0}/ANNOUNCE.rst

@@ -1,10 +1,7 @@
-Announcing Python-Blosc2 3.3.3
+Announcing Python-Blosc2 3.4.0
 ==============================
 
-This is a bugfix release, with some minor optimizations. We improved the
-correct chaining of *string* lazy expressions (there is still room for
-improvement).  Also, a new version of C-Blosc2 (v2.7.2.dev) that fixes
-mmap issues on Windows is used; thanks to @JanSellner for the fix.
+This release adds significant new functionality in the form of ``concatenate``. We support general concatenation of ndarrays, and offer an optimised path with significant speedups for the case of concatenating arrays with compatible chunk and blockshapes. In addition, there are bug fixes and more functionality for slicing of lazyexprs, and the possibility to jit compile user-defined functions which operate on pandas objects using the blosc2 engine.
 
 You can think of Python-Blosc2 3.x as an extension of NumPy/numexpr that:
 

{blosc2-3.3.3 → blosc2-3.4.0}/CMakeLists.txt

@@ -50,7 +50,7 @@ else()
     include(FetchContent)
     FetchContent_Declare(blosc2
         GIT_REPOSITORY https://github.com/Blosc/c-blosc2
-        GIT_TAG 3f9c085c0c0f357b140fb5a50d1f8df06de3383a  # v2.17.1 + mmap fix
+        GIT_TAG 1c2f8bb0c914c43e23b751fbcf6642cd7aec09db  # v2.18.0 (concatenate added)
     )
     FetchContent_MakeAvailable(blosc2)
     include_directories("${blosc2_SOURCE_DIR}/include")

{blosc2-3.3.3 → blosc2-3.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: blosc2
-Version: 3.3.3
+Version: 3.4.0
 Summary: A fast & compressed ndarray library with a flexible compute engine.
 Author-Email: Blosc Development Team <blosc@blosc.org>
 Maintainer-Email: Blosc Development Team <blosc@blosc.org>

{blosc2-3.3.3 → blosc2-3.4.0}/README_DEVELOPERS.md

@@ -22,6 +22,20 @@ You are done!
     pip install .   # add -e for editable mode
 ```
 
+There are situations where you may want to build the C-Blosc2 library separately, for example, when debugging issues in the C library. In that case, let's assume you have the C-Blosc2 library installed in `/usr/local`:
+
+```bash
+CMAKE_PREFIX_PATH=/usr/local USE_SYSTEM_BLOSC2=1 pip install -e .
+```
+
+and then, you can run the tests with:
+
+```bash
+LD_LIBRARY_PATH=/usr/local/lib pytest
+```
+
+[replace `LD_LIBRARY_PATH` with the appropriate environment variable for your system, such as `DYLD_LIBRARY_PATH` on macOS or `PATH` on Windows, if necessary].
+
 That's it! You can now proceed to the testing section.
 
 ## Testing

{blosc2-3.3.3 → blosc2-3.4.0}/RELEASE_NOTES.md

@@ -1,4 +1,34 @@
 # Release notes
+## Changes from 3.4.0 to 3.4.1
+
+XXX version-specific blurb XXX
+
+## Changes from 3.3.4 to 3.4.0
+
+* Added C-level ``concatenate`` function in response to community request. When possible, uses an optimised path which avoids decompression and recompression, giving a significant performance boost. See PR #423.
+
+* Slicing has been added to string-based lazyexprs, so that one may use
+  expressions like `expr[1:3] +1` to compute a slice of the expression. This is useful
+  for getting a sub-expression of a larger expression, and it works with both
+  string-based and lazy expressions. See PR #417.
+
+* Relatedly, the behaviour of the `slice` parameter in the `compute()` method of `LazyExpr` has been made more consistent and is now better documented, so that results are as expected. See PR #419.
+
+* UDF support for pandas has been added to allow for the use of ``blosc2.jit``. See PR #418. Thanks to [@datapythonista](https://github.com/datapythonista) for the implementation!
+
+## Changes from 3.3.3 to 3.3.4
+
+* Expand possibilities for chaining string-based lazy expressions to incorporate
+  data types which do not have shape attribute, e.g. int, float etc.
+  See #406 and PR #411.
+
+* Enable slicing within string-based lazy expressions. See PR #414.
+
+* Improved casting for string-based lazy expressions.
+
+* Documentation improvements, see PR #410.
+
+* Compatibility fixes for working with `h5py` files.
 
 ## Changes from 3.3.2 to 3.3.3
 

{blosc2-3.3.3 → blosc2-3.4.0}/doc/python-blosc2.rst

@@ -6,7 +6,7 @@
 
     <p style="text-align: center; color: black; background-color: rgba(230, 169, 9, 0.65);">
         <a href="https://github.com/Blosc/python-blosc2/blob/main/RELEASE_NOTES.md"
-           style="font-size: 1.5em;">Version 3.3.2 released on 2025-05-01!</a>
+           style="font-size: 1.5em;">Version 3.3.4 released on 2025-05-22!</a>
         <span style="display: inline-block; width: 20px;"></span>
         <span style="font-family: monospace;">pip install blosc2 -U</span>
     </p>

{blosc2-3.3.3 → blosc2-3.4.0}/doc/reference/classes.rst

@@ -12,3 +12,4 @@ Main Classes
     proxy
     proxysource
     proxyndsource
+    simpleproxy

{blosc2-3.3.3 → blosc2-3.4.0}/doc/reference/ndarray.rst

@@ -72,6 +72,7 @@ Constructors
 
     asarray
     copy
+    concatenate
     empty
     frombuffer
     fromiter

{blosc2-3.3.3 → blosc2-3.4.0}/doc/reference/proxysource.rst

@@ -4,8 +4,8 @@ ProxySource
 ===========
 
 Base interface for all supported sources in :ref:`Proxy` and are not NDim objects.
-For example, a file, a memory buffer, a network resource, etc.  For NDims, see
-:ref:`ProxyNDSource`.
+For example, a file, a memory buffer, a network resource, etc.  For n-dimemsional
+ones, see :ref:`ProxyNDSource`.
 
 .. currentmodule:: blosc2.ProxySource
 

blosc2-3.4.0/doc/reference/simpleproxy.rst

@@ -0,0 +1,30 @@
+.. _SimpleProxy:
+
+SimpleProxy
+===========
+
+Simple proxy for a NumPy array (or similar) that can be used with the Blosc2 compute engine.
+
+This only supports the __getitem__ method. No caching is performed.
+
+.. currentmodule:: blosc2.SimpleProxy
+
+Methods
+-------
+
+.. autosummary::
+    :toctree: autofiles/simpleproxy
+    :nosignatures:
+
+    __init__
+    __getitem__
+
+Attributes
+----------
+
+.. autosummary::
+    :toctree: autofiles/simpleproxy
+
+    shape
+    dtype
+    src

{blosc2-3.3.3 → blosc2-3.4.0}/doc/reference/storage.rst

@@ -1,3 +1,5 @@
+.. _CompStorParams:
+
 Compression, decompression and storage parameters
 =================================================
 

blosc2-3.4.0/examples/blosc2_hdf5_compression.py

@@ -0,0 +1,95 @@
+#######################################################################
+# Copyright (c) 2019-present, Blosc Development Team <blosc@blosc.org>
+# All rights reserved.
+#
+# This source code is licensed under a BSD-style license (found in the
+# LICENSE file in the root directory of this source tree)
+#######################################################################
+
+# This shows how to convert a generic .h5 file to a custom blosc2-compressed .h5 file
+# The blosc2 plugin in hdf5plugin doesn't support custom block shapes, and so one
+# has to go a different route for more bespoke compression
+
+import os
+
+import h5py
+import hdf5plugin
+
+import blosc2
+
+clevel = 5  # compression level, e.g., 0-9, where 0 is no compression and 9 is maximum compression
+fname_in = "kevlar.h5"  # input file with the kevlar dataset
+fname_out = "kevlar-blosc2.h5"
+if not os.path.exists(fname_in):
+    raise FileNotFoundError(
+        f"Input file {fname_in} does not exist\n"
+        "Please download it from the kevlar repository at:"
+        " http://www.silx.org/pub/pyFAI/pyFAI_UM_2020/data_ID13/kevlar.h5"
+    )
+
+# Example 1
+# hdf5plugin supports limited blosc2 compression with certain codecs
+cname = "zstd"
+with h5py.File(fname_in, "r") as fr:
+    dset = fr["/entry/data/data"][:]
+    with h5py.File(fname_out, "w") as fw:
+        g = fw.create_group("/data")
+        b2comp = hdf5plugin.Blosc2(cname=cname, clevel=clevel, filters=hdf5plugin.Blosc2.BITSHUFFLE)
+        dset_out = g.create_dataset(
+            f"cname-{cname}",
+            data=dset,
+            dtype=dset.dtype,
+            chunks=(1,) + dset.shape[1:],  # chunk size of 1 frame
+            **b2comp,
+        )
+    print("Successfully compressed file with hdf5plugin")
+
+# Example 2
+# For other codecs (e.g grok) or for more custom compression such as with user-defined block shapes, one
+# has to use a more involved route
+blocks = (50, 80, 80)
+chunks = (200, 240, 240)
+cparams = {
+    "codec": blosc2.Codec.LZ4,
+    "filters": [blosc2.Filter.BITSHUFFLE],
+    "splitmode": blosc2.SplitMode.NEVER_SPLIT,
+    "clevel": clevel,
+}
+
+if os.path.exists("dset.b2nd"):  # don't reload dset to blosc2 if already done so once
+    b2im = blosc2.open(urlpath="dset.b2nd", mode="r")
+    s, d = b2im.shape, b2im.dtype
+else:
+    with h5py.File(fname_in, "r") as fr:  # load file and process to blosc2 array
+        dset = fr["/entry/data/data"][:]
+        b2im = blosc2.asarray(
+            dset, chunks=chunks, blocks=blocks, cparams=cparams, urlpath="dset.b2nd", mode="w"
+        )
+        d = dset.dtype
+        del dset
+
+# Write to .h5 file #
+with h5py.File(fname_out, "w") as fw:
+    g = fw.create_group("/data")
+    b2comp = hdf5plugin.Blosc2()  # just for identification, no compression algorithm specified
+    dset_out = g.create_dataset(
+        "cname-customlz4",
+        s,
+        d,
+        chunks=chunks,  # chunk size of 1 frame
+        **b2comp,
+    )
+    # Write individual blosc2 chunks directly to hdf5
+    # hdf5 requires a cframe, which is only available via blosc2 schunks (not chunks)
+    for info in b2im.iterchunks_info():
+        ncoords = tuple(n * chunks[i] for i, n in enumerate(info.coords))
+        aux = blosc2.empty(
+            shape=b2im.chunks, chunks=b2im.chunks, blocks=b2im.blocks, dtype=b2im.dtype
+        )  # very cheap memory allocation
+        aux.schunk.insert_chunk(
+            0, b2im.get_chunk(info.nchunk)
+        )  # insert chunk into blosc2 array so we have schunk wrapper (no decompression required)
+        dset_out.id.write_direct_chunk(
+            ncoords, aux.schunk.to_cframe()
+        )  # convert schunk to cframe and write to hdf5
+    print("Successfully compressed file with custom parameters")

{blosc2-3.3.3 → blosc2-3.4.0}/examples/ndarray/jit-numpy-funcs.py

@@ -18,7 +18,7 @@ import blosc2
 # Create some sample data
 a = blosc2.linspace(0, 1, 10 * 100, dtype="float32", shape=(10, 100))
 b = blosc2.linspace(1, 2, 10 * 100, dtype="float32", shape=(10, 100))
-c = blosc2.linspace(-10, 10, 10, dtype="float32", shape=(100,))
+c = blosc2.linspace(-10, 10, 100, dtype="float32", shape=(100,))
 
 
 # Example 1: Basic usage of the jit decorator with reduction

blosc2-3.4.0/examples/ndarray/lazyexpr_where_indexing.py

@@ -0,0 +1,42 @@
+# Imports
+
+import numpy as np
+
+import blosc2
+
+N = 1000
+it = ((-x + 1, x - 2, 0.1 * x) for x in range(N))
+sa = blosc2.fromiter(
+    it, dtype=[("A", "i4"), ("B", "f4"), ("C", "f8")], shape=(N,), urlpath="sa-1M.b2nd", mode="w"
+)
+expr = sa["(A < B)"]
+A = sa["A"][:]
+B = sa["B"][:]
+C = sa["C"][:]
+temp = sa[:]
+indices = A < B
+idx = np.argmax(indices)
+
+# One might think that expr[:10] gives the first 10 elements of the evaluated expression, but this is not the case.
+# It actually computes the expression on the first 10 elements of the operands; since for some elements the condition
+# is False, the result will be shorter than 10 elements.
+# Returns less than 10 elements in general
+sliced = expr.compute(slice(0, 10))
+gotitem = expr[:10]
+np.testing.assert_array_equal(sliced[:], gotitem)
+np.testing.assert_array_equal(gotitem, temp[:10][indices[:10]])  # Equivalent syntax
+# Actually this makes sense since one can understand this as a request to compute on a portion of operands.
+# If one desires a portion of the result, one should compute the whole expression and then slice it.
+
+# Get first element for which condition is true
+sliced = expr.compute(idx)
+gotitem = expr[idx]
+# Arrays of one element
+np.testing.assert_array_equal(sliced[()], gotitem)
+np.testing.assert_array_equal(gotitem, temp[idx])
+
+# Should return void arrays here.
+sliced = expr.compute(0)
+gotitem = expr[0]
+np.testing.assert_array_equal(sliced[()], gotitem)
+np.testing.assert_array_equal(gotitem, temp[0])

{blosc2-3.3.3 → blosc2-3.4.0}/pyproject.toml

@@ -41,7 +41,7 @@ dependencies = [
     "py-cpuinfo; platform_machine != 'wasm32'",
     "requests",
 ]
-version = "3.3.3"
+version = "3.4.0"
 
 
 [project.optional-dependencies]

{blosc2-3.3.3 → blosc2-3.4.0}/src/blosc2/__init__.py

@@ -238,6 +238,7 @@ from .ndarray import (
     sort,
     reshape,
     copy,
+    concatenate,
     empty,
     frombuffer,
     fromiter,

{blosc2-3.3.3 → blosc2-3.4.0}/src/blosc2/blosc2_ext.pyx

@@ -507,6 +507,8 @@ cdef extern from "b2nd.h":
     int b2nd_squeeze_index(b2nd_array_t *array, const c_bool *index)
     int b2nd_resize(b2nd_array_t *array, const int64_t *new_shape, const int64_t *start)
     int b2nd_copy(b2nd_context_t *ctx, b2nd_array_t *src, b2nd_array_t **array)
+    int b2nd_concatenate(b2nd_context_t *ctx, b2nd_array_t *src1, b2nd_array_t *src2,
+                         int8_t axis, c_bool copy, b2nd_array_t **array)
     int b2nd_from_schunk(blosc2_schunk *schunk, b2nd_array_t **array)
 
     void blosc2_unidim_to_multidim(uint8_t ndim, int64_t *shape, int64_t i, int64_t *index)
@@ -2595,8 +2597,33 @@ cdef class NDArray:
 
 
 cdef b2nd_context_t* create_b2nd_context(shape, chunks, blocks, dtype, kwargs):
-    # This is used only in constructors, dtype will always have NumPy format
-    dtype = np.dtype(dtype)
+    if isinstance(dtype, list) and len(dtype) > 0 and isinstance(dtype[0], tuple):
+        # Extract just the field names and basic dtype info
+        fields = []
+        for field in dtype:
+            name = field[0]
+            field_dtype = field[1]
+
+            # Handle different field formats:
+            # 1. ('name', ('|S10', {'h5py_encoding': 'ascii'})) - h5py style
+            # 2. ('name', '<i4') - standard NumPy style
+            # 3. ('name', '<i4', (shape,)) - NumPy with shape info
+
+            if isinstance(field_dtype, tuple) and len(field_dtype) > 0:
+                # h5py nested representation with metadata dict
+                field_dtype = field_dtype[0]
+
+            # Check if we have shape information as third element
+            if len(field) > 2 and field[2] is not None:
+                # Include the shape information
+                fields.append((name, field_dtype, field[2]))
+            else:
+                fields.append((name, field_dtype))
+
+        dtype = np.dtype(fields)
+    else:
+        dtype = np.dtype(dtype)
+
     typesize = dtype.itemsize
     if 'cparams' in kwargs:
         kwargs['cparams']['typesize'] = typesize
@@ -2842,3 +2869,26 @@ def schunk_get_slice_nchunks(schunk: SChunk, key):
         res[i] = chunks_idx[i]
     free(chunks_idx)
     return res
+
+
+def concatenate(arr1: NDArray, arr2: NDArray, axis: int, **kwargs):
+    """
+    Concatenate two NDArray objects along a specified axis.
+    """
+    cdef c_bool copy = kwargs.pop("copy", True)
+    cdef b2nd_context_t *ctx = create_b2nd_context(arr1.shape, arr1.chunks, arr1.blocks, arr1.dtype, kwargs)
+    if ctx == NULL:
+        raise RuntimeError("Error while creating the context for concatenation")
+
+    cdef b2nd_array_t *array
+    _check_rc(b2nd_concatenate(ctx, arr1.array, arr2.array, axis, copy, &array),
+              "Error while concatenating the arrays")
+    _check_rc(b2nd_free_ctx(ctx), "Error while freeing the context")
+
+    if copy:
+        # We have copied the concatenated data into a new array
+        return blosc2.NDArray(_schunk=PyCapsule_New(array.sc, <char *> "blosc2_schunk*", NULL),
+                              _array=PyCapsule_New(array, <char *> "b2nd_array_t*", NULL))
+    else:
+        # Return the first array, which now contains the concatenated data
+        return arr1

blosc2-3.4.0/src/blosc2/exceptions.py

@@ -0,0 +1,7 @@
+class MissingOperands(ValueError):
+    def __init__(self, expr, missing_ops):
+        self.expr = expr
+        self.missing_ops = missing_ops
+
+        message = f'Lazy expression "{expr}" with missing operands: {missing_ops}'
+        super().__init__(message)