blosc2 3.3.4__tar.gz → 3.5.0__tar.gz

{blosc2-3.3.4 → blosc2-3.5.0}/.pre-commit-config.yaml

@@ -16,7 +16,7 @@ repos:
       - id: trailing-whitespace
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.11.8
+    rev: v0.11.12
     hooks:
       - id: ruff
         args: ["--fix", "--show-fixes"]

{blosc2-3.3.4 → blosc2-3.5.0}/ANNOUNCE.rst

@@ -1,11 +1,10 @@
-Announcing Python-Blosc2 3.3.4
+Announcing Python-Blosc2 3.5.0
 ==============================
 
-This is a bugfix release, with some minor optimizations. We further improved the
-correct chaining of *string* lazy expressions (to allow operands with more
-diverse data types). In addition, both indexing and where expressions are now
-supported within *string* lazy expressions. Finally, casting rules have
-been improved to be more consistent with NumPy.
+This release adds significant new functionality in the form of ``stack()``.
+It allows you to stack multiple Blosc2 NDarrays along a new axis, similar
+to NumPy's ``np.stack()``. This is particularly useful for creating
+higher-dimensional arrays from lower-dimensional ones.
 
 You can think of Python-Blosc2 3.x as an extension of NumPy/numexpr that:
 

{blosc2-3.3.4 → blosc2-3.5.0}/CMakeLists.txt

@@ -50,7 +50,7 @@ else()
     include(FetchContent)
     FetchContent_Declare(blosc2
         GIT_REPOSITORY https://github.com/Blosc/c-blosc2
-        GIT_TAG 4ef3c7440a85632e6c8b6c5d2a9e651e45569fc1  # v2.17.1 + mmap fix
+        GIT_TAG abb9cd973c2d2d2d068e13934641dcbdedb06752  # v2.19.0 (expand_dims added)
     )
     FetchContent_MakeAvailable(blosc2)
     include_directories("${blosc2_SOURCE_DIR}/include")

{blosc2-3.3.4 → blosc2-3.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: blosc2
-Version: 3.3.4
+Version: 3.5.0
 Summary: A fast & compressed ndarray library with a flexible compute engine.
 Author-Email: Blosc Development Team <blosc@blosc.org>
 Maintainer-Email: Blosc Development Team <blosc@blosc.org>

{blosc2-3.3.4 → blosc2-3.5.0}/README_DEVELOPERS.md

@@ -22,6 +22,20 @@ You are done!
     pip install .   # add -e for editable mode
 ```
 
+There are situations where you may want to build the C-Blosc2 library separately, for example, when debugging issues in the C library. In that case, let's assume you have the C-Blosc2 library installed in `/usr/local`:
+
+```bash
+CMAKE_PREFIX_PATH=/usr/local USE_SYSTEM_BLOSC2=1 pip install -e .
+```
+
+and then, you can run the tests with:
+
+```bash
+LD_LIBRARY_PATH=/usr/local/lib pytest
+```
+
+[replace `LD_LIBRARY_PATH` with the appropriate environment variable for your system, such as `DYLD_LIBRARY_PATH` on macOS or `PATH` on Windows, if necessary].
+
 That's it! You can now proceed to the testing section.
 
 ## Testing

{blosc2-3.3.4 → blosc2-3.5.0}/RELEASE_NOTES.md

@@ -1,8 +1,27 @@
 # Release notes
 
-## Changes from 3.3.4 to 3.3.5
+## Changes from 3.4.0 to 3.5.0
 
-XXX version-specific blurb XXX
+* New `blosc2.stack()` function for stacking multiple arrays along a new axis.
+  Useful for creating multi-dimensional arrays from multiple 1D arrays.
+  See PR #427. Thanks to [@lshaw8317](Luke Shaw) for the implementation!
+
+* New `blosc2.expand_dims()` function for expanding the dimensions of an array.
+  This is useful for adding a new axis to an array, similar to NumPy's `np.expand_dims()`.
+  See PR #427. Thanks to [@lshaw8317](Luke Shaw) for the implementation!
+
+## Changes from 3.3.4 to 3.4.0
+
+* Added C-level ``concatenate`` function in response to community request. When possible, uses an optimised path which avoids decompression and recompression, giving a significant performance boost. See PR #423.
+
+* Slicing has been added to string-based lazyexprs, so that one may use
+  expressions like `expr[1:3] +1` to compute a slice of the expression. This is useful
+  for getting a sub-expression of a larger expression, and it works with both
+  string-based and lazy expressions. See PR #417.
+
+* Relatedly, the behaviour of the `slice` parameter in the `compute()` method of `LazyExpr` has been made more consistent and is now better documented, so that results are as expected. See PR #419.
+
+* UDF support for pandas has been added to allow for the use of ``blosc2.jit``. See PR #418. Thanks to [@datapythonista](https://github.com/datapythonista) for the implementation!
 
 ## Changes from 3.3.3 to 3.3.4
 

{blosc2-3.3.4 → blosc2-3.5.0}/RELEASING.rst

@@ -110,7 +110,8 @@ Checking packaging
 
     [upload e.g.:] blosc2-3.2.0-cp312-cp312-pyodide_2024_0_wasm32.whl
 
-
+The wheels may be downloaded by going to "Actions->Python wheels for WASM" and selecting the completed workflow run for the version you are releasing.
+  Then, go to the "Artifacts" dropdown and download the WASM wheel file.
 Announcing
 ----------
 

{blosc2-3.3.4 → blosc2-3.5.0}/doc/python-blosc2.rst

@@ -6,7 +6,7 @@
 
     <p style="text-align: center; color: black; background-color: rgba(230, 169, 9, 0.65);">
         <a href="https://github.com/Blosc/python-blosc2/blob/main/RELEASE_NOTES.md"
-           style="font-size: 1.5em;">Version 3.3.3 released on 2025-05-14!</a>
+           style="font-size: 1.5em;">Version 3.4.0 released on 2025-06-13!</a>
         <span style="display: inline-block; width: 20px;"></span>
         <span style="font-family: monospace;">pip install blosc2 -U</span>
     </p>

{blosc2-3.3.4 → blosc2-3.5.0}/doc/reference/ndarray.rst

@@ -72,7 +72,9 @@ Constructors
 
     asarray
     copy
+    concatenate
     empty
+    expand_dims
     frombuffer
     fromiter
     nans
@@ -85,3 +87,4 @@ Constructors
     linspace
     eye
     reshape
+    stack

blosc2-3.5.0/examples/blosc2_hdf5_compression.py

@@ -0,0 +1,95 @@
+#######################################################################
+# Copyright (c) 2019-present, Blosc Development Team <blosc@blosc.org>
+# All rights reserved.
+#
+# This source code is licensed under a BSD-style license (found in the
+# LICENSE file in the root directory of this source tree)
+#######################################################################
+
+# This shows how to convert a generic .h5 file to a custom blosc2-compressed .h5 file
+# The blosc2 plugin in hdf5plugin doesn't support custom block shapes, and so one
+# has to go a different route for more bespoke compression
+
+import os
+
+import h5py
+import hdf5plugin
+
+import blosc2
+
+clevel = 5  # compression level, e.g., 0-9, where 0 is no compression and 9 is maximum compression
+fname_in = "kevlar.h5"  # input file with the kevlar dataset
+fname_out = "kevlar-blosc2.h5"
+if not os.path.exists(fname_in):
+    raise FileNotFoundError(
+        f"Input file {fname_in} does not exist\n"
+        "Please download it from the kevlar repository at:"
+        " http://www.silx.org/pub/pyFAI/pyFAI_UM_2020/data_ID13/kevlar.h5"
+    )
+
+# Example 1
+# hdf5plugin supports limited blosc2 compression with certain codecs
+cname = "zstd"
+with h5py.File(fname_in, "r") as fr:
+    dset = fr["/entry/data/data"][:]
+    with h5py.File(fname_out, "w") as fw:
+        g = fw.create_group("/data")
+        b2comp = hdf5plugin.Blosc2(cname=cname, clevel=clevel, filters=hdf5plugin.Blosc2.BITSHUFFLE)
+        dset_out = g.create_dataset(
+            f"cname-{cname}",
+            data=dset,
+            dtype=dset.dtype,
+            chunks=(1,) + dset.shape[1:],  # chunk size of 1 frame
+            **b2comp,
+        )
+    print("Successfully compressed file with hdf5plugin")
+
+# Example 2
+# For other codecs (e.g grok) or for more custom compression such as with user-defined block shapes, one
+# has to use a more involved route
+blocks = (50, 80, 80)
+chunks = (200, 240, 240)
+cparams = {
+    "codec": blosc2.Codec.LZ4,
+    "filters": [blosc2.Filter.BITSHUFFLE],
+    "splitmode": blosc2.SplitMode.NEVER_SPLIT,
+    "clevel": clevel,
+}
+
+if os.path.exists("dset.b2nd"):  # don't reload dset to blosc2 if already done so once
+    b2im = blosc2.open(urlpath="dset.b2nd", mode="r")
+    s, d = b2im.shape, b2im.dtype
+else:
+    with h5py.File(fname_in, "r") as fr:  # load file and process to blosc2 array
+        dset = fr["/entry/data/data"][:]
+        b2im = blosc2.asarray(
+            dset, chunks=chunks, blocks=blocks, cparams=cparams, urlpath="dset.b2nd", mode="w"
+        )
+        d = dset.dtype
+        del dset
+
+# Write to .h5 file #
+with h5py.File(fname_out, "w") as fw:
+    g = fw.create_group("/data")
+    b2comp = hdf5plugin.Blosc2()  # just for identification, no compression algorithm specified
+    dset_out = g.create_dataset(
+        "cname-customlz4",
+        s,
+        d,
+        chunks=chunks,  # chunk size of 1 frame
+        **b2comp,
+    )
+    # Write individual blosc2 chunks directly to hdf5
+    # hdf5 requires a cframe, which is only available via blosc2 schunks (not chunks)
+    for info in b2im.iterchunks_info():
+        ncoords = tuple(n * chunks[i] for i, n in enumerate(info.coords))
+        aux = blosc2.empty(
+            shape=b2im.chunks, chunks=b2im.chunks, blocks=b2im.blocks, dtype=b2im.dtype
+        )  # very cheap memory allocation
+        aux.schunk.insert_chunk(
+            0, b2im.get_chunk(info.nchunk)
+        )  # insert chunk into blosc2 array so we have schunk wrapper (no decompression required)
+        dset_out.id.write_direct_chunk(
+            ncoords, aux.schunk.to_cframe()
+        )  # convert schunk to cframe and write to hdf5
+    print("Successfully compressed file with custom parameters")

blosc2-3.5.0/examples/ndarray/lazyexpr_where_indexing.py

@@ -0,0 +1,42 @@
+# Imports
+
+import numpy as np
+
+import blosc2
+
+N = 1000
+it = ((-x + 1, x - 2, 0.1 * x) for x in range(N))
+sa = blosc2.fromiter(
+    it, dtype=[("A", "i4"), ("B", "f4"), ("C", "f8")], shape=(N,), urlpath="sa-1M.b2nd", mode="w"
+)
+expr = sa["(A < B)"]
+A = sa["A"][:]
+B = sa["B"][:]
+C = sa["C"][:]
+temp = sa[:]
+indices = A < B
+idx = np.argmax(indices)
+
+# One might think that expr[:10] gives the first 10 elements of the evaluated expression, but this is not the case.
+# It actually computes the expression on the first 10 elements of the operands; since for some elements the condition
+# is False, the result will be shorter than 10 elements.
+# Returns less than 10 elements in general
+sliced = expr.compute(slice(0, 10))
+gotitem = expr[:10]
+np.testing.assert_array_equal(sliced[:], gotitem)
+np.testing.assert_array_equal(gotitem, temp[:10][indices[:10]])  # Equivalent syntax
+# Actually this makes sense since one can understand this as a request to compute on a portion of operands.
+# If one desires a portion of the result, one should compute the whole expression and then slice it.
+
+# Get first element for which condition is true
+sliced = expr.compute(idx)
+gotitem = expr[idx]
+# Arrays of one element
+np.testing.assert_array_equal(sliced[()], gotitem)
+np.testing.assert_array_equal(gotitem, temp[idx])
+
+# Should return void arrays here.
+sliced = expr.compute(0)
+gotitem = expr[0]
+np.testing.assert_array_equal(sliced[()], gotitem)
+np.testing.assert_array_equal(gotitem, temp[0])

{blosc2-3.3.4 → blosc2-3.5.0}/pyproject.toml

@@ -41,7 +41,7 @@ dependencies = [
     "py-cpuinfo; platform_machine != 'wasm32'",
     "requests",
 ]
-version = "3.3.4"
+version = "3.5.0"
 
 
 [project.optional-dependencies]

{blosc2-3.3.4 → blosc2-3.5.0}/src/blosc2/__init__.py

@@ -238,6 +238,8 @@ from .ndarray import (
     sort,
     reshape,
     copy,
+    concatenate,
+    expand_dims,
     empty,
     frombuffer,
     fromiter,
@@ -252,6 +254,7 @@ from .ndarray import (
     permute_dims,
     transpose,
     matrix_transpose,
+    stack,
 )
 
 from .c2array import c2context, C2Array, URLPath

{blosc2-3.3.4 → blosc2-3.5.0}/src/blosc2/blosc2_ext.pyx

@@ -507,6 +507,9 @@ cdef extern from "b2nd.h":
     int b2nd_squeeze_index(b2nd_array_t *array, const c_bool *index)
     int b2nd_resize(b2nd_array_t *array, const int64_t *new_shape, const int64_t *start)
     int b2nd_copy(b2nd_context_t *ctx, b2nd_array_t *src, b2nd_array_t **array)
+    int b2nd_concatenate(b2nd_context_t *ctx, b2nd_array_t *src1, b2nd_array_t *src2,
+                         int8_t axis, c_bool copy, b2nd_array_t **array)
+    int b2nd_expand_dims(const b2nd_array_t *array, b2nd_array_t ** view, const int8_t axis)
     int b2nd_from_schunk(blosc2_schunk *schunk, b2nd_array_t **array)
 
     void blosc2_unidim_to_multidim(uint8_t ndim, int64_t *shape, int64_t i, int64_t *index)
@@ -2867,3 +2870,37 @@ def schunk_get_slice_nchunks(schunk: SChunk, key):
         res[i] = chunks_idx[i]
     free(chunks_idx)
     return res
+
+
+def concatenate(arr1: NDArray, arr2: NDArray, axis: int, **kwargs):
+    """
+    Concatenate two NDArray objects along a specified axis.
+    """
+    cdef c_bool copy = kwargs.pop("copy", True)
+    cdef b2nd_context_t *ctx = create_b2nd_context(arr1.shape, arr1.chunks, arr1.blocks, arr1.dtype, kwargs)
+    if ctx == NULL:
+        raise RuntimeError("Error while creating the context for concatenation")
+
+    cdef b2nd_array_t *array
+    _check_rc(b2nd_concatenate(ctx, arr1.array, arr2.array, axis, copy, &array),
+              "Error while concatenating the arrays")
+    _check_rc(b2nd_free_ctx(ctx), "Error while freeing the context")
+
+    if copy:
+        # We have copied the concatenated data into a new array
+        return blosc2.NDArray(_schunk=PyCapsule_New(array.sc, <char *> "blosc2_schunk*", NULL),
+                              _array=PyCapsule_New(array, <char *> "b2nd_array_t*", NULL))
+    else:
+        # Return the first array, which now contains the concatenated data
+        return arr1
+
+def expand_dims(arr1: NDArray, axis: int):
+    """
+    Add new dummy axis to NDArray object at specified dimension.
+    """
+    cdef b2nd_array_t *view
+    _check_rc(b2nd_expand_dims(arr1.array, &view, axis),
+              "Error while concatenating the arrays")
+
+    return blosc2.NDArray(_schunk=PyCapsule_New(view.sc, <char *> "blosc2_schunk*", NULL),
+                              _array=PyCapsule_New(view, <char *> "b2nd_array_t*", NULL))

{blosc2-3.3.4 → blosc2-3.5.0}/src/blosc2/core.py

@@ -1092,9 +1092,7 @@ def detect_number_of_cores() -> int:
 # Dictionaries for the maps between compressor names and libs
 codecs = compressor_list(plugins=True)
 # Map for compression libraries and versions
-clib_versions = {}
-for codec in compressor_list(plugins=False):
-    clib_versions[codec.name] = clib_info(codec)[1].decode("utf-8")
+clib_versions = {codec.name: clib_info(codec)[1].decode("utf-8") for codec in compressor_list(plugins=False)}
 
 
 def os_release_pretty_name():
@@ -1422,7 +1420,6 @@ def compute_partition(nitems, maxshape, minpart=None):
         if rsize <= max_items:
             # rsize = rsize if size % rsize == 0 else nearest_divisor(size, rsize)
             rsize = rsize if size % rsize == 0 else blosc2_ext.nearest_divisor(size, rsize)
-            partition[-(i + 1)] = rsize
         else:
             rsize = max(max_items, minsize)
             # new_rsize = rsize if size % rsize == 0 else nearest_divisor(size, rsize, strict=True)
@@ -1430,7 +1427,7 @@ def compute_partition(nitems, maxshape, minpart=None):
             # If the new rsize is not too far from the original rsize, use it
             if rsize // 2 < new_rsize < rsize * 2:
                 rsize = new_rsize
-            partition[-(i + 1)] = rsize
+        partition[-(i + 1)] = rsize
         max_items //= rsize
 
     return partition

blosc2-3.5.0/src/blosc2/exceptions.py

@@ -0,0 +1,7 @@
+class MissingOperands(ValueError):
+    def __init__(self, expr, missing_ops):
+        self.expr = expr
+        self.missing_ops = missing_ops
+
+        message = f'Lazy expression "{expr}" with missing operands: {missing_ops}'
+        super().__init__(message)

{blosc2-3.3.4 → blosc2-3.5.0}/src/blosc2/helpers.py

@@ -32,7 +32,7 @@ def _inherit_doc_parameter(parent_func, parameter, replacements=None):
                     # Next parameter starts, stop copying lines
                     break
                 matching_lines.append(line)
-        assert len(matching_lines) > 0, (
+        assert matching_lines, (
             f"Could not extract the parameter {parameter} from the docstring of {parent_func.__name__}"
         )