blosc2 3.3.4__tar.gz → 3.4.0__tar.gz

{blosc2-3.3.4 → blosc2-3.4.0}/.pre-commit-config.yaml

@@ -16,7 +16,7 @@ repos:
       - id: trailing-whitespace
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.11.8
+    rev: v0.11.12
     hooks:
       - id: ruff
         args: ["--fix", "--show-fixes"]

{blosc2-3.3.4 → blosc2-3.4.0}/ANNOUNCE.rst

@@ -1,11 +1,7 @@
-Announcing Python-Blosc2 3.3.4
+Announcing Python-Blosc2 3.4.0
 ==============================
 
-This is a bugfix release, with some minor optimizations. We further improved the
-correct chaining of *string* lazy expressions (to allow operands with more
-diverse data types). In addition, both indexing and where expressions are now
-supported within *string* lazy expressions. Finally, casting rules have
-been improved to be more consistent with NumPy.
+This release adds significant new functionality in the form of ``concatenate``. We support general concatenation of ndarrays, and offer an optimised path with significant speedups for the case of concatenating arrays with compatible chunk and blockshapes. In addition, there are bug fixes and more functionality for slicing of lazyexprs, and the possibility to jit compile user-defined functions which operate on pandas objects using the blosc2 engine.
 
 You can think of Python-Blosc2 3.x as an extension of NumPy/numexpr that:
 

{blosc2-3.3.4 → blosc2-3.4.0}/CMakeLists.txt

@@ -50,7 +50,7 @@ else()
     include(FetchContent)
     FetchContent_Declare(blosc2
         GIT_REPOSITORY https://github.com/Blosc/c-blosc2
-        GIT_TAG 4ef3c7440a85632e6c8b6c5d2a9e651e45569fc1  # v2.17.1 + mmap fix
+        GIT_TAG 1c2f8bb0c914c43e23b751fbcf6642cd7aec09db  # v2.18.0 (concatenate added)
     )
     FetchContent_MakeAvailable(blosc2)
     include_directories("${blosc2_SOURCE_DIR}/include")

{blosc2-3.3.4 → blosc2-3.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: blosc2
-Version: 3.3.4
+Version: 3.4.0
 Summary: A fast & compressed ndarray library with a flexible compute engine.
 Author-Email: Blosc Development Team <blosc@blosc.org>
 Maintainer-Email: Blosc Development Team <blosc@blosc.org>

{blosc2-3.3.4 → blosc2-3.4.0}/README_DEVELOPERS.md

@@ -22,6 +22,20 @@ You are done!
     pip install .   # add -e for editable mode
 ```
 
+There are situations where you may want to build the C-Blosc2 library separately, for example, when debugging issues in the C library. In that case, let's assume you have the C-Blosc2 library installed in `/usr/local`:
+
+```bash
+CMAKE_PREFIX_PATH=/usr/local USE_SYSTEM_BLOSC2=1 pip install -e .
+```
+
+and then, you can run the tests with:
+
+```bash
+LD_LIBRARY_PATH=/usr/local/lib pytest
+```
+
+[replace `LD_LIBRARY_PATH` with the appropriate environment variable for your system, such as `DYLD_LIBRARY_PATH` on macOS or `PATH` on Windows, if necessary].
+
 That's it! You can now proceed to the testing section.
 
 ## Testing

{blosc2-3.3.4 → blosc2-3.4.0}/RELEASE_NOTES.md

@@ -1,9 +1,21 @@
 # Release notes
-
-## Changes from 3.3.4 to 3.3.5
+## Changes from 3.4.0 to 3.4.1
 
 XXX version-specific blurb XXX
 
+## Changes from 3.3.4 to 3.4.0
+
+* Added C-level ``concatenate`` function in response to community request. When possible, uses an optimised path which avoids decompression and recompression, giving a significant performance boost. See PR #423.
+
+* Slicing has been added to string-based lazyexprs, so that one may use
+  expressions like `expr[1:3] +1` to compute a slice of the expression. This is useful
+  for getting a sub-expression of a larger expression, and it works with both
+  string-based and lazy expressions. See PR #417.
+
+* Relatedly, the behaviour of the `slice` parameter in the `compute()` method of `LazyExpr` has been made more consistent and is now better documented, so that results are as expected. See PR #419.
+
+* UDF support for pandas has been added to allow for the use of ``blosc2.jit``. See PR #418. Thanks to [@datapythonista](https://github.com/datapythonista) for the implementation!
+
 ## Changes from 3.3.3 to 3.3.4
 
 * Expand possibilities for chaining string-based lazy expressions to incorporate

{blosc2-3.3.4 → blosc2-3.4.0}/doc/python-blosc2.rst

@@ -6,7 +6,7 @@
 
     <p style="text-align: center; color: black; background-color: rgba(230, 169, 9, 0.65);">
         <a href="https://github.com/Blosc/python-blosc2/blob/main/RELEASE_NOTES.md"
-           style="font-size: 1.5em;">Version 3.3.3 released on 2025-05-14!</a>
+           style="font-size: 1.5em;">Version 3.3.4 released on 2025-05-22!</a>
         <span style="display: inline-block; width: 20px;"></span>
         <span style="font-family: monospace;">pip install blosc2 -U</span>
     </p>

{blosc2-3.3.4 → blosc2-3.4.0}/doc/reference/ndarray.rst

@@ -72,6 +72,7 @@ Constructors
 
     asarray
     copy
+    concatenate
     empty
     frombuffer
     fromiter

blosc2-3.4.0/examples/blosc2_hdf5_compression.py

@@ -0,0 +1,95 @@
+#######################################################################
+# Copyright (c) 2019-present, Blosc Development Team <blosc@blosc.org>
+# All rights reserved.
+#
+# This source code is licensed under a BSD-style license (found in the
+# LICENSE file in the root directory of this source tree)
+#######################################################################
+
+# This shows how to convert a generic .h5 file to a custom blosc2-compressed .h5 file
+# The blosc2 plugin in hdf5plugin doesn't support custom block shapes, and so one
+# has to go a different route for more bespoke compression
+
+import os
+
+import h5py
+import hdf5plugin
+
+import blosc2
+
+clevel = 5  # compression level, e.g., 0-9, where 0 is no compression and 9 is maximum compression
+fname_in = "kevlar.h5"  # input file with the kevlar dataset
+fname_out = "kevlar-blosc2.h5"
+if not os.path.exists(fname_in):
+    raise FileNotFoundError(
+        f"Input file {fname_in} does not exist\n"
+        "Please download it from the kevlar repository at:"
+        " http://www.silx.org/pub/pyFAI/pyFAI_UM_2020/data_ID13/kevlar.h5"
+    )
+
+# Example 1
+# hdf5plugin supports limited blosc2 compression with certain codecs
+cname = "zstd"
+with h5py.File(fname_in, "r") as fr:
+    dset = fr["/entry/data/data"][:]
+    with h5py.File(fname_out, "w") as fw:
+        g = fw.create_group("/data")
+        b2comp = hdf5plugin.Blosc2(cname=cname, clevel=clevel, filters=hdf5plugin.Blosc2.BITSHUFFLE)
+        dset_out = g.create_dataset(
+            f"cname-{cname}",
+            data=dset,
+            dtype=dset.dtype,
+            chunks=(1,) + dset.shape[1:],  # chunk size of 1 frame
+            **b2comp,
+        )
+    print("Successfully compressed file with hdf5plugin")
+
+# Example 2
+# For other codecs (e.g grok) or for more custom compression such as with user-defined block shapes, one
+# has to use a more involved route
+blocks = (50, 80, 80)
+chunks = (200, 240, 240)
+cparams = {
+    "codec": blosc2.Codec.LZ4,
+    "filters": [blosc2.Filter.BITSHUFFLE],
+    "splitmode": blosc2.SplitMode.NEVER_SPLIT,
+    "clevel": clevel,
+}
+
+if os.path.exists("dset.b2nd"):  # don't reload dset to blosc2 if already done so once
+    b2im = blosc2.open(urlpath="dset.b2nd", mode="r")
+    s, d = b2im.shape, b2im.dtype
+else:
+    with h5py.File(fname_in, "r") as fr:  # load file and process to blosc2 array
+        dset = fr["/entry/data/data"][:]
+        b2im = blosc2.asarray(
+            dset, chunks=chunks, blocks=blocks, cparams=cparams, urlpath="dset.b2nd", mode="w"
+        )
+        d = dset.dtype
+        del dset
+
+# Write to .h5 file #
+with h5py.File(fname_out, "w") as fw:
+    g = fw.create_group("/data")
+    b2comp = hdf5plugin.Blosc2()  # just for identification, no compression algorithm specified
+    dset_out = g.create_dataset(
+        "cname-customlz4",
+        s,
+        d,
+        chunks=chunks,  # chunk size of 1 frame
+        **b2comp,
+    )
+    # Write individual blosc2 chunks directly to hdf5
+    # hdf5 requires a cframe, which is only available via blosc2 schunks (not chunks)
+    for info in b2im.iterchunks_info():
+        ncoords = tuple(n * chunks[i] for i, n in enumerate(info.coords))
+        aux = blosc2.empty(
+            shape=b2im.chunks, chunks=b2im.chunks, blocks=b2im.blocks, dtype=b2im.dtype
+        )  # very cheap memory allocation
+        aux.schunk.insert_chunk(
+            0, b2im.get_chunk(info.nchunk)
+        )  # insert chunk into blosc2 array so we have schunk wrapper (no decompression required)
+        dset_out.id.write_direct_chunk(
+            ncoords, aux.schunk.to_cframe()
+        )  # convert schunk to cframe and write to hdf5
+    print("Successfully compressed file with custom parameters")

blosc2-3.4.0/examples/ndarray/lazyexpr_where_indexing.py

@@ -0,0 +1,42 @@
+# Imports
+
+import numpy as np
+
+import blosc2
+
+N = 1000
+it = ((-x + 1, x - 2, 0.1 * x) for x in range(N))
+sa = blosc2.fromiter(
+    it, dtype=[("A", "i4"), ("B", "f4"), ("C", "f8")], shape=(N,), urlpath="sa-1M.b2nd", mode="w"
+)
+expr = sa["(A < B)"]
+A = sa["A"][:]
+B = sa["B"][:]
+C = sa["C"][:]
+temp = sa[:]
+indices = A < B
+idx = np.argmax(indices)
+
+# One might think that expr[:10] gives the first 10 elements of the evaluated expression, but this is not the case.
+# It actually computes the expression on the first 10 elements of the operands; since for some elements the condition
+# is False, the result will be shorter than 10 elements.
+# Returns less than 10 elements in general
+sliced = expr.compute(slice(0, 10))
+gotitem = expr[:10]
+np.testing.assert_array_equal(sliced[:], gotitem)
+np.testing.assert_array_equal(gotitem, temp[:10][indices[:10]])  # Equivalent syntax
+# Actually this makes sense since one can understand this as a request to compute on a portion of operands.
+# If one desires a portion of the result, one should compute the whole expression and then slice it.
+
+# Get first element for which condition is true
+sliced = expr.compute(idx)
+gotitem = expr[idx]
+# Arrays of one element
+np.testing.assert_array_equal(sliced[()], gotitem)
+np.testing.assert_array_equal(gotitem, temp[idx])
+
+# Should return void arrays here.
+sliced = expr.compute(0)
+gotitem = expr[0]
+np.testing.assert_array_equal(sliced[()], gotitem)
+np.testing.assert_array_equal(gotitem, temp[0])

{blosc2-3.3.4 → blosc2-3.4.0}/pyproject.toml

@@ -41,7 +41,7 @@ dependencies = [
     "py-cpuinfo; platform_machine != 'wasm32'",
     "requests",
 ]
-version = "3.3.4"
+version = "3.4.0"
 
 
 [project.optional-dependencies]

{blosc2-3.3.4 → blosc2-3.4.0}/src/blosc2/__init__.py

@@ -238,6 +238,7 @@ from .ndarray import (
     sort,
     reshape,
     copy,
+    concatenate,
     empty,
     frombuffer,
     fromiter,

{blosc2-3.3.4 → blosc2-3.4.0}/src/blosc2/blosc2_ext.pyx

@@ -507,6 +507,8 @@ cdef extern from "b2nd.h":
     int b2nd_squeeze_index(b2nd_array_t *array, const c_bool *index)
     int b2nd_resize(b2nd_array_t *array, const int64_t *new_shape, const int64_t *start)
     int b2nd_copy(b2nd_context_t *ctx, b2nd_array_t *src, b2nd_array_t **array)
+    int b2nd_concatenate(b2nd_context_t *ctx, b2nd_array_t *src1, b2nd_array_t *src2,
+                         int8_t axis, c_bool copy, b2nd_array_t **array)
     int b2nd_from_schunk(blosc2_schunk *schunk, b2nd_array_t **array)
 
     void blosc2_unidim_to_multidim(uint8_t ndim, int64_t *shape, int64_t i, int64_t *index)
@@ -2867,3 +2869,26 @@ def schunk_get_slice_nchunks(schunk: SChunk, key):
         res[i] = chunks_idx[i]
     free(chunks_idx)
     return res
+
+
+def concatenate(arr1: NDArray, arr2: NDArray, axis: int, **kwargs):
+    """
+    Concatenate two NDArray objects along a specified axis.
+    """
+    cdef c_bool copy = kwargs.pop("copy", True)
+    cdef b2nd_context_t *ctx = create_b2nd_context(arr1.shape, arr1.chunks, arr1.blocks, arr1.dtype, kwargs)
+    if ctx == NULL:
+        raise RuntimeError("Error while creating the context for concatenation")
+
+    cdef b2nd_array_t *array
+    _check_rc(b2nd_concatenate(ctx, arr1.array, arr2.array, axis, copy, &array),
+              "Error while concatenating the arrays")
+    _check_rc(b2nd_free_ctx(ctx), "Error while freeing the context")
+
+    if copy:
+        # We have copied the concatenated data into a new array
+        return blosc2.NDArray(_schunk=PyCapsule_New(array.sc, <char *> "blosc2_schunk*", NULL),
+                              _array=PyCapsule_New(array, <char *> "b2nd_array_t*", NULL))
+    else:
+        # Return the first array, which now contains the concatenated data
+        return arr1

blosc2-3.4.0/src/blosc2/exceptions.py

@@ -0,0 +1,7 @@
+class MissingOperands(ValueError):
+    def __init__(self, expr, missing_ops):
+        self.expr = expr
+        self.missing_ops = missing_ops
+
+        message = f'Lazy expression "{expr}" with missing operands: {missing_ops}'
+        super().__init__(message)

{blosc2-3.3.4 → blosc2-3.4.0}/src/blosc2/lazyexpr.py

@@ -10,6 +10,7 @@ from __future__ import annotations
 
 import ast
 import asyncio
+import builtins
 import concurrent.futures
 import copy
 import inspect
@@ -28,6 +29,8 @@ from typing import TYPE_CHECKING, Any
 
 from numpy.exceptions import ComplexWarning
 
+from . import exceptions
+
 if TYPE_CHECKING:
     from collections.abc import Callable, Sequence
 
@@ -277,8 +280,9 @@ class LazyArray(ABC):
         Parameters
         ----------
         item: slice, list of slices, optional
-            If not None, only the chunks that intersect with the slices
-            in items will be evaluated.
+            If provided, item is used to slice the operands *prior* to computation; not to retrieve specified slices of
+            the evaluated result. This difference between slicing operands and slicing the final expression
+            is important when reductions or a where clause are used in the expression.
 
         kwargs: Any, optional
             Keyword arguments that are supported by the :func:`empty` constructor.
@@ -327,7 +331,9 @@ class LazyArray(ABC):
         Parameters
         ----------
         item: int, slice or sequence of slices
-            The slice(s) to be retrieved. Note that step parameter is not yet honored.
+            If provided, item is used to slice the operands *prior* to computation; not to retrieve specified slices of
+            the evaluated result. This difference between slicing operands and slicing the final expression
+            is important when reductions or a where clause are used in the expression.
 
         Returns
         -------
@@ -465,6 +471,10 @@ class LazyArray(ABC):
             "version": 3,
         }
 
+    # Provide a way to serialize the LazyArray
+    def to_cframe(self):
+        return self.compute().to_cframe()
+
 
 def convert_inputs(inputs):
     if not inputs or len(inputs) == 0:
@@ -1373,7 +1383,8 @@ def slices_eval(  # noqa: C901
             for i, (c, s) in enumerate(zip(coords, chunks, strict=True))
         )
         # Check whether current slice_ intersects with _slice
-        if _slice is not None and _slice != ():
+        checker = _slice.item() if hasattr(_slice, "item") else _slice  # can't use != when _slice is np.int
+        if checker is not None and checker != ():
             # Ensure that _slice is of type slice
             key = ndindex.ndindex(_slice).expand(shape).raw
             _slice = tuple(k if isinstance(k, slice) else slice(k, k + 1, None) for k in key)
@@ -1503,19 +1514,7 @@ def slices_eval(  # noqa: C901
         else:
             raise ValueError("The where condition must be a tuple with one or two elements")
 
-    if orig_slice is not None:
-        if isinstance(out, np.ndarray):
-            out = out[orig_slice]
-            if _order is not None:
-                indices_ = indices_[orig_slice]
-        elif isinstance(out, blosc2.NDArray):
-            # It *seems* better to choose an automatic chunks and blocks for the output array
-            # out = out.slice(orig_slice, chunks=out.chunks, blocks=out.blocks)
-            out = out.slice(orig_slice)
-        else:
-            raise ValueError("The output array is not a NumPy array or a NDArray")
-
-    if where is not None and len(where) < 2:
+    if where is not None and len(where) < 2:  # Don't need to take orig_slice since filled up from 0 index
         if _order is not None:
             # argsort the result following _order
             new_order = np.argsort(out[:lenout])
@@ -1527,6 +1526,19 @@ def slices_eval(  # noqa: C901
         else:
             out.resize((lenout,))
 
+    else:  # Need to take orig_slice since filled up array according to slice_ for each chunk
+        if orig_slice is not None:
+            if isinstance(out, np.ndarray):
+                out = out[orig_slice]
+                if _order is not None:
+                    indices_ = indices_[orig_slice]
+            elif isinstance(out, blosc2.NDArray):
+                # It *seems* better to choose an automatic chunks and blocks for the output array
+                # out = out.slice(orig_slice, chunks=out.chunks, blocks=out.blocks)
+                out = out.slice(orig_slice)
+            else:
+                raise ValueError("The output array is not a NumPy array or a NDArray")
+
     return out
 
 
@@ -1822,7 +1834,8 @@ def chunked_eval(  # noqa: C901
     operands: dict
         A dictionary containing the operands for the expression.
     item: int, slice or sequence of slices, optional
-        The slice(s) to be retrieved. Note that step parameter is not honored yet.
+        The slice(s) of the operands to be used in computation. Note that step parameter is not honored yet.
+        Item is used to slice the operands PRIOR to computation.
     kwargs: Any, optional
         Additional keyword arguments supported by the :func:`empty` constructor.  In addition,
         the following keyword arguments are supported:
@@ -2597,7 +2610,7 @@ class LazyExpr(LazyArray):
             _globals = get_expr_globals(self.expression)
             lazy_expr = eval(self.expression, _globals, self.operands)
             if not isinstance(lazy_expr, blosc2.LazyExpr):
-                key, _ = process_key(item, self.shape)
+                key, mask = process_key(item, self.shape)
                 # An immediate evaluation happened (e.g. all operands are numpy arrays)
                 if hasattr(self, "_where_args"):
                     # We need to apply the where() operation
@@ -2614,7 +2627,11 @@ class LazyExpr(LazyArray):
                     # This is not exactly optimized, but it works for now
                     self._output[:] = lazy_expr[key]
                     return self._output
-                return lazy_expr[key]
+                arr = lazy_expr[key]
+                if builtins.sum(mask) > 0:
+                    # Correct shape to adjust to NumPy convention
+                    arr.shape = tuple(arr.shape[i] for i in range(len(mask)) if not mask[i])
+                return arr
 
             return chunked_eval(lazy_expr.expression, lazy_expr.operands, item, **kwargs)
 
@@ -2714,6 +2731,9 @@ class LazyExpr(LazyArray):
         kwargs = {"_getitem": True}
         return self.compute(item, **kwargs)
 
+    def slice(self, item):
+        return self.compute(item)  # should do a slice since _getitem = False
+
     def __str__(self):
         return f"{self.expression}"
 
@@ -3276,14 +3296,16 @@ def _open_lazyarray(array):
     operands = lazyarray["operands"]
     parent_path = Path(array.schunk.urlpath).parent
     operands_dict = {}
+    missing_ops = {}
     for key, value in operands.items():
         if isinstance(value, str):
             value = parent_path / value
             try:
                 op = blosc2.open(value)
             except FileNotFoundError:
-                op = None
-            operands_dict[key] = op
+                missing_ops[key] = value
+            else:
+                operands_dict[key] = op
         elif isinstance(value, dict):
             # C2Array
             operands_dict[key] = blosc2.C2Array(
@@ -3294,16 +3316,17 @@ def _open_lazyarray(array):
             raise TypeError("Error when retrieving the operands")
 
     expr = lazyarray["expression"]
-    try:
-        new_expr = LazyExpr._new_expr(expr, operands_dict, guess=True, out=None, where=None)
-        # Make the array info available for the user (only available when opened from disk)
-        new_expr.array = array
-        # We want to expose schunk too, so that .info() can be used on the LazyArray
-        new_expr.schunk = array.schunk
-    except RuntimeError:
-        # Something unexpected happened. Avoid guessing and return something that
-        # can be introspected.
-        new_expr = LazyExpr._new_expr(expr, operands_dict, guess=False, out=None, where=None)
+    if missing_ops:
+        exc = exceptions.MissingOperands(expr, missing_ops)
+        exc.expr = expr
+        exc.missing_ops = missing_ops
+        raise exc
+
+    new_expr = LazyExpr._new_expr(expr, operands_dict, guess=True, out=None, where=None)
+    # Make the array info available for the user (only available when opened from disk)
+    new_expr.array = array
+    # We want to expose schunk too, so that .info() can be used on the LazyArray
+    new_expr.schunk = array.schunk
     return new_expr
 
 

{blosc2-3.3.4 → blosc2-3.4.0}/src/blosc2/ndarray.py

@@ -3523,6 +3523,71 @@ def copy(array: NDArray, dtype: np.dtype | str = None, **kwargs: Any) -> NDArray
     return array.copy(dtype, **kwargs)
 
 
+def concatenate(arrays: list[NDArray], /, axis=0, **kwargs: Any) -> NDArray:  # noqa: C901
+    """Concatenate two arrays along a specified axis.
+
+    Parameters
+    ----------
+    arrays: list of :ref:`NDArray`
+        A list containing two or more NDArray instances to be concatenated.
+    axis: int, optional
+        The axis along which the arrays will be concatenated. Default is 0.
+
+    Other Parameters
+    ----------------
+    kwargs: dict, optional
+        Keyword arguments that are supported by the :func:`empty` constructor.
+
+    Returns
+    -------
+    out: :ref:`NDArray`
+        A new NDArray containing the concatenated data.
+
+    Examples
+    --------
+    >>> import blosc2
+    >>> import numpy as np
+    >>> arr1 = blosc2.arange(0, 5, dtype=np.int32)
+    >>> arr2 = blosc2.arange(5, 10, dtype=np.int32)
+    >>> result = blosc2.concatenate([arr1, arr2])
+    >>> print(result[:])
+    [0 1 2 3 4 5 6 7 8 9]
+    """
+    if len(arrays) < 2:
+        raise ValueError("At least two arrays are required for concatenation.")
+    arr1 = arrays[0]
+    if not isinstance(arr1, blosc2.NDArray):
+        raise TypeError("All inputs must be instances of blosc2.NDArray")
+    # Do a first pass for checking array compatibility
+    if axis < 0:
+        axis += arr1.ndim
+    if axis >= arr1.ndim:
+        raise ValueError(f"Axis {axis} is out of bounds for array of dimension {arr1.ndim}.")
+    for arr2 in arrays[1:]:
+        if not isinstance(arr2, blosc2.NDArray):
+            raise TypeError("All inputs must be instances of blosc2.NDArray")
+        if arr1.ndim != arr2.ndim:
+            raise ValueError("Both arrays must have the same number of dimensions for concatenation.")
+        if arr1.dtype != arr2.dtype:
+            raise ValueError("Both arrays must have the same dtype for concatenation.")
+        # Check that the shapes match, except for the concatenation axis
+        if arr1.shape[:axis] != arr2.shape[:axis] or arr1.shape[axis + 1 :] != arr2.shape[axis + 1 :]:
+            raise ValueError(
+                f"Shapes of the arrays do not match along the concatenation axis {axis}: "
+                f"{arr1.shape} vs {arr2.shape}"
+            )
+
+    kwargs = _check_ndarray_kwargs(**kwargs)
+    # Proceed with the actual concatenation
+    copy = True
+    for arr2 in arrays[1:]:
+        arr1 = blosc2_ext.concatenate(arr1, arr2, axis, copy=copy, **kwargs)
+        # arr1 is now the result of the concatenation, so we can now just enlarge it
+        copy = False
+
+    return arr1
+
+
 def save(array: NDArray, urlpath: str, contiguous=True, **kwargs: Any) -> None:
     """Save an array to a file.
 
@@ -3545,9 +3610,9 @@ def save(array: NDArray, urlpath: str, contiguous=True, **kwargs: Any) -> None:
     >>> import blosc2
     >>> import numpy as np
     >>> # Create an array
-    >>> array = np.arange(0, 100, dtype=np.int64).reshape(10, 10)
+    >>> array = blosc2.arange(0, 100, dtype=np.int64, shape=(10, 10))
     >>> # Save the array to a file
-    >>> blosc2.save(array, "array.b2")
+    >>> blosc2.save(array, "array.b2", mode="w")
     """
     array.save(urlpath, contiguous, **kwargs)
 

{blosc2-3.3.4 → blosc2-3.4.0}/src/blosc2/proxy.py

@@ -691,3 +691,58 @@ def jit(func=None, *, out=None, **kwargs):  # noqa: C901
         return decorator
     else:
         return decorator(func)
+
+
+class PandasUdfEngine:
+    @staticmethod
+    def _ensure_numpy_data(data):
+        if not isinstance(data, np.ndarray):
+            try:
+                data = data.values
+            except AttributeError as err:
+                raise ValueError(
+                    "blosc2.jit received an object of type {data.__name__}, which is not supported. "
+                    "Try casting your Series or DataFrame to a NumPy dtype."
+                ) from err
+        return data
+
+    @classmethod
+    def map(cls, data, func, args, kwargs, decorator, skip_na):
+        """
+        JIT a NumPy array element-wise. In the case of Blosc2, functions are
+        expected to be vectorized NumPy operations, so the function is called
+        with the NumPy array as the function parameter, instead of calling the
+        function once for each element.
+        """
+        raise NotImplementedError("The Blosc2 engine does not support map. Use apply instead.")
+
+    @classmethod
+    def apply(cls, data, func, args, kwargs, decorator, axis):
+        """
+        JIT a NumPy array by column or row. In the case of Blosc2, functions are
+        expected to be vectorized NumPy operations, so the function is called
+        with the NumPy array as the function parameter, instead of calling the
+        function once for each column or row.
+        """
+        data = cls._ensure_numpy_data(data)
+        func = decorator(func)
+        if data.ndim == 1 or axis is None:
+            # pandas Series.apply or pipe
+            return func(data, *args, **kwargs)
+        elif axis in (0, "index"):
+            # pandas apply(axis=0) column-wise
+            result = []
+            for row_idx in range(data.shape[1]):
+                result.append(func(data[:, row_idx], *args, **kwargs))
+            return np.vstack(result).transpose()
+        elif axis in (1, "columns"):
+            # pandas apply(axis=1) row-wise
+            result = []
+            for col_idx in range(data.shape[0]):
+                result.append(func(data[col_idx, :], *args, **kwargs))
+            return np.vstack(result)
+        else:
+            raise NotImplementedError(f"Unknown axis '{axis}'. Use one of 0, 1 or None.")
+
+
+jit.__pandas_udf__ = PandasUdfEngine

blosc2-3.4.0/src/blosc2/version.py

@@ -0,0 +1 @@
+__version__ = "3.4.0"