blosc2 2.6.2__tar.gz → 2.7.1__tar.gz

{blosc2-2.6.2 → blosc2-2.7.1}/ANNOUNCE.rst

@@ -1,10 +1,9 @@
-Announcing Python-Blosc2 2.6.2
+Announcing Python-Blosc2 2.7.1
 ==============================
 
-Updated to latest C-Blosc2 2.14.3. This was necessary to fix
-several CVEs in the C-Blosc2 library.  Also, the new version adds
-protection when platforms have just one CPU. This caused the
-internal number of threads to be 0, producing a division by zero.
+Updated to latest C-Blosc2 2.15.0. Also, `LazyExpr.evaluate()`
+is marked as deprecated (LazyExpr.eval() is replacing it) and some other fixes
+have been carried out.
 
 For more info, you can have a look at the release notes in:
 

{blosc2-2.6.2/blosc2.egg-info → blosc2-2.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: blosc2
-Version: 2.6.2
+Version: 2.7.1
 Summary: Python wrapper for the C-Blosc2 library
 Author-email: Blosc Development Team <blosc@blosc.org>
 Maintainer-email: Blosc Development Team <blosc@blosc.org>
@@ -23,7 +23,7 @@ Classifier: Programming Language :: Python :: 3.12
 Requires-Python: <4,>=3.10
 Description-Content-Type: text/x-rst
 License-File: LICENSE.txt
-Requires-Dist: numpy>=1.20.3
+Requires-Dist: numpy>=1.21
 Requires-Dist: ndindex>=1.4
 Requires-Dist: msgpack
 Requires-Dist: numexpr

{blosc2-2.6.2 → blosc2-2.7.1}/RELEASE_NOTES.md

@@ -1,5 +1,20 @@
 # Release notes
 
+## Changes from 2.7.0 to 2.7.1
+
+* Updated to latest C-Blosc2 2.15.1.
+  Fixes SIGKILL issues when using the `blosc2` library in old Intel CPUs.
+
+
+## Changes from 2.6.2 to 2.7.0
+
+* Updated to latest C-Blosc2 2.15.0.
+
+* Deprecated `LazyExpr.evaluate()`.
+
+* Fixed `_check_rc` function. See https://github.com/Blosc/python-blosc2/issues/187.
+
+
 ## Changes from 2.6.1 to 2.6.2
 
 * Protection when platforms have just one CPU. This caused the

blosc2-2.7.1/VERSION

@@ -0,0 +1 @@
+2.7.1

{blosc2-2.6.2 → blosc2-2.7.1}/blosc2/blosc2_ext.pyx

@@ -1906,7 +1906,7 @@ def register_filter(id, forward, backward, filter_name):
     if forward and backward:
         blosc2.ufilters_registry[id] = (forward, backward)
 
-def _check_rc(rc, message):
+cdef _check_rc(rc, message):
     if rc < 0:
         raise RuntimeError(message)
 

{blosc2-2.6.2 → blosc2-2.7.1}/blosc2/c-blosc2/ANNOUNCE.md

@@ -1,10 +1,13 @@
-# Announcing C-Blosc2 2.14.3
+# Announcing C-Blosc2 2.15.1
 A fast, compressed and persistent binary data store library for C.
 
 ## What is new?
 
-This is a maintenance release. We have fixed some additional security
-issues discovered by OSS-Fuzz.
+This is a maintenance release in which we are fixing calling instructions
+more advanced than available in current CPU, causing a SIGKILL.
+Furthermore, a new `b2nd_nans` function has been added. In addition,
+the internal LZ4 sources were updated to 1.10.0 and some
+other improvements were made.
 
 For more info, please see the release notes in:
 

blosc2-2.7.1/blosc2/c-blosc2/DEVELOPING-GUIDE.rst

@@ -0,0 +1,12 @@
+Some conventions used in C-Blosc2
+=================================
+
+* Use C99 designated initialization only in examples. Libraries should use C89 initialization, which is more portable, specially with C++ (designated initialization in C++ is supported only since C++20).
+
+* Use _new and _free for memory allocating constructors and destructors and _init and _destroy for non-memory allocating constructors and destructors.
+
+* Lines must not exceed 120 characters. If a line is too long, it must be broken into several lines.
+
+* Conditional bodies must always use braces, even if they are one-liners.  The only exception that can be is when the conditional is a single line and the body is a single line:
+
+  if (condition) whatever();

{blosc2-2.6.2 → blosc2-2.7.1}/blosc2/c-blosc2/README.rst

@@ -37,6 +37,7 @@ C-Blosc2 is the new major version of `C-Blosc <https://github.com/Blosc/c-blosc>
 
 See a 3 minutes  `introductory video to Blosc2 <https://www.youtube.com/watch?v=ER12R7FXosk>`_.
 
+
 Blosc2 NDim: an N-Dimensional store
 ===================================
 
@@ -119,6 +120,19 @@ More info about the `improved capabilities of C-Blosc2 can be found in this talk
 C-Blosc2 API and format have been frozen, and that means that there is guarantee that your programs will continue to work with future versions of the library, and that next releases will be able to read from persistent storage generated from previous releases (as of 2.0.0).
 
 
+Open format
+===========
+
+The Blosc2 format is open and documented in the next documents:
+
+* [The chunk; the basic building block](https://github.com/Blosc/c-blosc2/blob/main/README_CHUNK_FORMAT.rst)
+* [The cframe; this is made of different chunks in contiguous storage](https://github.com/Blosc/c-blosc2/blob/main/README_CFRAME_FORMAT.rst)
+* [The sframe; a variation of the cframe for sparse storage](https://github.com/Blosc/c-blosc2/blob/main/README_SFRAME_FORMAT.rst)
+* [The b2nd metalayer; info for the n-dimensional data container](https://github.com/Blosc/c-blosc2/blob/main/README_B2ND_METALAYER.rst)
+
+All these documents take less than 1000 lines of text, so they should be easy to read and understand.  In our opinion, this is very important for the long-term success of the library, as it allows for third-party implementations of the format, and also for the users to understand what is going on under the hood.
+
+
 Python wrapper
 ==============
 

{blosc2-2.6.2 → blosc2-2.7.1}/blosc2/c-blosc2/README_B2ND_METALAYER.rst

@@ -1,5 +1,5 @@
 b2nd metalayer
-++++++++++++++
+==============
 
 b2nd format is specified as a metalayer on top of a Blosc2 container for storing
 multidimensional information.  Specifically, this metalayer is named 'b2nd'

{blosc2-2.6.2 → blosc2-2.7.1}/blosc2/c-blosc2/README_CHUNK_FORMAT.rst

@@ -8,7 +8,7 @@ A regular chunk is composed of a header and a blocks section::
     +---------+--------+
 
 Also, there are the so-called lazy chunks that do not have the actual compressed data,
-but only metainformation about how to read it. Lazy chunks typically appear when reading
+but only meta-information about how to read it. Lazy chunks typically appear when reading
 data from persistent media.  A lazy chunk has header and bstarts sections in place and
 in addition, an additional trailer for allowing to read the data blocks::
 

blosc2-2.7.1/blosc2/c-blosc2/README_EXTENSION_FILENAMES.rst

@@ -0,0 +1,8 @@
+Extensions for Blosc2 filenames
+===============================
+
+Blosc2 has different file extensions for different purposes.  Here is a list of the currently supported ones:
+
+- `.b2frame` (but also `.b2f` or `.b2`) (Blosc2 frame): this is the main extension for storing `Blosc2 contiguous frames <README_CFRAME_FORMAT.rst>`_.
+
+- `.b2nd` (Blosc2 n-dimensional): this is just a contiguous frame file with `a metalayer for storing n-dimensional information <README_B2ND_METALAYER.rst>`_ like shape, chunkshape, blockshape and dtype.

{blosc2-2.6.2 → blosc2-2.7.1}/blosc2/c-blosc2/RELEASE_NOTES.md

@@ -1,11 +1,58 @@
 Release notes for C-Blosc2
 ==========================
 
+Changes from 2.15.1 to 2.15.2
+=============================
+
+#XXX version-specific blurb XXX#
+
+
+Changes from 2.15.0 to 2.15.1
+=============================
+
+* Do not pass `-m` flags when compiling `shuffle.c`. This prevents the
+  compiler from incidentally optimizing the code called independently
+  of the runtime CPU check to these instruction sets, effectively
+  causing `SIGILL` on other CPUs.  Fixes #621.  Thanks to @t20100 and @mgorny.
+
+* Internal LZ4 sources bumped to 1.10.0.
+
+* Allow direct loading of plugins by name, without relying on
+  the presence of python. Thanks to @boxerab.
+
+* Add `b2nd_nans` method (PR #624). Thanks to @waynegm.
+
+
+Changes from 2.14.4 to 2.15.0
+=============================
+
+* Removed some duplicated functions. See https://github.com/Blosc/c-blosc2/issues/503.
+
+* Added a new io mode to memory map files. This forced to change the `io_cb` read API.
+  See https://github.com/Blosc/c-blosc2/blob/main/tests/test_mmap.c to see an example on
+  how to use it.
+
+* Updated the `SOVERSION` to 4 due to the API change in `io_cb` read.
+
+* Added functions to get cparams, dparams, storage and io defaults respectively.
+
+* Internal zstd sources updated to 1.5.6.
+
+* Fixed a bug when setting a slice using prefilters.
+
+
+Changes from 2.14.3 to 2.14.4
+=============================
+
+* Bumped SONAME due to recent API changes. See https://github.com/Blosc/c-blosc2/issues/581.
+
+
 Changes from 2.14.2 to 2.14.3
 =============================
 
 * More fixes for internal fuzzer.
 
+
 Changes from 2.14.1 to 2.14.2
 =============================
 
@@ -19,6 +66,7 @@ Changes from 2.14.0 to 2.14.1
   This is because many linux distros do not have `python` as a
   symlink to `python3` anymore.
 
+
 Changes from 2.13.2 to 2.14.0
 =============================
 

{blosc2-2.6.2 → blosc2-2.7.1}/blosc2/c-blosc2/RELEASING.rst

@@ -9,6 +9,11 @@ Preliminaries
 
 - Check that *VERSION* symbols in include/blosc2.h contains the correct info.
 
+- If API/ABI changes, please increase the minor number (e.g. 2.15 -> 2.16) *and*
+  bump the SOVERSION in blosc/CMakeLists.txt.
+  When in doubt on when SOVERSION should change, see these nice guidelines:
+  https://github.com/conda-forge/c-blosc2-feedstock/issues/62#issuecomment-2049675391
+
 - Commit the changes with::
 
     $ git commit -a -m "Getting ready for release X.Y.Z"

blosc2-2.7.1/blosc2/c-blosc2/bench/aggregate_multiple.py

@@ -0,0 +1,101 @@
+from pathlib import Path
+import re
+import os
+import subprocess
+from typing import Union
+
+import numpy as np
+import argparse
+
+
+def run_command(command: str, return_output: bool = False) -> Union[tuple[list[float], str], tuple[list[float]]]:
+    bench_folder = Path(__file__).parent.parent / "build" / "bench"
+    if os.name == "nt":
+        bench_folder /= "Release"
+    res = subprocess.run(f"{bench_folder}/{command}", shell=True, capture_output=True)
+    assert res.returncode == 0, f"\nstdout: {res.stdout.decode('utf-8')}\nstderr: {res.stderr.decode('utf-8')}"
+
+    out = res.stdout.decode("utf-8")
+    numbers = []
+    for number in re.findall(r"-?\d+(?:\.\d+)?", out):
+        numbers.append(float(number))
+    assert len(numbers) > 0, f"Could not find any numbers in the output:\n{out}"
+    
+    if return_output:
+        return numbers, out
+    else:
+        return numbers
+
+
+class ReplaceNumbers:
+    def __init__(self, numbers: np.ndarray):
+        self.numbers = numbers
+        self.index = 0
+
+    def __call__(self, match: re.Match) -> str:
+        assert match.group() == "NUMBER"
+        number = self.numbers[self.index]
+        self.index += 1
+
+        if number.is_integer():
+            return str(int(number))
+        else:
+            return f"{number:.3f}"
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description=(
+            "Repeat multiple runs of a benchmark and aggregate all numbers in the output (the outputs from all runs must "
+            "have the same format)."
+        ),
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument(
+        "--command",
+        required=True,
+        type=str,
+        help="The command to execute (e.g. sframe_bench 1000 insert 1000 io_mmap).",
+    )
+    parser.add_argument(
+        "--warmups",
+        default=3,
+        required=False,
+        type=int,
+        help="Number of warmup runs.",
+    )
+    parser.add_argument(
+        "--runs",
+        default=30,
+        required=False,
+        type=int,
+        help="Number of runs to perform and aggregate the output from.",
+    )
+    args = parser.parse_args()
+
+    for i in range(args.warmups):
+        print(f"Warmup {i+1}/{args.warmups}", end="\r")
+        run_command(args.command)
+    print()
+
+    # Collect numbers from outputs of multiple runs
+    example_output = None
+    numbers = []
+    for i in range(args.runs):
+        print(f"Run {i+1}/{args.runs}", end="\r")
+
+        if example_output is None:
+            values, text = run_command(args.command, return_output=True)
+            example_output = text
+            numbers.append(values)
+        else:
+            numbers.append(run_command(args.command))
+    print()
+    
+    numbers = np.asarray(numbers)
+    numbers = np.min(numbers, axis=0)
+
+    assert example_output is not None, "No output was collected."
+    example_output = re.sub(r"-?\d+(?:\.\d+)?", "NUMBER", example_output)
+    aggregated_output = re.sub("NUMBER", ReplaceNumbers(numbers), example_output)
+    print(aggregated_output)

{blosc2-2.6.2 → blosc2-2.7.1}/blosc2/c-blosc2/bench/sframe_bench.c

@@ -40,6 +40,7 @@
 
 int nchunks = NCHUNKS;
 int iterations = 5;
+int io_type = BLOSC2_IO_FILESYSTEM;
 
 
 
@@ -249,7 +250,18 @@ void test_create_sframe_frame(char* operation) {
   blosc2_remove_urlpath(storage.urlpath);
   schunk_sframe = blosc2_schunk_new(&storage);
 
+  blosc2_stdio_mmap mmap_file = BLOSC2_STDIO_MMAP_DEFAULTS;
+  mmap_file.mode = "w+";
+  blosc2_io io_mmap = {.id = BLOSC2_IO_FILESYSTEM_MMAP, .name = "filesystem_mmap", .params = &mmap_file};
+
   blosc2_storage storage2 = {.contiguous=true, .urlpath="test_cframe.b2frame", .cparams=&cparams, .dparams=&dparams};
+  if (io_type == BLOSC2_IO_FILESYSTEM) {
+    storage2.io = (blosc2_io*)&BLOSC2_IO_DEFAULTS;
+  }
+  else if (io_type == BLOSC2_IO_FILESYSTEM_MMAP) {
+    storage2.io = &io_mmap;
+  }
+
   blosc2_remove_urlpath(storage2.urlpath);
   schunk_cframe = blosc2_schunk_new(&storage2);
 
@@ -350,8 +362,8 @@ if (operation != NULL) {
 int main(int argc, char* argv[]) {
   char* operation = NULL;
 
-  if (argc >= 5) {
-    printf("Usage: ./sframe_bench [nchunks] [insert | update | reorder] [num operations]\n");
+  if (argc >= 6) {
+    printf("Usage: ./sframe_bench [nchunks] [insert | update | reorder] [num operations] [io_file | io_mmap]\n");
     exit(1);
   }
   else if (argc >= 2) {
@@ -360,9 +372,19 @@ int main(int argc, char* argv[]) {
   if (argc >= 3) {
     operation = argv[2];
   }
-  if (argc == 4) {
+  if (argc >= 4) {
     iterations = (int)strtol(argv[3], NULL, 10);
   }
+  if (argc == 5) {
+    if (strcmp(argv[4], "io_file") == 0) {
+      io_type = BLOSC2_IO_FILESYSTEM;
+    } else if (strcmp(argv[4], "io_mmap") == 0) {
+      io_type = BLOSC2_IO_FILESYSTEM_MMAP;
+    } else {
+      printf("Invalid io type. Use io_file or io_mmap\n");
+      exit(1);
+    }
+  }
 
   test_create_sframe_frame(operation);
 

{blosc2-2.6.2 → blosc2-2.7.1}/blosc2/c-blosc2/blosc/CMakeLists.txt

@@ -29,7 +29,7 @@ if(BUILD_SHARED)
     endif()
     set_target_properties(blosc2_shared PROPERTIES
         VERSION ${version_string}
-        SOVERSION 2  # Change this when an ABI change happens
+        SOVERSION 4  # Change this when an ABI change happens
     )
     target_compile_definitions(blosc2_shared PUBLIC BLOSC_SHARED_LIBRARY)
     target_include_directories(blosc2_shared PUBLIC
@@ -90,7 +90,7 @@ if(LZ4_FOUND)
         target_include_directories(blosc_testing PUBLIC ${LZ4_INCLUDE_DIR})
     endif()
 else()
-    set(LZ4_LOCAL_DIR ${INTERNAL_LIBS}/lz4-1.9.4)
+    set(LZ4_LOCAL_DIR ${INTERNAL_LIBS}/lz4-1.10.0)
     if(BUILD_SHARED)
         target_include_directories(blosc2_shared PRIVATE ${LZ4_LOCAL_DIR})
     endif()
@@ -152,7 +152,7 @@ if(NOT DEACTIVATE_ZSTD)
             target_link_libraries(blosc_testing PUBLIC ${ZSTD_LIBRARY})
         endif()
     else()
-        set(ZSTD_LOCAL_DIR ${INTERNAL_LIBS}/zstd-1.5.5)
+        set(ZSTD_LOCAL_DIR ${INTERNAL_LIBS}/zstd-1.5.6)
         if(BUILD_SHARED)
             target_include_directories(blosc2_shared PRIVATE ${ZSTD_LOCAL_DIR} ${ZSTD_LOCAL_DIR}/common)
         endif()
@@ -300,17 +300,11 @@ if(COMPILER_SUPPORT_SSE2)
             set_source_files_properties(
                     shuffle-sse2.c bitshuffle-sse2.c blosclz.c fastcopy.c
                     PROPERTIES COMPILE_OPTIONS "/arch:SSE2")
-            set_property(
-                    SOURCE shuffle.c
-                    APPEND PROPERTY COMPILE_OPTIONS "/arch:SSE2")
         endif()
     else()
         set_source_files_properties(
                 shuffle-sse2.c bitshuffle-sse2.c blosclz.c fastcopy.c
                 PROPERTIES COMPILE_OPTIONS -msse2)
-        set_property(
-                SOURCE shuffle.c
-                APPEND PROPERTY COMPILE_OPTIONS -msse2)
         # Add SIMD flags for the bytedelta filter and Intel (it seems that ARM64 does not need these)
         set_source_files_properties(
                 ${PROJECT_SOURCE_DIR}/plugins/filters/bytedelta/bytedelta.c
@@ -330,16 +324,10 @@ if(COMPILER_SUPPORT_AVX2)
         set_source_files_properties(
                 shuffle-avx2.c bitshuffle-avx2.c
                 PROPERTIES COMPILE_OPTIONS "/arch:AVX2")
-        set_property(
-                SOURCE shuffle.c
-                APPEND PROPERTY COMPILE_OPTIONS "/arch:AVX2")
     else()
         set_source_files_properties(
                 shuffle-avx2.c bitshuffle-avx2.c
                 PROPERTIES COMPILE_OPTIONS -mavx2)
-        set_property(
-                SOURCE shuffle.c
-                APPEND PROPERTY COMPILE_OPTIONS -mavx2)
     endif()
 
     # Define a symbol for the shuffle-dispatch implementation
@@ -354,16 +342,10 @@ if(COMPILER_SUPPORT_AVX512)
         set_source_files_properties(
                 bitshuffle-avx512.c
 		PROPERTIES COMPILE_OPTIONS "/arch:AVX512")
-        set_property(
-                SOURCE shuffle.c
-		APPEND PROPERTY COMPILE_OPTIONS "/arch:AVX512")
     else()
         set_source_files_properties(
                 bitshuffle-avx512.c
                 PROPERTIES COMPILE_OPTIONS "-mavx512f;-mavx512bw")
-        set_property(
-                SOURCE shuffle.c
-                APPEND PROPERTY COMPILE_OPTIONS "-mavx512f;-mavx512bw")
     endif()
 
     # Define a symbol for the shuffle-dispatch implementation
@@ -377,17 +359,11 @@ if(COMPILER_SUPPORT_NEON)
     set_source_files_properties(
             shuffle-neon.c bitshuffle-neon.c
             PROPERTIES COMPILE_OPTIONS "-flax-vector-conversions")
-    set_property(
-            SOURCE shuffle.c
-            APPEND PROPERTY COMPILE_OPTIONS "-flax-vector-conversions")
     if(CMAKE_SYSTEM_PROCESSOR STREQUAL armv7l)
         # Only armv7l needs special -mfpu=neon flag; aarch64 doesn't.
       set_source_files_properties(
             shuffle-neon.c bitshuffle-neon.c
             PROPERTIES COMPILE_OPTIONS "-mfpu=neon;-flax-vector-conversions")
-      set_property(
-            SOURCE shuffle.c
-            APPEND PROPERTY COMPILE_OPTIONS "-mfpu=neon;-flax-vector-conversions")
     endif()
     # Define a symbol for the shuffle-dispatch implementation
     # so it knows NEON is supported even though that file is