blosc2 2.6.2__tar.gz → 2.7.0__tar.gz

{blosc2-2.6.2 → blosc2-2.7.0}/ANNOUNCE.rst

@@ -1,10 +1,9 @@
-Announcing Python-Blosc2 2.6.2
+Announcing Python-Blosc2 2.7.0
 ==============================
 
-Updated to latest C-Blosc2 2.14.3. This was necessary to fix
-several CVEs in the C-Blosc2 library.  Also, the new version adds
-protection when platforms have just one CPU. This caused the
-internal number of threads to be 0, producing a division by zero.
+Updated to latest C-Blosc2 2.15.0. Also, `LazyExpr.evaluate()`
+is marked as deprecated (LazyExpr.eval() is replacing it) and some other fixes
+have been carried out.
 
 For more info, you can have a look at the release notes in:
 

{blosc2-2.6.2/blosc2.egg-info → blosc2-2.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: blosc2
-Version: 2.6.2
+Version: 2.7.0
 Summary: Python wrapper for the C-Blosc2 library
 Author-email: Blosc Development Team <blosc@blosc.org>
 Maintainer-email: Blosc Development Team <blosc@blosc.org>

{blosc2-2.6.2 → blosc2-2.7.0}/RELEASE_NOTES.md

@@ -1,5 +1,14 @@
 # Release notes
 
+## Changes from 2.6.2 to 2.7.0
+
+* Updated to latest C-Blosc2 2.15.0.
+
+* Deprecated `LazyExpr.evaluate()`.
+
+* Fixed `_check_rc` function. See https://github.com/Blosc/python-blosc2/issues/187.
+
+
 ## Changes from 2.6.1 to 2.6.2
 
 * Protection when platforms have just one CPU. This caused the

blosc2-2.7.0/VERSION

@@ -0,0 +1 @@
+2.7.0

{blosc2-2.6.2 → blosc2-2.7.0}/blosc2/blosc2_ext.pyx

@@ -1906,7 +1906,7 @@ def register_filter(id, forward, backward, filter_name):
     if forward and backward:
         blosc2.ufilters_registry[id] = (forward, backward)
 
-def _check_rc(rc, message):
+cdef _check_rc(rc, message):
     if rc < 0:
         raise RuntimeError(message)
 

{blosc2-2.6.2 → blosc2-2.7.0}/blosc2/c-blosc2/ANNOUNCE.md

@@ -1,10 +1,12 @@
-# Announcing C-Blosc2 2.14.3
+# Announcing C-Blosc2 2.15.0
 A fast, compressed and persistent binary data store library for C.
 
 ## What is new?
 
-This is a maintenance release. We have fixed some additional security
-issues discovered by OSS-Fuzz.
+This is a minor release in which a new io mode
+was added to memory-map files. Furthermore, the `io_cb` read API was changed, so the 
+`SOVERSION` was bumped. In addition, the internal zstd sources were updated to 1.5.6 and some
+other improvements were made.
 
 For more info, please see the release notes in:
 

blosc2-2.7.0/blosc2/c-blosc2/DEVELOPING-GUIDE.rst

@@ -0,0 +1,12 @@
+Some conventions used in C-Blosc2
+=================================
+
+* Use C99 designated initialization only in examples. Libraries should use C89 initialization, which is more portable, specially with C++ (designated initialization in C++ is supported only since C++20).
+
+* Use _new and _free for memory allocating constructors and destructors and _init and _destroy for non-memory allocating constructors and destructors.
+
+* Lines must not exceed 120 characters. If a line is too long, it must be broken into several lines.
+
+* Conditional bodies must always use braces, even if they are one-liners.  The only exception that can be is when the conditional is a single line and the body is a single line:
+
+  if (condition) whatever();

{blosc2-2.6.2 → blosc2-2.7.0}/blosc2/c-blosc2/README.rst

@@ -37,6 +37,7 @@ C-Blosc2 is the new major version of `C-Blosc <https://github.com/Blosc/c-blosc>
 
 See a 3 minutes  `introductory video to Blosc2 <https://www.youtube.com/watch?v=ER12R7FXosk>`_.
 
+
 Blosc2 NDim: an N-Dimensional store
 ===================================
 
@@ -119,6 +120,19 @@ More info about the `improved capabilities of C-Blosc2 can be found in this talk
 C-Blosc2 API and format have been frozen, and that means that there is guarantee that your programs will continue to work with future versions of the library, and that next releases will be able to read from persistent storage generated from previous releases (as of 2.0.0).
 
 
+Open format
+===========
+
+The Blosc2 format is open and documented in the next documents:
+
+* [The chunk; the basic building block](https://github.com/Blosc/c-blosc2/blob/main/README_CHUNK_FORMAT.rst)
+* [The cframe; this is made of different chunks in contiguous storage](https://github.com/Blosc/c-blosc2/blob/main/README_CFRAME_FORMAT.rst)
+* [The sframe; a variation of the cframe for sparse storage](https://github.com/Blosc/c-blosc2/blob/main/README_SFRAME_FORMAT.rst)
+* [The b2nd metalayer; info for the n-dimensional data container](https://github.com/Blosc/c-blosc2/blob/main/README_B2ND_METALAYER.rst)
+
+All these documents take less than 1000 lines of text, so they should be easy to read and understand.  In our opinion, this is very important for the long-term success of the library, as it allows for third-party implementations of the format, and also for the users to understand what is going on under the hood.
+
+
 Python wrapper
 ==============
 

{blosc2-2.6.2 → blosc2-2.7.0}/blosc2/c-blosc2/RELEASE_NOTES.md

@@ -1,11 +1,36 @@
 Release notes for C-Blosc2
 ==========================
 
+Changes from 2.14.4 to 2.15.0
+=============================
+
+* Removed some duplicated functions. See https://github.com/Blosc/c-blosc2/issues/503.
+
+* Added a new io mode to memory map files. This forced to change the `io_cb` read API.
+  See https://github.com/Blosc/c-blosc2/blob/main/tests/test_mmap.c to see an example on
+  how to use it.
+
+* Updated the `SOVERSION` to 4 due to the API change in `io_cb` read.
+
+* Added functions to get cparams, dparams, storage and io defaults respectively.
+
+* Internal zstd sources updated to 1.5.6.
+
+* Fixed a bug when setting a slice using prefilters.
+
+
+Changes from 2.14.3 to 2.14.4
+=============================
+
+* Bumped SONAME due to recent API changes. See https://github.com/Blosc/c-blosc2/issues/581.
+
+
 Changes from 2.14.2 to 2.14.3
 =============================
 
 * More fixes for internal fuzzer.
 
+
 Changes from 2.14.1 to 2.14.2
 =============================
 
@@ -19,6 +44,7 @@ Changes from 2.14.0 to 2.14.1
   This is because many linux distros do not have `python` as a
   symlink to `python3` anymore.
 
+
 Changes from 2.13.2 to 2.14.0
 =============================
 

{blosc2-2.6.2 → blosc2-2.7.0}/blosc2/c-blosc2/RELEASING.rst

@@ -9,6 +9,11 @@ Preliminaries
 
 - Check that *VERSION* symbols in include/blosc2.h contains the correct info.
 
+- If API/ABI changes, please increase the minor number (e.g. 2.15 -> 2.16) *and*
+  bump the SOVERSION in blosc/CMakeLists.txt.
+  When in doubt on when SOVERSION should change, see these nice guidelines:
+  https://github.com/conda-forge/c-blosc2-feedstock/issues/62#issuecomment-2049675391
+
 - Commit the changes with::
 
     $ git commit -a -m "Getting ready for release X.Y.Z"

blosc2-2.7.0/blosc2/c-blosc2/bench/aggregate_multiple.py

@@ -0,0 +1,101 @@
+from pathlib import Path
+import re
+import os
+import subprocess
+from typing import Union
+
+import numpy as np
+import argparse
+
+
+def run_command(command: str, return_output: bool = False) -> Union[tuple[list[float], str], tuple[list[float]]]:
+    bench_folder = Path(__file__).parent.parent / "build" / "bench"
+    if os.name == "nt":
+        bench_folder /= "Release"
+    res = subprocess.run(f"{bench_folder}/{command}", shell=True, capture_output=True)
+    assert res.returncode == 0, f"\nstdout: {res.stdout.decode('utf-8')}\nstderr: {res.stderr.decode('utf-8')}"
+
+    out = res.stdout.decode("utf-8")
+    numbers = []
+    for number in re.findall(r"-?\d+(?:\.\d+)?", out):
+        numbers.append(float(number))
+    assert len(numbers) > 0, f"Could not find any numbers in the output:\n{out}"
+    
+    if return_output:
+        return numbers, out
+    else:
+        return numbers
+
+
+class ReplaceNumbers:
+    def __init__(self, numbers: np.ndarray):
+        self.numbers = numbers
+        self.index = 0
+
+    def __call__(self, match: re.Match) -> str:
+        assert match.group() == "NUMBER"
+        number = self.numbers[self.index]
+        self.index += 1
+
+        if number.is_integer():
+            return str(int(number))
+        else:
+            return f"{number:.3f}"
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description=(
+            "Repeat multiple runs of a benchmark and aggregate all numbers in the output (the outputs from all runs must "
+            "have the same format)."
+        ),
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument(
+        "--command",
+        required=True,
+        type=str,
+        help="The command to execute (e.g. sframe_bench 1000 insert 1000 io_mmap).",
+    )
+    parser.add_argument(
+        "--warmups",
+        default=3,
+        required=False,
+        type=int,
+        help="Number of warmup runs.",
+    )
+    parser.add_argument(
+        "--runs",
+        default=30,
+        required=False,
+        type=int,
+        help="Number of runs to perform and aggregate the output from.",
+    )
+    args = parser.parse_args()
+
+    for i in range(args.warmups):
+        print(f"Warmup {i+1}/{args.warmups}", end="\r")
+        run_command(args.command)
+    print()
+
+    # Collect numbers from outputs of multiple runs
+    example_output = None
+    numbers = []
+    for i in range(args.runs):
+        print(f"Run {i+1}/{args.runs}", end="\r")
+
+        if example_output is None:
+            values, text = run_command(args.command, return_output=True)
+            example_output = text
+            numbers.append(values)
+        else:
+            numbers.append(run_command(args.command))
+    print()
+    
+    numbers = np.asarray(numbers)
+    numbers = np.min(numbers, axis=0)
+
+    assert example_output is not None, "No output was collected."
+    example_output = re.sub(r"-?\d+(?:\.\d+)?", "NUMBER", example_output)
+    aggregated_output = re.sub("NUMBER", ReplaceNumbers(numbers), example_output)
+    print(aggregated_output)

{blosc2-2.6.2 → blosc2-2.7.0}/blosc2/c-blosc2/bench/sframe_bench.c

@@ -40,6 +40,7 @@
 
 int nchunks = NCHUNKS;
 int iterations = 5;
+int io_type = BLOSC2_IO_FILESYSTEM;
 
 
 
@@ -249,7 +250,18 @@ void test_create_sframe_frame(char* operation) {
   blosc2_remove_urlpath(storage.urlpath);
   schunk_sframe = blosc2_schunk_new(&storage);
 
+  blosc2_stdio_mmap mmap_file = BLOSC2_STDIO_MMAP_DEFAULTS;
+  mmap_file.mode = "w+";
+  blosc2_io io_mmap = {.id = BLOSC2_IO_FILESYSTEM_MMAP, .name = "filesystem_mmap", .params = &mmap_file};
+
   blosc2_storage storage2 = {.contiguous=true, .urlpath="test_cframe.b2frame", .cparams=&cparams, .dparams=&dparams};
+  if (io_type == BLOSC2_IO_FILESYSTEM) {
+    storage2.io = (blosc2_io*)&BLOSC2_IO_DEFAULTS;
+  }
+  else if (io_type == BLOSC2_IO_FILESYSTEM_MMAP) {
+    storage2.io = &io_mmap;
+  }
+
   blosc2_remove_urlpath(storage2.urlpath);
   schunk_cframe = blosc2_schunk_new(&storage2);
 
@@ -350,8 +362,8 @@ if (operation != NULL) {
 int main(int argc, char* argv[]) {
   char* operation = NULL;
 
-  if (argc >= 5) {
-    printf("Usage: ./sframe_bench [nchunks] [insert | update | reorder] [num operations]\n");
+  if (argc >= 6) {
+    printf("Usage: ./sframe_bench [nchunks] [insert | update | reorder] [num operations] [io_file | io_mmap]\n");
     exit(1);
   }
   else if (argc >= 2) {
@@ -360,9 +372,19 @@ int main(int argc, char* argv[]) {
   if (argc >= 3) {
     operation = argv[2];
   }
-  if (argc == 4) {
+  if (argc >= 4) {
     iterations = (int)strtol(argv[3], NULL, 10);
   }
+  if (argc == 5) {
+    if (strcmp(argv[4], "io_file") == 0) {
+      io_type = BLOSC2_IO_FILESYSTEM;
+    } else if (strcmp(argv[4], "io_mmap") == 0) {
+      io_type = BLOSC2_IO_FILESYSTEM_MMAP;
+    } else {
+      printf("Invalid io type. Use io_file or io_mmap\n");
+      exit(1);
+    }
+  }
 
   test_create_sframe_frame(operation);
 

{blosc2-2.6.2 → blosc2-2.7.0}/blosc2/c-blosc2/blosc/CMakeLists.txt

@@ -29,7 +29,7 @@ if(BUILD_SHARED)
     endif()
     set_target_properties(blosc2_shared PROPERTIES
         VERSION ${version_string}
-        SOVERSION 2  # Change this when an ABI change happens
+        SOVERSION 4  # Change this when an ABI change happens
     )
     target_compile_definitions(blosc2_shared PUBLIC BLOSC_SHARED_LIBRARY)
     target_include_directories(blosc2_shared PUBLIC
@@ -152,7 +152,7 @@ if(NOT DEACTIVATE_ZSTD)
             target_link_libraries(blosc_testing PUBLIC ${ZSTD_LIBRARY})
         endif()
     else()
-        set(ZSTD_LOCAL_DIR ${INTERNAL_LIBS}/zstd-1.5.5)
+        set(ZSTD_LOCAL_DIR ${INTERNAL_LIBS}/zstd-1.5.6)
         if(BUILD_SHARED)
             target_include_directories(blosc2_shared PRIVATE ${ZSTD_LOCAL_DIR} ${ZSTD_LOCAL_DIR}/common)
         endif()