blissdata 2.0.1__tar.gz → 2.1.0__tar.gz

blissdata-2.1.0/LICENSE

@@ -0,0 +1,166 @@
+                   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.
+

{blissdata-2.0.1 → blissdata-2.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: blissdata
-Version: 2.0.1
+Version: 2.1.0
 Summary: Bliss data streaming client
 Maintainer: BCU (ESRF)
 License-Expression: LGPL-3.0-or-later
@@ -12,11 +12,13 @@ Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
+License-File: LICENSE
 Requires-Dist: beartype
 Requires-Dist: fabio>=0.12
 Requires-Dist: h5py>=2.9
 Requires-Dist: msgpack
 Requires-Dist: numpy>=1.20
+Requires-Dist: numpydantic<1.6.10
 Requires-Dist: pydantic>=2.3.0
 Requires-Dist: redis>=4.6.0
 Requires-Dist: redis-om<1.0,>=0.3.1
@@ -25,7 +27,7 @@ Provides-Extra: tango
 Requires-Dist: pytango>=9.3.1; extra == "tango"
 Provides-Extra: lima2
 Requires-Dist: blissdata[tango]; extra == "lima2"
-Requires-Dist: lima2-client~=2.0; extra == "lima2"
+Requires-Dist: lima2-client~=4.0; extra == "lima2"
 Provides-Extra: test
 Requires-Dist: gevent>=20.9.0; extra == "test"
 Requires-Dist: pytest>=7; extra == "test"
@@ -43,6 +45,7 @@ Requires-Dist: mkdocs-material; extra == "dev"
 Requires-Dist: mkdocstrings[python]>=0.18; extra == "dev"
 Provides-Extra: beacon
 Requires-Dist: ruamel.yaml; extra == "beacon"
+Dynamic: license-file
 
 # Blissdata
 

{blissdata-2.0.1 → blissdata-2.1.0}/pyproject.toml

@@ -1,9 +1,10 @@
 [project]
 name = "blissdata"
-version = "2.0.1"
+version = "2.1.0"
 description = "Bliss data streaming client"
 readme = "README.md"
 license = "LGPL-3.0-or-later"
+license-files = ["LICENSE"]
 maintainers = [{name = "BCU (ESRF)"}]
 classifiers = [
     "Intended Audience :: Science/Research",
@@ -17,6 +18,7 @@ dependencies = [
     "h5py>=2.9",
     "msgpack",
     "numpy>=1.20",
+    "numpydantic<1.6.10",
     "pydantic>=2.3.0", # (not a direct dep) redis-om 0.3.2 doesn't support lower versions
     "redis>=4.6.0",
     "redis-om>=0.3.1,<1.0",
@@ -32,7 +34,7 @@ Changes = "https://gitlab.esrf.fr/bliss/blissdata/-/blob/master/CHANGELOG.md"
 tango = ["pytango>=9.3.1"]
 lima2 = [
     "blissdata[tango]",
-    "lima2-client~=2.0"
+    "lima2-client~=4.0"
 ]
 test = [
     "gevent>=20.9.0",

blissdata-2.1.0/src/blissdata/h5map/__init__.py

@@ -0,0 +1,21 @@
+from .data_items import (  # noqa: F401
+    DataItem,
+    DataItemType,
+    InlineRaw,
+    Inline,
+    InlineBase64,
+    Stream,
+)
+from .hdf5_items import (  # noqa: F401
+    HDF5Item,
+    HDF5ItemType,
+    Group,
+    Dataset,
+    ExternalBinaryDataset,
+    ExternalBinaryFile,
+    SoftLink,
+    ExternalLink,
+    VirtualDataset,
+    VirtualSource,
+    SpaceSelection,
+)

blissdata-2.1.0/src/blissdata/h5map/data_items.py

@@ -0,0 +1,184 @@
+import numpy as np
+from typing import Any, Annotated, Literal
+from pydantic import (
+    BaseModel,
+    RootModel,
+    Base64Bytes,
+    Field,
+    field_validator,
+    model_serializer,
+    Discriminator,
+    Tag,
+    TypeAdapter,
+)
+from numpydantic import NDArray
+
+
+class InlineRaw(RootModel):
+    """Literal values to be embedded into JSON directly (no 'type' field).
+    Accepts scalars or arrays of any shape with types that can be coerced using
+    the following table:
+     -------------- --------- ---------------
+    | Python input |   Json  | Python output |
+     -------------- --------- ---------------
+    | any integer  | integer | np.int64      |
+    | any floating | decimal | np.float64    |
+    | any boolean  | boolean | np.bool_      |
+    | any string   | string  | np.str_       |
+     -------------- --------- ---------------
+    For mixed types arrays, coercion follows this hierarchy (lowest to highest):
+        bool → int64 → float64 → str
+    Example:
+        [True, 123, 1.23] → [1.0, 123.0, 1.23]
+
+    Non-finite values (nan, -inf, inf) will be refused, one should use Inline
+    model instead.
+    """
+
+    root: NDArray[Any, np.bool_ | np.str_ | np.int64 | np.float64]
+
+    @field_validator("root", mode="before")
+    @classmethod
+    def coerce_numbers(cls, value):
+        array = np.asarray(value)
+        if np.issubdtype(array.dtype, np.floating):
+            if not np.isfinite(value).all():
+                raise ValueError("non-finite values (nan, -inf, inf) are not allowed.")
+            array = array.astype(np.float64)
+        elif np.issubdtype(array.dtype, np.integer):
+            array = array.astype(np.int64)
+        return array
+
+    @model_serializer(when_used="json", mode="wrap")
+    def fix_empty_string_serialization(self, nxt, info):
+        if np.issubdtype(self.root.dtype, np.str_) and not self.root.shape:
+            return str(self.root)
+        else:
+            return nxt(self, info)
+
+    def __eq__(self, other):
+        if other.__class__ is self.__class__:
+            return np.array_equal(self.root, other.root)
+        return NotImplemented
+
+    def decode(self):
+        value = self.model_dump()
+        if value.dtype.kind in "UT":
+            return value.tolist()
+        else:
+            return value
+
+
+# Numpy dtype names (uint8, float16, complex128...), but limited to numbers and
+# booleans.
+_DType = Annotated[
+    str,
+    Field(
+        pattern=r"^(bool|(int|uint)(8|16|32|64)|float(16|32|64|128)|complex(64|128|256))$"
+    ),
+]
+
+
+# Numpy's array interface protocol type strings, but limited to numbers
+# and booleans. This notation contains both dtype and endianness.
+# See Numpy array interface protocol definition here:
+#     https://numpy.org/doc/stable/reference/arrays.interface.html
+_BinaryDType = Annotated[
+    str, Field(pattern=r"^(\|?[iub]1|[><]([iuf][248]|[fc](8|16)|c32))$")
+]
+
+
+class _InlineValue(RootModel):
+    """Value for the 'Inline' model, can be any JSON value other than a mapping
+    or null value. Different from InlineRaw because it can be heterogeneous and
+    non-finite values (nan, inf, -inf) are coerced into string.
+    """
+
+    root: str | bool | int | float | list["_InlineValue"]
+
+    @field_validator("root", mode="before")
+    @classmethod
+    def coerce_numpy_values(cls, value):
+        def coerce_single(item):
+            if not np.issubdtype(type(item), np.number):
+                # str | bool
+                return item
+            elif not np.isfinite(item) or np.issubdtype(type(item), np.complexfloating):
+                # nan | inf | -inf | complex
+                return str(item)
+            elif isinstance(item, np.generic):
+                # numpy scalar
+                return item.item()
+            else:
+                # python scalar
+                return item
+
+        def coerce_all(value):
+            if isinstance(value, list):
+                return [coerce_all(i) for i in value]
+            elif isinstance(value, np.ndarray) and value.ndim > 1:
+                return value
+            elif isinstance(value, np.ndarray) and value.ndim > 0:
+                return [coerce_all(value[i]) for i in range(value.shape[0])]
+            else:
+                return coerce_single(value)
+
+        return coerce_all(value)
+
+
+class Inline(BaseModel):
+    """Explicitly typed JSON value to produce specific HDF5 type (int8, float32, etc).
+    Can be used to transmit string as float like "nan" or "inf".
+    """
+
+    type: Literal["inline"] = "inline"
+    dtype: _DType
+    value: _InlineValue
+
+    def decode(self):
+        return np.array(self.value.model_dump(), dtype=self.dtype)
+
+
+class InlineBase64(BaseModel):
+    """Base64 encoded binary value."""
+
+    type: Literal["inline_b64"] = "inline_b64"
+    dtype: _BinaryDType
+    shape: list[int] | None = None
+    value: Base64Bytes
+
+    def decode(self):
+        value = np.frombuffer(self.value, dtype=self.dtype)
+        if self.shape is not None:
+            return value.reshape(self.shape)
+        else:
+            return value
+
+
+class Stream(BaseModel):
+    type: Literal["stream"] = "stream"
+    stream: str
+
+    def decode(self):
+        raise TypeError("Stream model is just a reference, can't use decode()")
+
+
+def get_discriminator_value(v: Any) -> str:
+    if isinstance(v, dict):
+        return v.get("type")
+    if issubclass(type(v), BaseModel):
+        return getattr(v, "type", "raw")
+    return "raw"
+
+
+DataItemType = Annotated[
+    (
+        Annotated[Inline, Tag("inline")]
+        | Annotated[InlineBase64, Tag("inline_b64")]
+        | Annotated[Stream, Tag("stream")]
+        | Annotated[InlineRaw, Tag("raw")]
+    ),
+    Discriminator(get_discriminator_value),
+]
+
+DataItem = TypeAdapter(DataItemType)

blissdata-2.1.0/src/blissdata/h5map/hdf5_items.py

@@ -0,0 +1,164 @@
+from pydantic import (
+    BaseModel,
+    Field,
+    NonNegativeInt,
+    model_serializer,
+    model_validator,
+    field_validator,
+    Discriminator,
+    Tag,
+    TypeAdapter,
+)
+from typing import Any, Annotated, Literal
+from .data_items import DataItemType, _BinaryDType
+
+
+class Dataset(BaseModel):
+    type: Literal["dataset"] = "dataset"
+    attributes: dict[str, DataItemType] = Field(default_factory=dict)
+    value: DataItemType
+
+    @model_serializer(mode="wrap")
+    def serialize_value_only(self, _next):
+        if self.attributes:
+            return _next(self)
+        else:
+            return _next(self)["value"]
+
+    @model_validator(mode="before")
+    @classmethod
+    def validate_value_only(cls, data: Any) -> Any:
+        if isinstance(data, dict) and data.get("type", "dataset") == "dataset":
+            return data
+        else:
+            return {"type": "dataset", "value": data}
+
+
+class ExternalBinaryFile(BaseModel):
+    name: str
+    offset: NonNegativeInt
+    size: NonNegativeInt | None = None  # None means H5F_UNLIMITED
+
+
+class ExternalBinaryDataset(BaseModel):
+    type: Literal["external_binary_dataset"] = "external_binary_dataset"
+    attributes: dict[str, DataItemType] = Field(default_factory=dict)
+    dtype: _BinaryDType
+    shape: list[int]
+    files: list[ExternalBinaryFile] = Field(min_length=1)
+
+    @model_serializer(mode="wrap")
+    def serialize(self, _next):
+        serialized = _next(self)
+        if not self.attributes:
+            serialized.pop("attributes", None)
+        return serialized
+
+
+class SpaceSelection(BaseModel):
+    start: list[NonNegativeInt]
+    stride: list[NonNegativeInt] | None = None
+    count: list[NonNegativeInt]
+    block: list[NonNegativeInt] | None = None
+
+    @field_validator("stride", mode="after")
+    @classmethod
+    def hide_all_ones_stride(cls, value):
+        if value is None or all(x == 1 for x in value):
+            return None
+        else:
+            return value
+
+    @field_validator("block", mode="after")
+    @classmethod
+    def hide_all_ones_block(cls, value):
+        if value is None or all(x == 1 for x in value):
+            return None
+        else:
+            return value
+
+
+class VirtualSource(BaseModel):
+    vspace: SpaceSelection
+    src_file: str
+    src_dataset: str
+    src_space: SpaceSelection
+
+
+class VirtualDataset(BaseModel):
+    type: Literal["virtual_dataset"] = "virtual_dataset"
+    attributes: dict[str, DataItemType] = Field(default_factory=dict)
+    dtype: _BinaryDType
+    shape: list[int]
+    virtual_sources: list[VirtualSource] = Field(min_length=1)
+
+    @model_serializer(mode="wrap")
+    def serialize(self, _next):
+        serialized = _next(self)
+        if not self.attributes:
+            serialized.pop("attributes", None)
+        return serialized
+
+
+class SoftLink(BaseModel):
+    type: Literal["soft_link"] = "soft_link"
+    target_path: str
+
+
+class ExternalLink(BaseModel):
+    type: Literal["external_link"] = "external_link"
+    target_file: str
+    target_path: str
+
+
+def get_discriminator_value(v: Any) -> str:
+    h5_type = {
+        "group",
+        "dataset",
+        "external_binary_dataset",
+        "soft_link",
+        "external_link",
+        "virtual_dataset",
+    }
+    # A dataset can be represented by its value only, therefore data_item types
+    # such as "inline" or "stream" may be end up here. Because of this we check
+    # wether or not type is an h5_type, otherwise it is a dataset.
+    if isinstance(v, dict):
+        type = v.get("type")
+        return type if type in h5_type else "dataset"
+    elif isinstance(v, BaseModel):
+        type = getattr(v, "type", None)
+        return type if type in h5_type else "dataset"
+    else:
+        return "dataset"
+
+
+HDF5ItemType = Annotated[
+    (
+        Annotated["Group", Tag("group")]
+        | Annotated[Dataset, Tag("dataset")]
+        | Annotated[ExternalBinaryDataset, Tag("external_binary_dataset")]
+        | Annotated[SoftLink, Tag("soft_link")]
+        | Annotated[ExternalLink, Tag("external_link")]
+        | Annotated[VirtualDataset, Tag("virtual_dataset")]
+    ),
+    Discriminator(get_discriminator_value),
+]
+
+
+class Group(BaseModel):
+    type: Literal["group"] = "group"
+    attributes: dict[str, DataItemType] = Field(default_factory=dict)
+    children: dict[str, HDF5ItemType] = Field(default_factory=dict)
+
+    @model_serializer(mode="wrap")
+    def skip_empty_fields(self, _next):
+        serialized = _next(self)
+        if not self.attributes:
+            serialized.pop("attributes", None)
+        if not self.children:
+            serialized.pop("children", None)
+        return serialized
+
+
+HDF5Item = TypeAdapter(HDF5ItemType)