pelutils 3.8.3__tar.gz → 3.8.5__tar.gz

{pelutils-3.8.3 → pelutils-3.8.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pelutils
-Version: 3.8.3
+Version: 3.8.5
 Summary: The Swiss army knife of Python projects.
 Home-page: https://github.com/peleiden/pelutils
 Download-URL: https://pypi.org/project/pelutils/
@@ -90,15 +90,17 @@ while True:
 
 ## Data Serialisation
 
-The `DataStorage2` class is an extension of `pydantic.BaseModel` that incluces save and load functionality.
-It supports any data type, storing all data to a pretty JSON file. A class should simply inherit from
-`DataStorage2`. The stored JSON will be a dictionary-like structure with the whole nested structure.
-Any nested `BaseModel` is automatically converted to a dictionary. If a data type is reached which is not
-JSON serialisable (e.g. normal classes, pandas DataFramas, numpy arrays), they are encoded with `pickle`
-and `base64`. Inside the JSON file, a record is kept of the original type.
+The `DataStorage2` class is an extension of `pydantic.BaseModel` which adds convenient `safe` and `load` methods for trivial saving to and loading from JSON files.
+It can serialise *any* attribute on the model to JSON by pickling types which `pydantic` cannot natively serialise.
+To get use it, simply inherit from `DataStorage2`, like you would inherit from `BaseModel`.
 
-Very long lists utilise the full allowed line length before splitting, preventing the common JSON issue of
-having either very long lines (no indents) or excessively many lines with a single element on each.
+The produced JSON files take advantage of `pelutils.pretty_json(...)` to ensure good and human-readable formatting, almost no matter the data types.
+Very long lists utilise the full allowed line length before splitting.
+This prevents the common JSON issue of having either very long lines (no indents) or excessively many lines with a single, small element on each.
+
+It is possible to get only the serialised JSON objects as dicts instead of saving them directly to a file with the `DataStorage2.model_safe_dump()` method.
+This is practical if you want to nest the objects into another dict or list.
+`DataStorage2.model_safe_load(...)` does the inverse operation by creating a class instance from a JSON-serialised dict.
 
 Because the class inherits from `pydantic.BaseModel`, type checking is built directly into it.
 
@@ -154,7 +156,6 @@ This will produce `directory/to/save/in/BasedClass.json` with the following cont
   "baseder": {"based_string": "Hello there"}
 }
 ```
-The class is built on top of the also provided `pretty_json` which does the exact same as the built-in `json.dumps` but with prettier formatting of the JSON string.
 
 ## Config and Command-line Argument Parsing
 
@@ -274,7 +275,7 @@ log2.configure("path/to/save/log2.log")
 
 A few different numpy types are defined for the convenience of not having to remember what data types your arrays are - and also to satisfy your nasty type checkers.
 ```py
-from pelutils.types import AnyArray, FloatArray, IntArray
+from pelutils.types import AnyArray, BoolArray, ComplexArray, FloatArray, IntArray
 
 
 def function_which_takes_np_types(

{pelutils-3.8.3 → pelutils-3.8.5}/README.md

@@ -65,15 +65,17 @@ while True:
 
 ## Data Serialisation
 
-The `DataStorage2` class is an extension of `pydantic.BaseModel` that incluces save and load functionality.
-It supports any data type, storing all data to a pretty JSON file. A class should simply inherit from
-`DataStorage2`. The stored JSON will be a dictionary-like structure with the whole nested structure.
-Any nested `BaseModel` is automatically converted to a dictionary. If a data type is reached which is not
-JSON serialisable (e.g. normal classes, pandas DataFramas, numpy arrays), they are encoded with `pickle`
-and `base64`. Inside the JSON file, a record is kept of the original type.
+The `DataStorage2` class is an extension of `pydantic.BaseModel` which adds convenient `safe` and `load` methods for trivial saving to and loading from JSON files.
+It can serialise *any* attribute on the model to JSON by pickling types which `pydantic` cannot natively serialise.
+To get use it, simply inherit from `DataStorage2`, like you would inherit from `BaseModel`.
 
-Very long lists utilise the full allowed line length before splitting, preventing the common JSON issue of
-having either very long lines (no indents) or excessively many lines with a single element on each.
+The produced JSON files take advantage of `pelutils.pretty_json(...)` to ensure good and human-readable formatting, almost no matter the data types.
+Very long lists utilise the full allowed line length before splitting.
+This prevents the common JSON issue of having either very long lines (no indents) or excessively many lines with a single, small element on each.
+
+It is possible to get only the serialised JSON objects as dicts instead of saving them directly to a file with the `DataStorage2.model_safe_dump()` method.
+This is practical if you want to nest the objects into another dict or list.
+`DataStorage2.model_safe_load(...)` does the inverse operation by creating a class instance from a JSON-serialised dict.
 
 Because the class inherits from `pydantic.BaseModel`, type checking is built directly into it.
 
@@ -129,7 +131,6 @@ This will produce `directory/to/save/in/BasedClass.json` with the following cont
   "baseder": {"based_string": "Hello there"}
 }
 ```
-The class is built on top of the also provided `pretty_json` which does the exact same as the built-in `json.dumps` but with prettier formatting of the JSON string.
 
 ## Config and Command-line Argument Parsing
 
@@ -249,7 +250,7 @@ log2.configure("path/to/save/log2.log")
 
 A few different numpy types are defined for the convenience of not having to remember what data types your arrays are - and also to satisfy your nasty type checkers.
 ```py
-from pelutils.types import AnyArray, FloatArray, IntArray
+from pelutils.types import AnyArray, BoolArray, ComplexArray, FloatArray, IntArray
 
 
 def function_which_takes_np_types(

{pelutils-3.8.3 → pelutils-3.8.5}/pelutils/__init__.py

@@ -24,7 +24,6 @@ import numpy as np
 import psutil
 
 try:
-    # torch is only to be installed if pelutils[ds] has been installed
     import torch
 
     _has_torch = True

{pelutils-3.8.3 → pelutils-3.8.5}/pelutils/__version__.py

@@ -1,2 +1,2 @@
 # Do not put anything else in this file
-__version__ = "3.8.3"
+__version__ = "3.8.5"

{pelutils-3.8.3 → pelutils-3.8.5}/pelutils/datastorage2/__init__.py

@@ -2,7 +2,7 @@ from __future__ import annotations
 
 import json
 from pathlib import Path
-from typing import TypeVar
+from typing import Any, TypeVar
 
 from pydantic import BaseModel, ConfigDict
 
@@ -24,6 +24,19 @@ class DataStorage2(BaseModel):
 
     model_config = ConfigDict(arbitrary_types_allowed=True)
 
+    def model_safe_dump(self) -> dict[str, Any]:  # pyright: ignore[reportExplicitAny]
+        """Dump the model as a dictionary where pickled objects have been converted to b64 strings.
+
+        This is very similar to `self.model_dump(mode="json")` but it handles non-JSON-serialisable types.
+        """
+        return self.model_dump(mode="json", fallback=_pickle_encode)
+
+    @classmethod
+    def model_safe_load(cls: type[_T], safe_json: dict[str, Any]) -> _T:  # pyright: ignore[reportExplicitAny]
+        """Build an instance from a safe JSON. This method is the inverse of `model_safe_dump`."""
+        self_dict = _make_json_unsafe(safe_json)
+        return cls.model_validate(self_dict)
+
     @classmethod
     def _resolve_save_file(cls, directory: str | Path, filename: str | None = None) -> Path:
         if filename is None:
@@ -39,10 +52,9 @@ class DataStorage2(BaseModel):
         """
         savepath = self._resolve_save_file(directory, filename)
         savepath.parent.mkdir(parents=True, exist_ok=True)
-        self_dict = self.model_dump(mode="json", fallback=_pickle_encode)
         savepath.write_text(
             _pretty_json(
-                self_dict,
+                self.model_safe_dump(),
                 max_line_length=max_line_length,
                 indent=indent,
                 # Safe should not be needed here, as the fallback function in model_dump should ensure no issues
@@ -59,5 +71,4 @@ class DataStorage2(BaseModel):
         savepath = cls._resolve_save_file(directory, filename)
         with savepath.open(encoding=encoding) as f:
             self_dict = json.load(f)
-        self_dict = _make_json_unsafe(self_dict)
-        return cls.model_validate(self_dict)
+        return cls.model_safe_load(self_dict)

{pelutils-3.8.3 → pelutils-3.8.5}/pelutils/datastorage2/_pretty_json.py

@@ -9,6 +9,13 @@ import json
 import pickle
 from typing import Any
 
+try:
+    import torch
+
+    _has_torch = True
+except ModuleNotFoundError:
+    _has_torch = False
+
 # Sentinel prefix so consumers can detect pickled blobs.
 _PICKLE_PREFIX = "__pickled_b64__"
 
@@ -45,6 +52,10 @@ def _get_qualified_type_name(obj: object) -> str:
 
 def _pickle_encode(value: object) -> str:
     """Pickle *value*, base64-encode the bytes, and return a prefixed string."""
+    if _has_torch and isinstance(value, torch.Tensor):  # pyright: ignore[reportPossiblyUnboundVariable]
+        # If a view of torch Tensor is given, the whole tensor is pickled, not just the view
+        # Therefore, it is cloned first to ensure that only the view itself is pickled
+        value = value.clone()
     raw = pickle.dumps(value, protocol=pickle.HIGHEST_PROTOCOL)
     encoded = base64.b64encode(raw).decode("ascii")
     return f"{_PICKLE_PREFIX}:{_get_qualified_type_name(value)}:{encoded}"

{pelutils-3.8.3 → pelutils-3.8.5}/pelutils/types.py

@@ -5,5 +5,7 @@ import numpy.typing as npt
 
 # Common numpy types
 AnyArray = npt.NDArray[Any]  # pyright: ignore[reportExplicitAny]
+BoolArray = npt.NDArray[np.bool_]
+ComplexArray = npt.NDArray[np.complexfloating[Any]]  # pyright: ignore[reportExplicitAny, reportInvalidTypeArguments]
 FloatArray = npt.NDArray[np.floating[Any]]  # pyright: ignore[reportExplicitAny]
 IntArray = npt.NDArray[np.integer[Any]]  # pyright: ignore[reportExplicitAny]

{pelutils-3.8.3 → pelutils-3.8.5}/pelutils.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pelutils
-Version: 3.8.3
+Version: 3.8.5
 Summary: The Swiss army knife of Python projects.
 Home-page: https://github.com/peleiden/pelutils
 Download-URL: https://pypi.org/project/pelutils/
@@ -90,15 +90,17 @@ while True:
 
 ## Data Serialisation
 
-The `DataStorage2` class is an extension of `pydantic.BaseModel` that incluces save and load functionality.
-It supports any data type, storing all data to a pretty JSON file. A class should simply inherit from
-`DataStorage2`. The stored JSON will be a dictionary-like structure with the whole nested structure.
-Any nested `BaseModel` is automatically converted to a dictionary. If a data type is reached which is not
-JSON serialisable (e.g. normal classes, pandas DataFramas, numpy arrays), they are encoded with `pickle`
-and `base64`. Inside the JSON file, a record is kept of the original type.
+The `DataStorage2` class is an extension of `pydantic.BaseModel` which adds convenient `safe` and `load` methods for trivial saving to and loading from JSON files.
+It can serialise *any* attribute on the model to JSON by pickling types which `pydantic` cannot natively serialise.
+To get use it, simply inherit from `DataStorage2`, like you would inherit from `BaseModel`.
 
-Very long lists utilise the full allowed line length before splitting, preventing the common JSON issue of
-having either very long lines (no indents) or excessively many lines with a single element on each.
+The produced JSON files take advantage of `pelutils.pretty_json(...)` to ensure good and human-readable formatting, almost no matter the data types.
+Very long lists utilise the full allowed line length before splitting.
+This prevents the common JSON issue of having either very long lines (no indents) or excessively many lines with a single, small element on each.
+
+It is possible to get only the serialised JSON objects as dicts instead of saving them directly to a file with the `DataStorage2.model_safe_dump()` method.
+This is practical if you want to nest the objects into another dict or list.
+`DataStorage2.model_safe_load(...)` does the inverse operation by creating a class instance from a JSON-serialised dict.
 
 Because the class inherits from `pydantic.BaseModel`, type checking is built directly into it.
 
@@ -154,7 +156,6 @@ This will produce `directory/to/save/in/BasedClass.json` with the following cont
   "baseder": {"based_string": "Hello there"}
 }
 ```
-The class is built on top of the also provided `pretty_json` which does the exact same as the built-in `json.dumps` but with prettier formatting of the JSON string.
 
 ## Config and Command-line Argument Parsing
 
@@ -274,7 +275,7 @@ log2.configure("path/to/save/log2.log")
 
 A few different numpy types are defined for the convenience of not having to remember what data types your arrays are - and also to satisfy your nasty type checkers.
 ```py
-from pelutils.types import AnyArray, FloatArray, IntArray
+from pelutils.types import AnyArray, BoolArray, ComplexArray, FloatArray, IntArray
 
 
 def function_which_takes_np_types(

{pelutils-3.8.3 → pelutils-3.8.5}/tests/test_types.py

@@ -1,14 +1,16 @@
 import numpy as np
 
-from pelutils.types import AnyArray, FloatArray, IntArray
+from pelutils import AnyArray, BoolArray, ComplexArray, FloatArray, IntArray
 
 
 def function_which_takes_np_types(
     any_array: AnyArray,
+    bool_array: BoolArray,
+    complex_array: ComplexArray,
     float_array: FloatArray,
     int_array: IntArray,
-) -> FloatArray:
-    return any_array + float_array + int_array
+) -> ComplexArray:
+    return any_array + bool_array + complex_array + float_array + int_array
 
 
 def test_types():
@@ -17,6 +19,8 @@ def test_types():
     Just make sure that using the types doesn't break anything silly like function declarations.
     """
     any_array = np.arange(5, dtype=np.float16)
+    bool_array = np.arange(5).astype(bool)
+    complex_array = 1j * np.arange(5)
     float_array = np.arange(5, dtype=np.float32)
     int_array = np.arange(5)
-    function_which_takes_np_types(any_array, float_array, int_array)
+    function_which_takes_np_types(any_array, bool_array, complex_array, float_array, int_array)