persidict 0.4.1__tar.gz → 0.5.0__tar.gz

{persidict-0.4.1 → persidict-0.5.0}/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2020-2023  Vlad (Volodymyr) Pavlov, Illia Shestakov, Kai Zhao.
+Copyright (c) 2020-2024  Vlad (Volodymyr) Pavlov, Illia Shestakov, Kai Zhao.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

{persidict-0.4.1 → persidict-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: persidict
-Version: 0.4.1
+Version: 0.5.0
 Summary: Simple persistent key-value store for Python. Values are stored as files on a disk or as S3 objects on AWS cloud.
 Home-page: https://github.com/vladlpavlov/persidict
 Author: Vlad (Volodymyr) Pavlov
@@ -149,6 +149,7 @@ that simultaneously work with the same instance of a dictionary.
 
 * Keys must be sequences of URL/filename-safe non-empty strings.
 * Values must be pickleable Python objects.
+* You can constrain values to be an instance of a specific class.
 * Insertion order is not preserved.
 * You can not assign initial key-value pairs to a dictionary in its constructor.
 * `PersiDict` API has additional methods `delete_if_exists()`, `mtimestamp()`,
@@ -160,9 +161,18 @@ not available in Python dicts.
 `PersiDict` subclasses have a number of parameters that can be used 
 to impact behaviour of a dictionary. 
 
+* `base_class_for_values` - A base class for values stored in a dictionary.  
+If specified, it will be used to check types of values in the dictionary. 
+If not specified (if set to `None`), no type checking will be performed 
+and all types will be allowed.
 * `file_type` - a string that specifies the type of files used to store objects.
-Possible values are "json" and "lz4". Default value is "lz4". 
-Storing objects as JSON files is mostly supported for debugging purposes.
+If `file_type` has one of two values: "lz4" or "json", it defines 
+which file format will be used by the dictionary to store values. 
+For all other values of `file_type`, the file format will always be plain
+text. "lz4" or "json" allow to store arbitrary Python objects,
+while all other file_type-s only work with str objects; 
+it means `base_class_for_values` must be explicitly set to `str` 
+if `file_type` is not set to "lz4" or "json".
 * `immutable_items` - a boolean that specifies whether items in a dictionary 
 can be modified/deleted. It enables various distributed cache optimizations 
 for remote storage. True means an append-only dictionary. 
@@ -189,6 +199,7 @@ Binary installers for the latest released version are available at the Python pa
 
 * [jsonpickle](https://jsonpickle.github.io)
 * [joblib](https://joblib.readthedocs.io)
+* [lz4](https://python-lz4.readthedocs.io)
 * [pandas](https://pandas.pydata.org)
 * [numpy](https://numpy.org)
 * [boto3](https://boto3.readthedocs.io)

{persidict-0.4.1 → persidict-0.5.0}/README.md

@@ -120,6 +120,7 @@ that simultaneously work with the same instance of a dictionary.
 
 * Keys must be sequences of URL/filename-safe non-empty strings.
 * Values must be pickleable Python objects.
+* You can constrain values to be an instance of a specific class.
 * Insertion order is not preserved.
 * You can not assign initial key-value pairs to a dictionary in its constructor.
 * `PersiDict` API has additional methods `delete_if_exists()`, `mtimestamp()`,
@@ -131,9 +132,18 @@ not available in Python dicts.
 `PersiDict` subclasses have a number of parameters that can be used 
 to impact behaviour of a dictionary. 
 
+* `base_class_for_values` - A base class for values stored in a dictionary.  
+If specified, it will be used to check types of values in the dictionary. 
+If not specified (if set to `None`), no type checking will be performed 
+and all types will be allowed.
 * `file_type` - a string that specifies the type of files used to store objects.
-Possible values are "json" and "lz4". Default value is "lz4". 
-Storing objects as JSON files is mostly supported for debugging purposes.
+If `file_type` has one of two values: "lz4" or "json", it defines 
+which file format will be used by the dictionary to store values. 
+For all other values of `file_type`, the file format will always be plain
+text. "lz4" or "json" allow to store arbitrary Python objects,
+while all other file_type-s only work with str objects; 
+it means `base_class_for_values` must be explicitly set to `str` 
+if `file_type` is not set to "lz4" or "json".
 * `immutable_items` - a boolean that specifies whether items in a dictionary 
 can be modified/deleted. It enables various distributed cache optimizations 
 for remote storage. True means an append-only dictionary. 
@@ -160,6 +170,7 @@ Binary installers for the latest released version are available at the Python pa
 
 * [jsonpickle](https://jsonpickle.github.io)
 * [joblib](https://joblib.readthedocs.io)
+* [lz4](https://python-lz4.readthedocs.io)
 * [pandas](https://pandas.pydata.org)
 * [numpy](https://numpy.org)
 * [boto3](https://boto3.readthedocs.io)

{persidict-0.4.1 → persidict-0.5.0}/persidict/file_dir_dict.py

@@ -3,13 +3,14 @@
 This functionality is implemented by the class FileDirDict
 (inherited from PersiDict): a dictionary that
 stores key-value pairs as files on a local hard-drive.
-A key is used to compose a filename, while a value is stored
-as a binary or a json object in the file.
+A key is used to compose a filename, while a value is stored in the file
+as a binary, or as a json object, or as a plain text
+(depends on configuration parameters).
 """
 from __future__ import annotations
 
 import os
-from typing import Any
+from typing import Any, Optional, Union
 
 import joblib
 import jsonpickle
@@ -33,31 +34,43 @@ class FileDirDict(PersiDict):
     Insertion order is not preserved.
 
     FileDirDict can store objects in binary files or in human-readable
-    text files (using jsonpickles).
+    text files (either in jason format or as a plain text).
     """
 
     def __init__(self
                  , dir_name: str = "FileDirDict"
                  , file_type: str = "lz4"
                  , immutable_items:bool = False
-                 , digest_len:int = 8):
+                 , digest_len:int = 8
+                 , base_class_for_values: Optional[str] = None):
         """A constructor defines location of the store and file format to use.
 
         dir_name is a directory that will contain all the files in
         the FileDirDict. If the directory does not exist, it will be created.
 
-        file_type can take one of two values: "lz4" or "json".
-        It defines which file format will be used by FileDirDict
-        to store values.
+        base_class_for_values constraints the type of values that can be
+        stored in the dictionary. If specified, it will be used to
+        check types of values in the dictionary. If not specified,
+        no type checking will be performed and all types will be allowed.
+
+        file_type is extension, which will be used for all files in the dictionary.
+        If file_type has one of two values: "lz4" or "json", it defines
+        which file format will be used by FileDirDict to store values.
+        For all other values of file_type, the file format will always be plain
+        text. "lz4" or "json" allow to store arbitrary Python objects,
+        while all other file_type-s only work with str objects.
         """
 
         super().__init__(immutable_items = immutable_items
-                ,digest_len = digest_len)
+                ,digest_len = digest_len
+                ,base_class_for_values = base_class_for_values)
 
         self.file_type = file_type
 
-        assert file_type in {"json", "lz4"}, (
-            "file_type must be either lz4 or json")
+        if (base_class_for_values is None or
+                not issubclass(base_class_for_values,str)):
+            assert file_type in {"json", "lz4"}, ("For non-string values "
+                + "file_type must be either lz4 or json")
         assert not os.path.isfile(dir_name)
         if not os.path.isdir(dir_name):
             os.mkdir(dir_name)
@@ -80,10 +93,11 @@ class FileDirDict(PersiDict):
         """ Get number of key-value pairs in the dictionary."""
 
         num_files = 0
+        suffix = "." + self.file_type
         for subdir_info in os.walk(self.base_dir):
             files = subdir_info[2]
             files = [f_name for f_name in files
-                     if f_name.endswith(self.file_type)]
+                     if f_name.endswith(suffix)]
             num_files += len(files)
         return num_files
 
@@ -95,8 +109,9 @@ class FileDirDict(PersiDict):
 
         for subdir_info in os.walk(self.base_dir, topdown=False):
             (subdir_name, _, files) = subdir_info
+            suffix = "." + self.file_type
             for f in files:
-                if f.endswith(self.file_type):
+                if f.endswith(suffix):
                     os.remove(os.path.join(subdir_name, f))
             if (subdir_name != self.base_dir) and (
                     len(os.listdir(subdir_name)) == 0 ):
@@ -149,8 +164,12 @@ class FileDirDict(PersiDict):
         elif self.file_type == "json":
             with open(file_name, 'r') as f:
                 result = jsonpickle.loads(f.read())
+        elif issubclass(self.base_class_for_values, str):
+            with open(file_name, 'r') as f:
+                result = f.read()
         else:
-            raise ValueError("file_type must be either lz4 or json")
+            raise ValueError("When base_class_for_values is not str,"
+                             + " file_type must be either lz4 or json")
         return result
 
     def _save_to_file(self, file_name:str, value:Any) -> None:
@@ -162,8 +181,12 @@ class FileDirDict(PersiDict):
         elif self.file_type == "json":
             with open(file_name, 'w') as f:
                 f.write(jsonpickle.dumps(value, indent=4))
+        elif issubclass(self.base_class_for_values, str):
+            with open(file_name, 'w') as f:
+                f.write(value)
         else:
-            raise ValueError("file_type must be either lz4 or json")
+            raise ValueError("When base_class_for_values is not str,"
+                            + " file_type must be either lz4 or json")
 
     def __contains__(self, key:PersiDictKey) -> bool:
         """True if the dictionary has the specified key, else False. """
@@ -182,6 +205,11 @@ class FileDirDict(PersiDict):
 
     def __setitem__(self, key:PersiDictKey, value:Any):
         """Set self[key] to value."""
+        if self.base_class_for_values is not None:
+            if not isinstance(value, self.base_class_for_values):
+                raise TypeError(
+                    f"Value must be of type {self.base_class_for_values}")
+
         key = SafeStrTuple(key)
         filename = self._build_full_path(key, create_subdirs=True)
         if self.immutable_items:
@@ -218,9 +246,10 @@ class FileDirDict(PersiDict):
             return tuple(result)
 
         def step():
+            suffix = "." + self.file_type
             for dir_name, _, files in walk_results:
                 for f in files:
-                    if f.endswith(self.file_type):
+                    if f.endswith(suffix):
                         prefix_key = os.path.relpath(
                             dir_name, start=self.base_dir)
 

{persidict-0.4.1 → persidict-0.5.0}/persidict/persi_dict.py

@@ -22,7 +22,7 @@ from __future__ import annotations
 
 from abc import abstractmethod
 import random
-from typing import Any, Dict, Union, Sequence
+from typing import Any, Dict, Union, Sequence, Optional
 from collections.abc import MutableMapping
 
 from .safe_str_tuple import SafeStrTuple
@@ -42,7 +42,8 @@ class PersiDict(MutableMapping):
 
     An abstract base class for key-value stores. It accepts keys in a form of
     SafeStrSequence - a URL/filename-safe sequence of strings.
-    It imposes no restrictions on types of values in the key-value pairs.
+    It assumes no restrictions on types of values in the key-value pairs,
+    but allows users to impose such restrictions.
 
     The API for the class resembles the API of Python's built-in Dict
     (see https://docs.python.org/3/library/stdtypes.html#mapping-types-dict)
@@ -68,6 +69,12 @@ class PersiDict(MutableMapping):
                  of persistent dictionaries with case-insensitive
                  (even if case-preserving) filesystems, such as MacOS HFS.
 
+    base_class_for_values: Any
+                    A base class for values stored in the dictionary.
+                    If specified, it will be used to check types of values
+                    in the dictionary. If not specified, no type checking
+                    will be performed and all types will be allowed.
+
     """
     # TODO: refactor to support variable length of min_digest_len
     digest_len:int
@@ -76,10 +83,12 @@ class PersiDict(MutableMapping):
     def __init__(self
                  , immutable_items:bool
                  , digest_len:int = 8
+                 , base_class_for_values:Any = None
                  , *args, **kwargas):
         assert digest_len >= 0
         self.digest_len = int(digest_len)
         self.immutable_items = bool(immutable_items)
+        self.base_class_for_values = base_class_for_values
 
 
     def __repr__(self):
@@ -88,6 +97,7 @@ class PersiDict(MutableMapping):
         repr_str += repr(dict(self.items()))
         repr_str += f", immutable_items={self.immutable_items}"
         repr_str += f", digest_len={self.digest_len}"
+        repr_str += f", base_class_for_values={self.base_class_for_values}"
         repr_str += ")"
         return repr_str
 

{persidict-0.4.1 → persidict-0.5.0}/persidict/s3_dict.py

@@ -35,6 +35,7 @@ class S3Dict(PersiDict):
                  , file_type:str = "pkl"
                  , immutable_items:bool = False
                  , digest_len:int = 8
+                 , base_class_for_values:Any = None
                  ,*args ,**kwargs):
         """A constructor defines location of the store and object format to use.
 
@@ -46,9 +47,17 @@ class S3Dict(PersiDict):
 
         dir_name is a local directory that will be used to store tmp files.
 
-        file_type can take one of two values: "pkl" or "json".
-        It defines which object format will be used by S3_Dict
-        to store values.
+        base_class_for_values constraints the type of values that can be
+        stored in the dictionary. If specified, it will be used to
+        check types of values in the dictionary. If not specified,
+        no type checking will be performed and all types will be allowed.
+
+        file_type is extension, which will be used for all files in the dictionary.
+        If file_type has one of two values: "lz4" or "json", it defines
+        which file format will be used by FileDirDict to store values.
+        For all other values of file_type, the file format will always be plain
+        text. "lz4" or "json" allow to store arbitrary Python objects,
+        while all other file_type-s only work with str objects.
         """
 
         super().__init__(immutable_items = immutable_items, digest_len = 0)
@@ -58,6 +67,7 @@ class S3Dict(PersiDict):
             dir_name = dir_name
             , file_type = file_type
             , immutable_items = immutable_items
+            , base_class_for_values=base_class_for_values
             , digest_len = digest_len)
 
         self.region = region
@@ -137,6 +147,11 @@ class S3Dict(PersiDict):
     def __setitem__(self, key:PersiDictKey, value:Any):
         """Set self[key] to value. """
 
+        if self.base_class_for_values is not None:
+            if not isinstance(value, self.base_class_for_values):
+                raise TypeError(
+                    f"Value must be of type {self.base_class_for_values}")
+
         key = SafeStrTuple(key)
         file_name = self.local_cache._build_full_path(key, create_subdirs=True)
         obj_name = self._build_full_objectname(key)

{persidict-0.4.1 → persidict-0.5.0}/persidict.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: persidict
-Version: 0.4.1
+Version: 0.5.0
 Summary: Simple persistent key-value store for Python. Values are stored as files on a disk or as S3 objects on AWS cloud.
 Home-page: https://github.com/vladlpavlov/persidict
 Author: Vlad (Volodymyr) Pavlov
@@ -149,6 +149,7 @@ that simultaneously work with the same instance of a dictionary.
 
 * Keys must be sequences of URL/filename-safe non-empty strings.
 * Values must be pickleable Python objects.
+* You can constrain values to be an instance of a specific class.
 * Insertion order is not preserved.
 * You can not assign initial key-value pairs to a dictionary in its constructor.
 * `PersiDict` API has additional methods `delete_if_exists()`, `mtimestamp()`,
@@ -160,9 +161,18 @@ not available in Python dicts.
 `PersiDict` subclasses have a number of parameters that can be used 
 to impact behaviour of a dictionary. 
 
+* `base_class_for_values` - A base class for values stored in a dictionary.  
+If specified, it will be used to check types of values in the dictionary. 
+If not specified (if set to `None`), no type checking will be performed 
+and all types will be allowed.
 * `file_type` - a string that specifies the type of files used to store objects.
-Possible values are "json" and "lz4". Default value is "lz4". 
-Storing objects as JSON files is mostly supported for debugging purposes.
+If `file_type` has one of two values: "lz4" or "json", it defines 
+which file format will be used by the dictionary to store values. 
+For all other values of `file_type`, the file format will always be plain
+text. "lz4" or "json" allow to store arbitrary Python objects,
+while all other file_type-s only work with str objects; 
+it means `base_class_for_values` must be explicitly set to `str` 
+if `file_type` is not set to "lz4" or "json".
 * `immutable_items` - a boolean that specifies whether items in a dictionary 
 can be modified/deleted. It enables various distributed cache optimizations 
 for remote storage. True means an append-only dictionary. 
@@ -189,6 +199,7 @@ Binary installers for the latest released version are available at the Python pa
 
 * [jsonpickle](https://jsonpickle.github.io)
 * [joblib](https://joblib.readthedocs.io)
+* [lz4](https://python-lz4.readthedocs.io)
 * [pandas](https://pandas.pydata.org)
 * [numpy](https://numpy.org)
 * [boto3](https://boto3.readthedocs.io)

{persidict-0.4.1 → persidict-0.5.0}/setup.py

@@ -5,7 +5,7 @@ with open("README.md", "r") as f:
 
 setuptools.setup(
     name="persidict"
-    ,version="0.4.1"
+    ,version="0.5.0"
     ,author="Vlad (Volodymyr) Pavlov"
     ,author_email="vlpavlov@ieee.org"
     ,description= "Simple persistent key-value store for Python. "