redis-dict 3.1.2__tar.gz → 3.2.1__tar.gz

{redis_dict-3.1.2/src/redis_dict.egg-info → redis_dict-3.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.2
 Name: redis-dict
-Version: 3.1.2
+Version: 3.2.1
 Summary: Dictionary with Redis as storage backend
 Author-email: Melvin Bijman <bijman.m.m@gmail.com>
 License: MIT
@@ -42,7 +42,7 @@ Requires-Dist: types-redis>=4.6.0; extra == "dev"
 Requires-Dist: typing_extensions>=4.5.0; extra == "dev"
 Requires-Dist: attrs==22.2.0; extra == "dev"
 Requires-Dist: cffi==1.15.1; extra == "dev"
-Requires-Dist: cryptography==43.0.1; extra == "dev"
+Requires-Dist: cryptography==44.0.1; extra == "dev"
 Requires-Dist: exceptiongroup==1.1.1; extra == "dev"
 Requires-Dist: future==0.18.3; extra == "dev"
 Requires-Dist: pycparser==2.21; extra == "dev"
@@ -122,6 +122,48 @@ In Redis our example looks like this.
 "str:hello world"
 ```
 
+## Types
+
+### standard types
+RedisDict supports a range of Python data types, from basic types to nested structures.
+Basic types are handled natively, while complex data types like lists and dictionaries, RedisDict uses JSON serialization, specifically avoiding [pickle](https://docs.python.org/3/library/pickle.html) due to its security vulnerabilities within distributed computing contexts.
+Although the library supports nested structures, the recommended best practice is to use RedisDict as a shallow dictionary.
+This approach optimizes Redis database performance and efficiency by ensuring that each set and get operation efficiently maps to Redis's key-value storage capabilities, while still preserving the library's Pythonic interface.
+Following types are supported: 
+`str, int, float, bool, NoneType, list, dict, tuple, set, datetime, date, time, timedelta, Decimal, complex, bytes, UUID, OrderedDict, defaultdict, frozenset`
+```python
+from uuid import UUID
+from decimal import Decimal
+from collections import OrderedDict, defaultdict
+from datetime import datetime, date, time, timedelta
+
+dic = RedisDict()
+
+dic["string"] = "Hello World"
+dic["number"] = 42
+dic["float"] = 3.14
+dic["bool"] = True
+dic["None"] = None
+
+dic["list"] = [1, 2, 3]
+dic["dict"] = {"a": 1, "b": 2}
+dic["tuple"] = (1, 2, 3)
+dic["set"] = {1, 2, 3}
+
+dic["datetime"] = datetime.date(2024, 1, 1, 12, 30, 45)
+dic["date"] = date(2024, 1, 1)
+dic["time"] = time(12, 30, 45)
+dic["delta"] = timedelta(days=1, hours=2)
+
+dic["decimal"] = Decimal("3.14159")
+dic["complex"] = complex(1, 2)
+dic["bytes"] = bytes([72, 101, 108, 108, 111])
+dic["uuid"] = UUID('12345678-1234-5678-1234-567812345678')
+
+dic["ordered"] = OrderedDict([('a', 1), ('b', 2)])
+dic["default"] = defaultdict(int, {'a': 1, 'b': 2})
+dic["frozen"] = frozenset([1, 2, 3])
+```
 
 ### Namespaces
 Acting as an identifier for your dictionary across different systems, RedisDict employs namespaces for organized data management. When a namespace isn't specified, "main" becomes the default. Thus allowing for data organization across systems and projects with the same redis instance.
@@ -290,51 +332,6 @@ print(dic["d"])  # Output: 4
 For more advanced examples of RedisDict, please refer to the unit-test files in the repository. All features and functionalities are thoroughly tested in [unit tests (here)](https://github.com/Attumm/redis-dict/blob/main/tests/unit/tests.py#L1) Or take a look at load test for batching [load test](https://github.com/Attumm/redis-dict/blob/main/tests/load/load_test.py#L1).
 The unit-tests can be as used as a starting point.
 
-## Types
-
-### standard types
-RedisDict supports a range of Python data types, from basic types to nested structures.
-Basic types are handled natively, while complex data types like lists and dictionaries, RedisDict uses JSON serialization, specifically avoiding `pickle` due to its [security vulnerabilities](https://docs.python.org/3/library/pickle.html) in distributed computing contexts.
-Although the library supports nested structures, the recommended best practice is to use RedisDict as a shallow dictionary.
-This approach optimizes Redis database performance and efficiency by ensuring that each set and get operation efficiently maps to Redis's key-value storage capabilities, while still preserving the library's Pythonic interface.
-Following types are supported: 
-`str, int, float, bool, NoneType, list, dict, tuple, set, datetime, date, time, timedelta, Decimal, complex, bytes, UUID, OrderedDict, defaultdict, frozenset`
-```python
-from uuid import UUID
-from decimal import Decimal
-from collections import OrderedDict, defaultdict
-from datetime import datetime, date, time, timedelta
-
-dic = RedisDict()
-
-dic["string"] = "Hello World"
-dic["number"] = 42
-dic["float"] = 3.14
-dic["bool"] = True
-dic["None"] = None
-
-dic["list"] = [1, 2, 3]
-dic["dict"] = {"a": 1, "b": 2}
-dic["tuple"] = (1, 2, 3)
-dic["set"] = {1, 2, 3}
-
-dic["datetime"] = datetime.date(2024, 1, 1, 12, 30, 45)
-dic["date"] = date(2024, 1, 1)
-dic["time"] = time(12, 30, 45)
-dic["delta"] = timedelta(days=1, hours=2)
-
-dic["decimal"] = Decimal("3.14159")
-dic["complex"] = complex(1, 2)
-dic["bytes"] = bytes([72, 101, 108, 108, 111])
-dic["uuid"] = UUID('12345678-1234-5678-1234-567812345678')
-
-dic["ordered"] = OrderedDict([('a', 1), ('b', 2)])
-dic["default"] = defaultdict(int, {'a': 1, 'b': 2})
-dic["frozen"] = frozenset([1, 2, 3])
-```
-
-
-
 ### Nested types
 Nested Types
 RedisDict supports nested structures with mixed types through JSON serialization. The feature works by utilizing JSON encoding and decoding under the hood. While this represents an upgrade in functionality, the feature is not fully implemented and should be used with caution. For optimal performance, using shallow dictionaries is recommended.
@@ -394,6 +391,21 @@ assert result.name == person.name
 assert result.age == person.age
 ```
 
+### Insertion Order
+For insertion order, use the PythonRedisDict. This class is focused on Python dictionary behavior one-to-one.
+It will eventually become a drop-in replacement for dictionary. Currently, nested types and typed keys are not yet supported but will be added in the future.
+
+```python
+from redis_dict import PythonRedisDict
+
+dic = PythonRedisDict()
+dic["1"] = "one"
+dic["2"] = "two"
+dic["3"] = "three"
+
+assert list(dic.keys()) == ["1", "2", "3"]
+```
+
 For more information on [extending types](https://github.com/Attumm/redis-dict/blob/main/tests/unit/extend_types_tests.py).
 ### Redis Encryption
 Setup guide for configuring and utilizing encrypted Redis TLS for redis-dict.
@@ -417,7 +429,7 @@ redis_config = {
     'port': 6380,
 }
 
-confid_dic = RedisDict(**redis_config)
+config_dic = RedisDict(**redis_config)
 ```
 
 ## Installation

{redis_dict-3.1.2 → redis_dict-3.2.1}/README.md

@@ -50,6 +50,48 @@ In Redis our example looks like this.
 "str:hello world"
 ```
 
+## Types
+
+### standard types
+RedisDict supports a range of Python data types, from basic types to nested structures.
+Basic types are handled natively, while complex data types like lists and dictionaries, RedisDict uses JSON serialization, specifically avoiding [pickle](https://docs.python.org/3/library/pickle.html) due to its security vulnerabilities within distributed computing contexts.
+Although the library supports nested structures, the recommended best practice is to use RedisDict as a shallow dictionary.
+This approach optimizes Redis database performance and efficiency by ensuring that each set and get operation efficiently maps to Redis's key-value storage capabilities, while still preserving the library's Pythonic interface.
+Following types are supported: 
+`str, int, float, bool, NoneType, list, dict, tuple, set, datetime, date, time, timedelta, Decimal, complex, bytes, UUID, OrderedDict, defaultdict, frozenset`
+```python
+from uuid import UUID
+from decimal import Decimal
+from collections import OrderedDict, defaultdict
+from datetime import datetime, date, time, timedelta
+
+dic = RedisDict()
+
+dic["string"] = "Hello World"
+dic["number"] = 42
+dic["float"] = 3.14
+dic["bool"] = True
+dic["None"] = None
+
+dic["list"] = [1, 2, 3]
+dic["dict"] = {"a": 1, "b": 2}
+dic["tuple"] = (1, 2, 3)
+dic["set"] = {1, 2, 3}
+
+dic["datetime"] = datetime.date(2024, 1, 1, 12, 30, 45)
+dic["date"] = date(2024, 1, 1)
+dic["time"] = time(12, 30, 45)
+dic["delta"] = timedelta(days=1, hours=2)
+
+dic["decimal"] = Decimal("3.14159")
+dic["complex"] = complex(1, 2)
+dic["bytes"] = bytes([72, 101, 108, 108, 111])
+dic["uuid"] = UUID('12345678-1234-5678-1234-567812345678')
+
+dic["ordered"] = OrderedDict([('a', 1), ('b', 2)])
+dic["default"] = defaultdict(int, {'a': 1, 'b': 2})
+dic["frozen"] = frozenset([1, 2, 3])
+```
 
 ### Namespaces
 Acting as an identifier for your dictionary across different systems, RedisDict employs namespaces for organized data management. When a namespace isn't specified, "main" becomes the default. Thus allowing for data organization across systems and projects with the same redis instance.
@@ -218,51 +260,6 @@ print(dic["d"])  # Output: 4
 For more advanced examples of RedisDict, please refer to the unit-test files in the repository. All features and functionalities are thoroughly tested in [unit tests (here)](https://github.com/Attumm/redis-dict/blob/main/tests/unit/tests.py#L1) Or take a look at load test for batching [load test](https://github.com/Attumm/redis-dict/blob/main/tests/load/load_test.py#L1).
 The unit-tests can be as used as a starting point.
 
-## Types
-
-### standard types
-RedisDict supports a range of Python data types, from basic types to nested structures.
-Basic types are handled natively, while complex data types like lists and dictionaries, RedisDict uses JSON serialization, specifically avoiding `pickle` due to its [security vulnerabilities](https://docs.python.org/3/library/pickle.html) in distributed computing contexts.
-Although the library supports nested structures, the recommended best practice is to use RedisDict as a shallow dictionary.
-This approach optimizes Redis database performance and efficiency by ensuring that each set and get operation efficiently maps to Redis's key-value storage capabilities, while still preserving the library's Pythonic interface.
-Following types are supported: 
-`str, int, float, bool, NoneType, list, dict, tuple, set, datetime, date, time, timedelta, Decimal, complex, bytes, UUID, OrderedDict, defaultdict, frozenset`
-```python
-from uuid import UUID
-from decimal import Decimal
-from collections import OrderedDict, defaultdict
-from datetime import datetime, date, time, timedelta
-
-dic = RedisDict()
-
-dic["string"] = "Hello World"
-dic["number"] = 42
-dic["float"] = 3.14
-dic["bool"] = True
-dic["None"] = None
-
-dic["list"] = [1, 2, 3]
-dic["dict"] = {"a": 1, "b": 2}
-dic["tuple"] = (1, 2, 3)
-dic["set"] = {1, 2, 3}
-
-dic["datetime"] = datetime.date(2024, 1, 1, 12, 30, 45)
-dic["date"] = date(2024, 1, 1)
-dic["time"] = time(12, 30, 45)
-dic["delta"] = timedelta(days=1, hours=2)
-
-dic["decimal"] = Decimal("3.14159")
-dic["complex"] = complex(1, 2)
-dic["bytes"] = bytes([72, 101, 108, 108, 111])
-dic["uuid"] = UUID('12345678-1234-5678-1234-567812345678')
-
-dic["ordered"] = OrderedDict([('a', 1), ('b', 2)])
-dic["default"] = defaultdict(int, {'a': 1, 'b': 2})
-dic["frozen"] = frozenset([1, 2, 3])
-```
-
-
-
 ### Nested types
 Nested Types
 RedisDict supports nested structures with mixed types through JSON serialization. The feature works by utilizing JSON encoding and decoding under the hood. While this represents an upgrade in functionality, the feature is not fully implemented and should be used with caution. For optimal performance, using shallow dictionaries is recommended.
@@ -322,6 +319,21 @@ assert result.name == person.name
 assert result.age == person.age
 ```
 
+### Insertion Order
+For insertion order, use the PythonRedisDict. This class is focused on Python dictionary behavior one-to-one.
+It will eventually become a drop-in replacement for dictionary. Currently, nested types and typed keys are not yet supported but will be added in the future.
+
+```python
+from redis_dict import PythonRedisDict
+
+dic = PythonRedisDict()
+dic["1"] = "one"
+dic["2"] = "two"
+dic["3"] = "three"
+
+assert list(dic.keys()) == ["1", "2", "3"]
+```
+
 For more information on [extending types](https://github.com/Attumm/redis-dict/blob/main/tests/unit/extend_types_tests.py).
 ### Redis Encryption
 Setup guide for configuring and utilizing encrypted Redis TLS for redis-dict.
@@ -345,7 +357,7 @@ redis_config = {
     'port': 6380,
 }
 
-confid_dic = RedisDict(**redis_config)
+config_dic = RedisDict(**redis_config)
 ```
 
 ## Installation

{redis_dict-3.1.2 → redis_dict-3.2.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "redis-dict"
-version = "3.1.2"
+version = "3.2.1"
 description = "Dictionary with Redis as storage backend"
 authors = [
     {name = "Melvin Bijman", email = "bijman.m.m@gmail.com"},
@@ -62,7 +62,7 @@ dev = [
 
     "attrs==22.2.0",
     "cffi==1.15.1",
-    "cryptography==43.0.1",
+    "cryptography==44.0.1",
     "exceptiongroup==1.1.1",
     "future==0.18.3",
     "pycparser==2.21",

{redis_dict-3.1.2 → redis_dict-3.2.1}/src/redis_dict/__init__.py

@@ -2,10 +2,12 @@
 from importlib.metadata import version, PackageNotFoundError
 
 from .core import RedisDict
+from .python_dict import PythonRedisDict
 from .type_management import decoding_registry, encoding_registry, RedisDictJSONEncoder, RedisDictJSONDecoder
 
 __all__ = [
     'RedisDict',
+    'PythonRedisDict',
     'decoding_registry',
     'encoding_registry',
     'RedisDictJSONEncoder',

{redis_dict-3.1.2 → redis_dict-3.2.1}/src/redis_dict/core.py

@@ -7,10 +7,10 @@ from collections.abc import Mapping
 
 from redis import StrictRedis
 
-from redis_dict.type_management import SENTINEL, EncodeFuncType, DecodeFuncType, EncodeType, DecodeType
-from redis_dict.type_management import _create_default_encode, _create_default_decode, _default_decoder
-from redis_dict.type_management import encoding_registry as enc_reg
-from redis_dict.type_management import decoding_registry as dec_reg
+from .type_management import SENTINEL, EncodeFuncType, DecodeFuncType, EncodeType, DecodeType
+from .type_management import _create_default_encode, _create_default_decode, _default_decoder
+from .type_management import encoding_registry as enc_reg
+from .type_management import decoding_registry as dec_reg
 
 
 # pylint: disable=R0902, R0904
@@ -40,12 +40,14 @@ class RedisDict:
     encoding_registry: EncodeType = enc_reg
     decoding_registry: DecodeType = dec_reg
 
+    # pylint: disable=R0913
     def __init__(self,
-                 namespace: str = 'main',
-                 expire: Union[int, timedelta, None] = None,
-                 preserve_expiration: Optional[bool] = False,
-                 redis: "Optional[StrictRedis[Any]]" = None,
-                 **redis_kwargs: Any) -> None:
+             namespace: str = 'main',
+             expire: Union[int, timedelta, None] = None,
+             preserve_expiration: Optional[bool] = False,
+             redis: "Optional[StrictRedis[Any]]" = None,
+             raise_key_error_delete: bool = False,
+             **redis_kwargs: Any) -> None:  # noqa: D202:R0913 pydocstyle clashes with Sphinx
         """
         Initialize a RedisDict instance.
 
@@ -56,12 +58,14 @@ class RedisDict:
             expire (Union[int, timedelta, None], optional): Expiration time for keys.
             preserve_expiration (Optional[bool], optional): Preserve expiration on key updates.
             redis (Optional[StrictRedis[Any]], optional): A Redis connection instance.
+            raise_key_error_delete (bool): Enable strict Python dict behavior raise if key not found when deleting.
             **redis_kwargs (Any): Additional kwargs for Redis connection if not provided.
         """
 
         self.namespace: str = namespace
         self.expire: Union[int, timedelta, None] = expire
         self.preserve_expiration: Optional[bool] = preserve_expiration
+        self.raise_key_error_delete: bool = raise_key_error_delete
         if redis:
             redis.connection_pool.connection_kwargs["decode_responses"] = True
 
@@ -74,6 +78,8 @@ class RedisDict:
         self._iter: Iterator[str] = iter([])
         self._max_string_size: int = 500 * 1024 * 1024  # 500mb
         self._temp_redis: Optional[StrictRedis[Any]] = None
+        self._insertion_order_key = f"redis-dict-insertion-order-{namespace}"
+        self._batch_size: int = 200
 
     def _format_key(self, key: str) -> str:
         """
@@ -85,9 +91,21 @@ class RedisDict:
         Returns:
             str: The formatted key with the namespace prefix.
         """
-        return f'{self.namespace}:{str(key)}'
+        return f'{self.namespace}:{key}'
 
-    def _valid_input(self, val: Any, val_type: str) -> bool:
+    def _parse_key(self, key: str) -> str:
+        """
+        Parse a formatted key with the namespace prefix and type.
+
+        Args:
+            key (str): The key to be parsed to type.
+
+        Returns:
+            str: The parsed key
+        """
+        return key[len(self.namespace) + 1:]
+
+    def _valid_input(self, value: Any) -> bool:
         """
         Check if the input value is valid based on the specified value type.
 
@@ -96,36 +114,35 @@ class RedisDict:
         length does not exceed the maximum allowed size (500 MB).
 
         Args:
-            val (Any): The input value to be validated.
-            val_type (str): The type of the input value ("str", "int", "float", or "bool").
+            value (Any): The input value to be validated.
 
         Returns:
             bool: True if the input value is valid, False otherwise.
         """
-        if val_type == "str":
-            return len(val) < self._max_string_size
+        store_type = type(value).__name__
+        if store_type == "str":
+            return len(value) < self._max_string_size
         return True
 
-    def _format_value(self, key: str, value: Any) -> str:
+    def _format_value(self, value: Any) -> str:
         """Format a valid value with the type and encoded representation of the value.
 
         Args:
-            key (str): The key of the value to be formatted.
             value (Any): The value to be encoded and formatted.
 
-        Raises:
-            ValueError: If the value or key fail validation.
-
         Returns:
             str: The formatted value with the type and encoded representation of the value.
         """
-        store_type, key = type(value).__name__, str(key)
-        if not self._valid_input(value, store_type) or not self._valid_input(key, "str"):
-            raise ValueError("Invalid input value or key size exceeded the maximum limit.")
+        store_type = type(value).__name__
         encoded_value = self.encoding_registry.get(store_type, lambda x: x)(value)  # type: ignore
-
         return f'{store_type}:{encoded_value}'
 
+    def _store_set(self, formatted_key: str, formatted_value: str) -> None:
+        if self.preserve_expiration and self.get_redis.exists(formatted_key):
+            self.redis.set(formatted_key, formatted_value, keepttl=True)
+        else:
+            self.redis.set(formatted_key, formatted_value, ex=self.expire)
+
     def _store(self, key: str, value: Any) -> None:
         """
         Store a value in Redis with the given key.
@@ -134,6 +151,9 @@ class RedisDict:
             key (str): The key to store the value.
             value (Any): The value to be stored.
 
+        Raises:
+            ValueError: If the value or key fail validation.
+
         Note: Validity checks could be refactored to allow for custom exceptions that inherit from ValueError,
         providing detailed information about why a specific validation failed.
         This would enable users to specify which validity checks should be executed, add custom validity functions,
@@ -142,12 +162,13 @@ class RedisDict:
         Allowing for simple dict set operation, but only cache data that makes sense.
 
         """
+        if not self._valid_input(value) or not self._valid_input(key):
+            raise ValueError("Invalid input value or key size exceeded the maximum limit.")
+
         formatted_key = self._format_key(key)
-        formatted_value = self._format_value(key, value)
-        if self.preserve_expiration and self.redis.exists(formatted_key):
-            self.redis.set(formatted_key, formatted_value, keepttl=True)
-        else:
-            self.redis.set(formatted_key, formatted_value, ex=self.expire)
+        formatted_value = self._format_value(value)
+
+        self._store_set(formatted_key, formatted_value)
 
     def _load(self, key: str) -> Tuple[bool, Any]:
         """
@@ -213,7 +234,7 @@ class RedisDict:
             decode: Optional[DecodeFuncType] = None,
             encoding_method_name: Optional[str] = None,
             decoding_method_name: Optional[str] = None,
-    ) -> None:
+    ) -> None: # noqa: D202 pydocstyle clashes with Sphinx
         """
         Extend RedisDict to support a custom type in the encode/decode mapping.
 
@@ -349,10 +370,25 @@ class RedisDict:
         """
         Delete the value associated with the given key, analogous to a dictionary.
 
+        For distributed systems, we intentionally don't raise KeyError when the key doesn't exist.
+        This ensures identical code running across different systems won't randomly fail
+        when another system already achieved the deletion goal (key not existing).
+
+        Warning:
+            Setting dict_compliant=True will raise KeyError when key doesn't exist.
+            This is not recommended for distributed systems as it can cause KeyErrors
+            that are hard to debug when multiple systems interact with the same keys.
+
         Args:
-            key (str): The key to delete the value.
+            key (str): The key to delete
+
+        Raises:
+            KeyError: Only if dict_compliant=True and key doesn't exist
         """
-        self.redis.delete(self._format_key(key))
+        formatted_key = self._format_key(key)
+        result = self.redis.delete(formatted_key)
+        if self.raise_key_error_delete and not result:
+            raise KeyError(key)
 
     def __contains__(self, key: str) -> bool:
         """
@@ -373,7 +409,7 @@ class RedisDict:
         Returns:
             int: The number of items in the RedisDict.
         """
-        return len(list(self._scan_keys()))
+        return sum(1 for _ in self._scan_keys(full_scan=True))
 
     def __iter__(self) -> Iterator[str]:
         """
@@ -470,12 +506,12 @@ class RedisDict:
         return self
 
     @classmethod
-    def __class_getitem__(cls: Type['RedisDict'], key: Any) -> Type['RedisDict']:
+    def __class_getitem__(cls: Type['RedisDict'], _key: Any) -> Type['RedisDict']:
         """
         Enable type hinting support like RedisDict[str, Any].
 
         Args:
-            key (Any): The type parameter(s) used in the type hint.
+            _key (Any): The type parameter(s) used in the type hint.
 
         Returns:
             Type[RedisDict]: The class itself, enabling type hint usage.
@@ -537,18 +573,19 @@ class RedisDict:
         """
         return f'{self.namespace}:{search_term}*'
 
-    def _scan_keys(self, search_term: str = '') -> Iterator[str]:
-        """
-        Scan for Redis keys matching the given search term.
+    def _scan_keys(self, search_term: str = '', full_scan: bool = False) -> Iterator[str]:
+        """Scan for Redis keys matching the given search term.
 
         Args:
             search_term (str): A search term to filter keys. Defaults to ''.
+            full_scan (bool): During full scan uses batches of self._batch_size by default 200
 
         Returns:
             Iterator[str]: An iterator of matching Redis keys.
         """
         search_query = self._create_iter_query(search_term)
-        return self.get_redis.scan_iter(match=search_query)
+        count = None if full_scan else self._batch_size
+        return self.get_redis.scan_iter(match=search_query, count=count)
 
     def get(self, key: str, default: Optional[Any] = None) -> Any:
         """Return the value for the given key if it exists, otherwise return the default value.
@@ -636,12 +673,24 @@ class RedisDict:
         Redis pipelining is employed to group multiple commands into a single request, minimizing
         network round-trip time, latency, and I/O load, thereby enhancing the overall performance.
 
-        It is important to highlight that the clear method can be safely executed within
-        the context of an initiated pipeline operation.
         """
         with self.pipeline():
-            for key in self:
-                del self[key]
+            for key in self._scan_keys(full_scan=True):
+                self.redis.delete(key)
+
+    def _pop(self, formatted_key: str) -> Any:
+        """
+        Remove the value associated with the given key and return it.
+
+        Or return the default value if the key is not found.
+
+        Args:
+            formatted_key (str): The formatted key to remove the value.
+
+        Returns:
+            Any: The value associated with the key or the default value.
+        """
+        return self.get_redis.execute_command("GETDEL", formatted_key)
 
     def pop(self, key: str, default: Union[Any, object] = SENTINEL) -> Any:
         """Analogous to a dictionary's pop method.
@@ -660,12 +709,11 @@ class RedisDict:
             KeyError: If the key is not found and no default value is provided.
         """
         formatted_key = self._format_key(key)
-        value = self.get_redis.execute_command("GETDEL", formatted_key)
+        value = self._pop(formatted_key)
         if value is None:
             if default is not SENTINEL:
                 return default
             raise KeyError(formatted_key)
-
         return self._transform(value)
 
     def popitem(self) -> Tuple[str, Any]:
@@ -673,6 +721,8 @@ class RedisDict:
 
         This method is analogous to the `popitem` method of a standard Python dictionary.
 
+        if dict_compliant set true stays true to In Python 3.7+, removes the last inserted item (LIFO order)
+
         Returns:
             tuple: A tuple containing a randomly chosen (key, value) pair.
 
@@ -688,22 +738,16 @@ class RedisDict:
             except KeyError:
                 continue
 
-    def setdefault(self, key: str, default_value: Optional[Any] = None) -> Any:
-        """Get value under key, and if not present set default value.
-
-        Return the value associated with the given key if it exists, otherwise set the value to the
-        default value and return it. Analogous to a dictionary's setdefault method.
+    def _create_set_get_command(self, formatted_key: str, formatted_value: str) -> Tuple[List[str], Dict[str, bool]]:
+        """Create SET command arguments and options for Redis. For setdefault operation.
 
         Args:
-            key (str): The key to retrieve the value.
-            default_value (Optional[Any], optional): The value to set if the key is not found.
+            formatted_key (str): The formatted Redis key.
+            formatted_value (str): The formatted value to be set.
 
         Returns:
-            Any: The value associated with the key or the default value.
+            Tuple[List[str], Dict[str, bool]]: A tuple containing the command arguments and options.
         """
-        formatted_key = self._format_key(key)
-        formatted_value = self._format_value(key, default_value)
-
         # Setting {"get": True} enables parsing of the redis result as "GET", instead of "SET" command
         options = {"get": True}
         args = ["SET", formatted_key, formatted_value, "NX", "GET"]
@@ -713,8 +757,27 @@ class RedisDict:
             expire_val = int(self.expire.total_seconds()) if isinstance(self.expire, timedelta) else self.expire
             expire_str = str(1) if expire_val <= 1 else str(expire_val)
             args.extend(["EX", expire_str])
+        return args, options
 
+    def setdefault(self, key: str, default_value: Optional[Any] = None) -> Any:
+        """Get value under key, and if not present set default value.
+
+        Return the value associated with the given key if it exists, otherwise set the value to the
+        default value and return it. Analogous to a dictionary's setdefault method.
+
+        Args:
+            key (str): The key to retrieve the value.
+            default_value (Optional[Any], optional): The value to set if the key is not found.
+
+        Returns:
+            Any: The value associated with the key or the default value.
+        """
+        formatted_key = self._format_key(key)
+        formatted_value = self._format_value(default_value)
+
+        args, options = self._create_set_get_command(formatted_key, formatted_value)
         result = self.get_redis.execute_command(*args, **options)
+
         if result is None:
             return default_value
 

redis_dict-3.2.1/src/redis_dict/python_dict.py

@@ -0,0 +1,331 @@
+"""Python Redis Dict module."""
+from typing import Any, Iterator, Tuple, Union, Optional, List, Dict
+
+import time
+from datetime import timedelta
+
+from redis import StrictRedis
+
+from .core import RedisDict
+
+
+class PythonRedisDict(RedisDict):
+    """Python dictionary with Redis as backend.
+
+    With support for advanced features, such as custom data types, pipelining, and key expiration.
+
+    This class focuses on having one-to-on behavior of a dictionary while using Redis as storage layer, allowing
+    for efficient storage and retrieval of key-value pairs. It supports various data types, including
+    strings, integers, floats, lists, dictionaries, tuples, sets, and user-defined types. The class
+    leverages the power of Redis pipelining to minimize network round-trip time, latency, and I/O load,
+    thereby optimizing performance for batch operations. Additionally, it allows for the management of
+    key expiration through the use of context managers.
+
+    The RedisDict class is designed to be analogous to a standard Python dictionary while providing
+    enhanced functionality, such as support for a wider range of data types and efficient batch operations.
+    It aims to offer a seamless and familiar interface for developers familiar with Python dictionaries,
+    enabling a smooth transition to a Redis-backed data store.
+
+    Extendable Types: You can extend RedisDict by adding or overriding encoding and decoding functions.
+    This functionality enables various use cases, such as managing encrypted data in Redis,
+    To implement this, simply create and register your custom encoding and decoding functions.
+    By delegating serialization to redis-dict, reduce complexity and have simple code in the codebase.
+    """
+
+    def __init__(self,
+                 namespace: str = 'main',
+                 expire: Union[int, timedelta, None] = None,
+                 preserve_expiration: Optional[bool] = False,
+                 redis: "Optional[StrictRedis[Any]]" = None,
+                 **redis_kwargs: Any) -> None:  # noqa: D202 pydocstyle clashes with Sphinx
+        """
+        Initialize a RedisDict instance.
+
+        Init the RedisDict instance.
+
+        Args:
+            namespace (str): A prefix for keys stored in Redis.
+            expire (Union[int, timedelta, None], optional): Expiration time for keys.
+            preserve_expiration (Optional[bool], optional): Preserve expiration on key updates.
+            redis (Optional[StrictRedis[Any]], optional): A Redis connection instance.
+            **redis_kwargs (Any): Additional kwargs for Redis connection if not provided.
+        """
+        super().__init__(
+            namespace=namespace,
+            expire=expire,
+            preserve_expiration=preserve_expiration,
+            redis=redis,
+            raise_key_error_delete=True,
+            **redis_kwargs
+        )
+        self._insertion_order_key = f"redis-dict-insertion-order-{namespace}"
+
+    def __delitem__(self, key: str) -> None:
+        """
+        Delete the value associated with the given key, analogous to a dictionary.
+
+        For distributed systems, we intentionally don't raise KeyError when the key doesn't exist.
+        This ensures identical code running across different systems won't randomly fail
+        when another system already achieved the deletion goal (key not existing).
+
+        Warning:
+            Setting dict_compliant=True will raise KeyError when key doesn't exist.
+            This is not recommended for distributed systems as it can cause KeyErrors
+            that are hard to debug when multiple systems interact with the same keys.
+
+        Args:
+            key (str): The key to delete
+
+        Raises:
+            KeyError: Only if dict_compliant=True and key doesn't exist
+        """
+        formatted_key = self._format_key(key)
+
+        result = self.redis.delete(formatted_key)
+        self._insertion_order_delete(formatted_key)
+        if not result:
+            raise KeyError(key)
+
+    def _store(self, key: str, value: Any) -> None:
+        """
+        Store a value in Redis with the given key.
+
+        Args:
+            key (str): The key to store the value.
+            value (Any): The value to be stored.
+
+        Raises:
+            ValueError: If the value or key fail validation.
+
+        Note: Validity checks could be refactored to allow for custom exceptions that inherit from ValueError,
+        providing detailed information about why a specific validation failed.
+        This would enable users to specify which validity checks should be executed, add custom validity functions,
+        and choose whether to fail on validation errors, or drop the data and only issue a warning and continue.
+        Example use case is caching, to cache data only when it's between min and max sizes.
+        Allowing for simple dict set operation, but only cache data that makes sense.
+
+        """
+        if not self._valid_input(value) or not self._valid_input(key):
+            raise ValueError("Invalid input value or key size exceeded the maximum limit.")
+
+        formatted_key = self._format_key(key)
+        formatted_value = self._format_value(value)
+
+        with self.pipeline():
+            self._insertion_order_add(formatted_key)
+            self._store_set(formatted_key, formatted_value)
+
+    def setdefault(self, key: str, default_value: Optional[Any] = None) -> Any:
+        """Get value under key, and if not present set default value.
+
+        Return the value associated with the given key if it exists, otherwise set the value to the
+        default value and return it. Analogous to a dictionary's setdefault method.
+
+        Args:
+            key (str): The key to retrieve the value.
+            default_value (Optional[Any], optional): The value to set if the key is not found.
+
+        Returns:
+            Any: The value associated with the key or the default value.
+        """
+        formatted_key = self._format_key(key)
+        formatted_value = self._format_value(default_value)
+
+        # Todo bind both commands
+        args, options = self._create_set_get_command(formatted_key, formatted_value)
+        result = self.get_redis.execute_command(*args, **options)
+        self._insertion_order_add(formatted_key)
+
+        if result is None:
+            return default_value
+
+        return self._transform(result)
+
+    def __len__(self) -> int:
+        """
+        Get the number of items in the RedisDict, analogous to a dictionary.
+
+        Returns:
+            int: The number of items in the RedisDict.
+        """
+        return self._insertion_order_len()
+
+    def _scan_keys(self, search_term: str = '', full_scan: bool = False) -> Iterator[str]:
+        return self._insertion_order_iter()
+
+    def clear(self) -> None:
+        """Remove all key-value pairs from the RedisDict in one batch operation using pipelining.
+
+        This method mimics the behavior of the `clear` method from a standard Python dictionary.
+        Redis pipelining is employed to group multiple commands into a single request, minimizing
+        network round-trip time, latency, and I/O load, thereby enhancing the overall performance.
+
+        """
+        with self.pipeline():
+            self._insertion_order_clear()
+            for key in self._scan_keys(full_scan=True):
+                self.redis.delete(key)
+
+    def popitem(self) -> Tuple[str, Any]:
+        """Remove and return a random (key, value) pair from the RedisDict as a tuple.
+
+        This method is analogous to the `popitem` method of a standard Python dictionary.
+
+        if dict_compliant set true stays true to In Python 3.7+, removes the last inserted item (LIFO order)
+
+        Returns:
+            tuple: A tuple containing a randomly chosen (key, value) pair.
+
+        Raises:
+            KeyError: If RedisDict is empty.
+        """
+        key = self._insertion_order_latest()
+        if key is None:
+            raise KeyError("popitem(): dictionary is empty")
+        return self._parse_key(key), self._transform(self._pop(key))
+
+    def _pop(self, formatted_key: str) -> Any:
+        """
+        Remove the value associated with the given key and return it.
+
+        Or return the default value if the key is not found.
+
+        Args:
+            formatted_key (str): The formatted key to remove the value.
+
+        Returns:
+            Any: The value associated with the key or the default value.
+        """
+        # TODO bind both commands
+        self._insertion_order_delete(formatted_key)
+        return self.get_redis.execute_command("GETDEL", formatted_key)
+
+    def multi_get(self, _key: str) -> List[Any]:
+        """
+        Not part of Python Redis Dict.
+
+        Args:
+            _key (str): Not used.
+
+        Raises:
+            NotImplementedError: Not part of Python Redis Dict.
+        """
+        raise NotImplementedError("Not part of PythonRedisDict")
+
+    def multi_chain_get(self, _keys: List[str]) -> List[Any]:
+        """
+        Not part of Python Redis Dict.
+
+        Args:
+            _keys (List[str]): Not used.
+
+        Raises:
+            NotImplementedError: Not part of Python Redis Dict.
+        """
+        raise NotImplementedError("Not part of PythonRedisDict")
+
+    def multi_dict(self, _key: str) -> Dict[str, Any]:
+        """
+        Not part of Python Redis Dict.
+
+        Args:
+            _key (str): Not used.
+
+        Raises:
+            NotImplementedError: Not part of Python Redis Dict.
+        """
+        raise NotImplementedError("Not part of PythonRedisDict")
+
+    def multi_del(self, _key: str) -> int:
+        """
+        Not part of Python Redis Dict.
+
+        Args:
+            _key (str): Not used.
+
+        Raises:
+            NotImplementedError: Not part of Python Redis Dict.
+        """
+        raise NotImplementedError("Not part of PythonRedisDict")
+
+    def _insertion_order_add(self, formatted_key: str) -> bool:
+        """Record a key's insertion into the dictionary.
+
+        This private method updates the insertion order tracking when a new key is added
+        to the dictionary.
+
+        Args:
+            formatted_key (str): The key being added to the dictionary.
+
+        Returns:
+            bool: True if the insertion order was updated, False otherwise.
+        """
+        return bool(self.redis.zadd(self._insertion_order_key, {formatted_key: time.time()}))
+
+    def _insertion_order_delete(self, formatted_key: str) -> bool:
+        """Remove a key from the insertion order tracking.
+
+        This private method updates the insertion order tracking when a key is removed
+        from the dictionary.
+
+        Args:
+            formatted_key (str): The key being removed from the dictionary.
+
+        Returns:
+            bool: True if the insertion order was updated, False otherwise.
+        """
+        return bool(self.redis.zrem(self._insertion_order_key, formatted_key))
+
+    def _insertion_order_iter(self) -> Iterator[str]:
+        """Create an iterator for dictionary keys in their insertion order.
+
+        This private method allows for iterating over the dictionary's keys in the order
+        they were inserted.
+
+        Yields:
+            str: Keys in their insertion order.
+        """
+        # TODO add full_scan boolean and search terms.
+        first = True
+        cursor = -1
+        while cursor != 0:
+            if first:
+                cursor = 0
+                first = False
+            cursor, data = self.get_redis.zscan(
+                name=self._insertion_order_key,
+                cursor=cursor,
+                count=1
+            )
+            yield from (item[0] for item in data)
+
+    def _insertion_order_clear(self) -> bool:
+        """Clear all insertion order information.
+
+        This private method resets the insertion order tracking for the dictionary.
+
+        Returns:
+            bool: True if the insertion order was successfully cleared, False otherwise.
+        """
+        return bool(self.redis.delete(self._insertion_order_key))
+
+    def _insertion_order_len(self) -> int:
+        """Get the number of keys in the insertion order tracking.
+
+        This private method returns the count of keys being tracked for insertion order.
+
+        Returns:
+            int: The number of keys in the insertion order tracking.
+        """
+        return self.get_redis.zcard(self._insertion_order_key)
+
+    def _insertion_order_latest(self) -> Union[str, None]:
+        """Get the most recently inserted key in the dictionary.
+
+        This private method retrieves the key that was most recently added to the dictionary.
+
+        Returns:
+            Union[str, None]: The most recently inserted key, or None if the dictionary is empty.
+        """
+        result = self.redis.zrange(self._insertion_order_key, -1, -1)
+        return result[0] if result else None

{redis_dict-3.1.2 → redis_dict-3.2.1/src/redis_dict.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.2
 Name: redis-dict
-Version: 3.1.2
+Version: 3.2.1
 Summary: Dictionary with Redis as storage backend
 Author-email: Melvin Bijman <bijman.m.m@gmail.com>
 License: MIT
@@ -42,7 +42,7 @@ Requires-Dist: types-redis>=4.6.0; extra == "dev"
 Requires-Dist: typing_extensions>=4.5.0; extra == "dev"
 Requires-Dist: attrs==22.2.0; extra == "dev"
 Requires-Dist: cffi==1.15.1; extra == "dev"
-Requires-Dist: cryptography==43.0.1; extra == "dev"
+Requires-Dist: cryptography==44.0.1; extra == "dev"
 Requires-Dist: exceptiongroup==1.1.1; extra == "dev"
 Requires-Dist: future==0.18.3; extra == "dev"
 Requires-Dist: pycparser==2.21; extra == "dev"
@@ -122,6 +122,48 @@ In Redis our example looks like this.
 "str:hello world"
 ```
 
+## Types
+
+### standard types
+RedisDict supports a range of Python data types, from basic types to nested structures.
+Basic types are handled natively, while complex data types like lists and dictionaries, RedisDict uses JSON serialization, specifically avoiding [pickle](https://docs.python.org/3/library/pickle.html) due to its security vulnerabilities within distributed computing contexts.
+Although the library supports nested structures, the recommended best practice is to use RedisDict as a shallow dictionary.
+This approach optimizes Redis database performance and efficiency by ensuring that each set and get operation efficiently maps to Redis's key-value storage capabilities, while still preserving the library's Pythonic interface.
+Following types are supported: 
+`str, int, float, bool, NoneType, list, dict, tuple, set, datetime, date, time, timedelta, Decimal, complex, bytes, UUID, OrderedDict, defaultdict, frozenset`
+```python
+from uuid import UUID
+from decimal import Decimal
+from collections import OrderedDict, defaultdict
+from datetime import datetime, date, time, timedelta
+
+dic = RedisDict()
+
+dic["string"] = "Hello World"
+dic["number"] = 42
+dic["float"] = 3.14
+dic["bool"] = True
+dic["None"] = None
+
+dic["list"] = [1, 2, 3]
+dic["dict"] = {"a": 1, "b": 2}
+dic["tuple"] = (1, 2, 3)
+dic["set"] = {1, 2, 3}
+
+dic["datetime"] = datetime.date(2024, 1, 1, 12, 30, 45)
+dic["date"] = date(2024, 1, 1)
+dic["time"] = time(12, 30, 45)
+dic["delta"] = timedelta(days=1, hours=2)
+
+dic["decimal"] = Decimal("3.14159")
+dic["complex"] = complex(1, 2)
+dic["bytes"] = bytes([72, 101, 108, 108, 111])
+dic["uuid"] = UUID('12345678-1234-5678-1234-567812345678')
+
+dic["ordered"] = OrderedDict([('a', 1), ('b', 2)])
+dic["default"] = defaultdict(int, {'a': 1, 'b': 2})
+dic["frozen"] = frozenset([1, 2, 3])
+```
 
 ### Namespaces
 Acting as an identifier for your dictionary across different systems, RedisDict employs namespaces for organized data management. When a namespace isn't specified, "main" becomes the default. Thus allowing for data organization across systems and projects with the same redis instance.
@@ -290,51 +332,6 @@ print(dic["d"])  # Output: 4
 For more advanced examples of RedisDict, please refer to the unit-test files in the repository. All features and functionalities are thoroughly tested in [unit tests (here)](https://github.com/Attumm/redis-dict/blob/main/tests/unit/tests.py#L1) Or take a look at load test for batching [load test](https://github.com/Attumm/redis-dict/blob/main/tests/load/load_test.py#L1).
 The unit-tests can be as used as a starting point.
 
-## Types
-
-### standard types
-RedisDict supports a range of Python data types, from basic types to nested structures.
-Basic types are handled natively, while complex data types like lists and dictionaries, RedisDict uses JSON serialization, specifically avoiding `pickle` due to its [security vulnerabilities](https://docs.python.org/3/library/pickle.html) in distributed computing contexts.
-Although the library supports nested structures, the recommended best practice is to use RedisDict as a shallow dictionary.
-This approach optimizes Redis database performance and efficiency by ensuring that each set and get operation efficiently maps to Redis's key-value storage capabilities, while still preserving the library's Pythonic interface.
-Following types are supported: 
-`str, int, float, bool, NoneType, list, dict, tuple, set, datetime, date, time, timedelta, Decimal, complex, bytes, UUID, OrderedDict, defaultdict, frozenset`
-```python
-from uuid import UUID
-from decimal import Decimal
-from collections import OrderedDict, defaultdict
-from datetime import datetime, date, time, timedelta
-
-dic = RedisDict()
-
-dic["string"] = "Hello World"
-dic["number"] = 42
-dic["float"] = 3.14
-dic["bool"] = True
-dic["None"] = None
-
-dic["list"] = [1, 2, 3]
-dic["dict"] = {"a": 1, "b": 2}
-dic["tuple"] = (1, 2, 3)
-dic["set"] = {1, 2, 3}
-
-dic["datetime"] = datetime.date(2024, 1, 1, 12, 30, 45)
-dic["date"] = date(2024, 1, 1)
-dic["time"] = time(12, 30, 45)
-dic["delta"] = timedelta(days=1, hours=2)
-
-dic["decimal"] = Decimal("3.14159")
-dic["complex"] = complex(1, 2)
-dic["bytes"] = bytes([72, 101, 108, 108, 111])
-dic["uuid"] = UUID('12345678-1234-5678-1234-567812345678')
-
-dic["ordered"] = OrderedDict([('a', 1), ('b', 2)])
-dic["default"] = defaultdict(int, {'a': 1, 'b': 2})
-dic["frozen"] = frozenset([1, 2, 3])
-```
-
-
-
 ### Nested types
 Nested Types
 RedisDict supports nested structures with mixed types through JSON serialization. The feature works by utilizing JSON encoding and decoding under the hood. While this represents an upgrade in functionality, the feature is not fully implemented and should be used with caution. For optimal performance, using shallow dictionaries is recommended.
@@ -394,6 +391,21 @@ assert result.name == person.name
 assert result.age == person.age
 ```
 
+### Insertion Order
+For insertion order, use the PythonRedisDict. This class is focused on Python dictionary behavior one-to-one.
+It will eventually become a drop-in replacement for dictionary. Currently, nested types and typed keys are not yet supported but will be added in the future.
+
+```python
+from redis_dict import PythonRedisDict
+
+dic = PythonRedisDict()
+dic["1"] = "one"
+dic["2"] = "two"
+dic["3"] = "three"
+
+assert list(dic.keys()) == ["1", "2", "3"]
+```
+
 For more information on [extending types](https://github.com/Attumm/redis-dict/blob/main/tests/unit/extend_types_tests.py).
 ### Redis Encryption
 Setup guide for configuring and utilizing encrypted Redis TLS for redis-dict.
@@ -417,7 +429,7 @@ redis_config = {
     'port': 6380,
 }
 
-confid_dic = RedisDict(**redis_config)
+config_dic = RedisDict(**redis_config)
 ```
 
 ## Installation

{redis_dict-3.1.2 → redis_dict-3.2.1}/src/redis_dict.egg-info/SOURCES.txt

@@ -4,6 +4,7 @@ pyproject.toml
 src/redis_dict/__init__.py
 src/redis_dict/core.py
 src/redis_dict/py.typed
+src/redis_dict/python_dict.py
 src/redis_dict/type_management.py
 src/redis_dict.egg-info/PKG-INFO
 src/redis_dict.egg-info/SOURCES.txt

{redis_dict-3.1.2 → redis_dict-3.2.1}/src/redis_dict.egg-info/requires.txt

@@ -14,7 +14,7 @@ types-redis>=4.6.0
 typing_extensions>=4.5.0
 attrs==22.2.0
 cffi==1.15.1
-cryptography==43.0.1
+cryptography==44.0.1
 exceptiongroup==1.1.1
 future==0.18.3
 pycparser==2.21