redis-dict 2.5.0__py3-none-any.whl → 2.6.0__py3-none-any.whl
Sign up to get free protection for your applications and to get access to all the features.
- {redis_dict-2.5.0.dist-info → redis_dict-2.6.0.dist-info}/METADATA +93 -22
- redis_dict-2.6.0.dist-info/RECORD +6 -0
- {redis_dict-2.5.0.dist-info → redis_dict-2.6.0.dist-info}/WHEEL +1 -1
- redis_dict.py +285 -53
- redis_dict-2.5.0.dist-info/RECORD +0 -6
- {redis_dict-2.5.0.dist-info → redis_dict-2.6.0.dist-info}/LICENSE +0 -0
- {redis_dict-2.5.0.dist-info → redis_dict-2.6.0.dist-info}/top_level.txt +0 -0
@@ -1,15 +1,24 @@
|
|
1
1
|
Metadata-Version: 2.1
|
2
2
|
Name: redis-dict
|
3
|
-
Version: 2.
|
3
|
+
Version: 2.6.0
|
4
4
|
Summary: Dictionary with Redis as storage backend
|
5
5
|
Home-page: https://github.com/Attumm/redisdict
|
6
6
|
Author: Melvin Bijman
|
7
7
|
Author-email: bijman.m.m@gmail.com
|
8
8
|
License: MIT
|
9
|
+
Keywords: redis python dictionary dict key-value key:value database caching distributed-computing dictionary-interface large-datasets scientific-computing data-persistence high-performance scalable pipelining batching big-data data-types distributed-algorithms encryption data-management
|
10
|
+
Platform: any
|
9
11
|
Classifier: Development Status :: 5 - Production/Stable
|
10
12
|
Classifier: Intended Audience :: Developers
|
13
|
+
Classifier: Intended Audience :: Information Technology
|
14
|
+
Classifier: Intended Audience :: Science/Research
|
15
|
+
Classifier: Topic :: Internet
|
16
|
+
Classifier: Topic :: Scientific/Engineering
|
11
17
|
Classifier: Topic :: Database
|
12
18
|
Classifier: Topic :: System :: Distributed Computing
|
19
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
20
|
+
Classifier: Topic :: Software Development :: Object Brokering
|
21
|
+
Classifier: Topic :: Database :: Database Engines/Servers
|
13
22
|
Classifier: License :: OSI Approved :: MIT License
|
14
23
|
Classifier: Programming Language :: Python :: 3
|
15
24
|
Classifier: Programming Language :: Python :: 3.6
|
@@ -24,13 +33,13 @@ License-File: LICENSE
|
|
24
33
|
Requires-Dist: redis
|
25
34
|
|
26
35
|
# Redis-dict
|
27
|
-
[](https://github.com/Attumm/redis-dict/actions/workflows/ci.yml)
|
37
|
+
[](https://codecov.io/gh/Attumm/redis-dict)
|
28
38
|
[](https://pepy.tech/project/redis-dict)
|
29
39
|
|
30
|
-
RedisDict is a Python library that
|
31
|
-
|
32
|
-
By leveraging Redis for efficient key-value storage, RedisDict allows for high-performance data management and is particularly useful for handling large datasets that may exceed local memory capacity.
|
40
|
+
RedisDict is a Python library that offers a convenient and familiar interface for interacting with Redis, treating it as if it were a Python dictionary. Its goal is to help developers write clean, Pythonic code while using Redis as a storage solution for seamless distributed computing. This simple yet powerful library utilizes Redis as a key-value store and supports various data types, including strings, integers, floats, booleans, lists, and dictionaries. Additionally, developers can extend RedisDict to work with custom objects.
|
33
41
|
|
42
|
+
The library includes utility functions for more complex use cases such as caching, batching, and more. By leveraging Redis for efficient key-value storage, RedisDict enables high-performance data management, maintaining efficiency even with large datasets and Redis instances.
|
34
43
|
|
35
44
|
## Features
|
36
45
|
|
@@ -41,20 +50,27 @@ By leveraging Redis for efficient key-value storage, RedisDict allows for high-p
|
|
41
50
|
* Efficiency and Scalability: RedisDict is designed for use with large datasets and is optimized for efficiency. It retrieves only the data needed for a particular operation, ensuring efficient memory usage and fast performance.
|
42
51
|
* Namespace Management: Provides simple and efficient namespace handling to help organize and manage data in Redis, streamlining data access and manipulation.
|
43
52
|
* Distributed Computing: With its ability to seamlessly connect to other instances or servers with access to the same Redis instance, RedisDict enables easy distributed computing.
|
44
|
-
* Custom data types: Add custom types
|
53
|
+
* Custom data: types: Add custom types encoding/decoding to store your data types.
|
54
|
+
* Encryption: allows for storing data encrypted, while retaining the simple dictionary interface.
|
45
55
|
|
46
56
|
## Example
|
47
57
|
Redis is an exceptionally fast database when used appropriately. RedisDict leverages Redis for efficient key-value storage, enabling high-performance data management.
|
48
58
|
|
49
|
-
```
|
50
|
-
|
59
|
+
```bash
|
60
|
+
pip install redis-dict
|
61
|
+
```
|
51
62
|
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
dic[
|
57
|
-
|
63
|
+
```python
|
64
|
+
>>> from redis_dict import RedisDict
|
65
|
+
>>> dic = RedisDict()
|
66
|
+
>>> dic['foo'] = 42
|
67
|
+
>>> dic['foo']
|
68
|
+
42
|
69
|
+
>>> 'foo' in dic
|
70
|
+
True
|
71
|
+
>>> dic["baz"] = "hello world"
|
72
|
+
>>> dic
|
73
|
+
{'foo': 42, 'baz': 'hello world'}
|
58
74
|
```
|
59
75
|
In Redis our example looks like this.
|
60
76
|
```
|
@@ -67,12 +83,12 @@ In Redis our example looks like this.
|
|
67
83
|
"str:hello world"
|
68
84
|
```
|
69
85
|
|
86
|
+
|
70
87
|
### Namespaces
|
71
|
-
Acting as an identifier for your dictionary across different systems, RedisDict employs namespaces for organized data management. When a namespace isn't specified, "main" becomes the default. Thus allowing for data organization
|
88
|
+
Acting as an identifier for your dictionary across different systems, RedisDict employs namespaces for organized data management. When a namespace isn't specified, "main" becomes the default. Thus allowing for data organization across systems and projects with the same redis instance.
|
72
89
|
|
73
90
|
This approach also minimizes the risk of key collisions between different applications, preventing hard-to-debug issues. By leveraging namespaces, RedisDict ensures a cleaner and more maintainable data management experience for developers working on multiple projects.
|
74
91
|
|
75
|
-
|
76
92
|
## Advanced Features
|
77
93
|
|
78
94
|
### Expiration
|
@@ -105,6 +121,8 @@ with dic.expire_at(seconds):
|
|
105
121
|
3. Updating keys while preserving the initial timeout In certain situations, there is a need to update the value while keeping the expiration intact. This is achievable by setting the 'preserve_expiration' to true.
|
106
122
|
|
107
123
|
```python
|
124
|
+
import time
|
125
|
+
|
108
126
|
dic = RedisDict(expire=10, preserve_expiration=True)
|
109
127
|
dic['gone'] = 'in ten seconds'
|
110
128
|
|
@@ -117,6 +135,7 @@ dic['gone'] = 'gone in 5 seconds'
|
|
117
135
|
Efficiently batch your requests using the Pipeline feature, which can be easily utilized with a context manager.
|
118
136
|
|
119
137
|
```python
|
138
|
+
from redis_dict import RedisDict
|
120
139
|
dic = RedisDict(namespace="example")
|
121
140
|
|
122
141
|
# one round trip to redis
|
@@ -146,13 +165,14 @@ print(dic["foo"]) # outputs "bar"
|
|
146
165
|
### Caching made simple
|
147
166
|
```python
|
148
167
|
import time
|
168
|
+
from datetime import timedelta
|
149
169
|
from redis_dict import RedisDict
|
150
170
|
|
151
171
|
def expensive_function(x):
|
152
|
-
time.sleep(
|
172
|
+
time.sleep(x)
|
153
173
|
return x * 2
|
154
174
|
|
155
|
-
cache = RedisDict(namespace="cache", expire=
|
175
|
+
cache = RedisDict(namespace="cache", expire=timedelta(minutes=60))
|
156
176
|
|
157
177
|
def cached_expensive_function(x):
|
158
178
|
if x not in cache:
|
@@ -160,7 +180,7 @@ def cached_expensive_function(x):
|
|
160
180
|
return cache[x]
|
161
181
|
|
162
182
|
start_time = time.time()
|
163
|
-
print(cached_expensive_function(5)) # Takes around
|
183
|
+
print(cached_expensive_function(5)) # Takes around 5 seconds to compute and caches the result.
|
164
184
|
print(f"Time taken: {time.time() - start_time:.2f} seconds")
|
165
185
|
|
166
186
|
start_time = time.time()
|
@@ -180,7 +200,7 @@ dic["name"] = "John Doe"
|
|
180
200
|
dic["age"] = 32
|
181
201
|
dic["city"] = "Amsterdam"
|
182
202
|
|
183
|
-
# Get value by key
|
203
|
+
# Get value by key, from any instance connected to the same redis/namespace
|
184
204
|
print(dic["name"]) # Output: John Doe
|
185
205
|
|
186
206
|
# Update value by key, got a year older
|
@@ -230,13 +250,64 @@ print(dic["d"]) # Output: 4
|
|
230
250
|
```
|
231
251
|
|
232
252
|
### Additional Examples
|
233
|
-
For more advanced examples of RedisDict, please refer to the unit-test files in the repository. All features and functionalities are thoroughly tested in [unit tests (here)](https://github.com/Attumm/redis-dict/blob/main/tests.py#L1) Or take a look at load test for batching [load test](https://github.com/Attumm/redis-dict/blob/main/load_test.py
|
253
|
+
For more advanced examples of RedisDict, please refer to the unit-test files in the repository. All features and functionalities are thoroughly tested in [unit tests (here)](https://github.com/Attumm/redis-dict/blob/main/tests.py#L1) Or take a look at load test for batching [load test](https://github.com/Attumm/redis-dict/blob/main/load_test.py#L1).
|
234
254
|
The unit-tests can be as used as a starting point.
|
235
255
|
|
256
|
+
### Extending Types
|
257
|
+
|
258
|
+
## Extending RedisDict with Custom Types
|
259
|
+
|
260
|
+
RedisDict supports custom type serialization. Here's how to add a new type:
|
261
|
+
|
262
|
+
|
263
|
+
```python
|
264
|
+
import json
|
265
|
+
from redis_dict import RedisDict
|
266
|
+
|
267
|
+
class Person:
|
268
|
+
def __init__(self, name, age):
|
269
|
+
self.name = name
|
270
|
+
self.age = age
|
271
|
+
|
272
|
+
def encode(self) -> str:
|
273
|
+
return json.dumps(self.__dict__)
|
274
|
+
|
275
|
+
@classmethod
|
276
|
+
def decode(cls, encoded_str: str) -> 'Person':
|
277
|
+
return cls(**json.loads(encoded_str))
|
278
|
+
|
279
|
+
redis_dict = RedisDict()
|
280
|
+
|
281
|
+
# Extend redis dict with the new type
|
282
|
+
redis_dict.extends_type(Person)
|
283
|
+
|
284
|
+
# RedisDict can now seamlessly handle Person instances.
|
285
|
+
person = Person(name="John", age=32)
|
286
|
+
redis_dict["person1"] = person
|
287
|
+
|
288
|
+
result = redis_dict["person1"]
|
289
|
+
|
290
|
+
assert result.name == person.name
|
291
|
+
assert result.age == person.age
|
292
|
+
```
|
293
|
+
|
294
|
+
```python
|
295
|
+
>>> from datetime import datetime
|
296
|
+
>>> redis_dict.extends_type(datetime, datetime.isoformat, datetime.fromisoformat)
|
297
|
+
>>> redis_dict["now"] = datetime.now()
|
298
|
+
>>> redis_dict
|
299
|
+
{'now': datetime.datetime(2024, 10, 14, 18, 41, 53, 493775)}
|
300
|
+
>>> redis_dict["now"]
|
301
|
+
datetime.datetime(2024, 10, 14, 18, 41, 53, 493775)
|
302
|
+
```
|
303
|
+
|
304
|
+
For more information on [extending types](https://github.com/Attumm/redis-dict/blob/main/extend_types_tests.py).
|
236
305
|
### Redis Encryption
|
237
|
-
Setup guide for configuring and utilizing encrypted Redis for redis-dict.
|
306
|
+
Setup guide for configuring and utilizing encrypted Redis TLS for redis-dict.
|
238
307
|
[Setup guide](https://github.com/Attumm/redis-dict/blob/main/encrypted_redis.MD)
|
239
308
|
|
309
|
+
### Redis Storage Encryption
|
310
|
+
For storing encrypted data values, it's possible to use extended types. Take a look at this [encrypted test](https://github.com/Attumm/redis-dict/blob/main/encrypt_tests.py).
|
240
311
|
|
241
312
|
### Tests
|
242
313
|
The RedisDict library includes a comprehensive suite of tests that ensure its correctness and resilience. The test suite covers various data types, edge cases, and error handling scenarios. It also employs the Hypothesis library for property-based testing, which provides fuzz testing to evaluate the implementation
|
@@ -0,0 +1,6 @@
|
|
1
|
+
redis_dict.py,sha256=50CSZ5dMBBbr-UU9BSvoGgBItD7uce8F5ty1lphaiUw,36901
|
2
|
+
redis_dict-2.6.0.dist-info/LICENSE,sha256=-QiLwYznh_vNUSz337k0faP9Jl0dgtCIHVZ39Uyl6cA,1070
|
3
|
+
redis_dict-2.6.0.dist-info/METADATA,sha256=6QJ93NO1RrVKSYbmbgtFKOazqtKrauPstB-H0hI1vDs,12564
|
4
|
+
redis_dict-2.6.0.dist-info/WHEEL,sha256=OVMc5UfuAQiSplgO0_WdW7vXVGAt9Hdd6qtN4HotdyA,91
|
5
|
+
redis_dict-2.6.0.dist-info/top_level.txt,sha256=Wyp5Xvq_imoxvu-c-Le1rbTZ3pYM5BF440H9YAcgBZ8,11
|
6
|
+
redis_dict-2.6.0.dist-info/RECORD,,
|
redis_dict.py
CHANGED
@@ -1,17 +1,101 @@
|
|
1
|
+
"""
|
2
|
+
redis_dict.py
|
3
|
+
|
4
|
+
RedisDict is a Python library that provides a convenient and familiar interface for
|
5
|
+
interacting with Redis as if it were a Python dictionary. The simple yet powerful library
|
6
|
+
enables you to manage key-value pairs in Redis using native Python syntax of dictionary. It supports
|
7
|
+
various data types, including strings, integers, floats, booleans, lists, and dictionaries,
|
8
|
+
and includes additional utility functions for more complex use cases.
|
9
|
+
|
10
|
+
By leveraging Redis for efficient key-value storage, RedisDict allows for high-performance
|
11
|
+
data management and is particularly useful for handling large datasets that may exceed local
|
12
|
+
memory capacity.
|
13
|
+
|
14
|
+
## Features
|
15
|
+
|
16
|
+
* **Dictionary-like interface**: Use familiar Python dictionary syntax to interact with Redis.
|
17
|
+
* **Data Type Support**: Comprehensive support for various data types, including strings,
|
18
|
+
integers, floats, booleans, lists, dictionaries, sets, and tuples.
|
19
|
+
* **Pipelining support**: Use pipelines for batch operations to improve performance.
|
20
|
+
* **Expiration Support**: Enables the setting of expiration times either globally or individually
|
21
|
+
per key, through the use of context managers.
|
22
|
+
* **Efficiency and Scalability**: RedisDict is designed for use with large datasets and is
|
23
|
+
optimized for efficiency. It retrieves only the data needed for a particular operation,
|
24
|
+
ensuring efficient memory usage and fast performance.
|
25
|
+
* **Namespace Management**: Provides simple and efficient namespace handling to help organize
|
26
|
+
and manage data in Redis, streamlining data access and manipulation.
|
27
|
+
* **Distributed Computing**: With its ability to seamlessly connect to other instances or
|
28
|
+
servers with access to the same Redis instance, RedisDict enables easy distributed computing.
|
29
|
+
* **Custom data types**: Add custom types and transformations to suit your specific needs.
|
30
|
+
|
31
|
+
New feature
|
32
|
+
|
33
|
+
Custom extendable Validity checks on keys, and values.to support redis-dict base exceptions with messages from
|
34
|
+
enabling detailed reporting on the reasons for specific validation failures. This refactor would allow users
|
35
|
+
to configure which validity checks to execute, integrate custom validation functions, and specify whether
|
36
|
+
to raise an error on validation failures or to drop the operation and log a warning.
|
37
|
+
|
38
|
+
For example, in a caching scenario, data should only be cached if it falls within defined minimum and
|
39
|
+
maximum size constraints. This approach enables straightforward dictionary set operations while ensuring
|
40
|
+
that only meaningful data is cached: values greater than 10 MB and less than 100 MB should be cached;
|
41
|
+
otherwise, they will be dropped.
|
42
|
+
|
43
|
+
>>> def my_custom_validity_check(value: str) -> None:
|
44
|
+
\"""
|
45
|
+
Validates the size of the input.
|
46
|
+
|
47
|
+
Args:
|
48
|
+
value (str): string to validate.
|
49
|
+
|
50
|
+
Raises:
|
51
|
+
RedisDictValidityException: If the length of the input is not within the allowed range.
|
52
|
+
\"""
|
53
|
+
min_size = 10 * 1024: # Minimum size: 10 KB
|
54
|
+
max_size = 10 * 1024 * 1024: # Maximum size: 10 MB
|
55
|
+
if len(value) < min_size
|
56
|
+
raise RedisDictValidityException(f"value is too small: {len(value)} bytes")
|
57
|
+
if len(value) > max_size
|
58
|
+
raise RedisDictValidityException(f"value is too large: {len(value)} bytes")
|
59
|
+
|
60
|
+
>>> cache = RedisDict(namespace='cache_valid_results_for_1_minute',
|
61
|
+
... expire=60,
|
62
|
+
... custom_valid_values_checks=[my_custom_validity_check], # extend with new valid check
|
63
|
+
... validity_exception_suppress=True) # when value is invalid, don't store, and don't raise an exception.
|
64
|
+
>>> too_small = "too small to cache"
|
65
|
+
>>> cache["1234"] = too_small # Since the value is below 10kb, thus there is no reason to cache the value.
|
66
|
+
>>> cache.get("1234") is None
|
67
|
+
>>> True
|
68
|
+
"""
|
1
69
|
import json
|
70
|
+
|
2
71
|
from datetime import timedelta
|
3
72
|
from typing import Any, Callable, Dict, Iterator, Set, List, Tuple, Union, Optional
|
4
|
-
from redis import StrictRedis
|
5
|
-
|
6
73
|
from contextlib import contextmanager
|
7
74
|
|
75
|
+
from redis import StrictRedis
|
76
|
+
|
8
77
|
SENTINEL = object()
|
9
78
|
|
10
|
-
|
11
|
-
|
79
|
+
EncodeFuncType = Callable[[Any], str]
|
80
|
+
DecodeFuncType = Callable[[str], Any]
|
81
|
+
|
82
|
+
EncodeType = Dict[str, EncodeFuncType]
|
83
|
+
DecodeType = Dict[str, DecodeFuncType]
|
84
|
+
|
85
|
+
|
86
|
+
def _create_default_encode(custom_encode_method: str) -> EncodeFuncType:
|
87
|
+
def default_encode(obj: Any) -> str:
|
88
|
+
return getattr(obj, custom_encode_method)() # type: ignore[no-any-return]
|
89
|
+
return default_encode
|
90
|
+
|
91
|
+
|
92
|
+
def _create_default_decode(cls: type, custom_decode_method: str) -> DecodeFuncType:
|
93
|
+
def default_decode(encoded_str: str) -> Any:
|
94
|
+
return getattr(cls, custom_decode_method)(encoded_str)
|
95
|
+
return default_decode
|
12
96
|
|
13
97
|
|
14
|
-
def
|
98
|
+
def _decode_tuple(val: str) -> Tuple[Any, ...]:
|
15
99
|
"""
|
16
100
|
Deserialize a JSON-formatted string to a tuple.
|
17
101
|
|
@@ -27,7 +111,7 @@ def _transform_tuple(val: str) -> Tuple[Any, ...]:
|
|
27
111
|
return tuple(json.loads(val))
|
28
112
|
|
29
113
|
|
30
|
-
def
|
114
|
+
def _encode_tuple(val: Tuple[Any, ...]) -> str:
|
31
115
|
"""
|
32
116
|
Serialize a tuple to a JSON-formatted string.
|
33
117
|
|
@@ -43,7 +127,7 @@ def _pre_transform_tuple(val: Tuple[Any, ...]) -> str:
|
|
43
127
|
return json.dumps(list(val))
|
44
128
|
|
45
129
|
|
46
|
-
def
|
130
|
+
def _decode_set(val: str) -> Set[Any]:
|
47
131
|
"""
|
48
132
|
Deserialize a JSON-formatted string to a set.
|
49
133
|
|
@@ -59,7 +143,7 @@ def _transform_set(val: str) -> Set[Any]:
|
|
59
143
|
return set(json.loads(val))
|
60
144
|
|
61
145
|
|
62
|
-
def
|
146
|
+
def _encode_set(val: Set[Any]) -> str:
|
63
147
|
"""
|
64
148
|
Serialize a set to a JSON-formatted string.
|
65
149
|
|
@@ -75,6 +159,7 @@ def _pre_transform_set(val: Set[Any]) -> str:
|
|
75
159
|
return json.dumps(list(val))
|
76
160
|
|
77
161
|
|
162
|
+
# pylint: disable=R0902, R0904
|
78
163
|
class RedisDict:
|
79
164
|
"""
|
80
165
|
A Redis-backed dictionary-like data structure with support for advanced features, such as
|
@@ -92,15 +177,20 @@ class RedisDict:
|
|
92
177
|
It aims to offer a seamless and familiar interface for developers familiar with Python dictionaries,
|
93
178
|
enabling a smooth transition to a Redis-backed data store.
|
94
179
|
|
180
|
+
Extendable Types: You can extend RedisDict by adding or overriding encoding and decoding functions.
|
181
|
+
This functionality enables various use cases, such as managing encrypted data in Redis,
|
182
|
+
To implement this, simply create and register your custom encoding and decoding functions.
|
183
|
+
By delegating serialization to redis-dict, reduce complexity and have simple code in the codebase.
|
184
|
+
|
95
185
|
Attributes:
|
96
|
-
|
97
|
-
|
186
|
+
decoding_registry (Dict[str, DecodeFuncType]): Mapping of decoding transformation functions based on type
|
187
|
+
encoding_registry (Dict[str, EncodeFuncType]): Mapping of encoding transformation functions based on type
|
98
188
|
namespace (str): A string used as a prefix for Redis keys to separate data in different namespaces.
|
99
189
|
expire (Union[int, None]): An optional expiration time for keys, in seconds.
|
100
190
|
|
101
191
|
"""
|
102
192
|
|
103
|
-
|
193
|
+
decoding_registry: DecodeType = {
|
104
194
|
type('').__name__: str,
|
105
195
|
type(1).__name__: int,
|
106
196
|
type(0.1).__name__: float,
|
@@ -109,15 +199,15 @@ class RedisDict:
|
|
109
199
|
|
110
200
|
"list": json.loads,
|
111
201
|
"dict": json.loads,
|
112
|
-
"tuple":
|
113
|
-
type(set()).__name__:
|
202
|
+
"tuple": _decode_tuple,
|
203
|
+
type(set()).__name__: _decode_set,
|
114
204
|
}
|
115
205
|
|
116
|
-
|
206
|
+
encoding_registry: EncodeType = {
|
117
207
|
"list": json.dumps,
|
118
208
|
"dict": json.dumps,
|
119
|
-
"tuple":
|
120
|
-
type(set()).__name__:
|
209
|
+
"tuple": _encode_tuple,
|
210
|
+
type(set()).__name__: _encode_set,
|
121
211
|
}
|
122
212
|
|
123
213
|
def __init__(self,
|
@@ -131,16 +221,23 @@ class RedisDict:
|
|
131
221
|
Args:
|
132
222
|
namespace (str, optional): A prefix for keys stored in Redis.
|
133
223
|
expire (int, timedelta, optional): Expiration time for keys in seconds.
|
134
|
-
preserve_expiration (bool, optional):
|
224
|
+
preserve_expiration (bool, optional): Preserve the expiration count when the key is updated.
|
135
225
|
**redis_kwargs: Additional keyword arguments passed to StrictRedis.
|
136
226
|
"""
|
137
|
-
|
227
|
+
|
138
228
|
self.namespace: str = namespace
|
139
229
|
self.expire: Union[int, timedelta, None] = expire
|
140
230
|
self.preserve_expiration: Optional[bool] = preserve_expiration
|
141
231
|
self.redis: StrictRedis[Any] = StrictRedis(decode_responses=True, **redis_kwargs)
|
142
232
|
self.get_redis: StrictRedis[Any] = self.redis
|
143
233
|
|
234
|
+
self.custom_encode_method = "encode"
|
235
|
+
self.custom_decode_method = "decode"
|
236
|
+
|
237
|
+
self._iter: Iterator[str] = iter([])
|
238
|
+
self._max_string_size: int = 500 * 1024 * 1024 # 500mb
|
239
|
+
self._temp_redis: Optional[StrictRedis[Any]] = None
|
240
|
+
|
144
241
|
def _format_key(self, key: str) -> str:
|
145
242
|
"""
|
146
243
|
Format a key with the namespace prefix.
|
@@ -151,7 +248,7 @@ class RedisDict:
|
|
151
248
|
Returns:
|
152
249
|
str: The formatted key with the namespace prefix.
|
153
250
|
"""
|
154
|
-
return '{}:{
|
251
|
+
return f'{self.namespace}:{str(key)}'
|
155
252
|
|
156
253
|
def _valid_input(self, val: Any, val_type: str) -> bool:
|
157
254
|
"""
|
@@ -169,7 +266,7 @@ class RedisDict:
|
|
169
266
|
bool: True if the input value is valid, False otherwise.
|
170
267
|
"""
|
171
268
|
if val_type == "str":
|
172
|
-
return len(val) <
|
269
|
+
return len(val) < self._max_string_size
|
173
270
|
return True
|
174
271
|
|
175
272
|
def _store(self, key: str, value: Any) -> None:
|
@@ -179,14 +276,21 @@ class RedisDict:
|
|
179
276
|
Args:
|
180
277
|
key (str): The key to store the value.
|
181
278
|
value (Any): The value to be stored.
|
279
|
+
|
280
|
+
Note: Validity checks could be refactored to allow for custom exceptions that inherit from ValueError,
|
281
|
+
providing detailed information about why a specific validation failed.
|
282
|
+
This would enable users to specify which validity checks should be executed, add custom validity functions,
|
283
|
+
and choose whether to fail on validation errors, or drop the data and only issue a warning and continue.
|
284
|
+
Example use case is caching, to cache data only when it's between min and max sizes.
|
285
|
+
Allowing for simple dict set operation, but only cache data that makes sense.
|
286
|
+
|
182
287
|
"""
|
183
288
|
store_type, key = type(value).__name__, str(key)
|
184
289
|
if not self._valid_input(value, store_type) or not self._valid_input(key, "str"):
|
185
|
-
# TODO When needed, make valid_input, pass the reason, or throw a exception.
|
186
290
|
raise ValueError("Invalid input value or key size exceeded the maximum limit.")
|
187
|
-
value = self.
|
291
|
+
value = self.encoding_registry.get(store_type, lambda x: x)(value) # type: ignore
|
188
292
|
|
189
|
-
store_value = '{}:{}'
|
293
|
+
store_value = f'{store_type}:{value}'
|
190
294
|
formatted_key = self._format_key(key)
|
191
295
|
|
192
296
|
if self.preserve_expiration and self.redis.exists(formatted_key):
|
@@ -207,8 +311,8 @@ class RedisDict:
|
|
207
311
|
result = self.get_redis.get(self._format_key(key))
|
208
312
|
if result is None:
|
209
313
|
return False, None
|
210
|
-
|
211
|
-
return True, self.
|
314
|
+
type_, value = result.split(':', 1)
|
315
|
+
return True, self.decoding_registry.get(type_, lambda x: x)(value)
|
212
316
|
|
213
317
|
def _transform(self, result: str) -> Any:
|
214
318
|
"""
|
@@ -220,18 +324,118 @@ class RedisDict:
|
|
220
324
|
Returns:
|
221
325
|
Any: The transformed Python object.
|
222
326
|
"""
|
223
|
-
|
224
|
-
return self.
|
327
|
+
type_, value = result.split(':', 1)
|
328
|
+
return self.decoding_registry.get(type_, lambda x: x)(value)
|
225
329
|
|
226
|
-
def
|
330
|
+
def new_type_compliance(
|
331
|
+
self,
|
332
|
+
class_type: type,
|
333
|
+
encode_method_name: Optional[str] = None,
|
334
|
+
decode_method_name: Optional[str] = None,
|
335
|
+
) -> None:
|
227
336
|
"""
|
228
|
-
|
337
|
+
Checks if a class complies with the required encoding and decoding methods.
|
229
338
|
|
230
339
|
Args:
|
231
|
-
|
232
|
-
|
340
|
+
class_type (type): The class to check for compliance.
|
341
|
+
encode_method_name (str, optional): Name of encoding method of the class for redis-dict custom types.
|
342
|
+
decode_method_name (str, optional): Name of decoding method of the class for redis-dict custom types.
|
343
|
+
|
344
|
+
Raises:
|
345
|
+
NotImplementedError: If the class does not implement the required methods when the respective check is True.
|
346
|
+
"""
|
347
|
+
if encode_method_name is not None:
|
348
|
+
if not (hasattr(class_type, encode_method_name) and callable(
|
349
|
+
getattr(class_type, encode_method_name))):
|
350
|
+
raise NotImplementedError(
|
351
|
+
f"Class {class_type.__name__} does not implement the required {encode_method_name} method.")
|
352
|
+
|
353
|
+
if decode_method_name is not None:
|
354
|
+
if not (hasattr(class_type, decode_method_name) and callable(
|
355
|
+
getattr(class_type, decode_method_name))):
|
356
|
+
raise NotImplementedError(
|
357
|
+
f"Class {class_type.__name__} does not implement the required {decode_method_name} class method.")
|
358
|
+
|
359
|
+
def extends_type(
|
360
|
+
self,
|
361
|
+
class_type: type,
|
362
|
+
encode: Optional[EncodeFuncType] = None,
|
363
|
+
decode: Optional[DecodeFuncType] = None,
|
364
|
+
encoding_method_name: Optional[str] = None,
|
365
|
+
decoding_method_name: Optional[str] = None,
|
366
|
+
) -> None:
|
367
|
+
"""
|
368
|
+
Extends RedisDict to support a custom type in the encode/decode mapping.
|
369
|
+
|
370
|
+
This method enables serialization of instances based on their type,
|
371
|
+
allowing for custom types, specialized storage formats, and more.
|
372
|
+
There are three ways to add custom types:
|
373
|
+
1. Have a class with an `encode` instance method and a `decode` class method.
|
374
|
+
2. Have a class and pass encoding and decoding functions, where
|
375
|
+
`encode` converts the class instance to a string, and
|
376
|
+
`decode` takes the string and recreates the class instance.
|
377
|
+
3. Have a class that already has serialization methods, that satisfies the:
|
378
|
+
EncodeFuncType = Callable[[Any], str]
|
379
|
+
DecodeFuncType = Callable[[str], Any]
|
380
|
+
|
381
|
+
`custom_encode_method`
|
382
|
+
`custom_decode_method` attributes.
|
383
|
+
|
384
|
+
Args:
|
385
|
+
class_type (Type[type]): The class `__name__` will become the key for the encoding and decoding functions.
|
386
|
+
encode (Optional[EncodeFuncType]): function that encodes an object into a storable string format.
|
387
|
+
This function should take an instance of `class_type` as input and return a string.
|
388
|
+
decode (Optional[DecodeFuncType]): function that decodes a string back into an object of `class_type`.
|
389
|
+
This function should take a string as input and return an instance of `class_type`.
|
390
|
+
encoding_method_name (str, optional): Name of encoding method of the class for redis-dict custom types.
|
391
|
+
decoding_method_name (str, optional): Name of decoding method of the class for redis-dict custom types.
|
392
|
+
|
393
|
+
If no encoding or decoding function is provided, default to use the `encode` and `decode` methods of the class.
|
394
|
+
|
395
|
+
The `encode` method should be an instance method that converts the object to a string.
|
396
|
+
The `decode` method should be a class method that takes a string and returns an instance of the class.
|
397
|
+
|
398
|
+
The method names for encoding and decoding can be changed by modifying the
|
399
|
+
- `custom_encode_method`
|
400
|
+
- `custom_decode_method`
|
401
|
+
attributes of the RedisDict instance
|
402
|
+
|
403
|
+
Example:
|
404
|
+
class Person:
|
405
|
+
def __init__(self, name, age):
|
406
|
+
self.name = name
|
407
|
+
self.age = age
|
408
|
+
|
409
|
+
def encode(self) -> str:
|
410
|
+
return json.dumps(self.__dict__)
|
411
|
+
|
412
|
+
@classmethod
|
413
|
+
def decode(cls, encoded_str: str) -> 'Person':
|
414
|
+
return cls(**json.loads(encoded_str))
|
415
|
+
|
416
|
+
redis_dict.extends_type(Person)
|
417
|
+
|
418
|
+
Note:
|
419
|
+
You can check for compliance of a class separately using the `new_type_compliance` method:
|
420
|
+
|
421
|
+
This method raises a NotImplementedError if either `encode` or `decode` is `None`
|
422
|
+
and the class does not implement the corresponding method.
|
233
423
|
"""
|
234
|
-
|
424
|
+
|
425
|
+
if encode is None or decode is None:
|
426
|
+
encode_method_name = encoding_method_name or self.custom_encode_method
|
427
|
+
if encode is None:
|
428
|
+
self.new_type_compliance(class_type, encode_method_name=encode_method_name)
|
429
|
+
encode = _create_default_encode(encode_method_name)
|
430
|
+
|
431
|
+
if decode is None:
|
432
|
+
decode_method_name = decoding_method_name or self.custom_decode_method
|
433
|
+
self.new_type_compliance(class_type, decode_method_name=decode_method_name)
|
434
|
+
decode = _create_default_decode(class_type, decode_method_name)
|
435
|
+
|
436
|
+
type_name = class_type.__name__
|
437
|
+
self.decoding_registry[type_name] = decode
|
438
|
+
self.encoding_registry[type_name] = encode
|
235
439
|
|
236
440
|
def __eq__(self, other: Any) -> bool:
|
237
441
|
"""
|
@@ -327,7 +531,7 @@ class RedisDict:
|
|
327
531
|
Returns:
|
328
532
|
Iterator[str]: An iterator over the keys of the RedisDict.
|
329
533
|
"""
|
330
|
-
self.
|
534
|
+
self._iter = self.iterkeys()
|
331
535
|
return self
|
332
536
|
|
333
537
|
def __repr__(self) -> str:
|
@@ -358,7 +562,7 @@ class RedisDict:
|
|
358
562
|
Raises:
|
359
563
|
StopIteration: If there are no more items.
|
360
564
|
"""
|
361
|
-
return next(self.
|
565
|
+
return next(self._iter)
|
362
566
|
|
363
567
|
def next(self) -> str:
|
364
568
|
"""
|
@@ -370,7 +574,29 @@ class RedisDict:
|
|
370
574
|
Raises:
|
371
575
|
StopIteration: If there are no more items.
|
372
576
|
"""
|
373
|
-
return self
|
577
|
+
return next(self)
|
578
|
+
|
579
|
+
def _create_iter_query(self, search_term: str) -> str:
|
580
|
+
"""
|
581
|
+
Create a Redis query string for iterating over keys based on the given search term.
|
582
|
+
|
583
|
+
This method constructs a query by prefixing the search term with the namespace
|
584
|
+
followed by a wildcard to facilitate scanning for keys that start with the
|
585
|
+
provided search term.
|
586
|
+
|
587
|
+
Args:
|
588
|
+
search_term (str): The term to search for in Redis keys.
|
589
|
+
|
590
|
+
Returns:
|
591
|
+
str: A formatted query string that can be used to find keys in Redis.
|
592
|
+
|
593
|
+
Example:
|
594
|
+
>>> d = RedisDict(namespace='foo')
|
595
|
+
>>> query = self._create_iter_query('bar')
|
596
|
+
>>> print(query)
|
597
|
+
'foo:bar*'
|
598
|
+
"""
|
599
|
+
return f'{self.namespace}:{search_term}*'
|
374
600
|
|
375
601
|
def _scan_keys(self, search_term: str = '') -> Iterator[str]:
|
376
602
|
"""
|
@@ -382,7 +608,8 @@ class RedisDict:
|
|
382
608
|
Returns:
|
383
609
|
Iterator[str]: An iterator of matching Redis keys.
|
384
610
|
"""
|
385
|
-
|
611
|
+
search_query = self._create_iter_query(search_term)
|
612
|
+
return self.get_redis.scan_iter(match=search_query)
|
386
613
|
|
387
614
|
def get(self, key: str, default: Optional[Any] = None) -> Any:
|
388
615
|
"""
|
@@ -403,17 +630,18 @@ class RedisDict:
|
|
403
630
|
|
404
631
|
def iterkeys(self) -> Iterator[str]:
|
405
632
|
"""
|
406
|
-
Note: for
|
633
|
+
Note: for python2 str is needed
|
407
634
|
"""
|
408
635
|
to_rm = len(self.namespace) + 1
|
409
636
|
return (str(item[to_rm:]) for item in self._scan_keys())
|
410
637
|
|
411
638
|
def key(self, search_term: str = '') -> Optional[str]:
|
412
639
|
"""
|
413
|
-
Note: for
|
640
|
+
Note: for python2 str is needed
|
414
641
|
"""
|
415
642
|
to_rm = len(self.namespace) + 1
|
416
|
-
|
643
|
+
search_query = self._create_iter_query(search_term)
|
644
|
+
_, data = self.get_redis.scan(match=search_query, count=1)
|
417
645
|
for item in data:
|
418
646
|
return str(item[to_rm:])
|
419
647
|
|
@@ -435,7 +663,7 @@ class RedisDict:
|
|
435
663
|
to_rm = len(self.namespace) + 1
|
436
664
|
for item in self._scan_keys():
|
437
665
|
try:
|
438
|
-
yield
|
666
|
+
yield str(item[to_rm:]), self[item[to_rm:]]
|
439
667
|
except KeyError:
|
440
668
|
pass
|
441
669
|
|
@@ -488,11 +716,12 @@ class RedisDict:
|
|
488
716
|
Redis pipelining is employed to group multiple commands into a single request, minimizing
|
489
717
|
network round-trip time, latency, and I/O load, thereby enhancing the overall performance.
|
490
718
|
|
491
|
-
It is important to highlight that the clear method can be safely executed within
|
719
|
+
It is important to highlight that the clear method can be safely executed within
|
720
|
+
the context of an initiated pipeline operation.
|
492
721
|
"""
|
493
722
|
with self.pipeline():
|
494
723
|
for key in self:
|
495
|
-
del
|
724
|
+
del self[key]
|
496
725
|
|
497
726
|
def pop(self, key: str, default: Union[Any, object] = SENTINEL) -> Any:
|
498
727
|
"""
|
@@ -516,7 +745,7 @@ class RedisDict:
|
|
516
745
|
return default
|
517
746
|
raise
|
518
747
|
|
519
|
-
del
|
748
|
+
del self[key]
|
520
749
|
return value
|
521
750
|
|
522
751
|
def popitem(self) -> Tuple[str, Any]:
|
@@ -546,7 +775,7 @@ class RedisDict:
|
|
546
775
|
|
547
776
|
Args:
|
548
777
|
key (str): The key to retrieve the value.
|
549
|
-
|
778
|
+
default_value (Optional[Any], optional): The value to set if the key is not found.
|
550
779
|
|
551
780
|
Returns:
|
552
781
|
Any: The value associated with the key or the default value.
|
@@ -575,7 +804,7 @@ class RedisDict:
|
|
575
804
|
Update the RedisDict with key-value pairs from the given mapping, analogous to a dictionary's update method.
|
576
805
|
|
577
806
|
Args:
|
578
|
-
|
807
|
+
dic (Mapping[str, Any]): A mapping containing key-value pairs to update the RedisDict.
|
579
808
|
"""
|
580
809
|
with self.pipeline():
|
581
810
|
for key, value in dic.items():
|
@@ -594,7 +823,8 @@ class RedisDict:
|
|
594
823
|
value (Optional[Any], optional): The value to be assigned to each key in the RedisDict. Defaults to None.
|
595
824
|
|
596
825
|
Returns:
|
597
|
-
RedisDict: The current RedisDict instance,
|
826
|
+
RedisDict: The current RedisDict instance,populated with the keys from the iterable and their
|
827
|
+
corresponding values.
|
598
828
|
"""
|
599
829
|
for key in iterable:
|
600
830
|
self[key] = value
|
@@ -640,7 +870,7 @@ class RedisDict:
|
|
640
870
|
Args:
|
641
871
|
iterable (List[str]): A list of keys representing the chain.
|
642
872
|
"""
|
643
|
-
|
873
|
+
del self[':'.join(iterable)]
|
644
874
|
|
645
875
|
# def expire_at(self, sec_epoch: int | timedelta) -> Iterator[None]:
|
646
876
|
# compatibility with Python 3.9 typing
|
@@ -668,13 +898,13 @@ class RedisDict:
|
|
668
898
|
ContextManager: A context manager to create a Redis pipeline batching all operations within the context.
|
669
899
|
"""
|
670
900
|
top_level = False
|
671
|
-
if self.
|
672
|
-
self.redis, self.
|
901
|
+
if self._temp_redis is None:
|
902
|
+
self.redis, self._temp_redis, top_level = self.redis.pipeline(), self.redis, True
|
673
903
|
try:
|
674
904
|
yield
|
675
905
|
finally:
|
676
906
|
if top_level:
|
677
|
-
_, self.
|
907
|
+
_, self._temp_redis, self.redis = self.redis.execute(), None, self._temp_redis # type: ignore
|
678
908
|
|
679
909
|
def multi_get(self, key: str) -> List[Any]:
|
680
910
|
"""
|
@@ -717,7 +947,9 @@ class RedisDict:
|
|
717
947
|
if len(keys) == 0:
|
718
948
|
return {}
|
719
949
|
to_rm = keys[0].rfind(':') + 1
|
720
|
-
return dict(
|
950
|
+
return dict(
|
951
|
+
zip([i[to_rm:] for i in keys], (self._transform(i) for i in self.redis.mget(keys) if i is not None))
|
952
|
+
)
|
721
953
|
|
722
954
|
def multi_del(self, key: str) -> int:
|
723
955
|
"""
|
@@ -746,13 +978,13 @@ class RedisDict:
|
|
746
978
|
def get_ttl(self, key: str) -> Optional[int]:
|
747
979
|
"""
|
748
980
|
Get the Time To Live (TTL) in seconds for a given key. If the key does not exist or does not have an
|
749
|
-
associated expire
|
981
|
+
associated `expire`, return None.
|
750
982
|
|
751
983
|
Args:
|
752
984
|
key (str): The key for which to get the TTL.
|
753
985
|
|
754
986
|
Returns:
|
755
|
-
Optional[int]: The TTL in seconds if the key exists and has an
|
987
|
+
Optional[int]: The TTL in seconds if the key exists and has an expiry set; otherwise, None.
|
756
988
|
"""
|
757
989
|
val = self.redis.ttl(self._format_key(key))
|
758
990
|
if val < 0:
|
@@ -1,6 +0,0 @@
|
|
1
|
-
redis_dict.py,sha256=xB36Vby3FIIuSoRSqfTT1XuVXi-ZpQPHmUQoUf_HOTg,25661
|
2
|
-
redis_dict-2.5.0.dist-info/LICENSE,sha256=-QiLwYznh_vNUSz337k0faP9Jl0dgtCIHVZ39Uyl6cA,1070
|
3
|
-
redis_dict-2.5.0.dist-info/METADATA,sha256=e0bB_4MTF5C7lHFt0eRPqBRB2QocOJjmNDs80gaur7w,9847
|
4
|
-
redis_dict-2.5.0.dist-info/WHEEL,sha256=oiQVh_5PnQM0E3gPdiz09WCNmwiHDMaGer_elqB3coM,92
|
5
|
-
redis_dict-2.5.0.dist-info/top_level.txt,sha256=Wyp5Xvq_imoxvu-c-Le1rbTZ3pYM5BF440H9YAcgBZ8,11
|
6
|
-
redis_dict-2.5.0.dist-info/RECORD,,
|
File without changes
|
File without changes
|