persidict 0.36.1__py3-none-any.whl → 0.36.2__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of persidict might be problematic. Click here for more details.
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.3
|
|
2
2
|
Name: persidict
|
|
3
|
-
Version: 0.36.
|
|
3
|
+
Version: 0.36.2
|
|
4
4
|
Summary: Simple persistent key-value store for Python. Values are stored as files on a disk or as S3 objects on AWS cloud.
|
|
5
5
|
Keywords: persistence,dicts,distributed,parallel
|
|
6
6
|
Author: Vlad (Volodymyr) Pavlov
|
|
@@ -38,7 +38,7 @@ Simple persistent dictionaries for distributed applications in Python.
|
|
|
38
38
|
|
|
39
39
|
## 1. What Is It?
|
|
40
40
|
|
|
41
|
-
|
|
41
|
+
`persidict` offers a simple persistent key-value store for Python.
|
|
42
42
|
It saves the content of the dictionary in a folder on a disk
|
|
43
43
|
or in an S3 bucket on AWS. Each value is stored as a separate file / S3 object.
|
|
44
44
|
Only text strings or sequences of strings are allowed as keys.
|
|
@@ -48,11 +48,26 @@ Unlike other persistent dictionaries (e.g. Python's native `shelve`),
|
|
|
48
48
|
where multiple instances of a program run concurrently across many machines,
|
|
49
49
|
accessing the same dictionary via a shared storage.
|
|
50
50
|
|
|
51
|
-
## 2.
|
|
51
|
+
## 2. Features
|
|
52
|
+
* **Persistent Storage**: Save dictionaries to the local filesystem
|
|
53
|
+
(`FileDirDict`) or AWS S3 (`S3Dict`).
|
|
54
|
+
* **Standard Dictionary API**: Use persidict objects like standard
|
|
55
|
+
Python dictionaries with methods like `__getitem__`, `__setitem__`,
|
|
56
|
+
`__delitem__`, `keys`, `values`, `items`, etc.
|
|
57
|
+
* **Distributed Computing Ready**: Designed for concurrent access
|
|
58
|
+
in distributed environments.
|
|
59
|
+
* **Flexible Serialization**: Store values as pickles (`pkl`),
|
|
60
|
+
JSON (`json`), or plain text.
|
|
61
|
+
* **Type Safety**: Optionally enforce that all values in a dictionary are
|
|
62
|
+
instances of a specific class.
|
|
63
|
+
* **Advanced Functionality**: Includes features like write-once dictionaries,
|
|
64
|
+
timestamping of entries, and tools for handling file-system-safe keys.
|
|
65
|
+
|
|
66
|
+
## 3. Usage
|
|
52
67
|
|
|
53
|
-
###
|
|
68
|
+
### 3.1 Storing Data on a Local Disk
|
|
54
69
|
|
|
55
|
-
The
|
|
70
|
+
The `FileDirDict` class saves your dictionary to a local folder.
|
|
56
71
|
Each key-value pair is stored as a separate file.
|
|
57
72
|
|
|
58
73
|
```python
|
|
@@ -84,7 +99,7 @@ print(f"Number of settings: {len(reloaded_settings)}")
|
|
|
84
99
|
print("username" in reloaded_settings)
|
|
85
100
|
# >>> True
|
|
86
101
|
```
|
|
87
|
-
###
|
|
102
|
+
### 3.2 Storing Data in the Cloud (AWS S3)
|
|
88
103
|
|
|
89
104
|
For distributed applications, you can use **`S3Dict`** to store data in
|
|
90
105
|
an AWS S3 bucket. The usage is identical, allowing you to switch
|
|
@@ -105,10 +120,9 @@ print(f"API Key: {cloud_config['api_key']}")
|
|
|
105
120
|
# >>> API Key: ABC-123-XYZ
|
|
106
121
|
```
|
|
107
122
|
|
|
123
|
+
## 4. Glossary
|
|
108
124
|
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
### 3.1 Core Concepts
|
|
125
|
+
### 4.1 Core Concepts
|
|
112
126
|
|
|
113
127
|
* **`PersiDict`**: The abstract base class that defines the common interface
|
|
114
128
|
for all persistent dictionaries in the package. It's the foundation
|
|
@@ -120,7 +134,7 @@ a single string, or a sequence of strings.
|
|
|
120
134
|
flat tuple of non-empty, URL/filename-safe strings, ensuring that
|
|
121
135
|
keys are consistent and safe for various storage backends.
|
|
122
136
|
|
|
123
|
-
###
|
|
137
|
+
### 4.2 Main Implementations
|
|
124
138
|
|
|
125
139
|
* **`FileDirDict`**: A primary, concrete implementation of `PersiDict`
|
|
126
140
|
that stores each key-value pair as a separate file in a local directory.
|
|
@@ -128,7 +142,7 @@ that stores each key-value pair as a separate file in a local directory.
|
|
|
128
142
|
which stores each key-value pair as an object in an AWS S3 bucket,
|
|
129
143
|
suitable for distributed environments.
|
|
130
144
|
|
|
131
|
-
###
|
|
145
|
+
### 4.3 Key Parameters
|
|
132
146
|
|
|
133
147
|
* **`file_type`**: A key parameter for `FileDirDict` and `S3Dict` that
|
|
134
148
|
determines the serialization format for values.
|
|
@@ -141,16 +155,23 @@ instances of a specific class.
|
|
|
141
155
|
"write-once," preventing any modification or deletion of existing items.
|
|
142
156
|
* **`digest_len`**: An integer that specifies the length of a hash suffix
|
|
143
157
|
added to key components to prevent collisions on case-insensitive file systems.
|
|
158
|
+
* **`base_dir`**: A string specifying the directory path where a `FileDirDict`
|
|
159
|
+
stores its files. For `S3Dict`, this directory is used to cache files locally.
|
|
160
|
+
* **`bucket_name`**: A string specifying the name of the S3 bucket where
|
|
161
|
+
an `S3Dict` stores its objects.
|
|
162
|
+
* **`region`**: An optional string specifying the AWS region for the S3 bucket.
|
|
144
163
|
|
|
145
|
-
###
|
|
164
|
+
### 4.4 Advanced Classes
|
|
146
165
|
|
|
147
166
|
* **`WriteOnceDict`**: A wrapper that enforces write-once behavior
|
|
148
|
-
on any `PersiDict`, ignoring subsequent writes to the same key.
|
|
167
|
+
on any `PersiDict`, ignoring subsequent writes to the same key.
|
|
168
|
+
It also allows for random consistency checks to ensure subsequent
|
|
169
|
+
writes to the same key always match the original value.
|
|
149
170
|
* **`OverlappingMultiDict`**: An advanced container that holds
|
|
150
171
|
multiple `PersiDict` instances sharing the same storage
|
|
151
172
|
but with different `file_type`s.
|
|
152
173
|
|
|
153
|
-
###
|
|
174
|
+
### 4.5 Special "Joker" Values
|
|
154
175
|
|
|
155
176
|
* **`Joker`**: The base class for special command-like values that
|
|
156
177
|
can be assigned to a key to trigger an action instead of storing a value.
|
|
@@ -159,41 +180,33 @@ ensures the existing value is not changed.
|
|
|
159
180
|
* **`DELETE_CURRENT`**: A "joker" value that deletes the key-value pair
|
|
160
181
|
from the dictionary when assigned to a key.
|
|
161
182
|
|
|
162
|
-
##
|
|
163
|
-
|
|
164
|
-
### 4.1 Similarities
|
|
183
|
+
## 5. Comparison With Python Built-in Dictionaries
|
|
165
184
|
|
|
166
|
-
|
|
185
|
+
### 5.1 Similarities
|
|
167
186
|
|
|
168
|
-
|
|
169
|
-
* You can iterate over keys, values, or items.
|
|
170
|
-
* You can check if a key is in the dictionary.
|
|
171
|
-
* You can check whether two dicts are equal
|
|
172
|
-
(meaning they contain the same key-value pairs).
|
|
173
|
-
* You can get the length of the dictionary.
|
|
174
|
-
* Methods `keys()`, `values()`, `items()`, `get()`, `clear()`
|
|
175
|
-
, `setdefault()`, `update()` etc. work as expected.
|
|
187
|
+
`PersiDict` subclasses can be used like regular Python dictionaries, supporting:
|
|
176
188
|
|
|
177
|
-
|
|
189
|
+
* Get, set, and delete operations with square brackets (`[]`).
|
|
190
|
+
* Iteration over keys, values, and items.
|
|
191
|
+
* Membership testing with `in`.
|
|
192
|
+
* Length checking with `len()`.
|
|
193
|
+
* Standard methods like `keys()`, `values()`, `items()`, `get()`, `clear()`
|
|
194
|
+
, `setdefault()`, and `update()`.
|
|
178
195
|
|
|
179
|
-
|
|
180
|
-
as well as make it possible to concurrently run programs
|
|
181
|
-
that simultaneously work with the same instance of a dictionary.
|
|
196
|
+
### 5.2 Differences
|
|
182
197
|
|
|
183
|
-
*
|
|
184
|
-
*
|
|
185
|
-
*
|
|
186
|
-
|
|
187
|
-
*
|
|
188
|
-
*
|
|
189
|
-
`
|
|
190
|
-
`
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
to avoid actually setting/updating a value. Or `DELETE_CURRENT` as
|
|
194
|
-
a fake new value to delete the previous value from a dictionary.
|
|
198
|
+
* **Persistence**: Data is saved between program executions.
|
|
199
|
+
* **Keys**: Keys must be strings or sequences of URL/filename-safe strings.
|
|
200
|
+
* **Values**: Values must be pickleable.
|
|
201
|
+
You can also constrain values to a specific class.
|
|
202
|
+
* **Order**: Insertion order is not preserved.
|
|
203
|
+
* **Additional Methods**: `PersiDict` provides extra methods not in the standard
|
|
204
|
+
dict API, such as `timestamp()`, `random_key()`, `newest_keys()`, `subdicts()`
|
|
205
|
+
, `delete_if_exists()`, `get_params()` and more.
|
|
206
|
+
* **Special Values**: Use `KEEP_CURRENT` to avoid updating a value
|
|
207
|
+
and `DELETE_CURRENT` to delete a value during an assignment.
|
|
195
208
|
|
|
196
|
-
##
|
|
209
|
+
## 6. Installation
|
|
197
210
|
|
|
198
211
|
The source code is hosted on GitHub at:
|
|
199
212
|
[https://github.com/pythagoras-dev/persidict](https://github.com/pythagoras-dev/persidict)
|
|
@@ -201,27 +214,50 @@ The source code is hosted on GitHub at:
|
|
|
201
214
|
Binary installers for the latest released version are available at the Python package index at:
|
|
202
215
|
[https://pypi.org/project/persidict](https://pypi.org/project/persidict)
|
|
203
216
|
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
217
|
+
You can install `persidict` using `pip` or your favorite package manager:
|
|
218
|
+
|
|
219
|
+
```bash
|
|
220
|
+
pip install persidict
|
|
207
221
|
```
|
|
208
222
|
|
|
209
|
-
|
|
223
|
+
To include the AWS S3 extra dependencies:
|
|
224
|
+
|
|
225
|
+
```bash
|
|
226
|
+
pip install persidict[aws]
|
|
210
227
|
```
|
|
211
|
-
|
|
228
|
+
|
|
229
|
+
For development, including test dependencies:
|
|
230
|
+
|
|
231
|
+
```bash
|
|
232
|
+
pip install persidict[dev]
|
|
212
233
|
```
|
|
213
234
|
|
|
214
|
-
##
|
|
235
|
+
## 7. Dependencies
|
|
236
|
+
|
|
237
|
+
`persidict` has the following core dependencies:
|
|
215
238
|
|
|
239
|
+
* [parameterizable](https://pypi.org/project/parameterizable/)
|
|
216
240
|
* [jsonpickle](https://jsonpickle.github.io)
|
|
217
241
|
* [joblib](https://joblib.readthedocs.io)
|
|
218
242
|
* [lz4](https://python-lz4.readthedocs.io)
|
|
219
243
|
* [pandas](https://pandas.pydata.org)
|
|
220
244
|
* [numpy](https://numpy.org)
|
|
245
|
+
* [deepdiff](https://zepworks.com/deepdiff)
|
|
246
|
+
|
|
247
|
+
For AWS S3 support (S3Dict), you will also need:
|
|
221
248
|
* [boto3](https://boto3.readthedocs.io)
|
|
249
|
+
|
|
250
|
+
For development and testing, the following packages are used:
|
|
222
251
|
* [pytest](https://pytest.org)
|
|
223
252
|
* [moto](http://getmoto.org)
|
|
224
253
|
|
|
225
|
-
##
|
|
254
|
+
## 8. Contributing
|
|
255
|
+
Contributions are welcome! Please see the contributing [guide](https://github.com/pythagoras-dev/persidict?tab=contributing-ov-file) for more details
|
|
256
|
+
on how to get started, run tests, and submit pull requests.
|
|
257
|
+
|
|
258
|
+
## 9. License
|
|
259
|
+
`persidict` is licensed under the MIT License. See the [LICENSE](https://github.com/pythagoras-dev/persidict?tab=MIT-1-ov-file) file for more details.
|
|
260
|
+
|
|
261
|
+
## 10. Key Contacts
|
|
226
262
|
|
|
227
263
|
* [Vlad (Volodymyr) Pavlov](https://www.linkedin.com/in/vlpavlov/)
|
|
@@ -9,6 +9,6 @@ persidict/safe_chars.py,sha256=9Qy24fu2dmiJOdmCF8mKZULfQaRp7H4oxfgDXeLgogI,1160
|
|
|
9
9
|
persidict/safe_str_tuple.py,sha256=YBTcYjUKIffznOawXb9xKjz4HaKdklrgyVtegJFmr5w,7202
|
|
10
10
|
persidict/safe_str_tuple_signing.py,sha256=RQAj4fnpRVaOe0KpwLler1UTaeNOgXCQpU3t80ixtxg,7493
|
|
11
11
|
persidict/write_once_dict.py,sha256=-lPQ_yuU62pczHT0BYO6SFbiZBKFq8Tj9ln3jCzNDzA,11443
|
|
12
|
-
persidict-0.36.
|
|
13
|
-
persidict-0.36.
|
|
14
|
-
persidict-0.36.
|
|
12
|
+
persidict-0.36.2.dist-info/WHEEL,sha256=Pi5uDq5Fdo_Rr-HD5h9BiPn9Et29Y9Sh8NhcJNnFU1c,79
|
|
13
|
+
persidict-0.36.2.dist-info/METADATA,sha256=KS86C2ZjXL6VsvpBkz6ah1xPl6XimlmmqywDdbnnfhs,10021
|
|
14
|
+
persidict-0.36.2.dist-info/RECORD,,
|
|
File without changes
|