persistedstate 24.2__tar.gz → 25.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- persistedstate-25.1/.gitignore +9 -0
- persistedstate-25.1/.pylintrc +16 -0
- persistedstate-25.1/CHANGELOG.md +47 -0
- {persistedstate-24.2/src/persistedstate.egg-info → persistedstate-25.1}/PKG-INFO +100 -112
- persistedstate-25.1/example.py +10 -0
- persistedstate-25.1/justfile +52 -0
- persistedstate-25.1/loggertest.py +14 -0
- persistedstate-25.1/mypy.ini +6 -0
- persistedstate-25.1/perftest.py +121 -0
- {persistedstate-24.2 → persistedstate-25.1}/pyproject.toml +60 -52
- persistedstate-25.1/test_mappedyaml.py +124 -0
- persistedstate-25.1/test_persistedstate.py +30 -0
- persistedstate-25.1/test_thread.py +46 -0
- persistedstate-24.2/PKG-INFO +0 -112
- persistedstate-24.2/setup.cfg +0 -4
- persistedstate-24.2/src/persistedstate.egg-info/SOURCES.txt +0 -10
- persistedstate-24.2/src/persistedstate.egg-info/dependency_links.txt +0 -1
- persistedstate-24.2/src/persistedstate.egg-info/requires.txt +0 -18
- persistedstate-24.2/src/persistedstate.egg-info/top_level.txt +0 -1
- {persistedstate-24.2 → persistedstate-25.1}/LICENSE +0 -0
- {persistedstate-24.2 → persistedstate-25.1}/README.md +0 -0
- {persistedstate-24.2 → persistedstate-25.1}/src/persistedstate/__init__.py +0 -0
- {persistedstate-24.2 → persistedstate-25.1}/src/persistedstate/py.typed +0 -0
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
[MAIN]
|
|
2
|
+
disable=
|
|
3
|
+
missing-docstring,
|
|
4
|
+
format,
|
|
5
|
+
logging-not-lazy,
|
|
6
|
+
protected-access,
|
|
7
|
+
logging-fstring-interpolation,
|
|
8
|
+
|
|
9
|
+
[persisstedstate]
|
|
10
|
+
disable=consider-using-with
|
|
11
|
+
|
|
12
|
+
[test_persistedstate]
|
|
13
|
+
disable=attribute-defined-outside-init
|
|
14
|
+
|
|
15
|
+
[perftest]
|
|
16
|
+
disable=no-member, invalid-name
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# 25.1
|
|
2
|
+
|
|
3
|
+
- Add Python 3.14 support
|
|
4
|
+
|
|
5
|
+
# 24.2
|
|
6
|
+
|
|
7
|
+
- Remove Python 3.8 support
|
|
8
|
+
|
|
9
|
+
# 24.1
|
|
10
|
+
|
|
11
|
+
- Official Python 3.13 support
|
|
12
|
+
|
|
13
|
+
# 23.9
|
|
14
|
+
|
|
15
|
+
- Changing the state object is thread safe now.
|
|
16
|
+
|
|
17
|
+
# 23.8
|
|
18
|
+
|
|
19
|
+
- Remove Python 3.7 support
|
|
20
|
+
- Add Python 3.12 support
|
|
21
|
+
|
|
22
|
+
# 23.6
|
|
23
|
+
|
|
24
|
+
- Fix crash when the PersistedState is subclassed or constructed from another module
|
|
25
|
+
- Make vacuum more failure tolerant. In some edge cases it was possible for the state at beginning to overlap the state at the end.
|
|
26
|
+
- Internal: use `pyproject.toml` instead of `setup.py`
|
|
27
|
+
|
|
28
|
+
# 23.5
|
|
29
|
+
|
|
30
|
+
- Map the whole YAML file, so modification of nested objects are also persisted immediately
|
|
31
|
+
|
|
32
|
+
# 23.4
|
|
33
|
+
|
|
34
|
+
- In-file vacuuming
|
|
35
|
+
|
|
36
|
+
# 23.3
|
|
37
|
+
|
|
38
|
+
- Use YAML file format, not just a subset of YAML
|
|
39
|
+
|
|
40
|
+
# 23.2
|
|
41
|
+
|
|
42
|
+
- Add long description to setup.py
|
|
43
|
+
|
|
44
|
+
# 23.1
|
|
45
|
+
|
|
46
|
+
- Initial public (beta) version
|
|
47
|
+
- Version schema: <year:2d>.<counter>
|
|
@@ -1,112 +1,100 @@
|
|
|
1
|
-
Metadata-Version: 2.
|
|
2
|
-
Name: persistedstate
|
|
3
|
-
Version:
|
|
4
|
-
Summary: Ultra fast persistent store for small states in mapped YAML file
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
Classifier:
|
|
11
|
-
Classifier: Intended Audience ::
|
|
12
|
-
Classifier:
|
|
13
|
-
Classifier:
|
|
14
|
-
Classifier:
|
|
15
|
-
Classifier: Programming Language :: Python :: 3
|
|
16
|
-
Classifier: Programming Language :: Python :: 3.
|
|
17
|
-
Classifier: Programming Language :: Python :: 3.
|
|
18
|
-
Classifier: Programming Language :: Python :: 3.
|
|
19
|
-
Classifier: Programming Language :: Python :: 3.
|
|
20
|
-
Classifier:
|
|
21
|
-
Classifier:
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
Requires-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
Counting to 10,000 (using Windows and Python 3.11):
|
|
102
|
-
|
|
103
|
-
```
|
|
104
|
-
PersistedState 0.193 sec
|
|
105
|
-
DiskCache 1.222 sec
|
|
106
|
-
SqliteDict 3.089 sec
|
|
107
|
-
PickleDb 8.251 sec
|
|
108
|
-
Lmdb 14.944 sec
|
|
109
|
-
Shelve 39.771 sec
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
The example seems to be silly, but this is very close to the use case it was developed for. For complex data structures or big amount of data I suggest using other libraries, like [DiskCache](https://grantjenks.com/docs/diskcache/). (The rule of thumb is when your state file is too big to be edited easily in your favorite text editor, you may think about using another key-value store library.)
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: persistedstate
|
|
3
|
+
Version: 25.1
|
|
4
|
+
Summary: Ultra fast persistent store for small states in mapped YAML file
|
|
5
|
+
Project-URL: Homepage, https://github.com/presidento/persistedstate
|
|
6
|
+
Project-URL: Changelog, https://github.com/presidento/persistedstate/blob/main/CHANGELOG.md
|
|
7
|
+
Author-email: Máté Farkas <fm@farkas-mate.hu>
|
|
8
|
+
License-Expression: MIT
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
11
|
+
Classifier: Intended Audience :: Developers
|
|
12
|
+
Classifier: Intended Audience :: System Administrators
|
|
13
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
14
|
+
Classifier: Operating System :: OS Independent
|
|
15
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
22
|
+
Classifier: Topic :: Utilities
|
|
23
|
+
Classifier: Typing :: Typed
|
|
24
|
+
Requires-Python: >=3.9
|
|
25
|
+
Requires-Dist: pyyaml
|
|
26
|
+
Description-Content-Type: text/markdown
|
|
27
|
+
|
|
28
|
+
# PersistedState
|
|
29
|
+
|
|
30
|
+
Simple and fast solution for persisting small states:
|
|
31
|
+
|
|
32
|
+
```python
|
|
33
|
+
from persistedstate import PersistedState
|
|
34
|
+
|
|
35
|
+
STATE = PersistedState("example.state", last_id=0)
|
|
36
|
+
|
|
37
|
+
for current_id in range(STATE.last_id + 1, 100_001):
|
|
38
|
+
print(f"Processing item #{current_id}")
|
|
39
|
+
STATE.last_id = current_id
|
|
40
|
+
|
|
41
|
+
print("Processing DONE.")
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
You can interrupt this script, and next time it will continue from the first unprocessed item.
|
|
45
|
+
As another example see [perftest.py](perftest.py).
|
|
46
|
+
|
|
47
|
+
## Mapped YAML state file
|
|
48
|
+
|
|
49
|
+
The use case is persisting small amount of data which can be edited easily with a text editor.
|
|
50
|
+
The database is an UTF-8 encoded YAML file, so it can be highlighted and edited manually if needed.
|
|
51
|
+
(It also uses YAML stream file format for journal.)
|
|
52
|
+
The YAML file is fully mapped to Python objects, so every change is synchronized to disk immediately. E.g. you can do this:
|
|
53
|
+
|
|
54
|
+
```python
|
|
55
|
+
STATE = PersistedState("state.yaml", processed_items=[])
|
|
56
|
+
STATE.setdefault("key", {})
|
|
57
|
+
STATE["key"].setdefault("nested", 2)
|
|
58
|
+
|
|
59
|
+
STATE.processed_items.append("<some item>")
|
|
60
|
+
STATE["key"]["nested"] += 1
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
## Failure tolerance
|
|
64
|
+
|
|
65
|
+
It uses Write-Ahead-Logging and atomic vacuum, so there will be no data loss.
|
|
66
|
+
|
|
67
|
+
## Thread safe
|
|
68
|
+
|
|
69
|
+
Changing the state is thread safe. You also can use the `._thread_lock` attribute to make atomic changes:
|
|
70
|
+
|
|
71
|
+
|
|
72
|
+
```python
|
|
73
|
+
def atomic_increment():
|
|
74
|
+
with STATE._thread_lock:
|
|
75
|
+
STATE.counter += 1
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## Performance
|
|
79
|
+
|
|
80
|
+
For its use case it outperforms existing key-value store modules.
|
|
81
|
+
For example incrementing a counter (for the details see [perftest.py](perftest.py)):
|
|
82
|
+
|
|
83
|
+
```python
|
|
84
|
+
state.counter = 0
|
|
85
|
+
for _ in range(COUNT_TO):
|
|
86
|
+
state.counter += 1
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Counting to 10,000 (using Windows and Python 3.11):
|
|
90
|
+
|
|
91
|
+
```
|
|
92
|
+
PersistedState 0.193 sec
|
|
93
|
+
DiskCache 1.222 sec
|
|
94
|
+
SqliteDict 3.089 sec
|
|
95
|
+
PickleDb 8.251 sec
|
|
96
|
+
Lmdb 14.944 sec
|
|
97
|
+
Shelve 39.771 sec
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
The example seems to be silly, but this is very close to the use case it was developed for. For complex data structures or big amount of data I suggest using other libraries, like [DiskCache](https://grantjenks.com/docs/diskcache/). (The rule of thumb is when your state file is too big to be edited easily in your favorite text editor, you may think about using another key-value store library.)
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
from persistedstate import PersistedState
|
|
3
|
+
|
|
4
|
+
STATE = PersistedState("tmp/example.state", last_id=0)
|
|
5
|
+
|
|
6
|
+
for current_id in range(STATE.last_id + 1, 100_001):
|
|
7
|
+
print(f"Processing #{current_id}")
|
|
8
|
+
STATE.last_id = current_id
|
|
9
|
+
|
|
10
|
+
print("Processing DONE.")
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
set shell := ["nu", "-c"]
|
|
2
|
+
|
|
3
|
+
# Bootstrap with all supported Python versions
|
|
4
|
+
bootstrap:
|
|
5
|
+
mkdir tmp
|
|
6
|
+
uv lock --upgrade
|
|
7
|
+
uv sync --all-groups
|
|
8
|
+
uv run tox run --notest
|
|
9
|
+
|
|
10
|
+
# Run python command with the default Python version
|
|
11
|
+
py *ARGS:
|
|
12
|
+
@uv run python -X dev {{ ARGS }}
|
|
13
|
+
|
|
14
|
+
# Run every check against source code
|
|
15
|
+
check: black mypy pylint test-all
|
|
16
|
+
|
|
17
|
+
# Check static typing
|
|
18
|
+
mypy:
|
|
19
|
+
# Checking mypy is temporary disabled
|
|
20
|
+
## just py -m mypy src
|
|
21
|
+
|
|
22
|
+
# Static code analysis with Pylint
|
|
23
|
+
pylint:
|
|
24
|
+
just py -m pylint src *.py
|
|
25
|
+
|
|
26
|
+
# Check code formatting
|
|
27
|
+
black:
|
|
28
|
+
just py -m black --check src *.py
|
|
29
|
+
|
|
30
|
+
# Test with default Python version
|
|
31
|
+
test:
|
|
32
|
+
just py -m pytest
|
|
33
|
+
|
|
34
|
+
# Test with all supported Python versions
|
|
35
|
+
test-all:
|
|
36
|
+
uv run tox run
|
|
37
|
+
|
|
38
|
+
# Remove compiled assets
|
|
39
|
+
clean:
|
|
40
|
+
rm --force --recursive --verbose build dist persistedstate.egg-info
|
|
41
|
+
|
|
42
|
+
# Run performance test
|
|
43
|
+
perftest:
|
|
44
|
+
just py perftest.py
|
|
45
|
+
|
|
46
|
+
# Build the whole project, create a release
|
|
47
|
+
build: clean bootstrap check
|
|
48
|
+
uv build
|
|
49
|
+
|
|
50
|
+
# Upload the release to PyPi
|
|
51
|
+
upload:
|
|
52
|
+
uv upload
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
import scripthelper
|
|
2
|
+
|
|
3
|
+
from persistedstate import PersistedState
|
|
4
|
+
|
|
5
|
+
scripthelper.bootstrap()
|
|
6
|
+
|
|
7
|
+
state = PersistedState("tmp/loggertest.state", string="Hello")
|
|
8
|
+
state.string = "Ahoi"
|
|
9
|
+
state.num = 42
|
|
10
|
+
state.bool = True
|
|
11
|
+
state.list = []
|
|
12
|
+
state.list.append({"a": "A"})
|
|
13
|
+
state.list[0]["a"] = "AA"
|
|
14
|
+
state.close()
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
import json
|
|
2
|
+
import pathlib
|
|
3
|
+
import shelve
|
|
4
|
+
import time
|
|
5
|
+
|
|
6
|
+
import pickledb
|
|
7
|
+
from diskcache import Cache
|
|
8
|
+
from lmdbm import Lmdb
|
|
9
|
+
from sqlitedict import SqliteDict
|
|
10
|
+
|
|
11
|
+
from persistedstate import PersistedState
|
|
12
|
+
|
|
13
|
+
COUNT_TO = 10_000
|
|
14
|
+
TMP_FOLDER = pathlib.Path("tmp/perftest")
|
|
15
|
+
STATE = PersistedState("tmp/perftest.state")
|
|
16
|
+
ITERATIONS = 5
|
|
17
|
+
|
|
18
|
+
TMP_FOLDER.mkdir(parents=True, exist_ok=True)
|
|
19
|
+
|
|
20
|
+
|
|
21
|
+
class BaseTest:
|
|
22
|
+
def do_the_count(self):
|
|
23
|
+
pass
|
|
24
|
+
|
|
25
|
+
def perftest(self, iteration):
|
|
26
|
+
test_name = self.__class__.__name__.replace("Test", "")
|
|
27
|
+
STATE.setdefault(test_name, [])
|
|
28
|
+
if len(STATE[test_name]) > iteration:
|
|
29
|
+
duration = STATE[test_name][iteration]
|
|
30
|
+
else:
|
|
31
|
+
start = time.perf_counter()
|
|
32
|
+
self.do_the_count()
|
|
33
|
+
end = time.perf_counter()
|
|
34
|
+
duration = end - start
|
|
35
|
+
STATE[test_name].append(duration)
|
|
36
|
+
print(f"{test_name:<15s} {duration:6.3f} sec")
|
|
37
|
+
|
|
38
|
+
def print_best_result(self):
|
|
39
|
+
test_name = self.__class__.__name__.replace("Test", "")
|
|
40
|
+
duration = min(STATE[test_name])
|
|
41
|
+
print(f"{test_name:<15s} {duration:6.3f} sec")
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
class PersistedStateTest(BaseTest):
|
|
45
|
+
def do_the_count(self):
|
|
46
|
+
file = TMP_FOLDER / "persisted.state"
|
|
47
|
+
file.unlink(missing_ok=True)
|
|
48
|
+
with PersistedState(file) as state:
|
|
49
|
+
state.counter = 0
|
|
50
|
+
for _ in range(COUNT_TO):
|
|
51
|
+
state.counter += 1
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
class BaseDictTest(BaseTest):
|
|
55
|
+
def do_the_count(self):
|
|
56
|
+
self.dict["counter"] = 0
|
|
57
|
+
for _ in range(COUNT_TO):
|
|
58
|
+
self.dict["counter"] += 1
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
class ShelveTest(BaseDictTest):
|
|
62
|
+
def __init__(self) -> None:
|
|
63
|
+
self.dict = shelve.open((TMP_FOLDER / "shelve.dat").as_posix())
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
class SqliteDictTest(BaseDictTest):
|
|
67
|
+
def __init__(self) -> None:
|
|
68
|
+
self.dict = SqliteDict(TMP_FOLDER / "sqlitedict.sqlite")
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
class DiskCacheTest(BaseDictTest):
|
|
72
|
+
def __init__(self) -> None:
|
|
73
|
+
self.dict = Cache(TMP_FOLDER / "diskcache")
|
|
74
|
+
|
|
75
|
+
|
|
76
|
+
class PickleDbTest(BaseTest):
|
|
77
|
+
def __init__(self) -> None:
|
|
78
|
+
self.db = pickledb.load(TMP_FOLDER / "pickle.db", auto_dump=True)
|
|
79
|
+
|
|
80
|
+
def do_the_count(self):
|
|
81
|
+
self.db.set("counter", 0)
|
|
82
|
+
for _ in range(COUNT_TO):
|
|
83
|
+
self.db.set("counter", self.db.get("counter") + 1)
|
|
84
|
+
|
|
85
|
+
|
|
86
|
+
class JsonLmdb(Lmdb):
|
|
87
|
+
def _pre_value(self, value):
|
|
88
|
+
return json.dumps(value).encode("utf-8")
|
|
89
|
+
|
|
90
|
+
def _post_value(self, value):
|
|
91
|
+
return json.loads(value.decode("utf-8"))
|
|
92
|
+
|
|
93
|
+
|
|
94
|
+
class LmdbTest(BaseDictTest):
|
|
95
|
+
def __init__(self):
|
|
96
|
+
self.dict = JsonLmdb.open((TMP_FOLDER / "lmdb").as_posix(), "c")
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
TEST_CLASSES = [
|
|
100
|
+
PersistedStateTest,
|
|
101
|
+
DiskCacheTest,
|
|
102
|
+
SqliteDictTest,
|
|
103
|
+
PickleDbTest,
|
|
104
|
+
LmdbTest,
|
|
105
|
+
ShelveTest,
|
|
106
|
+
]
|
|
107
|
+
|
|
108
|
+
|
|
109
|
+
def main():
|
|
110
|
+
STATE.pop("PersistedState", None)
|
|
111
|
+
print(f"Counting to {COUNT_TO}")
|
|
112
|
+
for iteration in range(ITERATIONS):
|
|
113
|
+
print(f"\nIteration #{iteration}\n")
|
|
114
|
+
for klass in TEST_CLASSES:
|
|
115
|
+
klass().perftest(iteration)
|
|
116
|
+
print("\nBest results:\n")
|
|
117
|
+
for klass in TEST_CLASSES:
|
|
118
|
+
klass().print_best_result()
|
|
119
|
+
|
|
120
|
+
|
|
121
|
+
main()
|
|
@@ -1,52 +1,60 @@
|
|
|
1
|
-
[project]
|
|
2
|
-
name = "persistedstate"
|
|
3
|
-
version = "
|
|
4
|
-
description = "Ultra fast persistent store for small states in mapped YAML file"
|
|
5
|
-
readme = "README.md"
|
|
6
|
-
license =
|
|
7
|
-
classifiers = [
|
|
8
|
-
"Development Status :: 5 - Production/Stable",
|
|
9
|
-
"Intended Audience :: Developers",
|
|
10
|
-
"Intended Audience :: System Administrators",
|
|
11
|
-
"License :: OSI Approved :: MIT License",
|
|
12
|
-
"Operating System :: OS Independent",
|
|
13
|
-
"Programming Language :: Python :: 3 :: Only",
|
|
14
|
-
"Programming Language :: Python :: 3.9",
|
|
15
|
-
"Programming Language :: Python :: 3.10",
|
|
16
|
-
"Programming Language :: Python :: 3.11",
|
|
17
|
-
"Programming Language :: Python :: 3.12",
|
|
18
|
-
"Programming Language :: Python :: 3.13",
|
|
19
|
-
"
|
|
20
|
-
"
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
[
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
"
|
|
29
|
-
"
|
|
30
|
-
"
|
|
31
|
-
"
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
"
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
"
|
|
39
|
-
"
|
|
40
|
-
"
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
1
|
+
[project]
|
|
2
|
+
name = "persistedstate"
|
|
3
|
+
version = "25.1"
|
|
4
|
+
description = "Ultra fast persistent store for small states in mapped YAML file"
|
|
5
|
+
readme = "README.md"
|
|
6
|
+
license = "MIT"
|
|
7
|
+
classifiers = [
|
|
8
|
+
"Development Status :: 5 - Production/Stable",
|
|
9
|
+
"Intended Audience :: Developers",
|
|
10
|
+
"Intended Audience :: System Administrators",
|
|
11
|
+
"License :: OSI Approved :: MIT License",
|
|
12
|
+
"Operating System :: OS Independent",
|
|
13
|
+
"Programming Language :: Python :: 3 :: Only",
|
|
14
|
+
"Programming Language :: Python :: 3.9",
|
|
15
|
+
"Programming Language :: Python :: 3.10",
|
|
16
|
+
"Programming Language :: Python :: 3.11",
|
|
17
|
+
"Programming Language :: Python :: 3.12",
|
|
18
|
+
"Programming Language :: Python :: 3.13",
|
|
19
|
+
"Programming Language :: Python :: 3.14",
|
|
20
|
+
"Topic :: Utilities",
|
|
21
|
+
"Typing :: Typed",
|
|
22
|
+
]
|
|
23
|
+
requires-python = ">=3.9"
|
|
24
|
+
dependencies = ["pyyaml"]
|
|
25
|
+
|
|
26
|
+
[dependency-groups]
|
|
27
|
+
dev = [
|
|
28
|
+
"black",
|
|
29
|
+
"mypy",
|
|
30
|
+
"pylint",
|
|
31
|
+
"scripthelper",
|
|
32
|
+
"tox",
|
|
33
|
+
"tox-uv>=1.25",
|
|
34
|
+
"types-PyYAML",
|
|
35
|
+
"pytest"
|
|
36
|
+
]
|
|
37
|
+
perftest = [
|
|
38
|
+
"diskcache",
|
|
39
|
+
"lmdbm",
|
|
40
|
+
"pickledb",
|
|
41
|
+
"sqlitedict",
|
|
42
|
+
]
|
|
43
|
+
|
|
44
|
+
[[project.authors]]
|
|
45
|
+
name = "Máté Farkas"
|
|
46
|
+
email = "fm@farkas-mate.hu"
|
|
47
|
+
|
|
48
|
+
[project.urls]
|
|
49
|
+
Homepage = "https://github.com/presidento/persistedstate"
|
|
50
|
+
Changelog = "https://github.com/presidento/persistedstate/blob/main/CHANGELOG.md"
|
|
51
|
+
|
|
52
|
+
[build-system]
|
|
53
|
+
requires = ["hatchling >= 1.26"]
|
|
54
|
+
build-backend = "hatchling.build"
|
|
55
|
+
|
|
56
|
+
[tool.tox]
|
|
57
|
+
env_list =["py39", "py310", "py311", "py312", "py313", "py314"]
|
|
58
|
+
|
|
59
|
+
[tool.tox.env_run_base]
|
|
60
|
+
commands = [["pytest"]]
|
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
import pathlib
|
|
2
|
+
import textwrap
|
|
3
|
+
from abc import ABC, abstractmethod
|
|
4
|
+
|
|
5
|
+
from persistedstate import PersistedState
|
|
6
|
+
|
|
7
|
+
TMP_PATH = pathlib.Path("tmp")
|
|
8
|
+
BACKUP_PATH = TMP_PATH / "unittest-backup"
|
|
9
|
+
BACKUP_PATH.mkdir(exist_ok=True, parents=True)
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
class Base(ABC):
|
|
13
|
+
file1 = pathlib.Path("tmp/test1.state")
|
|
14
|
+
file2 = pathlib.Path("tmp/test2.state")
|
|
15
|
+
input_data = ""
|
|
16
|
+
|
|
17
|
+
def test_change(self):
|
|
18
|
+
self.file1.write_text(
|
|
19
|
+
textwrap.dedent(self.input_data).strip(), encoding="utf-8"
|
|
20
|
+
)
|
|
21
|
+
with PersistedState(self.file1) as state:
|
|
22
|
+
self.change(state)
|
|
23
|
+
changed_data = self.file1.read_text(encoding="utf-8")
|
|
24
|
+
(BACKUP_PATH / self.__class__.__name__).with_suffix(".yaml").write_text(
|
|
25
|
+
changed_data, encoding="utf-8"
|
|
26
|
+
)
|
|
27
|
+
self.file2.write_text(changed_data, encoding="utf-8")
|
|
28
|
+
with PersistedState(self.file2) as state:
|
|
29
|
+
self.assertions(state)
|
|
30
|
+
|
|
31
|
+
@abstractmethod
|
|
32
|
+
def change(self, state):
|
|
33
|
+
pass
|
|
34
|
+
|
|
35
|
+
@abstractmethod
|
|
36
|
+
def assertions(self, state):
|
|
37
|
+
pass
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
class TestBasicTestWithoutChange(Base):
|
|
41
|
+
input_data = """
|
|
42
|
+
string: Peach
|
|
43
|
+
num: 42
|
|
44
|
+
bool: true
|
|
45
|
+
dict:
|
|
46
|
+
array:
|
|
47
|
+
- 793
|
|
48
|
+
"""
|
|
49
|
+
|
|
50
|
+
def change(self, state):
|
|
51
|
+
pass
|
|
52
|
+
|
|
53
|
+
def assertions(self, state):
|
|
54
|
+
assert state.string == "Peach"
|
|
55
|
+
assert state.num == 42
|
|
56
|
+
assert state.dict["array"][0] == 793
|
|
57
|
+
assert state.bool
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
class TestTopLevelDict(Base):
|
|
61
|
+
input_data = """
|
|
62
|
+
a: a
|
|
63
|
+
b: b
|
|
64
|
+
c: c
|
|
65
|
+
"""
|
|
66
|
+
|
|
67
|
+
def change(self, state):
|
|
68
|
+
state.a = "AA"
|
|
69
|
+
del state["b"]
|
|
70
|
+
|
|
71
|
+
def assertions(self, state):
|
|
72
|
+
assert state.a == "AA"
|
|
73
|
+
assert "b" not in state
|
|
74
|
+
assert state.c == "c"
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
class TestNestedDict(Base):
|
|
78
|
+
input_data = """
|
|
79
|
+
top:
|
|
80
|
+
a: a
|
|
81
|
+
b: b
|
|
82
|
+
c: c
|
|
83
|
+
"""
|
|
84
|
+
|
|
85
|
+
def change(self, state):
|
|
86
|
+
state.top["a"] = "AA"
|
|
87
|
+
del state.top["b"]
|
|
88
|
+
|
|
89
|
+
def assertions(self, state):
|
|
90
|
+
assert state.top["a"] == "AA"
|
|
91
|
+
assert "b" not in state.top
|
|
92
|
+
assert state.top["c"] == "c"
|
|
93
|
+
|
|
94
|
+
|
|
95
|
+
class TestList(Base):
|
|
96
|
+
input_data = """
|
|
97
|
+
list:
|
|
98
|
+
- 20
|
|
99
|
+
- 30
|
|
100
|
+
- 40
|
|
101
|
+
"""
|
|
102
|
+
|
|
103
|
+
def change(self, state):
|
|
104
|
+
del state.list[1]
|
|
105
|
+
state.list.insert(0, 10)
|
|
106
|
+
state.list.append(50)
|
|
107
|
+
|
|
108
|
+
def assertions(self, state):
|
|
109
|
+
assert [10, 20, 40, 50] == list(state.list)
|
|
110
|
+
|
|
111
|
+
|
|
112
|
+
class TestMultiChange(Base):
|
|
113
|
+
def change(self, state):
|
|
114
|
+
state.list = [{"a": "A"}, 1, 2]
|
|
115
|
+
state.list[0]["b"] = "B"
|
|
116
|
+
del state.list[0]["a"]
|
|
117
|
+
state.list[2] += 10
|
|
118
|
+
state.list.append({"c": "C"})
|
|
119
|
+
state.list[3]["c"] = "CC"
|
|
120
|
+
|
|
121
|
+
def assertions(self, state):
|
|
122
|
+
assert dict(state.list[0]) == {"b": "B"}
|
|
123
|
+
assert state.list[2] == 12
|
|
124
|
+
assert state.list[3]["c"] == "CC"
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
import pathlib
|
|
2
|
+
import textwrap
|
|
3
|
+
|
|
4
|
+
from persistedstate import PersistedState
|
|
5
|
+
|
|
6
|
+
|
|
7
|
+
class TestAttributes:
|
|
8
|
+
def setup_method(self) -> None:
|
|
9
|
+
self.filepath = pathlib.Path("tmp/test.state")
|
|
10
|
+
self.filepath.unlink(missing_ok=True)
|
|
11
|
+
|
|
12
|
+
def test_default_values(self):
|
|
13
|
+
with PersistedState(self.filepath, num=42, string="Hello") as state:
|
|
14
|
+
assert state.num == 42
|
|
15
|
+
assert state.string == "Hello"
|
|
16
|
+
|
|
17
|
+
def test_persisting_attributes(self):
|
|
18
|
+
with PersistedState(self.filepath, string="Hello") as state:
|
|
19
|
+
state.bool = True
|
|
20
|
+
state.string = "Ahoi"
|
|
21
|
+
with PersistedState(self.filepath) as state:
|
|
22
|
+
assert state.bool
|
|
23
|
+
assert state.string == "Ahoi"
|
|
24
|
+
expected_state = textwrap.dedent(
|
|
25
|
+
"""
|
|
26
|
+
bool: true
|
|
27
|
+
string: Ahoi
|
|
28
|
+
"""
|
|
29
|
+
).strip()
|
|
30
|
+
assert expected_state == self.filepath.read_text(encoding="utf-8").strip()
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
import concurrent.futures
|
|
2
|
+
import pathlib
|
|
3
|
+
import shutil
|
|
4
|
+
import threading
|
|
5
|
+
|
|
6
|
+
from src.persistedstate import PersistedState
|
|
7
|
+
|
|
8
|
+
NUM_OF_THREADS = 100
|
|
9
|
+
STEPS_OF_EACH_THREAD = 10
|
|
10
|
+
STATE_FILE = pathlib.Path("tmp/threads.state")
|
|
11
|
+
TEMP_FILE = STATE_FILE.with_suffix(".tmp")
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
class TestThreads:
|
|
15
|
+
def setup_method(self):
|
|
16
|
+
self.filepath = STATE_FILE
|
|
17
|
+
self.filepath.unlink(missing_ok=True)
|
|
18
|
+
self.state = None
|
|
19
|
+
|
|
20
|
+
def test_thread_usage(self):
|
|
21
|
+
self.open_state()
|
|
22
|
+
with concurrent.futures.ThreadPoolExecutor(
|
|
23
|
+
max_workers=NUM_OF_THREADS
|
|
24
|
+
) as executor:
|
|
25
|
+
futures = [
|
|
26
|
+
executor.submit(self.threaded_function) for _ in range(NUM_OF_THREADS)
|
|
27
|
+
]
|
|
28
|
+
concurrent.futures.wait(futures)
|
|
29
|
+
shutil.copy(STATE_FILE, TEMP_FILE)
|
|
30
|
+
self.close_state()
|
|
31
|
+
STATE_FILE.unlink()
|
|
32
|
+
shutil.copy(TEMP_FILE, STATE_FILE)
|
|
33
|
+
|
|
34
|
+
self.open_state()
|
|
35
|
+
assert len(self.state.list) == NUM_OF_THREADS * STEPS_OF_EACH_THREAD
|
|
36
|
+
self.close_state()
|
|
37
|
+
|
|
38
|
+
def open_state(self):
|
|
39
|
+
self.state = PersistedState(self.filepath, index=0, list=[])
|
|
40
|
+
|
|
41
|
+
def close_state(self):
|
|
42
|
+
self.state.close()
|
|
43
|
+
|
|
44
|
+
def threaded_function(self):
|
|
45
|
+
for _ in range(STEPS_OF_EACH_THREAD):
|
|
46
|
+
self.state.list.append(threading.get_ident())
|
persistedstate-24.2/PKG-INFO
DELETED
|
@@ -1,112 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.1
|
|
2
|
-
Name: persistedstate
|
|
3
|
-
Version: 24.2
|
|
4
|
-
Summary: Ultra fast persistent store for small states in mapped YAML file
|
|
5
|
-
Author-email: Máté Farkas <fm@farkas-mate.hu>
|
|
6
|
-
License: MIT License
|
|
7
|
-
Project-URL: Homepage, https://github.com/presidento/persistedstate
|
|
8
|
-
Project-URL: Changelog, https://github.com/presidento/persistedstate/blob/main/CHANGELOG.md
|
|
9
|
-
Classifier: Development Status :: 5 - Production/Stable
|
|
10
|
-
Classifier: Intended Audience :: Developers
|
|
11
|
-
Classifier: Intended Audience :: System Administrators
|
|
12
|
-
Classifier: License :: OSI Approved :: MIT License
|
|
13
|
-
Classifier: Operating System :: OS Independent
|
|
14
|
-
Classifier: Programming Language :: Python :: 3 :: Only
|
|
15
|
-
Classifier: Programming Language :: Python :: 3.9
|
|
16
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
-
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
-
Classifier: Topic :: Utilities
|
|
21
|
-
Classifier: Typing :: Typed
|
|
22
|
-
Description-Content-Type: text/markdown
|
|
23
|
-
License-File: LICENSE
|
|
24
|
-
Requires-Dist: pyyaml
|
|
25
|
-
Provides-Extra: dev
|
|
26
|
-
Requires-Dist: black; extra == "dev"
|
|
27
|
-
Requires-Dist: mypy; extra == "dev"
|
|
28
|
-
Requires-Dist: pylint; extra == "dev"
|
|
29
|
-
Requires-Dist: scripthelper; extra == "dev"
|
|
30
|
-
Requires-Dist: twine; extra == "dev"
|
|
31
|
-
Requires-Dist: types-PyYAML; extra == "dev"
|
|
32
|
-
Provides-Extra: test
|
|
33
|
-
Requires-Dist: pytest; extra == "test"
|
|
34
|
-
Provides-Extra: perftest
|
|
35
|
-
Requires-Dist: diskcache; extra == "perftest"
|
|
36
|
-
Requires-Dist: lmdbm; extra == "perftest"
|
|
37
|
-
Requires-Dist: pickledb; extra == "perftest"
|
|
38
|
-
Requires-Dist: sqlitedict; extra == "perftest"
|
|
39
|
-
|
|
40
|
-
# PersistedState
|
|
41
|
-
|
|
42
|
-
Simple and fast solution for persisting small states:
|
|
43
|
-
|
|
44
|
-
```python
|
|
45
|
-
from persistedstate import PersistedState
|
|
46
|
-
|
|
47
|
-
STATE = PersistedState("example.state", last_id=0)
|
|
48
|
-
|
|
49
|
-
for current_id in range(STATE.last_id + 1, 100_001):
|
|
50
|
-
print(f"Processing item #{current_id}")
|
|
51
|
-
STATE.last_id = current_id
|
|
52
|
-
|
|
53
|
-
print("Processing DONE.")
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
You can interrupt this script, and next time it will continue from the first unprocessed item.
|
|
57
|
-
As another example see [perftest.py](perftest.py).
|
|
58
|
-
|
|
59
|
-
## Mapped YAML state file
|
|
60
|
-
|
|
61
|
-
The use case is persisting small amount of data which can be edited easily with a text editor.
|
|
62
|
-
The database is an UTF-8 encoded YAML file, so it can be highlighted and edited manually if needed.
|
|
63
|
-
(It also uses YAML stream file format for journal.)
|
|
64
|
-
The YAML file is fully mapped to Python objects, so every change is synchronized to disk immediately. E.g. you can do this:
|
|
65
|
-
|
|
66
|
-
```python
|
|
67
|
-
STATE = PersistedState("state.yaml", processed_items=[])
|
|
68
|
-
STATE.setdefault("key", {})
|
|
69
|
-
STATE["key"].setdefault("nested", 2)
|
|
70
|
-
|
|
71
|
-
STATE.processed_items.append("<some item>")
|
|
72
|
-
STATE["key"]["nested"] += 1
|
|
73
|
-
```
|
|
74
|
-
|
|
75
|
-
## Failure tolerance
|
|
76
|
-
|
|
77
|
-
It uses Write-Ahead-Logging and atomic vacuum, so there will be no data loss.
|
|
78
|
-
|
|
79
|
-
## Thread safe
|
|
80
|
-
|
|
81
|
-
Changing the state is thread safe. You also can use the `._thread_lock` attribute to make atomic changes:
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
```python
|
|
85
|
-
def atomic_increment():
|
|
86
|
-
with STATE._thread_lock:
|
|
87
|
-
STATE.counter += 1
|
|
88
|
-
```
|
|
89
|
-
|
|
90
|
-
## Performance
|
|
91
|
-
|
|
92
|
-
For its use case it outperforms existing key-value store modules.
|
|
93
|
-
For example incrementing a counter (for the details see [perftest.py](perftest.py)):
|
|
94
|
-
|
|
95
|
-
```python
|
|
96
|
-
state.counter = 0
|
|
97
|
-
for _ in range(COUNT_TO):
|
|
98
|
-
state.counter += 1
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
Counting to 10,000 (using Windows and Python 3.11):
|
|
102
|
-
|
|
103
|
-
```
|
|
104
|
-
PersistedState 0.193 sec
|
|
105
|
-
DiskCache 1.222 sec
|
|
106
|
-
SqliteDict 3.089 sec
|
|
107
|
-
PickleDb 8.251 sec
|
|
108
|
-
Lmdb 14.944 sec
|
|
109
|
-
Shelve 39.771 sec
|
|
110
|
-
```
|
|
111
|
-
|
|
112
|
-
The example seems to be silly, but this is very close to the use case it was developed for. For complex data structures or big amount of data I suggest using other libraries, like [DiskCache](https://grantjenks.com/docs/diskcache/). (The rule of thumb is when your state file is too big to be edited easily in your favorite text editor, you may think about using another key-value store library.)
|
persistedstate-24.2/setup.cfg
DELETED
|
@@ -1,10 +0,0 @@
|
|
|
1
|
-
LICENSE
|
|
2
|
-
README.md
|
|
3
|
-
pyproject.toml
|
|
4
|
-
src/persistedstate/__init__.py
|
|
5
|
-
src/persistedstate/py.typed
|
|
6
|
-
src/persistedstate.egg-info/PKG-INFO
|
|
7
|
-
src/persistedstate.egg-info/SOURCES.txt
|
|
8
|
-
src/persistedstate.egg-info/dependency_links.txt
|
|
9
|
-
src/persistedstate.egg-info/requires.txt
|
|
10
|
-
src/persistedstate.egg-info/top_level.txt
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
persistedstate
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|