persistedstate 24.2__tar.gz → 25.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,9 @@
1
+ __pycache__
2
+ /.tox/
3
+ /tmp/
4
+ /build/
5
+ /dist/
6
+ *.egg-info/
7
+
8
+ # Always test with the latest-greatest versions
9
+ uv.lock
@@ -0,0 +1,16 @@
1
+ [MAIN]
2
+ disable=
3
+ missing-docstring,
4
+ format,
5
+ logging-not-lazy,
6
+ protected-access,
7
+ logging-fstring-interpolation,
8
+
9
+ [persisstedstate]
10
+ disable=consider-using-with
11
+
12
+ [test_persistedstate]
13
+ disable=attribute-defined-outside-init
14
+
15
+ [perftest]
16
+ disable=no-member, invalid-name
@@ -0,0 +1,47 @@
1
+ # 25.1
2
+
3
+ - Add Python 3.14 support
4
+
5
+ # 24.2
6
+
7
+ - Remove Python 3.8 support
8
+
9
+ # 24.1
10
+
11
+ - Official Python 3.13 support
12
+
13
+ # 23.9
14
+
15
+ - Changing the state object is thread safe now.
16
+
17
+ # 23.8
18
+
19
+ - Remove Python 3.7 support
20
+ - Add Python 3.12 support
21
+
22
+ # 23.6
23
+
24
+ - Fix crash when the PersistedState is subclassed or constructed from another module
25
+ - Make vacuum more failure tolerant. In some edge cases it was possible for the state at beginning to overlap the state at the end.
26
+ - Internal: use `pyproject.toml` instead of `setup.py`
27
+
28
+ # 23.5
29
+
30
+ - Map the whole YAML file, so modification of nested objects are also persisted immediately
31
+
32
+ # 23.4
33
+
34
+ - In-file vacuuming
35
+
36
+ # 23.3
37
+
38
+ - Use YAML file format, not just a subset of YAML
39
+
40
+ # 23.2
41
+
42
+ - Add long description to setup.py
43
+
44
+ # 23.1
45
+
46
+ - Initial public (beta) version
47
+ - Version schema: <year:2d>.<counter>
@@ -1,112 +1,100 @@
1
- Metadata-Version: 2.1
2
- Name: persistedstate
3
- Version: 24.2
4
- Summary: Ultra fast persistent store for small states in mapped YAML file
5
- Author-email: Máté Farkas <fm@farkas-mate.hu>
6
- License: MIT License
7
- Project-URL: Homepage, https://github.com/presidento/persistedstate
8
- Project-URL: Changelog, https://github.com/presidento/persistedstate/blob/main/CHANGELOG.md
9
- Classifier: Development Status :: 5 - Production/Stable
10
- Classifier: Intended Audience :: Developers
11
- Classifier: Intended Audience :: System Administrators
12
- Classifier: License :: OSI Approved :: MIT License
13
- Classifier: Operating System :: OS Independent
14
- Classifier: Programming Language :: Python :: 3 :: Only
15
- Classifier: Programming Language :: Python :: 3.9
16
- Classifier: Programming Language :: Python :: 3.10
17
- Classifier: Programming Language :: Python :: 3.11
18
- Classifier: Programming Language :: Python :: 3.12
19
- Classifier: Programming Language :: Python :: 3.13
20
- Classifier: Topic :: Utilities
21
- Classifier: Typing :: Typed
22
- Description-Content-Type: text/markdown
23
- License-File: LICENSE
24
- Requires-Dist: pyyaml
25
- Provides-Extra: dev
26
- Requires-Dist: black; extra == "dev"
27
- Requires-Dist: mypy; extra == "dev"
28
- Requires-Dist: pylint; extra == "dev"
29
- Requires-Dist: scripthelper; extra == "dev"
30
- Requires-Dist: twine; extra == "dev"
31
- Requires-Dist: types-PyYAML; extra == "dev"
32
- Provides-Extra: test
33
- Requires-Dist: pytest; extra == "test"
34
- Provides-Extra: perftest
35
- Requires-Dist: diskcache; extra == "perftest"
36
- Requires-Dist: lmdbm; extra == "perftest"
37
- Requires-Dist: pickledb; extra == "perftest"
38
- Requires-Dist: sqlitedict; extra == "perftest"
39
-
40
- # PersistedState
41
-
42
- Simple and fast solution for persisting small states:
43
-
44
- ```python
45
- from persistedstate import PersistedState
46
-
47
- STATE = PersistedState("example.state", last_id=0)
48
-
49
- for current_id in range(STATE.last_id + 1, 100_001):
50
- print(f"Processing item #{current_id}")
51
- STATE.last_id = current_id
52
-
53
- print("Processing DONE.")
54
- ```
55
-
56
- You can interrupt this script, and next time it will continue from the first unprocessed item.
57
- As another example see [perftest.py](perftest.py).
58
-
59
- ## Mapped YAML state file
60
-
61
- The use case is persisting small amount of data which can be edited easily with a text editor.
62
- The database is an UTF-8 encoded YAML file, so it can be highlighted and edited manually if needed.
63
- (It also uses YAML stream file format for journal.)
64
- The YAML file is fully mapped to Python objects, so every change is synchronized to disk immediately. E.g. you can do this:
65
-
66
- ```python
67
- STATE = PersistedState("state.yaml", processed_items=[])
68
- STATE.setdefault("key", {})
69
- STATE["key"].setdefault("nested", 2)
70
-
71
- STATE.processed_items.append("<some item>")
72
- STATE["key"]["nested"] += 1
73
- ```
74
-
75
- ## Failure tolerance
76
-
77
- It uses Write-Ahead-Logging and atomic vacuum, so there will be no data loss.
78
-
79
- ## Thread safe
80
-
81
- Changing the state is thread safe. You also can use the `._thread_lock` attribute to make atomic changes:
82
-
83
-
84
- ```python
85
- def atomic_increment():
86
- with STATE._thread_lock:
87
- STATE.counter += 1
88
- ```
89
-
90
- ## Performance
91
-
92
- For its use case it outperforms existing key-value store modules.
93
- For example incrementing a counter (for the details see [perftest.py](perftest.py)):
94
-
95
- ```python
96
- state.counter = 0
97
- for _ in range(COUNT_TO):
98
- state.counter += 1
99
- ```
100
-
101
- Counting to 10,000 (using Windows and Python 3.11):
102
-
103
- ```
104
- PersistedState 0.193 sec
105
- DiskCache 1.222 sec
106
- SqliteDict 3.089 sec
107
- PickleDb 8.251 sec
108
- Lmdb 14.944 sec
109
- Shelve 39.771 sec
110
- ```
111
-
112
- The example seems to be silly, but this is very close to the use case it was developed for. For complex data structures or big amount of data I suggest using other libraries, like [DiskCache](https://grantjenks.com/docs/diskcache/). (The rule of thumb is when your state file is too big to be edited easily in your favorite text editor, you may think about using another key-value store library.)
1
+ Metadata-Version: 2.4
2
+ Name: persistedstate
3
+ Version: 25.1
4
+ Summary: Ultra fast persistent store for small states in mapped YAML file
5
+ Project-URL: Homepage, https://github.com/presidento/persistedstate
6
+ Project-URL: Changelog, https://github.com/presidento/persistedstate/blob/main/CHANGELOG.md
7
+ Author-email: Máté Farkas <fm@farkas-mate.hu>
8
+ License-Expression: MIT
9
+ License-File: LICENSE
10
+ Classifier: Development Status :: 5 - Production/Stable
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: Intended Audience :: System Administrators
13
+ Classifier: License :: OSI Approved :: MIT License
14
+ Classifier: Operating System :: OS Independent
15
+ Classifier: Programming Language :: Python :: 3 :: Only
16
+ Classifier: Programming Language :: Python :: 3.9
17
+ Classifier: Programming Language :: Python :: 3.10
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Programming Language :: Python :: 3.14
22
+ Classifier: Topic :: Utilities
23
+ Classifier: Typing :: Typed
24
+ Requires-Python: >=3.9
25
+ Requires-Dist: pyyaml
26
+ Description-Content-Type: text/markdown
27
+
28
+ # PersistedState
29
+
30
+ Simple and fast solution for persisting small states:
31
+
32
+ ```python
33
+ from persistedstate import PersistedState
34
+
35
+ STATE = PersistedState("example.state", last_id=0)
36
+
37
+ for current_id in range(STATE.last_id + 1, 100_001):
38
+ print(f"Processing item #{current_id}")
39
+ STATE.last_id = current_id
40
+
41
+ print("Processing DONE.")
42
+ ```
43
+
44
+ You can interrupt this script, and next time it will continue from the first unprocessed item.
45
+ As another example see [perftest.py](perftest.py).
46
+
47
+ ## Mapped YAML state file
48
+
49
+ The use case is persisting small amount of data which can be edited easily with a text editor.
50
+ The database is an UTF-8 encoded YAML file, so it can be highlighted and edited manually if needed.
51
+ (It also uses YAML stream file format for journal.)
52
+ The YAML file is fully mapped to Python objects, so every change is synchronized to disk immediately. E.g. you can do this:
53
+
54
+ ```python
55
+ STATE = PersistedState("state.yaml", processed_items=[])
56
+ STATE.setdefault("key", {})
57
+ STATE["key"].setdefault("nested", 2)
58
+
59
+ STATE.processed_items.append("<some item>")
60
+ STATE["key"]["nested"] += 1
61
+ ```
62
+
63
+ ## Failure tolerance
64
+
65
+ It uses Write-Ahead-Logging and atomic vacuum, so there will be no data loss.
66
+
67
+ ## Thread safe
68
+
69
+ Changing the state is thread safe. You also can use the `._thread_lock` attribute to make atomic changes:
70
+
71
+
72
+ ```python
73
+ def atomic_increment():
74
+ with STATE._thread_lock:
75
+ STATE.counter += 1
76
+ ```
77
+
78
+ ## Performance
79
+
80
+ For its use case it outperforms existing key-value store modules.
81
+ For example incrementing a counter (for the details see [perftest.py](perftest.py)):
82
+
83
+ ```python
84
+ state.counter = 0
85
+ for _ in range(COUNT_TO):
86
+ state.counter += 1
87
+ ```
88
+
89
+ Counting to 10,000 (using Windows and Python 3.11):
90
+
91
+ ```
92
+ PersistedState 0.193 sec
93
+ DiskCache 1.222 sec
94
+ SqliteDict 3.089 sec
95
+ PickleDb 8.251 sec
96
+ Lmdb 14.944 sec
97
+ Shelve 39.771 sec
98
+ ```
99
+
100
+ The example seems to be silly, but this is very close to the use case it was developed for. For complex data structures or big amount of data I suggest using other libraries, like [DiskCache](https://grantjenks.com/docs/diskcache/). (The rule of thumb is when your state file is too big to be edited easily in your favorite text editor, you may think about using another key-value store library.)
@@ -0,0 +1,10 @@
1
+ #!/usr/bin/env python3
2
+ from persistedstate import PersistedState
3
+
4
+ STATE = PersistedState("tmp/example.state", last_id=0)
5
+
6
+ for current_id in range(STATE.last_id + 1, 100_001):
7
+ print(f"Processing #{current_id}")
8
+ STATE.last_id = current_id
9
+
10
+ print("Processing DONE.")
@@ -0,0 +1,52 @@
1
+ set shell := ["nu", "-c"]
2
+
3
+ # Bootstrap with all supported Python versions
4
+ bootstrap:
5
+ mkdir tmp
6
+ uv lock --upgrade
7
+ uv sync --all-groups
8
+ uv run tox run --notest
9
+
10
+ # Run python command with the default Python version
11
+ py *ARGS:
12
+ @uv run python -X dev {{ ARGS }}
13
+
14
+ # Run every check against source code
15
+ check: black mypy pylint test-all
16
+
17
+ # Check static typing
18
+ mypy:
19
+ # Checking mypy is temporary disabled
20
+ ## just py -m mypy src
21
+
22
+ # Static code analysis with Pylint
23
+ pylint:
24
+ just py -m pylint src *.py
25
+
26
+ # Check code formatting
27
+ black:
28
+ just py -m black --check src *.py
29
+
30
+ # Test with default Python version
31
+ test:
32
+ just py -m pytest
33
+
34
+ # Test with all supported Python versions
35
+ test-all:
36
+ uv run tox run
37
+
38
+ # Remove compiled assets
39
+ clean:
40
+ rm --force --recursive --verbose build dist persistedstate.egg-info
41
+
42
+ # Run performance test
43
+ perftest:
44
+ just py perftest.py
45
+
46
+ # Build the whole project, create a release
47
+ build: clean bootstrap check
48
+ uv build
49
+
50
+ # Upload the release to PyPi
51
+ upload:
52
+ uv upload
@@ -0,0 +1,14 @@
1
+ import scripthelper
2
+
3
+ from persistedstate import PersistedState
4
+
5
+ scripthelper.bootstrap()
6
+
7
+ state = PersistedState("tmp/loggertest.state", string="Hello")
8
+ state.string = "Ahoi"
9
+ state.num = 42
10
+ state.bool = True
11
+ state.list = []
12
+ state.list.append({"a": "A"})
13
+ state.list[0]["a"] = "AA"
14
+ state.close()
@@ -0,0 +1,6 @@
1
+ [mypy]
2
+ ignore_missing_imports = True
3
+ no_site_packages = True
4
+
5
+ [mypy-persistedstate]
6
+ disable_error_code=override, attr-defined
@@ -0,0 +1,121 @@
1
+ import json
2
+ import pathlib
3
+ import shelve
4
+ import time
5
+
6
+ import pickledb
7
+ from diskcache import Cache
8
+ from lmdbm import Lmdb
9
+ from sqlitedict import SqliteDict
10
+
11
+ from persistedstate import PersistedState
12
+
13
+ COUNT_TO = 10_000
14
+ TMP_FOLDER = pathlib.Path("tmp/perftest")
15
+ STATE = PersistedState("tmp/perftest.state")
16
+ ITERATIONS = 5
17
+
18
+ TMP_FOLDER.mkdir(parents=True, exist_ok=True)
19
+
20
+
21
+ class BaseTest:
22
+ def do_the_count(self):
23
+ pass
24
+
25
+ def perftest(self, iteration):
26
+ test_name = self.__class__.__name__.replace("Test", "")
27
+ STATE.setdefault(test_name, [])
28
+ if len(STATE[test_name]) > iteration:
29
+ duration = STATE[test_name][iteration]
30
+ else:
31
+ start = time.perf_counter()
32
+ self.do_the_count()
33
+ end = time.perf_counter()
34
+ duration = end - start
35
+ STATE[test_name].append(duration)
36
+ print(f"{test_name:<15s} {duration:6.3f} sec")
37
+
38
+ def print_best_result(self):
39
+ test_name = self.__class__.__name__.replace("Test", "")
40
+ duration = min(STATE[test_name])
41
+ print(f"{test_name:<15s} {duration:6.3f} sec")
42
+
43
+
44
+ class PersistedStateTest(BaseTest):
45
+ def do_the_count(self):
46
+ file = TMP_FOLDER / "persisted.state"
47
+ file.unlink(missing_ok=True)
48
+ with PersistedState(file) as state:
49
+ state.counter = 0
50
+ for _ in range(COUNT_TO):
51
+ state.counter += 1
52
+
53
+
54
+ class BaseDictTest(BaseTest):
55
+ def do_the_count(self):
56
+ self.dict["counter"] = 0
57
+ for _ in range(COUNT_TO):
58
+ self.dict["counter"] += 1
59
+
60
+
61
+ class ShelveTest(BaseDictTest):
62
+ def __init__(self) -> None:
63
+ self.dict = shelve.open((TMP_FOLDER / "shelve.dat").as_posix())
64
+
65
+
66
+ class SqliteDictTest(BaseDictTest):
67
+ def __init__(self) -> None:
68
+ self.dict = SqliteDict(TMP_FOLDER / "sqlitedict.sqlite")
69
+
70
+
71
+ class DiskCacheTest(BaseDictTest):
72
+ def __init__(self) -> None:
73
+ self.dict = Cache(TMP_FOLDER / "diskcache")
74
+
75
+
76
+ class PickleDbTest(BaseTest):
77
+ def __init__(self) -> None:
78
+ self.db = pickledb.load(TMP_FOLDER / "pickle.db", auto_dump=True)
79
+
80
+ def do_the_count(self):
81
+ self.db.set("counter", 0)
82
+ for _ in range(COUNT_TO):
83
+ self.db.set("counter", self.db.get("counter") + 1)
84
+
85
+
86
+ class JsonLmdb(Lmdb):
87
+ def _pre_value(self, value):
88
+ return json.dumps(value).encode("utf-8")
89
+
90
+ def _post_value(self, value):
91
+ return json.loads(value.decode("utf-8"))
92
+
93
+
94
+ class LmdbTest(BaseDictTest):
95
+ def __init__(self):
96
+ self.dict = JsonLmdb.open((TMP_FOLDER / "lmdb").as_posix(), "c")
97
+
98
+
99
+ TEST_CLASSES = [
100
+ PersistedStateTest,
101
+ DiskCacheTest,
102
+ SqliteDictTest,
103
+ PickleDbTest,
104
+ LmdbTest,
105
+ ShelveTest,
106
+ ]
107
+
108
+
109
+ def main():
110
+ STATE.pop("PersistedState", None)
111
+ print(f"Counting to {COUNT_TO}")
112
+ for iteration in range(ITERATIONS):
113
+ print(f"\nIteration #{iteration}\n")
114
+ for klass in TEST_CLASSES:
115
+ klass().perftest(iteration)
116
+ print("\nBest results:\n")
117
+ for klass in TEST_CLASSES:
118
+ klass().print_best_result()
119
+
120
+
121
+ main()
@@ -1,52 +1,60 @@
1
- [project]
2
- name = "persistedstate"
3
- version = "24.2"
4
- description = "Ultra fast persistent store for small states in mapped YAML file"
5
- readme = "README.md"
6
- license = {text = "MIT License"}
7
- classifiers = [
8
- "Development Status :: 5 - Production/Stable",
9
- "Intended Audience :: Developers",
10
- "Intended Audience :: System Administrators",
11
- "License :: OSI Approved :: MIT License",
12
- "Operating System :: OS Independent",
13
- "Programming Language :: Python :: 3 :: Only",
14
- "Programming Language :: Python :: 3.9",
15
- "Programming Language :: Python :: 3.10",
16
- "Programming Language :: Python :: 3.11",
17
- "Programming Language :: Python :: 3.12",
18
- "Programming Language :: Python :: 3.13",
19
- "Topic :: Utilities",
20
- "Typing :: Typed",
21
- ]
22
- dependencies = ["pyyaml"]
23
-
24
- [project.optional-dependencies]
25
- dev = [
26
- "black",
27
- "mypy",
28
- "pylint",
29
- "scripthelper",
30
- "twine",
31
- "types-PyYAML",
32
- ]
33
- test = [
34
- "pytest"
35
- ]
36
- perftest = [
37
- "diskcache",
38
- "lmdbm",
39
- "pickledb",
40
- "sqlitedict",
41
- ]
42
-
43
- [[project.authors]]
44
- name = "Máté Farkas"
45
- email = "fm@farkas-mate.hu"
46
-
47
- [project.urls]
48
- Homepage = "https://github.com/presidento/persistedstate"
49
- Changelog = "https://github.com/presidento/persistedstate/blob/main/CHANGELOG.md"
50
-
51
- [tool.setuptools]
52
- package-data = {persistedstate = ["py.typed"]}
1
+ [project]
2
+ name = "persistedstate"
3
+ version = "25.1"
4
+ description = "Ultra fast persistent store for small states in mapped YAML file"
5
+ readme = "README.md"
6
+ license = "MIT"
7
+ classifiers = [
8
+ "Development Status :: 5 - Production/Stable",
9
+ "Intended Audience :: Developers",
10
+ "Intended Audience :: System Administrators",
11
+ "License :: OSI Approved :: MIT License",
12
+ "Operating System :: OS Independent",
13
+ "Programming Language :: Python :: 3 :: Only",
14
+ "Programming Language :: Python :: 3.9",
15
+ "Programming Language :: Python :: 3.10",
16
+ "Programming Language :: Python :: 3.11",
17
+ "Programming Language :: Python :: 3.12",
18
+ "Programming Language :: Python :: 3.13",
19
+ "Programming Language :: Python :: 3.14",
20
+ "Topic :: Utilities",
21
+ "Typing :: Typed",
22
+ ]
23
+ requires-python = ">=3.9"
24
+ dependencies = ["pyyaml"]
25
+
26
+ [dependency-groups]
27
+ dev = [
28
+ "black",
29
+ "mypy",
30
+ "pylint",
31
+ "scripthelper",
32
+ "tox",
33
+ "tox-uv>=1.25",
34
+ "types-PyYAML",
35
+ "pytest"
36
+ ]
37
+ perftest = [
38
+ "diskcache",
39
+ "lmdbm",
40
+ "pickledb",
41
+ "sqlitedict",
42
+ ]
43
+
44
+ [[project.authors]]
45
+ name = "Máté Farkas"
46
+ email = "fm@farkas-mate.hu"
47
+
48
+ [project.urls]
49
+ Homepage = "https://github.com/presidento/persistedstate"
50
+ Changelog = "https://github.com/presidento/persistedstate/blob/main/CHANGELOG.md"
51
+
52
+ [build-system]
53
+ requires = ["hatchling >= 1.26"]
54
+ build-backend = "hatchling.build"
55
+
56
+ [tool.tox]
57
+ env_list =["py39", "py310", "py311", "py312", "py313", "py314"]
58
+
59
+ [tool.tox.env_run_base]
60
+ commands = [["pytest"]]
@@ -0,0 +1,124 @@
1
+ import pathlib
2
+ import textwrap
3
+ from abc import ABC, abstractmethod
4
+
5
+ from persistedstate import PersistedState
6
+
7
+ TMP_PATH = pathlib.Path("tmp")
8
+ BACKUP_PATH = TMP_PATH / "unittest-backup"
9
+ BACKUP_PATH.mkdir(exist_ok=True, parents=True)
10
+
11
+
12
+ class Base(ABC):
13
+ file1 = pathlib.Path("tmp/test1.state")
14
+ file2 = pathlib.Path("tmp/test2.state")
15
+ input_data = ""
16
+
17
+ def test_change(self):
18
+ self.file1.write_text(
19
+ textwrap.dedent(self.input_data).strip(), encoding="utf-8"
20
+ )
21
+ with PersistedState(self.file1) as state:
22
+ self.change(state)
23
+ changed_data = self.file1.read_text(encoding="utf-8")
24
+ (BACKUP_PATH / self.__class__.__name__).with_suffix(".yaml").write_text(
25
+ changed_data, encoding="utf-8"
26
+ )
27
+ self.file2.write_text(changed_data, encoding="utf-8")
28
+ with PersistedState(self.file2) as state:
29
+ self.assertions(state)
30
+
31
+ @abstractmethod
32
+ def change(self, state):
33
+ pass
34
+
35
+ @abstractmethod
36
+ def assertions(self, state):
37
+ pass
38
+
39
+
40
+ class TestBasicTestWithoutChange(Base):
41
+ input_data = """
42
+ string: Peach
43
+ num: 42
44
+ bool: true
45
+ dict:
46
+ array:
47
+ - 793
48
+ """
49
+
50
+ def change(self, state):
51
+ pass
52
+
53
+ def assertions(self, state):
54
+ assert state.string == "Peach"
55
+ assert state.num == 42
56
+ assert state.dict["array"][0] == 793
57
+ assert state.bool
58
+
59
+
60
+ class TestTopLevelDict(Base):
61
+ input_data = """
62
+ a: a
63
+ b: b
64
+ c: c
65
+ """
66
+
67
+ def change(self, state):
68
+ state.a = "AA"
69
+ del state["b"]
70
+
71
+ def assertions(self, state):
72
+ assert state.a == "AA"
73
+ assert "b" not in state
74
+ assert state.c == "c"
75
+
76
+
77
+ class TestNestedDict(Base):
78
+ input_data = """
79
+ top:
80
+ a: a
81
+ b: b
82
+ c: c
83
+ """
84
+
85
+ def change(self, state):
86
+ state.top["a"] = "AA"
87
+ del state.top["b"]
88
+
89
+ def assertions(self, state):
90
+ assert state.top["a"] == "AA"
91
+ assert "b" not in state.top
92
+ assert state.top["c"] == "c"
93
+
94
+
95
+ class TestList(Base):
96
+ input_data = """
97
+ list:
98
+ - 20
99
+ - 30
100
+ - 40
101
+ """
102
+
103
+ def change(self, state):
104
+ del state.list[1]
105
+ state.list.insert(0, 10)
106
+ state.list.append(50)
107
+
108
+ def assertions(self, state):
109
+ assert [10, 20, 40, 50] == list(state.list)
110
+
111
+
112
+ class TestMultiChange(Base):
113
+ def change(self, state):
114
+ state.list = [{"a": "A"}, 1, 2]
115
+ state.list[0]["b"] = "B"
116
+ del state.list[0]["a"]
117
+ state.list[2] += 10
118
+ state.list.append({"c": "C"})
119
+ state.list[3]["c"] = "CC"
120
+
121
+ def assertions(self, state):
122
+ assert dict(state.list[0]) == {"b": "B"}
123
+ assert state.list[2] == 12
124
+ assert state.list[3]["c"] == "CC"
@@ -0,0 +1,30 @@
1
+ import pathlib
2
+ import textwrap
3
+
4
+ from persistedstate import PersistedState
5
+
6
+
7
+ class TestAttributes:
8
+ def setup_method(self) -> None:
9
+ self.filepath = pathlib.Path("tmp/test.state")
10
+ self.filepath.unlink(missing_ok=True)
11
+
12
+ def test_default_values(self):
13
+ with PersistedState(self.filepath, num=42, string="Hello") as state:
14
+ assert state.num == 42
15
+ assert state.string == "Hello"
16
+
17
+ def test_persisting_attributes(self):
18
+ with PersistedState(self.filepath, string="Hello") as state:
19
+ state.bool = True
20
+ state.string = "Ahoi"
21
+ with PersistedState(self.filepath) as state:
22
+ assert state.bool
23
+ assert state.string == "Ahoi"
24
+ expected_state = textwrap.dedent(
25
+ """
26
+ bool: true
27
+ string: Ahoi
28
+ """
29
+ ).strip()
30
+ assert expected_state == self.filepath.read_text(encoding="utf-8").strip()
@@ -0,0 +1,46 @@
1
+ import concurrent.futures
2
+ import pathlib
3
+ import shutil
4
+ import threading
5
+
6
+ from src.persistedstate import PersistedState
7
+
8
+ NUM_OF_THREADS = 100
9
+ STEPS_OF_EACH_THREAD = 10
10
+ STATE_FILE = pathlib.Path("tmp/threads.state")
11
+ TEMP_FILE = STATE_FILE.with_suffix(".tmp")
12
+
13
+
14
+ class TestThreads:
15
+ def setup_method(self):
16
+ self.filepath = STATE_FILE
17
+ self.filepath.unlink(missing_ok=True)
18
+ self.state = None
19
+
20
+ def test_thread_usage(self):
21
+ self.open_state()
22
+ with concurrent.futures.ThreadPoolExecutor(
23
+ max_workers=NUM_OF_THREADS
24
+ ) as executor:
25
+ futures = [
26
+ executor.submit(self.threaded_function) for _ in range(NUM_OF_THREADS)
27
+ ]
28
+ concurrent.futures.wait(futures)
29
+ shutil.copy(STATE_FILE, TEMP_FILE)
30
+ self.close_state()
31
+ STATE_FILE.unlink()
32
+ shutil.copy(TEMP_FILE, STATE_FILE)
33
+
34
+ self.open_state()
35
+ assert len(self.state.list) == NUM_OF_THREADS * STEPS_OF_EACH_THREAD
36
+ self.close_state()
37
+
38
+ def open_state(self):
39
+ self.state = PersistedState(self.filepath, index=0, list=[])
40
+
41
+ def close_state(self):
42
+ self.state.close()
43
+
44
+ def threaded_function(self):
45
+ for _ in range(STEPS_OF_EACH_THREAD):
46
+ self.state.list.append(threading.get_ident())
@@ -1,112 +0,0 @@
1
- Metadata-Version: 2.1
2
- Name: persistedstate
3
- Version: 24.2
4
- Summary: Ultra fast persistent store for small states in mapped YAML file
5
- Author-email: Máté Farkas <fm@farkas-mate.hu>
6
- License: MIT License
7
- Project-URL: Homepage, https://github.com/presidento/persistedstate
8
- Project-URL: Changelog, https://github.com/presidento/persistedstate/blob/main/CHANGELOG.md
9
- Classifier: Development Status :: 5 - Production/Stable
10
- Classifier: Intended Audience :: Developers
11
- Classifier: Intended Audience :: System Administrators
12
- Classifier: License :: OSI Approved :: MIT License
13
- Classifier: Operating System :: OS Independent
14
- Classifier: Programming Language :: Python :: 3 :: Only
15
- Classifier: Programming Language :: Python :: 3.9
16
- Classifier: Programming Language :: Python :: 3.10
17
- Classifier: Programming Language :: Python :: 3.11
18
- Classifier: Programming Language :: Python :: 3.12
19
- Classifier: Programming Language :: Python :: 3.13
20
- Classifier: Topic :: Utilities
21
- Classifier: Typing :: Typed
22
- Description-Content-Type: text/markdown
23
- License-File: LICENSE
24
- Requires-Dist: pyyaml
25
- Provides-Extra: dev
26
- Requires-Dist: black; extra == "dev"
27
- Requires-Dist: mypy; extra == "dev"
28
- Requires-Dist: pylint; extra == "dev"
29
- Requires-Dist: scripthelper; extra == "dev"
30
- Requires-Dist: twine; extra == "dev"
31
- Requires-Dist: types-PyYAML; extra == "dev"
32
- Provides-Extra: test
33
- Requires-Dist: pytest; extra == "test"
34
- Provides-Extra: perftest
35
- Requires-Dist: diskcache; extra == "perftest"
36
- Requires-Dist: lmdbm; extra == "perftest"
37
- Requires-Dist: pickledb; extra == "perftest"
38
- Requires-Dist: sqlitedict; extra == "perftest"
39
-
40
- # PersistedState
41
-
42
- Simple and fast solution for persisting small states:
43
-
44
- ```python
45
- from persistedstate import PersistedState
46
-
47
- STATE = PersistedState("example.state", last_id=0)
48
-
49
- for current_id in range(STATE.last_id + 1, 100_001):
50
- print(f"Processing item #{current_id}")
51
- STATE.last_id = current_id
52
-
53
- print("Processing DONE.")
54
- ```
55
-
56
- You can interrupt this script, and next time it will continue from the first unprocessed item.
57
- As another example see [perftest.py](perftest.py).
58
-
59
- ## Mapped YAML state file
60
-
61
- The use case is persisting small amount of data which can be edited easily with a text editor.
62
- The database is an UTF-8 encoded YAML file, so it can be highlighted and edited manually if needed.
63
- (It also uses YAML stream file format for journal.)
64
- The YAML file is fully mapped to Python objects, so every change is synchronized to disk immediately. E.g. you can do this:
65
-
66
- ```python
67
- STATE = PersistedState("state.yaml", processed_items=[])
68
- STATE.setdefault("key", {})
69
- STATE["key"].setdefault("nested", 2)
70
-
71
- STATE.processed_items.append("<some item>")
72
- STATE["key"]["nested"] += 1
73
- ```
74
-
75
- ## Failure tolerance
76
-
77
- It uses Write-Ahead-Logging and atomic vacuum, so there will be no data loss.
78
-
79
- ## Thread safe
80
-
81
- Changing the state is thread safe. You also can use the `._thread_lock` attribute to make atomic changes:
82
-
83
-
84
- ```python
85
- def atomic_increment():
86
- with STATE._thread_lock:
87
- STATE.counter += 1
88
- ```
89
-
90
- ## Performance
91
-
92
- For its use case it outperforms existing key-value store modules.
93
- For example incrementing a counter (for the details see [perftest.py](perftest.py)):
94
-
95
- ```python
96
- state.counter = 0
97
- for _ in range(COUNT_TO):
98
- state.counter += 1
99
- ```
100
-
101
- Counting to 10,000 (using Windows and Python 3.11):
102
-
103
- ```
104
- PersistedState 0.193 sec
105
- DiskCache 1.222 sec
106
- SqliteDict 3.089 sec
107
- PickleDb 8.251 sec
108
- Lmdb 14.944 sec
109
- Shelve 39.771 sec
110
- ```
111
-
112
- The example seems to be silly, but this is very close to the use case it was developed for. For complex data structures or big amount of data I suggest using other libraries, like [DiskCache](https://grantjenks.com/docs/diskcache/). (The rule of thumb is when your state file is too big to be edited easily in your favorite text editor, you may think about using another key-value store library.)
@@ -1,4 +0,0 @@
1
- [egg_info]
2
- tag_build =
3
- tag_date = 0
4
-
@@ -1,10 +0,0 @@
1
- LICENSE
2
- README.md
3
- pyproject.toml
4
- src/persistedstate/__init__.py
5
- src/persistedstate/py.typed
6
- src/persistedstate.egg-info/PKG-INFO
7
- src/persistedstate.egg-info/SOURCES.txt
8
- src/persistedstate.egg-info/dependency_links.txt
9
- src/persistedstate.egg-info/requires.txt
10
- src/persistedstate.egg-info/top_level.txt
@@ -1,18 +0,0 @@
1
- pyyaml
2
-
3
- [dev]
4
- black
5
- mypy
6
- pylint
7
- scripthelper
8
- twine
9
- types-PyYAML
10
-
11
- [perftest]
12
- diskcache
13
- lmdbm
14
- pickledb
15
- sqlitedict
16
-
17
- [test]
18
- pytest
@@ -1 +0,0 @@
1
- persistedstate
File without changes
File without changes