checkpointer 2.11.2__py3-none-any.whl → 2.13.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checkpointer/__init__.py +4 -4
- checkpointer/checkpoint.py +125 -107
- checkpointer/fn_ident.py +129 -71
- checkpointer/fn_string.py +77 -0
- checkpointer/import_mappings.py +47 -0
- checkpointer/object_hash.py +37 -29
- checkpointer/storages/__init__.py +6 -4
- checkpointer/storages/storage.py +5 -4
- checkpointer/types.py +30 -2
- checkpointer/utils.py +84 -54
- checkpointer-2.13.0.dist-info/METADATA +260 -0
- checkpointer-2.13.0.dist-info/RECORD +18 -0
- checkpointer-2.13.0.dist-info/licenses/ATTRIBUTION.md +33 -0
- {checkpointer-2.11.2.dist-info → checkpointer-2.13.0.dist-info}/licenses/LICENSE +2 -0
- checkpointer/test_checkpointer.py +0 -168
- checkpointer-2.11.2.dist-info/METADATA +0 -240
- checkpointer-2.11.2.dist-info/RECORD +0 -16
- {checkpointer-2.11.2.dist-info → checkpointer-2.13.0.dist-info}/WHEEL +0 -0
@@ -1,240 +0,0 @@
|
|
1
|
-
Metadata-Version: 2.4
|
2
|
-
Name: checkpointer
|
3
|
-
Version: 2.11.2
|
4
|
-
Summary: checkpointer adds code-aware caching to Python functions, maintaining correctness and speeding up execution as your code changes.
|
5
|
-
Project-URL: Repository, https://github.com/Reddan/checkpointer.git
|
6
|
-
Author: Hampus Hallman
|
7
|
-
License: Copyright 2018-2025 Hampus Hallman
|
8
|
-
|
9
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
10
|
-
|
11
|
-
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
12
|
-
|
13
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
14
|
-
License-File: LICENSE
|
15
|
-
Classifier: Programming Language :: Python :: 3.11
|
16
|
-
Classifier: Programming Language :: Python :: 3.12
|
17
|
-
Classifier: Programming Language :: Python :: 3.13
|
18
|
-
Requires-Python: >=3.11
|
19
|
-
Description-Content-Type: text/markdown
|
20
|
-
|
21
|
-
# checkpointer · [](https://github.com/Reddan/checkpointer/blob/master/LICENSE) [](https://pypi.org/project/checkpointer/) [](https://pypi.org/project/checkpointer/)
|
22
|
-
|
23
|
-
`checkpointer` is a Python library offering a decorator-based API for memoizing (caching) function results with code-aware cache invalidation. It works with sync and async functions, supports multiple storage backends, and refreshes caches automatically when your code or dependencies change - helping you maintain correctness, speed up execution, and smooth out your workflows by skipping redundant, costly operations.
|
24
|
-
|
25
|
-
## 📦 Installation
|
26
|
-
|
27
|
-
```bash
|
28
|
-
pip install checkpointer
|
29
|
-
```
|
30
|
-
|
31
|
-
## 🚀 Quick Start
|
32
|
-
|
33
|
-
Apply the `@checkpoint` decorator to any function:
|
34
|
-
|
35
|
-
```python
|
36
|
-
from checkpointer import checkpoint
|
37
|
-
|
38
|
-
@checkpoint
|
39
|
-
def expensive_function(x: int) -> int:
|
40
|
-
print("Computing...")
|
41
|
-
return x ** 2
|
42
|
-
|
43
|
-
result = expensive_function(4) # Computes and stores the result
|
44
|
-
result = expensive_function(4) # Loads from the cache
|
45
|
-
```
|
46
|
-
|
47
|
-
## 🧠 How It Works
|
48
|
-
|
49
|
-
When a `@checkpoint`-decorated function is called, `checkpointer` first computes a unique identifier (hash) for the call. This hash is derived from the function's source code, its dependencies, and the arguments passed.
|
50
|
-
|
51
|
-
It then tries to retrieve a cached result using this ID. If a valid cached result is found, it's returned immediately. Otherwise, the original function executes, its result is stored, and then returned.
|
52
|
-
|
53
|
-
Cache validity is determined by this function's hash, which automatically updates if:
|
54
|
-
|
55
|
-
* **Function Code Changes**: The decorated function's source code is modified.
|
56
|
-
* **Dependencies Change**: Any user-defined function in its dependency tree (direct or indirect, even across modules or not decorated) is modified.
|
57
|
-
* **Captured Variables Change** (with `capture=True`): Global or closure-based variables used within the function are altered.
|
58
|
-
|
59
|
-
**Example: Dependency Invalidation**
|
60
|
-
|
61
|
-
```python
|
62
|
-
def multiply(a, b):
|
63
|
-
return a * b
|
64
|
-
|
65
|
-
@checkpoint
|
66
|
-
def helper(x):
|
67
|
-
# Depends on `multiply`
|
68
|
-
return multiply(x + 1, 2)
|
69
|
-
|
70
|
-
@checkpoint
|
71
|
-
def compute(a, b):
|
72
|
-
# Depends on `helper` and `multiply`
|
73
|
-
return helper(a) + helper(b)
|
74
|
-
```
|
75
|
-
|
76
|
-
If `multiply` is modified, caches for both `helper` and `compute` are automatically invalidated and recomputed.
|
77
|
-
|
78
|
-
## 💡 Usage
|
79
|
-
|
80
|
-
Once a function is decorated with `@checkpoint`, you can interact with its caching behavior using the following methods:
|
81
|
-
|
82
|
-
* **`expensive_function(...)`**:\
|
83
|
-
Call the function normally. This will either compute and cache the result or load it from the cache if available.
|
84
|
-
|
85
|
-
* **`expensive_function.rerun(...)`**:\
|
86
|
-
Forces the original function to execute, compute a new result, and overwrite any existing cached value for the given arguments.
|
87
|
-
|
88
|
-
* **`expensive_function.fn(...)`**:\
|
89
|
-
Calls the original, undecorated function directly, bypassing the cache entirely. This is particularly useful within recursive functions to prevent caching intermediate steps.
|
90
|
-
|
91
|
-
* **`expensive_function.get(...)`**:\
|
92
|
-
Attempts to retrieve the cached result for the given arguments without executing the original function. Raises `CheckpointError` if no valid cached result exists.
|
93
|
-
|
94
|
-
* **`expensive_function.exists(...)`**:\
|
95
|
-
Checks if a cached result exists for the given arguments without attempting to compute or load it. Returns `True` if a valid checkpoint exists, `False` otherwise.
|
96
|
-
|
97
|
-
* **`expensive_function.delete(...)`**:\
|
98
|
-
Removes the cached entry for the specified arguments.
|
99
|
-
|
100
|
-
* **`expensive_function.reinit()`**:\
|
101
|
-
Recalculates the function's internal hash. This is primarily used when `capture=True` and you need to update the cache based on changes to external variables within the same Python session.
|
102
|
-
|
103
|
-
## ⚙️ Configuration & Customization
|
104
|
-
|
105
|
-
The `@checkpoint` decorator accepts the following parameters to customize its behavior:
|
106
|
-
|
107
|
-
* **`format`** (Type: `str` or `checkpointer.Storage`, Default: `"pickle"`)\
|
108
|
-
Defines the storage backend to use. Built-in options are `"pickle"` (disk-based, persistent) and `"memory"` (in-memory, non-persistent). You can also provide a custom `Storage` class.
|
109
|
-
|
110
|
-
* **`root_path`** (Type: `str` or `pathlib.Path` or `None`, Default: `~/.cache/checkpoints`)\
|
111
|
-
The base directory for storing disk-based checkpoints. This parameter is only relevant when `format` is set to `"pickle"`.
|
112
|
-
|
113
|
-
* **`when`** (Type: `bool`, Default: `True`)\
|
114
|
-
A boolean flag to enable or disable checkpointing for the decorated function. This is particularly useful for toggling caching based on environment variables (e.g., `when=os.environ.get("ENABLE_CACHING", "false").lower() == "true"`).
|
115
|
-
|
116
|
-
* **`capture`** (Type: `bool`, Default: `False`)\
|
117
|
-
If set to `True`, `checkpointer` includes global or closure-based variables used by the function in its hash calculation. This ensures that changes to these external variables also trigger cache invalidation and recomputation.
|
118
|
-
|
119
|
-
* **`should_expire`** (Type: `Callable[[datetime.datetime], bool]`, Default: `None`)\
|
120
|
-
A custom callable that receives the `datetime` timestamp of a cached result. It should return `True` if the cached result is considered expired and needs recomputation, or `False` otherwise.
|
121
|
-
|
122
|
-
* **`fn_hash_from`** (Type: `Any`, Default: `None`)\
|
123
|
-
This allows you to override the automatically computed function hash, giving you explicit control over when the function's cache should be invalidated. You can pass any object relevant to your invalidation logic (e.g., version strings, config parameters). The object you provide will be hashed internally by `checkpointer`.
|
124
|
-
|
125
|
-
* **`verbosity`** (Type: `int` (`0`, `1`, or `2`), Default: `1`)\
|
126
|
-
Controls the level of logging output from `checkpointer`.
|
127
|
-
* `0`: No output.
|
128
|
-
* `1`: Shows when functions are computed and cached.
|
129
|
-
* `2`: Also shows when cached results are remembered (loaded from cache).
|
130
|
-
|
131
|
-
## 🔬 Customize Argument Hashing
|
132
|
-
|
133
|
-
You can customize how individual function arguments are hashed without changing their actual values when passed in.
|
134
|
-
|
135
|
-
* **`Annotated[T, HashBy[fn]]`**:\
|
136
|
-
Hashes the argument by applying `fn(argument)` before hashing. This enables custom normalization (e.g., sorting lists to ignore order) or optimized hashing for complex types, improving cache hit rates or speeding up hashing.
|
137
|
-
|
138
|
-
* **`NoHash[T]`**:\
|
139
|
-
Completely excludes the argument from hashing, so changes to it won’t trigger cache invalidation.
|
140
|
-
|
141
|
-
**Example:**
|
142
|
-
|
143
|
-
```python
|
144
|
-
from typing import Annotated
|
145
|
-
from checkpointer import checkpoint, HashBy, NoHash
|
146
|
-
from pathlib import Path
|
147
|
-
import logging
|
148
|
-
|
149
|
-
def file_bytes(path: Path) -> bytes:
|
150
|
-
return path.read_bytes()
|
151
|
-
|
152
|
-
@checkpoint
|
153
|
-
def process(
|
154
|
-
numbers: Annotated[list[int], HashBy[sorted]], # Hash by sorted list
|
155
|
-
data_file: Annotated[Path, HashBy[file_bytes]], # Hash by file content
|
156
|
-
log: NoHash[logging.Logger], # Exclude logger from hashing
|
157
|
-
):
|
158
|
-
...
|
159
|
-
```
|
160
|
-
|
161
|
-
In this example, the cache key for `numbers` ignores order, `data_file` is hashed based on its contents rather than path, and changes to `log` don’t affect caching.
|
162
|
-
|
163
|
-
## 🗄️ Custom Storage Backends
|
164
|
-
|
165
|
-
For integration with databases, cloud storage, or custom serialization, implement your own storage backend by inheriting from `checkpointer.Storage` and implementing its abstract methods.
|
166
|
-
|
167
|
-
Within custom storage methods, `call_hash` identifies calls by arguments. Use `self.fn_id()` to get the function's unique identity (name + hash/version), crucial for organizing stored checkpoints (e.g., by function version). Access global `Checkpointer` config via `self.checkpointer`.
|
168
|
-
|
169
|
-
**Example: Custom Storage Backend**
|
170
|
-
|
171
|
-
```python
|
172
|
-
from checkpointer import checkpoint, Storage
|
173
|
-
from datetime import datetime
|
174
|
-
|
175
|
-
class MyCustomStorage(Storage):
|
176
|
-
def exists(self, call_hash):
|
177
|
-
# Example: Constructing a path based on function ID and call ID
|
178
|
-
fn_dir = self.checkpointer.root_path / self.fn_id()
|
179
|
-
return (fn_dir / call_hash).exists()
|
180
|
-
|
181
|
-
def store(self, call_hash, data):
|
182
|
-
... # Store the serialized data for `call_hash`
|
183
|
-
return data # This method must return the data back to checkpointer
|
184
|
-
|
185
|
-
def checkpoint_date(self, call_hash): ...
|
186
|
-
def load(self, call_hash): ...
|
187
|
-
def delete(self, call_hash): ...
|
188
|
-
|
189
|
-
@checkpoint(format=MyCustomStorage)
|
190
|
-
def custom_cached_function(x: int):
|
191
|
-
return x ** 2
|
192
|
-
```
|
193
|
-
|
194
|
-
## 🧱 Layered Caching
|
195
|
-
|
196
|
-
You can apply multiple `@checkpoint` decorators to a single function to create layered caching strategies. `checkpointer` processes these decorators from bottom to top, meaning the decorator closest to the function definition is evaluated first.
|
197
|
-
|
198
|
-
This is useful for scenarios like combining a fast, ephemeral cache (e.g., in-memory) with a persistent, slower cache (e.g., disk-based).
|
199
|
-
|
200
|
-
**Example: Memory Cache over Disk Cache**
|
201
|
-
|
202
|
-
```python
|
203
|
-
from checkpointer import checkpoint
|
204
|
-
|
205
|
-
@checkpoint(format="memory") # Layer 2: Fast, ephemeral in-memory cache
|
206
|
-
@checkpoint(format="pickle") # Layer 1: Persistent disk cache
|
207
|
-
def some_expensive_operation():
|
208
|
-
print("Performing a time-consuming operation...")
|
209
|
-
return sum(i for i in range(10**7))
|
210
|
-
```
|
211
|
-
|
212
|
-
## ⚡ Async Support
|
213
|
-
|
214
|
-
`checkpointer` works seamlessly with Python's `asyncio` and other async runtimes.
|
215
|
-
|
216
|
-
```python
|
217
|
-
import asyncio
|
218
|
-
from checkpointer import checkpoint
|
219
|
-
|
220
|
-
@checkpoint
|
221
|
-
async def async_compute_sum(a: int, b: int) -> int:
|
222
|
-
print(f"Asynchronously computing {a} + {b}...")
|
223
|
-
await asyncio.sleep(1)
|
224
|
-
return a + b
|
225
|
-
|
226
|
-
async def main():
|
227
|
-
# First call computes and caches
|
228
|
-
result1 = await async_compute_sum(3, 7)
|
229
|
-
print(f"Result 1: {result1}")
|
230
|
-
|
231
|
-
# Second call loads from cache
|
232
|
-
result2 = await async_compute_sum(3, 7)
|
233
|
-
print(f"Result 2: {result2}")
|
234
|
-
|
235
|
-
# Retrieve from cache without re-running the async function
|
236
|
-
result3 = async_compute_sum.get(3, 7)
|
237
|
-
print(f"Result 3 (from cache): {result3}")
|
238
|
-
|
239
|
-
asyncio.run(main())
|
240
|
-
```
|
@@ -1,16 +0,0 @@
|
|
1
|
-
checkpointer/__init__.py,sha256=ayjFyHwvl_HRHwocY-hOJvAx0Ko5X9IMZrNT4CwfoMU,824
|
2
|
-
checkpointer/checkpoint.py,sha256=jUTImaeAMde2skReH8DxmlTaUe8XzL1uKRSkbS1-N80,9523
|
3
|
-
checkpointer/fn_ident.py,sha256=-5XbovQowVyYCFc7JdT9z1NoIEiL8h9fi7alF_34Ils,4470
|
4
|
-
checkpointer/object_hash.py,sha256=YlyFupQrg3V2mpzTLfOqpqlZWhoSCHliScQ4cKd36T0,8133
|
5
|
-
checkpointer/print_checkpoint.py,sha256=uUQ493fJCaB4nhp4Ox60govSCiBTIPbBX15zt2QiRGo,1356
|
6
|
-
checkpointer/test_checkpointer.py,sha256=-EvsMMNOOiIxhTcG97LLX0jUMWp534ko7qCKDSFWiA0,3802
|
7
|
-
checkpointer/types.py,sha256=n1CxJsh7c_o72pFEfSbZ8cZgMeSNfAbLWUwrca8-zNo,449
|
8
|
-
checkpointer/utils.py,sha256=CXoofmu6nxY5uW7oPUqdH31qffi3jQn_sc1qXbzxCsU,2137
|
9
|
-
checkpointer/storages/__init__.py,sha256=en32nTUltpCSgz8RVGS_leIHC1Y1G89IqG1ZqAb6qUo,236
|
10
|
-
checkpointer/storages/memory_storage.py,sha256=aQRSOmAfS0UudubCpv8cdfu2ycM8mlsO9tFMcD2kmgo,1133
|
11
|
-
checkpointer/storages/pickle_storage.py,sha256=je1LM2lTSs5yzm25Apg5tJ9jU9T6nXCgD9SlqQRIFaM,1652
|
12
|
-
checkpointer/storages/storage.py,sha256=5Jel7VlmCG9gnIbZFKT1NrEiAePz8ZD8hfsD-tYEiP4,905
|
13
|
-
checkpointer-2.11.2.dist-info/METADATA,sha256=5nwfJ3M_F-RjJLCr49fMaXdjbl5yk1dKZt6Fk7WCy_k,11630
|
14
|
-
checkpointer-2.11.2.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
|
15
|
-
checkpointer-2.11.2.dist-info/licenses/LICENSE,sha256=9xVsdtv_-uSyY9Xl9yujwAPm4-mjcCLeVy-ljwXEWbo,1059
|
16
|
-
checkpointer-2.11.2.dist-info/RECORD,,
|
File without changes
|