checkpointer 2.0.2__tar.gz → 2.5.0__tar.gz

{checkpointer-2.0.2 → checkpointer-2.5.0}/LICENSE

@@ -1,4 +1,4 @@
-Copyright 2024 Hampus Hallman
+Copyright 2018-2025 Hampus Hallman
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 

{checkpointer-2.0.2 → checkpointer-2.5.0}/PKG-INFO

@@ -1,25 +1,25 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.4
 Name: checkpointer
-Version: 2.0.2
+Version: 2.5.0
 Summary: A Python library for memoizing function results with support for multiple storage backends, async runtimes, and automatic cache invalidation
 Project-URL: Repository, https://github.com/Reddan/checkpointer.git
 Author: Hampus Hallman
-License: Copyright 2024 Hampus Hallman
+License: Copyright 2018-2025 Hampus Hallman
         
         Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
         
         The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
         
         THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+License-File: LICENSE
 Requires-Python: >=3.12
-Requires-Dist: relib
 Description-Content-Type: text/markdown
 
 # checkpointer · [![License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/Reddan/checkpointer/blob/master/LICENSE) [![pypi](https://img.shields.io/pypi/v/checkpointer)](https://pypi.org/project/checkpointer/) [![Python 3.12](https://img.shields.io/badge/python-3.12-blue)](https://pypi.org/project/checkpointer/)
 
 `checkpointer` is a Python library for memoizing function results. It provides a decorator-based API with support for multiple storage backends. Use it for computationally expensive operations where caching can save time, or during development to avoid waiting for redundant computations.
 
-Adding or removing `@checkpoint` doesn't change how your code works, and it can be applied to any function, including ones you've already written, without altering their behavior or introducing side effects. The original function remains unchanged and can still be called directly when needed.
+Adding or removing `@checkpoint` doesn't change how your code works. You can apply it to any function, including ones you've already written, without altering their behavior or introducing side effects. The original function remains unchanged and can still be called directly when needed.
 
 ### Key Features:
 - 🗂️ **Multiple Storage Backends**: Built-in support for in-memory and pickle-based storage, or create your own.
@@ -27,6 +27,7 @@ Adding or removing `@checkpoint` doesn't change how your code works, and it can
 - 🔄 **Async and Sync Compatibility**: Works with synchronous functions and any Python async runtime (e.g., `asyncio`, `Trio`, `Curio`).
 - ⏲️ **Custom Expiration Logic**: Automatically invalidate old checkpoints.
 - 📂 **Flexible Path Configuration**: Control where checkpoints are stored.
+- 📦 **Captured Variables Handling**: Optionally include captured variables in cache invalidation.
 
 ---
 
@@ -59,8 +60,10 @@ result = expensive_function(4)  # Loads from the cache
 When you use `@checkpoint`, the function's **arguments** (`args`, `kwargs`) are hashed to create a unique identifier for each call. This identifier is used to store and retrieve cached results. If the same arguments are passed again, `checkpointer` loads the cached result instead of recomputing.
 
 Additionally, `checkpointer` ensures that caches are invalidated when a function's implementation or any of its dependencies change. Each function is assigned a hash based on:
+
 1. **Its source code**: Changes to the function's code update its hash.
 2. **Dependent functions**: If a function calls others, changes in those dependencies will also update the hash.
+3. **Captured variables**: (Optional) If `capture=True`, changes to captured variables and global variables will also update the hash.
 
 ### Example: Cache Invalidation
 
@@ -105,7 +108,7 @@ Layer caches by stacking checkpoints:
 @dev_checkpoint  # Adds caching during development
 def some_expensive_function():
     print("Performing a time-consuming operation...")
-    return sum(i * i for i in range(10**6))
+    return sum(i * i for i in range(10**8))
 ```
 
 - **In development**: Both `dev_checkpoint` and `memory` caches are active.
@@ -115,7 +118,17 @@ def some_expensive_function():
 
 ## Usage
 
+### Basic Invocation and Caching
+
+Call the decorated function as usual. On the first call, the result is computed and stored in the cache. Subsequent calls with the same arguments load the result from the cache:
+
+```python
+result = expensive_function(4)  # Computes and stores the result
+result = expensive_function(4)  # Loads the result from the cache
+```
+
 ### Force Recalculation
+
 Force a recalculation and overwrite the stored checkpoint:
 
 ```python
@@ -123,6 +136,7 @@ result = expensive_function.rerun(4)
 ```
 
 ### Call the Original Function
+
 Use `fn` to directly call the original, undecorated function:
 
 ```python
@@ -132,12 +146,25 @@ result = expensive_function.fn(4)
 This is especially useful **inside recursive functions** to avoid redundant caching of intermediate steps while still caching the final result.
 
 ### Retrieve Stored Checkpoints
+
 Access cached results without recalculating:
 
 ```python
 stored_result = expensive_function.get(4)
 ```
 
+### Refresh Function Hash
+
+When using `capture=True`, changes to captured variables are included in the function's hash to determine cache invalidation. However, `checkpointer` does not automatically update this hash during a running Python session—it recalculates between sessions or when you explicitly refresh it.
+
+Use the `reinit` method to manually refresh the function's hash within the same session:
+
+```python
+expensive_function.reinit()
+```
+
+This forces `checkpointer` to recalculate the hash of `expensive_function`, considering any changes to captured variables. It's useful when you've modified external variables that the function depends on and want the cache to reflect these changes immediately.
+
 ---
 
 ## Storage Backends
@@ -154,11 +181,11 @@ You can specify a storage backend using either its name (`"pickle"` or `"memory"
 ```python
 from checkpointer import checkpoint, PickleStorage, MemoryStorage
 
-@checkpoint(format="pickle")  # Equivalent to format=PickleStorage
+@checkpoint(format="pickle")  # Short for format=PickleStorage
 def disk_cached(x: int) -> int:
     return x ** 2
 
-@checkpoint(format="memory")  # Equivalent to format=MemoryStorage
+@checkpoint(format="memory")  # Short for format=MemoryStorage
 def memory_cached(x: int) -> int:
     return x * 10
 ```
@@ -174,9 +201,9 @@ from checkpointer import checkpoint, Storage
 from datetime import datetime
 
 class CustomStorage(Storage):
+    def store(self, path, data): ...  # Save the checkpoint data
     def exists(self, path) -> bool: ...  # Check if a checkpoint exists at the given path
     def checkpoint_date(self, path) -> datetime: ...  # Return the date the checkpoint was created
-    def store(self, path, data): ...  # Save the checkpoint data
     def load(self, path): ...  # Return the checkpoint data
     def delete(self, path): ...  # Delete the checkpoint
 
@@ -191,14 +218,15 @@ Using a custom backend lets you tailor storage to your application, whether it i
 
 ## Configuration Options ⚙️
 
-| Option         | Type                                | Default     | Description                                 |
-|----------------|-------------------------------------|-------------|---------------------------------------------|
-| `format`       | `"pickle"`, `"memory"`, `Storage`   | `"pickle"`  | Storage backend format.                     |
-| `root_path`    | `Path`, `str`, or `None`            | User Cache  | Root directory for storing checkpoints.     |
-| `when`         | `bool`                              | `True`      | Enable or disable checkpointing.            |
-| `verbosity`    | `0` or `1`                          | `1`         | Logging verbosity.                          |
-| `path`         | `Callable[..., str]`                | `None`      | Custom path for checkpoint storage.         |
-| `should_expire`| `Callable[[datetime], bool]`        | `None`      | Custom expiration logic.                    |
+| Option          | Type                              | Default              | Description                                    |
+|-----------------|-----------------------------------|----------------------|------------------------------------------------|
+| `capture`       | `bool`                            | `False`              | Include captured variables in function hashes. |
+| `format`        | `"pickle"`, `"memory"`, `Storage` | `"pickle"`           | Storage backend format.                        |
+| `root_path`     | `Path`, `str`, or `None`          | ~/.cache/checkpoints | Root directory for storing checkpoints.        |
+| `when`          | `bool`                            | `True`               | Enable or disable checkpointing.               |
+| `verbosity`     | `0` or `1`                        | `1`                  | Logging verbosity.                             |
+| `path`          | `Callable[..., str]`              | `None`               | Custom path for checkpoint storage.            |
+| `should_expire` | `Callable[[datetime], bool]`      | `None`               | Custom expiration logic.                       |
 
 ---
 
@@ -220,13 +248,13 @@ async def async_compute_sum(a: int, b: int) -> int:
 
 async def main():
     result1 = compute_square(5)
-    print(result1)
+    print(result1)  # Outputs 25
 
     result2 = await async_compute_sum(3, 7)
-    print(result2)
+    print(result2)  # Outputs 10
 
-    result3 = async_compute_sum.get(3, 7)
-    print(result3)
+    result3 = await async_compute_sum.get(3, 7)
+    print(result3)  # Outputs 10
 
 asyncio.run(main())
 ```

{checkpointer-2.0.2 → checkpointer-2.5.0}/README.md

@@ -2,7 +2,7 @@
 
 `checkpointer` is a Python library for memoizing function results. It provides a decorator-based API with support for multiple storage backends. Use it for computationally expensive operations where caching can save time, or during development to avoid waiting for redundant computations.
 
-Adding or removing `@checkpoint` doesn't change how your code works, and it can be applied to any function, including ones you've already written, without altering their behavior or introducing side effects. The original function remains unchanged and can still be called directly when needed.
+Adding or removing `@checkpoint` doesn't change how your code works. You can apply it to any function, including ones you've already written, without altering their behavior or introducing side effects. The original function remains unchanged and can still be called directly when needed.
 
 ### Key Features:
 - 🗂️ **Multiple Storage Backends**: Built-in support for in-memory and pickle-based storage, or create your own.
@@ -10,6 +10,7 @@ Adding or removing `@checkpoint` doesn't change how your code works, and it can
 - 🔄 **Async and Sync Compatibility**: Works with synchronous functions and any Python async runtime (e.g., `asyncio`, `Trio`, `Curio`).
 - ⏲️ **Custom Expiration Logic**: Automatically invalidate old checkpoints.
 - 📂 **Flexible Path Configuration**: Control where checkpoints are stored.
+- 📦 **Captured Variables Handling**: Optionally include captured variables in cache invalidation.
 
 ---
 
@@ -42,8 +43,10 @@ result = expensive_function(4)  # Loads from the cache
 When you use `@checkpoint`, the function's **arguments** (`args`, `kwargs`) are hashed to create a unique identifier for each call. This identifier is used to store and retrieve cached results. If the same arguments are passed again, `checkpointer` loads the cached result instead of recomputing.
 
 Additionally, `checkpointer` ensures that caches are invalidated when a function's implementation or any of its dependencies change. Each function is assigned a hash based on:
+
 1. **Its source code**: Changes to the function's code update its hash.
 2. **Dependent functions**: If a function calls others, changes in those dependencies will also update the hash.
+3. **Captured variables**: (Optional) If `capture=True`, changes to captured variables and global variables will also update the hash.
 
 ### Example: Cache Invalidation
 
@@ -88,7 +91,7 @@ Layer caches by stacking checkpoints:
 @dev_checkpoint  # Adds caching during development
 def some_expensive_function():
     print("Performing a time-consuming operation...")
-    return sum(i * i for i in range(10**6))
+    return sum(i * i for i in range(10**8))
 ```
 
 - **In development**: Both `dev_checkpoint` and `memory` caches are active.
@@ -98,7 +101,17 @@ def some_expensive_function():
 
 ## Usage
 
+### Basic Invocation and Caching
+
+Call the decorated function as usual. On the first call, the result is computed and stored in the cache. Subsequent calls with the same arguments load the result from the cache:
+
+```python
+result = expensive_function(4)  # Computes and stores the result
+result = expensive_function(4)  # Loads the result from the cache
+```
+
 ### Force Recalculation
+
 Force a recalculation and overwrite the stored checkpoint:
 
 ```python
@@ -106,6 +119,7 @@ result = expensive_function.rerun(4)
 ```
 
 ### Call the Original Function
+
 Use `fn` to directly call the original, undecorated function:
 
 ```python
@@ -115,12 +129,25 @@ result = expensive_function.fn(4)
 This is especially useful **inside recursive functions** to avoid redundant caching of intermediate steps while still caching the final result.
 
 ### Retrieve Stored Checkpoints
+
 Access cached results without recalculating:
 
 ```python
 stored_result = expensive_function.get(4)
 ```
 
+### Refresh Function Hash
+
+When using `capture=True`, changes to captured variables are included in the function's hash to determine cache invalidation. However, `checkpointer` does not automatically update this hash during a running Python session—it recalculates between sessions or when you explicitly refresh it.
+
+Use the `reinit` method to manually refresh the function's hash within the same session:
+
+```python
+expensive_function.reinit()
+```
+
+This forces `checkpointer` to recalculate the hash of `expensive_function`, considering any changes to captured variables. It's useful when you've modified external variables that the function depends on and want the cache to reflect these changes immediately.
+
 ---
 
 ## Storage Backends
@@ -137,11 +164,11 @@ You can specify a storage backend using either its name (`"pickle"` or `"memory"
 ```python
 from checkpointer import checkpoint, PickleStorage, MemoryStorage
 
-@checkpoint(format="pickle")  # Equivalent to format=PickleStorage
+@checkpoint(format="pickle")  # Short for format=PickleStorage
 def disk_cached(x: int) -> int:
     return x ** 2
 
-@checkpoint(format="memory")  # Equivalent to format=MemoryStorage
+@checkpoint(format="memory")  # Short for format=MemoryStorage
 def memory_cached(x: int) -> int:
     return x * 10
 ```
@@ -157,9 +184,9 @@ from checkpointer import checkpoint, Storage
 from datetime import datetime
 
 class CustomStorage(Storage):
+    def store(self, path, data): ...  # Save the checkpoint data
     def exists(self, path) -> bool: ...  # Check if a checkpoint exists at the given path
     def checkpoint_date(self, path) -> datetime: ...  # Return the date the checkpoint was created
-    def store(self, path, data): ...  # Save the checkpoint data
     def load(self, path): ...  # Return the checkpoint data
     def delete(self, path): ...  # Delete the checkpoint
 
@@ -174,14 +201,15 @@ Using a custom backend lets you tailor storage to your application, whether it i
 
 ## Configuration Options ⚙️
 
-| Option         | Type                                | Default     | Description                                 |
-|----------------|-------------------------------------|-------------|---------------------------------------------|
-| `format`       | `"pickle"`, `"memory"`, `Storage`   | `"pickle"`  | Storage backend format.                     |
-| `root_path`    | `Path`, `str`, or `None`            | User Cache  | Root directory for storing checkpoints.     |
-| `when`         | `bool`                              | `True`      | Enable or disable checkpointing.            |
-| `verbosity`    | `0` or `1`                          | `1`         | Logging verbosity.                          |
-| `path`         | `Callable[..., str]`                | `None`      | Custom path for checkpoint storage.         |
-| `should_expire`| `Callable[[datetime], bool]`        | `None`      | Custom expiration logic.                    |
+| Option          | Type                              | Default              | Description                                    |
+|-----------------|-----------------------------------|----------------------|------------------------------------------------|
+| `capture`       | `bool`                            | `False`              | Include captured variables in function hashes. |
+| `format`        | `"pickle"`, `"memory"`, `Storage` | `"pickle"`           | Storage backend format.                        |
+| `root_path`     | `Path`, `str`, or `None`          | ~/.cache/checkpoints | Root directory for storing checkpoints.        |
+| `when`          | `bool`                            | `True`               | Enable or disable checkpointing.               |
+| `verbosity`     | `0` or `1`                        | `1`                  | Logging verbosity.                             |
+| `path`          | `Callable[..., str]`              | `None`               | Custom path for checkpoint storage.            |
+| `should_expire` | `Callable[[datetime], bool]`      | `None`               | Custom expiration logic.                       |
 
 ---
 
@@ -203,13 +231,13 @@ async def async_compute_sum(a: int, b: int) -> int:
 
 async def main():
     result1 = compute_square(5)
-    print(result1)
+    print(result1)  # Outputs 25
 
     result2 = await async_compute_sum(3, 7)
-    print(result2)
+    print(result2)  # Outputs 10
 
-    result3 = async_compute_sum.get(3, 7)
-    print(result3)
+    result3 = await async_compute_sum.get(3, 7)
+    print(result3)  # Outputs 10
 
 asyncio.run(main())
 ```

checkpointer-2.5.0/checkpointer/__init__.py

@@ -0,0 +1,20 @@
+import gc
+import tempfile
+from typing import Callable
+from .checkpoint import Checkpointer, CheckpointError, CheckpointFn
+from .object_hash import ObjectHash
+from .storages import MemoryStorage, PickleStorage, Storage
+
+create_checkpointer = Checkpointer
+checkpoint = Checkpointer()
+capture_checkpoint = Checkpointer(capture=True)
+memory_checkpoint = Checkpointer(format="memory", verbosity=0)
+tmp_checkpoint = Checkpointer(root_path=tempfile.gettempdir() + "/checkpoints")
+
+def cleanup_all(invalidated=True, expired=True):
+  for obj in gc.get_objects():
+    if isinstance(obj, CheckpointFn):
+      obj.cleanup(invalidated=invalidated, expired=expired)
+
+def get_function_hash(fn: Callable, capture=False) -> str:
+  return CheckpointFn(Checkpointer(capture=capture), fn).fn_hash

{checkpointer-2.0.2 → checkpointer-2.5.0}/checkpointer/checkpoint.py

@@ -1,22 +1,19 @@
 from __future__ import annotations
 import inspect
-import relib.hashing as hashing
-from typing import Generic, TypeVar, Type, TypedDict, Callable, Unpack, Literal, Any, cast, overload
-from pathlib import Path
+import re
 from datetime import datetime
 from functools import update_wrapper
-from .types import Storage
-from .function_body import get_function_hash
-from .utils import unwrap_fn, sync_resolve_coroutine
-from .storages.pickle_storage import PickleStorage
-from .storages.memory_storage import MemoryStorage
-from .storages.bcolz_storage import BcolzStorage
+from pathlib import Path
+from typing import Any, Callable, Generic, Iterable, Literal, Type, TypedDict, TypeVar, Unpack, cast, overload
+from .fn_ident import get_fn_ident
+from .object_hash import ObjectHash
 from .print_checkpoint import print_checkpoint
+from .storages import STORAGE_MAP, Storage
+from .utils import resolved_awaitable, sync_resolve_coroutine, unwrap_fn
 
 Fn = TypeVar("Fn", bound=Callable)
 
 DEFAULT_DIR = Path.home() / ".cache/checkpoints"
-STORAGE_MAP: dict[str, Type[Storage]] = {"memory": MemoryStorage, "pickle": PickleStorage, "bcolz": BcolzStorage}
 
 class CheckpointError(Exception):
   pass
@@ -28,6 +25,7 @@ class CheckpointerOpts(TypedDict, total=False):
   verbosity: Literal[0, 1]
   path: Callable[..., str] | None
   should_expire: Callable[[datetime], bool] | None
+  capture: bool
 
 class Checkpointer:
   def __init__(self, **opts: Unpack[CheckpointerOpts]):
@@ -37,6 +35,7 @@ class Checkpointer:
     self.verbosity = opts.get("verbosity", 1)
     self.path = opts.get("path")
     self.should_expire = opts.get("should_expire")
+    self.capture = opts.get("capture", False)
 
   @overload
   def __call__(self, fn: Fn, **override_opts: Unpack[CheckpointerOpts]) -> CheckpointFn[Fn]: ...
@@ -51,20 +50,47 @@ class Checkpointer:
 
 class CheckpointFn(Generic[Fn]):
   def __init__(self, checkpointer: Checkpointer, fn: Fn):
-    wrapped = unwrap_fn(fn)
-    file_name = Path(wrapped.__code__.co_filename).name
-    update_wrapper(cast(Callable, self), wrapped)
-    storage = STORAGE_MAP[checkpointer.format] if isinstance(checkpointer.format, str) else checkpointer.format
     self.checkpointer = checkpointer
     self.fn = fn
-    self.fn_hash = get_function_hash(wrapped)
-    self.fn_id = f"{file_name}/{wrapped.__name__}"
+
+  def _set_ident(self, force=False):
+    if not hasattr(self, "fn_hash_raw") or force:
+      self.fn_hash_raw, self.depends = get_fn_ident(unwrap_fn(self.fn), self.checkpointer.capture)
+    return self
+
+  def _lazyinit(self):
+    wrapped = unwrap_fn(self.fn)
+    fn_file = Path(wrapped.__code__.co_filename).name
+    fn_name = re.sub(r"[^\w.]", "", wrapped.__qualname__)
+    update_wrapper(cast(Callable, self), wrapped)
+    store_format = self.checkpointer.format
+    Storage = STORAGE_MAP[store_format] if isinstance(store_format, str) else store_format
+    deep_hashes = [child._set_ident().fn_hash_raw for child in iterate_checkpoint_fns(self)]
+    self.fn_hash = str(ObjectHash().update_hash(self.fn_hash_raw, iter=deep_hashes))
+    self.fn_subdir = f"{fn_file}/{fn_name}/{self.fn_hash[:16]}"
     self.is_async = inspect.iscoroutinefunction(wrapped)
-    self.storage = storage(checkpointer)
+    self.storage = Storage(self)
+    self.cleanup = self.storage.cleanup
+
+  def __getattribute__(self, name: str) -> Any:
+    return object.__getattribute__(self, "_getattribute")(name)
+
+  def _getattribute(self, name: str) -> Any:
+    setattr(self, "_getattribute", super().__getattribute__)
+    self._lazyinit()
+    return self._getattribute(name)
+
+  def reinit(self, recursive=False):
+    pointfns = list(iterate_checkpoint_fns(self)) if recursive else [self]
+    for pointfn in pointfns:
+      pointfn._set_ident(True)
+    for pointfn in pointfns:
+      pointfn._lazyinit()
 
   def get_checkpoint_id(self, args: tuple, kw: dict) -> str:
     if not callable(self.checkpointer.path):
-      return f"{self.fn_id}/{hashing.hash([self.fn_hash, args, kw or 0])}"
+      call_hash = ObjectHash(self.fn_hash, args, kw, digest_size=16)
+      return f"{self.fn_subdir}/{call_hash}"
     checkpoint_id = self.checkpointer.path(*args, **kw)
     if not isinstance(checkpoint_id, str):
       raise CheckpointError(f"path function must return a string, got {type(checkpoint_id)}")
@@ -73,13 +99,13 @@ class CheckpointFn(Generic[Fn]):
   async def _store_on_demand(self, args: tuple, kw: dict, rerun: bool):
     checkpoint_id = self.get_checkpoint_id(args, kw)
     checkpoint_path = self.checkpointer.root_path / checkpoint_id
-    should_log = self.checkpointer.verbosity > 0
+    verbose = self.checkpointer.verbosity > 0
     refresh = rerun \
       or not self.storage.exists(checkpoint_path) \
       or (self.checkpointer.should_expire and self.checkpointer.should_expire(self.storage.checkpoint_date(checkpoint_path)))
 
     if refresh:
-      print_checkpoint(should_log, "MEMORIZING", checkpoint_id, "blue")
+      print_checkpoint(verbose, "MEMORIZING", checkpoint_id, "blue")
       data = self.fn(*args, **kw)
       if inspect.iscoroutine(data):
         data = await data
@@ -88,12 +114,12 @@ class CheckpointFn(Generic[Fn]):
 
     try:
       data = self.storage.load(checkpoint_path)
-      print_checkpoint(should_log, "REMEMBERED", checkpoint_id, "green")
+      print_checkpoint(verbose, "REMEMBERED", checkpoint_id, "green")
       return data
     except (EOFError, FileNotFoundError):
-      print_checkpoint(should_log, "CORRUPTED", checkpoint_id, "yellow")
-      self.storage.delete(checkpoint_path)
-      return await self._store_on_demand(args, kw, rerun)
+      pass
+    print_checkpoint(verbose, "CORRUPTED", checkpoint_id, "yellow")
+    return await self._store_on_demand(args, kw, True)
 
   def _call(self, args: tuple, kw: dict, rerun=False):
     if not self.checkpointer.when:
@@ -101,12 +127,29 @@ class CheckpointFn(Generic[Fn]):
     coroutine = self._store_on_demand(args, kw, rerun)
     return coroutine if self.is_async else sync_resolve_coroutine(coroutine)
 
+  def _get(self, args, kw) -> Any:
+    checkpoint_path = self.checkpointer.root_path / self.get_checkpoint_id(args, kw)
+    try:
+      val = self.storage.load(checkpoint_path)
+      return resolved_awaitable(val) if self.is_async else val
+    except Exception as ex:
+      raise CheckpointError("Could not load checkpoint") from ex
+
+  def exists(self, *args: tuple, **kw: dict) -> bool:
+    return self.storage.exists(self.checkpointer.root_path / self.get_checkpoint_id(args, kw))
+
   __call__: Fn = cast(Fn, lambda self, *args, **kw: self._call(args, kw))
   rerun: Fn = cast(Fn, lambda self, *args, **kw: self._call(args, kw, True))
+  get: Fn = cast(Fn, lambda self, *args, **kw: self._get(args, kw))
 
-  def get(self, *args, **kw) -> Any:
-    checkpoint_path = self.checkpointer.root_path / self.get_checkpoint_id(args, kw)
-    try:
-      return self.storage.load(checkpoint_path)
-    except:
-      raise CheckpointError("Could not load checkpoint")
+  def __repr__(self) -> str:
+    return f"<CheckpointFn {self.fn.__name__} {self.fn_hash[:6]}>"
+
+def iterate_checkpoint_fns(pointfn: CheckpointFn, visited: set[CheckpointFn] = set()) -> Iterable[CheckpointFn]:
+  visited = visited or set()
+  if pointfn not in visited:
+    yield pointfn
+    visited.add(pointfn)
+    for depend in pointfn.depends:
+      if isinstance(depend, CheckpointFn):
+        yield from iterate_checkpoint_fns(depend, visited)

checkpointer-2.5.0/checkpointer/fn_ident.py

@@ -0,0 +1,94 @@
+import dis
+import inspect
+from collections.abc import Callable
+from itertools import takewhile
+from pathlib import Path
+from types import CodeType, FunctionType, MethodType
+from typing import Any, Generator, Type, TypeGuard
+from .object_hash import ObjectHash
+from .utils import AttrDict, distinct, get_cell_contents, iterate_and_upcoming, transpose, unwrap_fn
+
+cwd = Path.cwd()
+
+def is_class(obj) -> TypeGuard[Type]:
+  # isinstance works too, but needlessly triggers __getattribute__
+  return issubclass(type(obj), type)
+
+def extract_classvars(code: CodeType, scope_vars: AttrDict) -> dict[str, dict[str, Type]]:
+  attr_path: tuple[str, ...] = ()
+  scope_obj = None
+  classvars: dict[str, dict[str, Type]] = {}
+  for instr, upcoming_instrs in iterate_and_upcoming(dis.get_instructions(code)):
+    if instr.opname in scope_vars and not attr_path:
+      attrs = takewhile(lambda instr: instr.opname == "LOAD_ATTR", upcoming_instrs)
+      attr_path = (instr.opname, instr.argval, *(str(x.argval) for x in attrs))
+    elif instr.opname == "CALL":
+      obj = scope_vars.get_at(attr_path)
+      attr_path = ()
+      if is_class(obj):
+        scope_obj = obj
+    elif instr.opname in ("STORE_FAST", "STORE_DEREF") and scope_obj:
+      load_key = instr.opname.replace("STORE", "LOAD")
+      classvars.setdefault(load_key, {})[instr.argval] = scope_obj
+      scope_obj = None
+  return classvars
+
+def extract_scope_values(code: CodeType, scope_vars: AttrDict) -> Generator[tuple[tuple[str, ...], Any], None, None]:
+  classvars = extract_classvars(code, scope_vars)
+  scope_vars = scope_vars.set({k: scope_vars[k].set(v) for k, v in classvars.items()})
+  for instr, upcoming_instrs in iterate_and_upcoming(dis.get_instructions(code)):
+    if instr.opname in scope_vars:
+      attrs = takewhile(lambda instr: instr.opname == "LOAD_ATTR", upcoming_instrs)
+      attr_path: tuple[str, ...] = (instr.opname, instr.argval, *(str(x.argval) for x in attrs))
+      val = scope_vars.get_at(attr_path)
+      if val is not None:
+        yield attr_path, val
+  for const in code.co_consts:
+    if isinstance(const, CodeType):
+      yield from extract_scope_values(const, scope_vars)
+
+def get_self_value(fn: Callable) -> type | object | None:
+  if isinstance(fn, MethodType):
+    return fn.__self__
+  parts = tuple(fn.__qualname__.split(".")[:-1])
+  cls = parts and AttrDict(fn.__globals__).get_at(parts)
+  if is_class(cls):
+    return cls
+
+def get_fn_captured_vals(fn: Callable) -> list[Any]:
+  self_value = get_self_value(fn)
+  scope_vars = AttrDict({
+    "LOAD_FAST": AttrDict({"self": self_value} if self_value else {}),
+    "LOAD_DEREF": AttrDict(get_cell_contents(fn)),
+    "LOAD_GLOBAL": AttrDict(fn.__globals__),
+  })
+  vals = dict(extract_scope_values(fn.__code__, scope_vars))
+  return list(vals.values())
+
+def is_user_fn(candidate_fn) -> TypeGuard[Callable]:
+  if not isinstance(candidate_fn, (FunctionType, MethodType)):
+    return False
+  fn_path = Path(inspect.getfile(candidate_fn)).resolve()
+  return cwd in fn_path.parents and ".venv" not in fn_path.parts
+
+def get_depend_fns(fn: Callable, capture: bool, captured_vals_by_fn: dict[Callable, list[Any]] = {}) -> dict[Callable, list[Any]]:
+  from .checkpoint import CheckpointFn
+  captured_vals_by_fn = captured_vals_by_fn or {}
+  captured_vals = get_fn_captured_vals(fn)
+  captured_vals_by_fn[fn] = [val for val in captured_vals if not callable(val)] * capture
+  child_fns = (unwrap_fn(val, checkpoint_fn=True) for val in captured_vals if callable(val))
+  for child_fn in child_fns:
+    if isinstance(child_fn, CheckpointFn):
+      captured_vals_by_fn[child_fn] = []
+    elif child_fn not in captured_vals_by_fn and is_user_fn(child_fn):
+      get_depend_fns(child_fn, capture, captured_vals_by_fn)
+  return captured_vals_by_fn
+
+def get_fn_ident(fn: Callable, capture: bool) -> tuple[str, list[Callable]]:
+  from .checkpoint import CheckpointFn
+  captured_vals_by_fn = get_depend_fns(fn, capture)
+  depends, depend_captured_vals = transpose(captured_vals_by_fn.items(), 2)
+  depends = distinct(fn.__func__ if isinstance(fn, MethodType) else fn for fn in depends)
+  unwrapped_depends = [fn for fn in depends if not isinstance(fn, CheckpointFn)]
+  fn_hash = str(ObjectHash(fn, unwrapped_depends).update(depend_captured_vals, tolerate_errors=True))
+  return fn_hash, depends