checkpointer 2.0.0__tar.gz → 2.0.2__tar.gz

{checkpointer-2.0.0 → checkpointer-2.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: checkpointer
-Version: 2.0.0
+Version: 2.0.2
 Summary: A Python library for memoizing function results with support for multiple storage backends, async runtimes, and automatic cache invalidation
 Project-URL: Repository, https://github.com/Reddan/checkpointer.git
 Author: Hampus Hallman
@@ -17,41 +17,16 @@ Description-Content-Type: text/markdown
 
 # checkpointer · [![License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/Reddan/checkpointer/blob/master/LICENSE) [![pypi](https://img.shields.io/pypi/v/checkpointer)](https://pypi.org/project/checkpointer/) [![Python 3.12](https://img.shields.io/badge/python-3.12-blue)](https://pypi.org/project/checkpointer/)
 
-`checkpointer` is a Python library for memoizing function results. It simplifies caching by providing a decorator-based API and supports various storage backends. It's designed for computationally expensive operations where caching can save time, or during development to avoid waiting for redundant computations. 🚀
+`checkpointer` is a Python library for memoizing function results. It provides a decorator-based API with support for multiple storage backends. Use it for computationally expensive operations where caching can save time, or during development to avoid waiting for redundant computations.
 
 Adding or removing `@checkpoint` doesn't change how your code works, and it can be applied to any function, including ones you've already written, without altering their behavior or introducing side effects. The original function remains unchanged and can still be called directly when needed.
 
 ### Key Features:
-- **Multiple Storage Backends**: Supports in-memory, pickle, or your own custom storage.
-- **Simple Decorator API**: Apply `@checkpoint` to functions.
-- **Async and Sync Compatibility**: Works with synchronous functions and any Python async runtime (e.g., `asyncio`, `Trio`, `Curio`).
-- **Custom Expiration Logic**: Automatically invalidate old checkpoints.
-- **Flexible Path Configuration**: Control where checkpoints are stored.
-
-### How It Works
-
-When you use `@checkpoint`, the function's **arguments** (`args`, `kwargs`) are hashed to create a unique identifier for each call. This identifier is used to store and retrieve cached results. If the same arguments are passed again, `checkpointer` will return the cached result instead of recomputing.
-
-Additionally, `checkpointer` ensures that caches are invalidated when a function’s implementation or any of its dependencies change. Each function is assigned a hash based on:
-1. **Its source code**: Changes to the function’s code update its hash.
-2. **Dependent functions**: If a function calls others, changes to those will also update the hash.
-
-### Example: Cache Invalidation by Function Dependencies
-
-```python
-def multiply(a, b):
-    return a * b
-
-@checkpoint
-def helper(x):
-    return multiply(x + 1, 2)
-
-@checkpoint
-def compute(a, b):
-    return helper(a) + helper(b)
-```
-
-If you change `multiply`, the checkpoints for both `helper` and `compute` will be invalidated and recomputed.
+- 🗂️ **Multiple Storage Backends**: Built-in support for in-memory and pickle-based storage, or create your own.
+- 🎯 **Simple Decorator API**: Apply `@checkpoint` to functions without boilerplate.
+- 🔄 **Async and Sync Compatibility**: Works with synchronous functions and any Python async runtime (e.g., `asyncio`, `Trio`, `Curio`).
+- ⏲️ **Custom Expiration Logic**: Automatically invalidate old checkpoints.
+- 📂 **Flexible Path Configuration**: Control where checkpoints are stored.
 
 ---
 
@@ -63,7 +38,7 @@ pip install checkpointer
 
 ---
 
-## Quick Start
+## Quick Start 🚀
 
 ```python
 from checkpointer import checkpoint
@@ -73,95 +48,91 @@ def expensive_function(x: int) -> int:
     print("Computing...")
     return x ** 2
 
-result = expensive_function(4)  # Computes and stores result
-result = expensive_function(4)  # Loads from checkpoint
+result = expensive_function(4)  # Computes and stores the result
+result = expensive_function(4)  # Loads from the cache
 ```
 
 ---
 
-## Parameterization
+## How It Works
+
+When you use `@checkpoint`, the function's **arguments** (`args`, `kwargs`) are hashed to create a unique identifier for each call. This identifier is used to store and retrieve cached results. If the same arguments are passed again, `checkpointer` loads the cached result instead of recomputing.
 
-### Global Configuration
+Additionally, `checkpointer` ensures that caches are invalidated when a function's implementation or any of its dependencies change. Each function is assigned a hash based on:
+1. **Its source code**: Changes to the function's code update its hash.
+2. **Dependent functions**: If a function calls others, changes in those dependencies will also update the hash.
 
-You can configure a custom `Checkpointer`:
+### Example: Cache Invalidation
 
 ```python
-from checkpointer import checkpoint
+def multiply(a, b):
+    return a * b
 
-checkpoint = checkpoint(format="memory", root_path="/tmp/checkpoints")
+@checkpoint
+def helper(x):
+    return multiply(x + 1, 2)
+
+@checkpoint
+def compute(a, b):
+    return helper(a) + helper(b)
 ```
 
-Extend this configuration by calling itself again:
+If you modify `multiply`, caches for both `helper` and `compute` are invalidated and recomputed.
 
-```python
-extended_checkpoint = checkpoint(format="pickle", verbosity=0)
-```
+---
 
-### Per-Function Customization
+## Parameterization
 
-```python
-@checkpoint(format="pickle", verbosity=0)
-def my_function(x, y):
-    return x + y
-```
+### Custom Configuration
 
-### Combining Configurations
+Set up a `Checkpointer` instance with custom settings, and extend it by calling itself with overrides:
 
 ```python
-checkpoint = checkpoint(format="memory", verbosity=1)
-quiet_checkpoint = checkpoint(verbosity=0)
-pickle_checkpoint = checkpoint(format="pickle", root_path="/tmp/pickle_checkpoints")
-
-@checkpoint
-def compute_square(n: int) -> int:
-    return n ** 2
+from checkpointer import checkpoint
 
-@quiet_checkpoint
-def compute_quietly(n: int) -> int:
-    return n ** 3
+IS_DEVELOPMENT = True  # Toggle based on your environment
 
-@pickle_checkpoint
-def compute_sum(a: int, b: int) -> int:
-    return a + b
+tmp_checkpoint = checkpoint(root_path="/tmp/checkpoints")
+dev_checkpoint = tmp_checkpoint(when=IS_DEVELOPMENT)  # Adds development-specific behavior
 ```
 
-### Layered Caching
+### Per-Function Customization & Layered Caching
 
-```python
-IS_DEVELOPMENT = True  # Toggle based on environment
+Layer caches by stacking checkpoints:
 
-dev_checkpoint = checkpoint(when=IS_DEVELOPMENT)
-
-@checkpoint(format="memory")
-@dev_checkpoint
+```python
+@checkpoint(format="memory")  # Always use memory storage
+@dev_checkpoint  # Adds caching during development
 def some_expensive_function():
     print("Performing a time-consuming operation...")
     return sum(i * i for i in range(10**6))
 ```
 
-- In development: Both `dev_checkpoint` and `memory` caches are active.
-- In production: Only the `memory` cache is active.
+- **In development**: Both `dev_checkpoint` and `memory` caches are active.
+- **In production**: Only the `memory` cache is active.
 
 ---
 
 ## Usage
 
 ### Force Recalculation
-Use `rerun` to force a recalculation and overwrite the stored checkpoint:
+Force a recalculation and overwrite the stored checkpoint:
 
 ```python
 result = expensive_function.rerun(4)
 ```
 
-### Bypass Checkpointer
+### Call the Original Function
 Use `fn` to directly call the original, undecorated function:
 
 ```python
 result = expensive_function.fn(4)
 ```
 
+This is especially useful **inside recursive functions** to avoid redundant caching of intermediate steps while still caching the final result.
+
 ### Retrieve Stored Checkpoints
-Access stored results without recalculating:
+Access cached results without recalculating:
 
 ```python
 stored_result = expensive_function.get(4)
@@ -169,7 +140,56 @@ stored_result = expensive_function.get(4)
 
 ---
 
-## Configuration Options
+## Storage Backends
+
+`checkpointer` works with both built-in and custom storage backends, so you can use what's provided or roll your own as needed.
+
+### Built-In Backends
+
+1. **PickleStorage**: Stores checkpoints on disk using Python's `pickle`.
+2. **MemoryStorage**: Keeps checkpoints in memory for non-persistent, fast caching.
+
+You can specify a storage backend using either its name (`"pickle"` or `"memory"`) or its corresponding class (`PickleStorage` or `MemoryStorage`) in the `format` parameter:
+
+```python
+from checkpointer import checkpoint, PickleStorage, MemoryStorage
+
+@checkpoint(format="pickle")  # Equivalent to format=PickleStorage
+def disk_cached(x: int) -> int:
+    return x ** 2
+
+@checkpoint(format="memory")  # Equivalent to format=MemoryStorage
+def memory_cached(x: int) -> int:
+    return x * 10
+```
+
+### Custom Storage Backends
+
+Create a custom storage backend by inheriting from the `Storage` class and implementing its methods. Access configuration options through the `self.checkpointer` attribute, an instance of `Checkpointer`.
+
+#### Example: Custom Storage Backend
+
+```python
+from checkpointer import checkpoint, Storage
+from datetime import datetime
+
+class CustomStorage(Storage):
+    def exists(self, path) -> bool: ...  # Check if a checkpoint exists at the given path
+    def checkpoint_date(self, path) -> datetime: ...  # Return the date the checkpoint was created
+    def store(self, path, data): ...  # Save the checkpoint data
+    def load(self, path): ...  # Return the checkpoint data
+    def delete(self, path): ...  # Delete the checkpoint
+
+@checkpoint(format=CustomStorage)
+def custom_cached(x: int):
+    return x ** 2
+```
+
+Using a custom backend lets you tailor storage to your application, whether it involves databases, cloud storage, or custom file formats.
+
+---
+
+## Configuration Options ⚙️
 
 | Option         | Type                                | Default     | Description                                 |
 |----------------|-------------------------------------|-------------|---------------------------------------------|
@@ -177,12 +197,12 @@ stored_result = expensive_function.get(4)
 | `root_path`    | `Path`, `str`, or `None`            | User Cache  | Root directory for storing checkpoints.     |
 | `when`         | `bool`                              | `True`      | Enable or disable checkpointing.            |
 | `verbosity`    | `0` or `1`                          | `1`         | Logging verbosity.                          |
-| `path`         | `str` or `Callable[..., str]`       | `None`      | Custom path for checkpoint storage.         |
+| `path`         | `Callable[..., str]`                | `None`      | Custom path for checkpoint storage.         |
 | `should_expire`| `Callable[[datetime], bool]`        | `None`      | Custom expiration logic.                    |
 
 ---
 
-## Full Example
+## Full Example 🛠️
 
 ```python
 import asyncio

checkpointer-2.0.2/README.md

@@ -0,0 +1,215 @@
+# checkpointer · [![License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/Reddan/checkpointer/blob/master/LICENSE) [![pypi](https://img.shields.io/pypi/v/checkpointer)](https://pypi.org/project/checkpointer/) [![Python 3.12](https://img.shields.io/badge/python-3.12-blue)](https://pypi.org/project/checkpointer/)
+
+`checkpointer` is a Python library for memoizing function results. It provides a decorator-based API with support for multiple storage backends. Use it for computationally expensive operations where caching can save time, or during development to avoid waiting for redundant computations.
+
+Adding or removing `@checkpoint` doesn't change how your code works, and it can be applied to any function, including ones you've already written, without altering their behavior or introducing side effects. The original function remains unchanged and can still be called directly when needed.
+
+### Key Features:
+- 🗂️ **Multiple Storage Backends**: Built-in support for in-memory and pickle-based storage, or create your own.
+- 🎯 **Simple Decorator API**: Apply `@checkpoint` to functions without boilerplate.
+- 🔄 **Async and Sync Compatibility**: Works with synchronous functions and any Python async runtime (e.g., `asyncio`, `Trio`, `Curio`).
+- ⏲️ **Custom Expiration Logic**: Automatically invalidate old checkpoints.
+- 📂 **Flexible Path Configuration**: Control where checkpoints are stored.
+
+---
+
+## Installation
+
+```bash
+pip install checkpointer
+```
+
+---
+
+## Quick Start 🚀
+
+```python
+from checkpointer import checkpoint
+
+@checkpoint
+def expensive_function(x: int) -> int:
+    print("Computing...")
+    return x ** 2
+
+result = expensive_function(4)  # Computes and stores the result
+result = expensive_function(4)  # Loads from the cache
+```
+
+---
+
+## How It Works
+
+When you use `@checkpoint`, the function's **arguments** (`args`, `kwargs`) are hashed to create a unique identifier for each call. This identifier is used to store and retrieve cached results. If the same arguments are passed again, `checkpointer` loads the cached result instead of recomputing.
+
+Additionally, `checkpointer` ensures that caches are invalidated when a function's implementation or any of its dependencies change. Each function is assigned a hash based on:
+1. **Its source code**: Changes to the function's code update its hash.
+2. **Dependent functions**: If a function calls others, changes in those dependencies will also update the hash.
+
+### Example: Cache Invalidation
+
+```python
+def multiply(a, b):
+    return a * b
+
+@checkpoint
+def helper(x):
+    return multiply(x + 1, 2)
+
+@checkpoint
+def compute(a, b):
+    return helper(a) + helper(b)
+```
+
+If you modify `multiply`, caches for both `helper` and `compute` are invalidated and recomputed.
+
+---
+
+## Parameterization
+
+### Custom Configuration
+
+Set up a `Checkpointer` instance with custom settings, and extend it by calling itself with overrides:
+
+```python
+from checkpointer import checkpoint
+
+IS_DEVELOPMENT = True  # Toggle based on your environment
+
+tmp_checkpoint = checkpoint(root_path="/tmp/checkpoints")
+dev_checkpoint = tmp_checkpoint(when=IS_DEVELOPMENT)  # Adds development-specific behavior
+```
+
+### Per-Function Customization & Layered Caching
+
+Layer caches by stacking checkpoints:
+
+```python
+@checkpoint(format="memory")  # Always use memory storage
+@dev_checkpoint  # Adds caching during development
+def some_expensive_function():
+    print("Performing a time-consuming operation...")
+    return sum(i * i for i in range(10**6))
+```
+
+- **In development**: Both `dev_checkpoint` and `memory` caches are active.
+- **In production**: Only the `memory` cache is active.
+
+---
+
+## Usage
+
+### Force Recalculation
+Force a recalculation and overwrite the stored checkpoint:
+
+```python
+result = expensive_function.rerun(4)
+```
+
+### Call the Original Function
+Use `fn` to directly call the original, undecorated function:
+
+```python
+result = expensive_function.fn(4)
+```
+
+This is especially useful **inside recursive functions** to avoid redundant caching of intermediate steps while still caching the final result.
+
+### Retrieve Stored Checkpoints
+Access cached results without recalculating:
+
+```python
+stored_result = expensive_function.get(4)
+```
+
+---
+
+## Storage Backends
+
+`checkpointer` works with both built-in and custom storage backends, so you can use what's provided or roll your own as needed.
+
+### Built-In Backends
+
+1. **PickleStorage**: Stores checkpoints on disk using Python's `pickle`.
+2. **MemoryStorage**: Keeps checkpoints in memory for non-persistent, fast caching.
+
+You can specify a storage backend using either its name (`"pickle"` or `"memory"`) or its corresponding class (`PickleStorage` or `MemoryStorage`) in the `format` parameter:
+
+```python
+from checkpointer import checkpoint, PickleStorage, MemoryStorage
+
+@checkpoint(format="pickle")  # Equivalent to format=PickleStorage
+def disk_cached(x: int) -> int:
+    return x ** 2
+
+@checkpoint(format="memory")  # Equivalent to format=MemoryStorage
+def memory_cached(x: int) -> int:
+    return x * 10
+```
+
+### Custom Storage Backends
+
+Create a custom storage backend by inheriting from the `Storage` class and implementing its methods. Access configuration options through the `self.checkpointer` attribute, an instance of `Checkpointer`.
+
+#### Example: Custom Storage Backend
+
+```python
+from checkpointer import checkpoint, Storage
+from datetime import datetime
+
+class CustomStorage(Storage):
+    def exists(self, path) -> bool: ...  # Check if a checkpoint exists at the given path
+    def checkpoint_date(self, path) -> datetime: ...  # Return the date the checkpoint was created
+    def store(self, path, data): ...  # Save the checkpoint data
+    def load(self, path): ...  # Return the checkpoint data
+    def delete(self, path): ...  # Delete the checkpoint
+
+@checkpoint(format=CustomStorage)
+def custom_cached(x: int):
+    return x ** 2
+```
+
+Using a custom backend lets you tailor storage to your application, whether it involves databases, cloud storage, or custom file formats.
+
+---
+
+## Configuration Options ⚙️
+
+| Option         | Type                                | Default     | Description                                 |
+|----------------|-------------------------------------|-------------|---------------------------------------------|
+| `format`       | `"pickle"`, `"memory"`, `Storage`   | `"pickle"`  | Storage backend format.                     |
+| `root_path`    | `Path`, `str`, or `None`            | User Cache  | Root directory for storing checkpoints.     |
+| `when`         | `bool`                              | `True`      | Enable or disable checkpointing.            |
+| `verbosity`    | `0` or `1`                          | `1`         | Logging verbosity.                          |
+| `path`         | `Callable[..., str]`                | `None`      | Custom path for checkpoint storage.         |
+| `should_expire`| `Callable[[datetime], bool]`        | `None`      | Custom expiration logic.                    |
+
+---
+
+## Full Example 🛠️
+
+```python
+import asyncio
+from checkpointer import checkpoint
+
+@checkpoint
+def compute_square(n: int) -> int:
+    print(f"Computing {n}^2...")
+    return n ** 2
+
+@checkpoint(format="memory")
+async def async_compute_sum(a: int, b: int) -> int:
+    await asyncio.sleep(1)
+    return a + b
+
+async def main():
+    result1 = compute_square(5)
+    print(result1)
+
+    result2 = await async_compute_sum(3, 7)
+    print(result2)
+
+    result3 = async_compute_sum.get(3, 7)
+    print(result3)
+
+asyncio.run(main())
+```

checkpointer-2.0.2/checkpointer/__init__.py

@@ -0,0 +1,9 @@
+from .checkpoint import Checkpointer, CheckpointFn, CheckpointError
+from .types import Storage
+from .function_body import get_function_hash
+import tempfile
+
+create_checkpointer = Checkpointer
+checkpoint = Checkpointer()
+memory_checkpoint = Checkpointer(format="memory", verbosity=0)
+tmp_checkpoint = Checkpointer(root_path=tempfile.gettempdir() + "/checkpoints")

{checkpointer-2.0.0 → checkpointer-2.0.2}/checkpointer/checkpoint.py

@@ -1,9 +1,9 @@
+from __future__ import annotations
 import inspect
 import relib.hashing as hashing
-from typing import Generic, TypeVar, TypedDict, Unpack, Literal, Union, Any, cast, overload
-from collections.abc import Callable
-from datetime import datetime
+from typing import Generic, TypeVar, Type, TypedDict, Callable, Unpack, Literal, Any, cast, overload
 from pathlib import Path
+from datetime import datetime
 from functools import update_wrapper
 from .types import Storage
 from .function_body import get_function_hash
@@ -16,25 +16,18 @@ from .print_checkpoint import print_checkpoint
 Fn = TypeVar("Fn", bound=Callable)
 
 DEFAULT_DIR = Path.home() / ".cache/checkpoints"
-STORAGE_MAP = {"memory": MemoryStorage, "pickle": PickleStorage, "bcolz": BcolzStorage}
+STORAGE_MAP: dict[str, Type[Storage]] = {"memory": MemoryStorage, "pickle": PickleStorage, "bcolz": BcolzStorage}
 
 class CheckpointError(Exception):
   pass
 
-class CheckpointReadFail(CheckpointError):
-  pass
-
-StorageType = Literal["pickle", "memory", "bcolz"] | Storage
-CheckpointPath = str | Callable[..., str] | None
-ShouldExpire = Callable[[datetime], bool]
-
 class CheckpointerOpts(TypedDict, total=False):
-  format: StorageType
+  format: Type[Storage] | Literal["pickle", "memory", "bcolz"]
   root_path: Path | str | None
   when: bool
   verbosity: Literal[0, 1]
-  path: CheckpointPath
-  should_expire: ShouldExpire
+  path: Callable[..., str] | None
+  should_expire: Callable[[datetime], bool] | None
 
 class Checkpointer:
   def __init__(self, **opts: Unpack[CheckpointerOpts]):
@@ -45,14 +38,11 @@ class Checkpointer:
     self.path = opts.get("path")
     self.should_expire = opts.get("should_expire")
 
-  def get_storage(self) -> Storage:
-    return STORAGE_MAP[self.format] if isinstance(self.format, str) else self.format
-
   @overload
-  def __call__(self, fn: Fn, **override_opts: Unpack[CheckpointerOpts]) -> "CheckpointFn[Fn]": ...
+  def __call__(self, fn: Fn, **override_opts: Unpack[CheckpointerOpts]) -> CheckpointFn[Fn]: ...
   @overload
-  def __call__(self, fn=None, **override_opts: Unpack[CheckpointerOpts]) -> "Checkpointer": ...
-  def __call__(self, fn: Fn | None=None, **override_opts: Unpack[CheckpointerOpts]) -> Union["Checkpointer", "CheckpointFn[Fn]"]:
+  def __call__(self, fn: None=None, **override_opts: Unpack[CheckpointerOpts]) -> Checkpointer: ...
+  def __call__(self, fn: Fn | None=None, **override_opts: Unpack[CheckpointerOpts]) -> Checkpointer | CheckpointFn[Fn]:
     if override_opts:
       opts = CheckpointerOpts(**{**self.__dict__, **override_opts})
       return Checkpointer(**opts)(fn)
@@ -64,52 +54,51 @@ class CheckpointFn(Generic[Fn]):
     wrapped = unwrap_fn(fn)
     file_name = Path(wrapped.__code__.co_filename).name
     update_wrapper(cast(Callable, self), wrapped)
+    storage = STORAGE_MAP[checkpointer.format] if isinstance(checkpointer.format, str) else checkpointer.format
     self.checkpointer = checkpointer
     self.fn = fn
     self.fn_hash = get_function_hash(wrapped)
     self.fn_id = f"{file_name}/{wrapped.__name__}"
-    self.is_async = inspect.iscoroutinefunction(fn)
+    self.is_async = inspect.iscoroutinefunction(wrapped)
+    self.storage = storage(checkpointer)
 
   def get_checkpoint_id(self, args: tuple, kw: dict) -> str:
-    match self.checkpointer.path:
-      case str() as path:
-        return path
-      case Callable() as path:
-        p = path(*args, **kw)
-        assert isinstance(p, str), "path function must return a string"
-        return p
-      case _:
-        return f"{self.fn_id}/{hashing.hash([self.fn_hash, args, kw or 0])}"
-
-  async def _store_on_demand(self, args: tuple, kw: dict, force: bool):
+    if not callable(self.checkpointer.path):
+      return f"{self.fn_id}/{hashing.hash([self.fn_hash, args, kw or 0])}"
+    checkpoint_id = self.checkpointer.path(*args, **kw)
+    if not isinstance(checkpoint_id, str):
+      raise CheckpointError(f"path function must return a string, got {type(checkpoint_id)}")
+    return checkpoint_id
+
+  async def _store_on_demand(self, args: tuple, kw: dict, rerun: bool):
     checkpoint_id = self.get_checkpoint_id(args, kw)
     checkpoint_path = self.checkpointer.root_path / checkpoint_id
-    storage = self.checkpointer.get_storage()
-    should_log = storage is not MemoryStorage and self.checkpointer.verbosity > 0
-    refresh = force \
-      or storage.is_expired(checkpoint_path) \
-      or (self.checkpointer.should_expire and storage.should_expire(checkpoint_path, self.checkpointer.should_expire))
+    should_log = self.checkpointer.verbosity > 0
+    refresh = rerun \
+      or not self.storage.exists(checkpoint_path) \
+      or (self.checkpointer.should_expire and self.checkpointer.should_expire(self.storage.checkpoint_date(checkpoint_path)))
 
     if refresh:
       print_checkpoint(should_log, "MEMORIZING", checkpoint_id, "blue")
       data = self.fn(*args, **kw)
       if inspect.iscoroutine(data):
         data = await data
-      return storage.store_data(checkpoint_path, data)
+      self.storage.store(checkpoint_path, data)
+      return data
 
     try:
-      data = storage.load_data(checkpoint_path)
+      data = self.storage.load(checkpoint_path)
       print_checkpoint(should_log, "REMEMBERED", checkpoint_id, "green")
       return data
     except (EOFError, FileNotFoundError):
       print_checkpoint(should_log, "CORRUPTED", checkpoint_id, "yellow")
-      storage.delete_data(checkpoint_path)
-      return await self._store_on_demand(args, kw, force)
+      self.storage.delete(checkpoint_path)
+      return await self._store_on_demand(args, kw, rerun)
 
-  def _call(self, args: tuple, kw: dict, force=False):
+  def _call(self, args: tuple, kw: dict, rerun=False):
     if not self.checkpointer.when:
       return self.fn(*args, **kw)
-    coroutine = self._store_on_demand(args, kw, force)
+    coroutine = self._store_on_demand(args, kw, rerun)
     return coroutine if self.is_async else sync_resolve_coroutine(coroutine)
 
   __call__: Fn = cast(Fn, lambda self, *args, **kw: self._call(args, kw))
@@ -118,6 +107,6 @@ class CheckpointFn(Generic[Fn]):
   def get(self, *args, **kw) -> Any:
     checkpoint_path = self.checkpointer.root_path / self.get_checkpoint_id(args, kw)
     try:
-      return self.checkpointer.get_storage().load_data(checkpoint_path)
+      return self.storage.load(checkpoint_path)
     except:
-      raise CheckpointReadFail()
+      raise CheckpointError("Could not load checkpoint")

{checkpointer-2.0.0 → checkpointer-2.0.2}/checkpointer/print_checkpoint.py

@@ -44,7 +44,7 @@ def colored_(text: str, color: Color | None = None, on_color: Color | None = Non
     text = f"\033[{COLOR_MAP[on_color] + 10}m{text}"
   return text + "\033[0m"
 
-noop = lambda *args, **_: args[0]
+noop = lambda text, *a, **k: text
 colored = colored_ if allow_color() else noop
 
 def print_checkpoint(should_log: bool, title: str, text: str, color: Color):

{checkpointer-2.0.0 → checkpointer-2.0.2}/checkpointer/storages/bcolz_storage.py

@@ -18,35 +18,21 @@ def get_data_type_str(x):
 def get_metapath(path: Path):
   return path.with_name(f"{path.name}_meta")
 
-def get_collection_timestamp(path: Path):
-  import bcolz
-  metapath = get_metapath(path)
-  meta_data = bcolz.open(metapath)[:][0]
-  return meta_data["created"]
-
 def insert_data(path: Path, data):
   import bcolz
   c = bcolz.carray(data, rootdir=path, mode="w")
   c.flush()
 
 class BcolzStorage(Storage):
-  @staticmethod
-  def is_expired(path):
-    try:
-      get_collection_timestamp(path)
-      return False
-    except (FileNotFoundError, EOFError):
-      return True
+  def exists(self, path):
+    return path.exists()
 
-  @staticmethod
-  def should_expire(path, expire_fn):
-    return expire_fn(get_collection_timestamp(path))
+  def checkpoint_date(self, path):
+    return datetime.fromtimestamp(path.stat().st_mtime)
 
-  @staticmethod
-  def store_data(path, data):
+  def store(self, path, data):
     metapath = get_metapath(path)
     path.parent.mkdir(parents=True, exist_ok=True)
-    created = datetime.now()
     data_type_str = get_data_type_str(data)
     if data_type_str == "tuple":
       fields = list(range(len(data)))
@@ -54,18 +40,16 @@ class BcolzStorage(Storage):
       fields = sorted(data.keys())
     else:
       fields = []
-    meta_data = {"created": created, "data_type_str": data_type_str, "fields": fields}
+    meta_data = {"data_type_str": data_type_str, "fields": fields}
     insert_data(metapath, meta_data)
     if data_type_str in ["tuple", "dict"]:
       for i in range(len(fields)):
         child_path = Path(f"{path} ({i})")
-        BcolzStorage.store_data(child_path, data[fields[i]])
+        self.store(child_path, data[fields[i]])
     else:
       insert_data(path, data)
-    return data
 
-  @staticmethod
-  def load_data(path):
+  def load(self, path):
     import bcolz
     metapath = get_metapath(path)
     meta_data = bcolz.open(metapath)[:][0]
@@ -73,7 +57,7 @@ class BcolzStorage(Storage):
     if data_type_str in ["tuple", "dict"]:
       fields = meta_data["fields"]
       partitions = range(len(fields))
-      data = [BcolzStorage.load_data(Path(f"{path} ({i})")) for i in partitions]
+      data = [self.load(Path(f"{path} ({i})")) for i in partitions]
       if data_type_str == "tuple":
         return tuple(data)
       else:
@@ -87,8 +71,7 @@ class BcolzStorage(Storage):
       else:
         return data[:]
 
-  @staticmethod
-  def delete_data(path):
+  def delete(self, path):
     # NOTE: Not recursive
     metapath = get_metapath(path)
     try:

checkpointer-2.0.2/checkpointer/storages/memory_storage.py

@@ -0,0 +1,25 @@
+from typing import Any
+from pathlib import Path
+from datetime import datetime
+from ..types import Storage
+
+item_map: dict[str, tuple[datetime, Any]] = {}
+
+class MemoryStorage(Storage):
+  def get_short_path(self, path: Path):
+    return str(path.relative_to(self.checkpointer.root_path))
+
+  def exists(self, path):
+    return self.get_short_path(path) in item_map
+
+  def checkpoint_date(self, path):
+    return item_map[self.get_short_path(path)][0]
+
+  def store(self, path, data):
+    item_map[self.get_short_path(path)] = (datetime.now(), data)
+
+  def load(self, path):
+    return item_map[self.get_short_path(path)][1]
+
+  def delete(self, path):
+    del item_map[self.get_short_path(path)]

checkpointer-2.0.2/checkpointer/storages/pickle_storage.py

@@ -0,0 +1,31 @@
+import pickle
+from pathlib import Path
+from datetime import datetime
+from ..types import Storage
+
+def get_path(path: Path):
+  return path.with_name(f"{path.name}.pkl")
+
+class PickleStorage(Storage):
+  def exists(self, path):
+    return get_path(path).exists()
+
+  def checkpoint_date(self, path):
+    return datetime.fromtimestamp(get_path(path).stat().st_mtime)
+
+  def store(self, path, data):
+    full_path = get_path(path)
+    full_path.parent.mkdir(parents=True, exist_ok=True)
+    with full_path.open("wb") as file:
+      pickle.dump(data, file, -1)
+
+  def load(self, path):
+    full_path = get_path(path)
+    with full_path.open("rb") as file:
+      return pickle.load(file)
+
+  def delete(self, path):
+    try:
+      get_path(path).unlink()
+    except FileNotFoundError:
+      pass

checkpointer-2.0.2/checkpointer/types.py

@@ -0,0 +1,23 @@
+from __future__ import annotations
+from typing import Any, TYPE_CHECKING
+from pathlib import Path
+from datetime import datetime
+
+if TYPE_CHECKING:
+  from .checkpoint import Checkpointer
+
+class Storage:
+  checkpointer: Checkpointer
+
+  def __init__(self, checkpointer: Checkpointer):
+    self.checkpointer = checkpointer
+
+  def exists(self, path: Path) -> bool: ...
+
+  def checkpoint_date(self, path: Path) -> datetime: ...
+
+  def store(self, path: Path, data: Any) -> None: ...
+
+  def load(self, path: Path) -> Any: ...
+
+  def delete(self, path: Path) -> None: ...

{checkpointer-2.0.0 → checkpointer-2.0.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "checkpointer"
-version = "2.0.0"
+version = "2.0.2"
 requires-python = ">=3.12"
 dependencies = [
   "relib",

{checkpointer-2.0.0 → checkpointer-2.0.2}/uv.lock

@@ -3,7 +3,7 @@ requires-python = ">=3.12"
 
 [[package]]
 name = "checkpointer"
-version = "2.0.0"
+version = "2.0.2"
 source = { editable = "." }
 dependencies = [
     { name = "relib" },

checkpointer-2.0.0/README.md

@@ -1,195 +0,0 @@
-# checkpointer · [![License](https://img.shields.io/badge/license-MIT-blue)](https://github.com/Reddan/checkpointer/blob/master/LICENSE) [![pypi](https://img.shields.io/pypi/v/checkpointer)](https://pypi.org/project/checkpointer/) [![Python 3.12](https://img.shields.io/badge/python-3.12-blue)](https://pypi.org/project/checkpointer/)
-
-`checkpointer` is a Python library for memoizing function results. It simplifies caching by providing a decorator-based API and supports various storage backends. It's designed for computationally expensive operations where caching can save time, or during development to avoid waiting for redundant computations. 🚀
-
-Adding or removing `@checkpoint` doesn't change how your code works, and it can be applied to any function, including ones you've already written, without altering their behavior or introducing side effects. The original function remains unchanged and can still be called directly when needed.
-
-### Key Features:
-- **Multiple Storage Backends**: Supports in-memory, pickle, or your own custom storage.
-- **Simple Decorator API**: Apply `@checkpoint` to functions.
-- **Async and Sync Compatibility**: Works with synchronous functions and any Python async runtime (e.g., `asyncio`, `Trio`, `Curio`).
-- **Custom Expiration Logic**: Automatically invalidate old checkpoints.
-- **Flexible Path Configuration**: Control where checkpoints are stored.
-
-### How It Works
-
-When you use `@checkpoint`, the function's **arguments** (`args`, `kwargs`) are hashed to create a unique identifier for each call. This identifier is used to store and retrieve cached results. If the same arguments are passed again, `checkpointer` will return the cached result instead of recomputing.
-
-Additionally, `checkpointer` ensures that caches are invalidated when a function’s implementation or any of its dependencies change. Each function is assigned a hash based on:
-1. **Its source code**: Changes to the function’s code update its hash.
-2. **Dependent functions**: If a function calls others, changes to those will also update the hash.
-
-### Example: Cache Invalidation by Function Dependencies
-
-```python
-def multiply(a, b):
-    return a * b
-
-@checkpoint
-def helper(x):
-    return multiply(x + 1, 2)
-
-@checkpoint
-def compute(a, b):
-    return helper(a) + helper(b)
-```
-
-If you change `multiply`, the checkpoints for both `helper` and `compute` will be invalidated and recomputed.
-
----
-
-## Installation
-
-```bash
-pip install checkpointer
-```
-
----
-
-## Quick Start
-
-```python
-from checkpointer import checkpoint
-
-@checkpoint
-def expensive_function(x: int) -> int:
-    print("Computing...")
-    return x ** 2
-
-result = expensive_function(4)  # Computes and stores result
-result = expensive_function(4)  # Loads from checkpoint
-```
-
----
-
-## Parameterization
-
-### Global Configuration
-
-You can configure a custom `Checkpointer`:
-
-```python
-from checkpointer import checkpoint
-
-checkpoint = checkpoint(format="memory", root_path="/tmp/checkpoints")
-```
-
-Extend this configuration by calling itself again:
-
-```python
-extended_checkpoint = checkpoint(format="pickle", verbosity=0)
-```
-
-### Per-Function Customization
-
-```python
-@checkpoint(format="pickle", verbosity=0)
-def my_function(x, y):
-    return x + y
-```
-
-### Combining Configurations
-
-```python
-checkpoint = checkpoint(format="memory", verbosity=1)
-quiet_checkpoint = checkpoint(verbosity=0)
-pickle_checkpoint = checkpoint(format="pickle", root_path="/tmp/pickle_checkpoints")
-
-@checkpoint
-def compute_square(n: int) -> int:
-    return n ** 2
-
-@quiet_checkpoint
-def compute_quietly(n: int) -> int:
-    return n ** 3
-
-@pickle_checkpoint
-def compute_sum(a: int, b: int) -> int:
-    return a + b
-```
-
-### Layered Caching
-
-```python
-IS_DEVELOPMENT = True  # Toggle based on environment
-
-dev_checkpoint = checkpoint(when=IS_DEVELOPMENT)
-
-@checkpoint(format="memory")
-@dev_checkpoint
-def some_expensive_function():
-    print("Performing a time-consuming operation...")
-    return sum(i * i for i in range(10**6))
-```
-
-- In development: Both `dev_checkpoint` and `memory` caches are active.
-- In production: Only the `memory` cache is active.
-
----
-
-## Usage
-
-### Force Recalculation
-Use `rerun` to force a recalculation and overwrite the stored checkpoint:
-
-```python
-result = expensive_function.rerun(4)
-```
-
-### Bypass Checkpointer
-Use `fn` to directly call the original, undecorated function:
-
-```python
-result = expensive_function.fn(4)
-```
-
-### Retrieve Stored Checkpoints
-Access stored results without recalculating:
-
-```python
-stored_result = expensive_function.get(4)
-```
-
----
-
-## Configuration Options
-
-| Option         | Type                                | Default     | Description                                 |
-|----------------|-------------------------------------|-------------|---------------------------------------------|
-| `format`       | `"pickle"`, `"memory"`, `Storage`   | `"pickle"`  | Storage backend format.                     |
-| `root_path`    | `Path`, `str`, or `None`            | User Cache  | Root directory for storing checkpoints.     |
-| `when`         | `bool`                              | `True`      | Enable or disable checkpointing.            |
-| `verbosity`    | `0` or `1`                          | `1`         | Logging verbosity.                          |
-| `path`         | `str` or `Callable[..., str]`       | `None`      | Custom path for checkpoint storage.         |
-| `should_expire`| `Callable[[datetime], bool]`        | `None`      | Custom expiration logic.                    |
-
----
-
-## Full Example
-
-```python
-import asyncio
-from checkpointer import checkpoint
-
-@checkpoint
-def compute_square(n: int) -> int:
-    print(f"Computing {n}^2...")
-    return n ** 2
-
-@checkpoint(format="memory")
-async def async_compute_sum(a: int, b: int) -> int:
-    await asyncio.sleep(1)
-    return a + b
-
-async def main():
-    result1 = compute_square(5)
-    print(result1)
-
-    result2 = await async_compute_sum(3, 7)
-    print(result2)
-
-    result3 = async_compute_sum.get(3, 7)
-    print(result3)
-
-asyncio.run(main())
-```

checkpointer-2.0.0/checkpointer/__init__.py

@@ -1,9 +0,0 @@
-from .checkpoint import Checkpointer, CheckpointFn
-from .checkpoint import CheckpointError, CheckpointReadFail
-from .types import Storage
-from .function_body import get_function_hash
-
-create_checkpointer = Checkpointer
-checkpoint = Checkpointer()
-memory_checkpoint = Checkpointer(format="memory")
-tmp_checkpoint = Checkpointer(root_path="/tmp/checkpoints")

checkpointer-2.0.0/checkpointer/storages/memory_storage.py

@@ -1,29 +0,0 @@
-from datetime import datetime
-from ..types import Storage
-
-store = {}
-date_stored = {}
-
-class MemoryStorage(Storage):
-  @staticmethod
-  def is_expired(path):
-    return path not in store
-
-  @staticmethod
-  def should_expire(path, expire_fn):
-    return expire_fn(date_stored[path])
-
-  @staticmethod
-  def store_data(path, data):
-    store[path] = data
-    date_stored[path] = datetime.now()
-    return data
-
-  @staticmethod
-  def load_data(path):
-    return store[path]
-
-  @staticmethod
-  def delete_data(path):
-    del store[path]
-    del date_stored[path]

checkpointer-2.0.0/checkpointer/storages/pickle_storage.py

@@ -1,55 +0,0 @@
-import pickle
-from pathlib import Path
-from datetime import datetime
-from ..types import Storage
-
-def get_paths(path: Path):
-  meta_full_path = path.with_name(f"{path.name}_meta.pkl")
-  pkl_full_path = path.with_name(f"{path.name}.pkl")
-  return meta_full_path, pkl_full_path
-
-def get_collection_timestamp(path: Path):
-  meta_full_path, _ = get_paths(path)
-  with meta_full_path.open("rb") as file:
-    meta_data = pickle.load(file)
-    return meta_data["created"]
-
-class PickleStorage(Storage):
-  @staticmethod
-  def is_expired(path):
-    try:
-      get_collection_timestamp(path)
-      return False
-    except (FileNotFoundError, EOFError):
-      return True
-
-  @staticmethod
-  def should_expire(path, expire_fn):
-    return expire_fn(get_collection_timestamp(path))
-
-  @staticmethod
-  def store_data(path, data):
-    created = datetime.now()
-    meta_data = {"created": created} # TODO: this should just be a JSON or binary dump of the unix timestamp and other metadata - not pickle
-    meta_full_path, pkl_full_path = get_paths(path)
-    pkl_full_path.parent.mkdir(parents=True, exist_ok=True)
-    with pkl_full_path.open("wb") as file:
-      pickle.dump(data, file, -1)
-    with meta_full_path.open("wb") as file:
-      pickle.dump(meta_data, file, -1)
-    return data
-
-  @staticmethod
-  def load_data(path):
-    _, full_path = get_paths(path)
-    with full_path.open("rb") as file:
-      return pickle.load(file)
-
-  @staticmethod
-  def delete_data(path):
-    meta_full_path, pkl_full_path = get_paths(path)
-    try:
-      meta_full_path.unlink()
-      pkl_full_path.unlink()
-    except FileNotFoundError:
-      pass

checkpointer-2.0.0/checkpointer/types.py

@@ -1,19 +0,0 @@
-from typing import Callable, Protocol, Any
-from pathlib import Path
-from datetime import datetime
-
-class Storage(Protocol):
-  @staticmethod
-  def is_expired(path: Path) -> bool: ...
-
-  @staticmethod
-  def should_expire(path: Path, expire_fn: Callable[[datetime], bool]) -> bool: ...
-
-  @staticmethod
-  def store_data(path: Path, data: Any) -> Any: ...
-
-  @staticmethod
-  def load_data(path: Path) -> Any: ...
-
-  @staticmethod
-  def delete_data(path: Path) -> None: ...