memodisk 0.0.6.dev0__tar.gz → 0.1.1.dev0__tar.gz

{memodisk-0.0.6.dev0 → memodisk-0.1.1.dev0}/.github/workflows/python-publish.yml

@@ -14,7 +14,7 @@ jobs:
   verify-tests:
     uses: ./.github/workflows/run-tests.yml
     with:
-      python-version: '["3.10"]'  # Run with a single version to be faster
+      python-version: '["3.12", "3.13", "3.14"]'
   
   pypi-publish:
     name: upload release to PyPI
@@ -31,7 +31,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@v3
         with:
-          python-version: '3.10'
+          python-version: '3.12'
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip

{memodisk-0.0.6.dev0 → memodisk-0.1.1.dev0}/.github/workflows/run-tests.yml

@@ -6,7 +6,7 @@ on:
       python-version:
         required: false
         type: string
-        default: '["3.10"]'
+        default: '["3.12", "3.13", "3.14"]'
         description: 'Python versions to test with'
 
 jobs:

{memodisk-0.0.6.dev0 → memodisk-0.1.1.dev0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: memodisk
-Version: 0.0.6.dev0
+Version: 0.1.1.dev0
 Summary: A python module to memoize function results on disk with python code and data dependencies tracking
 Author-email: Martin de La Gorce <martin.delagorce@gmail.com>
 Maintainer-email: Martin de La Gorce <martin.delagorce@gmail.com>
@@ -34,15 +34,15 @@ Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3.10
-Requires-Python: ==3.10.*
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: typing-extensions>=4.0.0
 Provides-Extra: dev
 Requires-Dist: pytest>=6.0; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
-Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: ruff>=0.15.0; extra == "dev"
 Requires-Dist: mypy; extra == "dev"
 Requires-Dist: pre-commit; extra == "dev"
 Requires-Dist: build; extra == "dev"
@@ -51,30 +51,30 @@ Provides-Extra: test
 Requires-Dist: pytest>=6.0; extra == "test"
 Requires-Dist: pytest-cov; extra == "test"
 Requires-Dist: numpy>1.24.4; extra == "test"
-Requires-Dist: numba>=0.61; extra == "test"
+Requires-Dist: numba>=0.65; extra == "test"
 Requires-Dist: opencv-python; extra == "test"
 Dynamic: license-file
 
 # memodisk
 
-**memodisk** is a python module to memoize function results on disk with python code and data dependencies tracking.
+A Python package to cache function results on disk with dependency changes tracking.
 
 ![Python package](https://github.com/martinResearch/memodisk/workflows/Python%20package/badge.svg)
 
 ## Goal
 
-This package provides a python decorator to save on disk and reuse the results of functions that are long to execute. This can be referred to as *persistent memoization*.
+This package provides a Python decorator to save on disk and reuse the results of functions that are long to execute. This can be referred to as *persistent memoization*.
 
 The result of a decorated function is loaded from disk if and only if
 
 * the function has been called previously with the same input argument
-* there are no changes in the python code dependencies or data dependency files.
+* there are no changes in the Python code dependencies or data dependency files.
 
-The use of the second condition differentiates this library from most other python persistent memoization libraries. It is a useful feature when prototyping code with frequent code changes.
+The use of the second condition differentiates this library from most other Python persistent memoization libraries. It is a useful feature when prototyping code with frequent code changes.
 
 ## Warning
 
-This is a prototype with limited testing and works with python 3.10 only. There could be security risks related to the use of Pickle. Some data, code or global variables dependencies could be not detected leading the memoization to return stale results (see the limitations section). If you find failure modes that are not listed in the limitations, please create an issue with a minimal code example to reproduce the problem.
+This is a prototype with limited testing and currently targets Python 3.12+. There could be security risks related to the use of Pickle. Some data, code or global variables dependencies could be not detected leading the memoization to return stale results (see the limitations section). If you find failure modes that are not listed in the limitations, please create an issue with a minimal code example to reproduce the problem.
 
 ## Installation
 
@@ -87,9 +87,11 @@ pip install memodisk
 Using the memoization is as simple as adding the `@memoize` decorator to the function you want to use memoization with. In general you want to use memoization on a function whose execution time is longer than the time it takes to check that this function has already been called with the same argument (compute the input hashes) and load the result from disk.
 To get the largest speedups thanks to memoization, it might be necessary to refactor the code to move the parts that are long to execute into functions that take limited size inputs.
 
+The decorator also accepts options such as `mode=...`, `external_process_mode=...`, `ignore_code_changes=True`, `condition=...`, `serializer=...`, `argument_hasher=...`, and `store_call_arguments=True`. The `condition` callable receives the same arguments as the memoized function and allows caching only for selected invocations. The `serializer` option customizes how cached results are written and read back, while argument hashing remains unchanged. The `argument_hasher` option customizes cache-key generation for `args` and `kwargs`, which is useful for very large inputs or objects that are not picklable by default. The `store_call_arguments` option writes a companion pickle file containing the original `args` and `kwargs` so a cached call can be replayed later. The helper `load_cached_call_arguments(...)` reads those stored arguments back from a dependencies JSON file. The `external_process_mode` option controls subprocess and `os.system` handling: `manual` disables automatic executable tracking, `direct` records the launched executable path but marks subprocess coverage as incomplete, and `strict` raises when traced code launches external processes without complete dependency coverage. The `mode` policy also governs common ambient time APIs such as `time.time()` and `datetime.datetime.now()`: `safe` recomputes and skips cache writes, `strict` raises, and `optimistic` allows caching.
+
 ### Example 1: code dependency changes
 
-let run the code in [example_1.py](./tests/example_1.py) several times.
+let's run the code in [example_1.py](./tests/example_1.py) several times.
 
 ```python
 from memodisk import memoize
@@ -128,7 +130,7 @@ fun_b(5) = 77
 
 As you can see the function fun_b is executed only once. The second time the function is called, the result is loaded from a file on disk and the function is not executed.
 
-Let now edit the file and replace `x * x * 3` by `x * x * 2` and execute again. As expected we now get
+Let's now edit the file and replace `x * x * 3` by `x * x * 2` and execute again. As expected we now get
 
 ```
 executing fun_b
@@ -140,7 +142,7 @@ The change in the body of the function `fun_a`, that is a dependency of `fun_b`,
 
 ### Example 2: data dependency changes
 
-The example [example_2.py](./tests/example_2.py) illustrates how to keep track of data dependencies access using the build-in `open` function.
+The example [example_2.py](./tests/example_2.py) illustrates how to keep track of data dependencies access using the built-in `open` function.
 
 ```python
 from memodisk import memoize
@@ -167,13 +169,13 @@ if __name__ == "__main__":
     assert load_file() == "b"
 ```
 
-When we call `save_file("b")` we overwrite the data in `data_file.txt`. This change in the file content gets detected when calling `load_file` for the second time. This is done by automatically replacing the built-in python function *open* with a wrapper around this function that keeps track of files that are accessed for reading.
+When we call `save_file("b")` we overwrite the data in `data_file.txt`. This change in the file content gets detected when calling `load_file` for the second time. This is done by automatically replacing the built-in Python function *open* with a wrapper around this function that keeps track of files that are accessed for reading.
 
 ### Example 3: file access monkey patching
 
-The built-in `open` function is not the only way the code can access data. For example images can be loaded using opencv's `imread` function. If some data is loaded with another function than the build-in `open` function then the data dependency will not be automatically detected.
+The built-in `open` function is not the only way the code can access data. For example images can be loaded using opencv's `imread` function. If some data is loaded with another function than the built-in `open` function then the data dependency will not be automatically detected.
 
-We provide a function `add_data_dependency` that the user can call from his code next to the line of code that loads the data, with the path of the file that contains the data as input. However this can be error prone as it is very easy to forget calling `add_data_dependency` in some places.
+We provide a function `add_data_dependency` that the user can call from their code next to the line of code that loads the data, with the path of the file that contains the data as input. However this can be error prone as it is very easy to forget calling `add_data_dependency` in some places.
 
 We provide a less error-prone mechanism, through a functor called `DataLoaderWrapper`. The functor allows the user to replace any function accessing data (opencv's `imread` function for example) by a wrapper around this function so that it automatically calls the function add_data_dependency each time `imread` is used. This in-memory modification is called *monkey-patching* and is done in [example_3.py](./tests/example_3.py) using the `DataLoaderWrapper` functor.
 
@@ -187,17 +189,22 @@ cv2.imread = DataLoaderWrapper(cv2.imread)
 
 ## How this works
 
-For each function that is decorated with the `memoize` decorator we keep track of all the python functions it depends on at runtime. Similarly to what is done in the [coverage](https://coverage.readthedocs.io/en/6.0.2/), we provide through `sys.settrace` a callback function to the python interpreter that gets called each time the interpreter calls a new function or executes a new line of code. Doing run time analysis allows us to keep as dependencies only the functions that are required to evaluate the memoize function for the specific arguments. In contrast, static analysis would yield unnecessary dependencies by detecting dependencies for all possible code paths, even the one not executed for the specific set of input arguments. In order to keep the list of dependency files reasonable we exclude from this list of dependencies the functions defined in files under the python lib folder, assuming these will not get modified. The user can also provide an additional list of files he wants to exclude.
+For each function that is decorated with the `memoize` decorator we keep track of all the Python functions it depends on at runtime. Similarly to what is done in the [coverage](https://coverage.readthedocs.io/en/6.0.2/), we register a callback with Python's runtime monitoring API (`sys.monitoring` in Python 3.12+) so the interpreter notifies memodisk when Python functions start executing. Doing runtime analysis allows us to keep as dependencies only the functions that are required to evaluate the memoize function for the specific arguments. In contrast, static analysis would yield unnecessary dependencies by detecting dependencies for all possible code paths, even the one not executed for the specific set of input arguments. In order to keep the list of dependency files reasonable we exclude from this list of dependencies the functions defined in files under the Python lib folder, assuming these will not get modified. The user can also provide an additional list of files to exclude.
 
-For each function listed in the dependencies we compute a hash from its bytecode
-Using the bytecode instead of the function body text allows the user to modify the comments or the formatting a the function without invalidating the previously cached results for this function or the functions that depends on it. We could use instead the hash of the Abstract Syntax Tree of the function (see the [ast module](https://docs.python.org/3/library/ast.html)), but that would rely on the assumption that the code source is not modified during execution of the script (unless there is a way to get access to the AST from when the execution was started). One current limitation resulting from hashing the bytecode is that the python debugger modifies the bytecode when adding breakpoint, which leads to cache misses. This could be potentially resolved by filtering out the lines added by the python debugger before computing the hash of a function.
+For each function listed in the dependencies we compute a hash from its bytecode.
+Using the bytecode instead of the function body text allows the user to modify the comments or the formatting of the function without invalidating the previously cached results for this function or the functions that depend on it. We could use instead the hash of the Abstract Syntax Tree of the function (see the [ast module](https://docs.python.org/3/library/ast.html)), but that would rely on the assumption that the code source is not modified during execution of the script (unless there is a way to get access to the AST from when the execution was started). On current Python 3.12+ with `debugpy`, line breakpoints did not reproduce a bytecode hash change in our tests.
+We also store the source file modification time for each code dependency so cache validation can skip recomputing bytecode hashes when the dependency file has not changed.
+For globals or imported callables backed by packages installed in `site-packages`, we also store the distribution version so upgrading a dependency invalidates stale cache entries even when the package code itself is not traced as a user dependency.
+For compiled extension modules that are already loaded under those same top-level packages, we also store the extension file modification times so in-place `.pyd`/`.so`/`.dll` swaps invalidate the cache.
 
 We keep track of the data dependencies by storing the last modification date of the files that have been opened in read mode.
-The code dependencies hashes and data dependencies last modification dates are saved in a human readable json file in the folder `memodisk_cache` in the user's temp folder (this can be modified at the module level by changing the model variable `disk_cache_dir` ) while the result of the function is pickled in a binary file.
+When traced code launches external processes through `subprocess` or `os.system`, memodisk can also store the resolved top-level executable path as a data dependency so changing that executable invalidates stale cache entries. This is only direct executable tracking: memodisk does not attempt to discover the full runtime dependency graph of that process such as transitively loaded `.dll`/`.so` files, plugins, shell expansion, or other environment-dependent behavior. Cache entries also record whether external-process tracking was complete. In `safe` mode, incomplete external-process coverage forces recomputation instead of serving a cached result, and in `strict` mode it raises.
+When traced code calls common ambient time APIs such as `time.time()`, `time.time_ns()`, `datetime.datetime.now()`, `datetime.datetime.utcnow()`, `datetime.datetime.today()`, or `datetime.date.today()`, memodisk records that the result depended on ambient time. This tracking is intentionally treated as incomplete: in `safe` mode memodisk recomputes instead of serving or writing a cache entry, and in `strict` mode it raises. The robust way to memoize time-sensitive code is to pass the relevant timestamp or date in as an explicit function argument.
+The code dependencies hashes and data dependencies last modification dates are saved in a human readable json file in the folder `memodisk_cache` in the user's temp folder (this can be modified at the module level by changing the module variable `disk_cache_dir` ) while the result of the function is saved in a binary file using pickle by default or a custom serializer when provided.
 The names of the two generated files differ only by the extension (json and pkl) and are formatted
 by default as `{function_name}_{arguments_hash}.json` and `{function_name}_{arguments_hash}.pkl`.
 There are some subtle details to take into consideration when accessing data from a file in order to guarantee that the caching will not provide stale results.
-The data dependency "version" is obtained by recording the  modification date of the accessed file. The modification date resolution is limited and it is possible that a file gets modified during the lapse of time during which the modification date is not incremented. We guard against this by locking the file for a time that is greater than the modification date quantization step.
+The data dependency "version" is obtained by recording the modification date of the accessed file. The modification date resolution is limited and it is possible that a file gets modified during the lapse of time during which the modification date is not incremented. We guard against this by locking the file for a time that is greater than the modification date quantization step.
 
 The hash of the arguments is obtained by pickling the arguments. This can be slow if the input is large or made of many small objects.
 
@@ -213,7 +220,10 @@ Here is an example of a generated dependencies json file:
             "filename": "D:\\repos\\memodisk\\tests\\test_data_dependency_change.py",
             "bytecode_hash": "f7b971c6ea7997dc5c5222f74cec2a249e8680293511c6a8350b621643af2d07",
             "global_vars": {},
-            "closure_vars": {}
+            "closure_vars": {},
+            "package_versions": {},
+            "compiled_dependencies": {},
+            "file_last_modified_date_str": "2026-04-04 15:33:14.682488"
         }
     ],
     "data": [
@@ -222,35 +232,39 @@ Here is an example of a generated dependencies json file:
             "last_modified_date_str": "2020-03-04 15:33:14.682488"
         }
     ],
-    "random_states": null
+    "random_states": null,
+    "external_processes": [],
+    "ambient_time_sources": [],
+    "ambient_environment_sources": [],
+    "ignore_code_changes": false,
+    "result_serializer": "pickle",
+    "argument_hasher": null,
+    "store_call_arguments": false,
+    "call_arguments_file": null
 }
 ```
 
-We do not try to detect changes in the global variables but an error will be raised if any of the functions listed in the dependencies uses global variables.
+We do not try to detect changes in global variables but an error will be raised if any of the functions listed in the dependencies uses global variables.
 
 ## Using numpy's random generator
 
-A function that uses the default numpy generator is problematic from caching two reasons: 1) its output depends on the random generator state that is not provided as an explicit input to the function 2) the function modifies the state of the random generator for the functions that get called after it.
-When retrieving a cached results for such a function we want to use the state of the random generator when entering the function in the hash of the inputs and after retrieveing cached results we want to set the random state to the same state as the one we would get by running the the cached function. 
+A function that uses the default numpy generator is problematic for caching for two reasons: 1) its output depends on the random generator state that is not provided as an explicit input to the function 2) the function modifies the state of the random generator for the functions that get called after it.
+When retrieving cached results for such a function we want to use the state of the random generator when entering the function in the hash of the inputs and after retrieving cached results we want to set the random state to the same state as the one we would get by running the cached function. 
 The use of the random generator is detected by comparing the state of the random generator state before and after executing the function.
 The input state and output state of the random generator are saved in the json file and the memoized result is loaded in subsequent run only if the random state is identical to the one saved in the json file i.e. when entering the function at the first run, the result is loaded from the pickle file and the random state is modified to match the random state after execution of the function at the first run.
 
-This mechanism can fail in some cases, if a function access the random generator but restore the generator in the same state as it was when entring the function for example and thus we recommend to avoid using the default "global" numpy random generator, but instead to use instances of `numpy.random.Generator` that are passed as arguments to the functions that use the random number in order to reduce the risk of getting stale results from the memoize decorator.
+This mechanism can fail in some cases, if a function accesses the random generator but restores the generator in the same state as it was when entering the function for example and thus we recommend to avoid using the default "global" numpy random generator, but instead to use instances of `numpy.random.Generator` that are passed as arguments to the functions that use the random number in order to reduce the risk of getting stale results from the memoize decorator.
 
 If the same function is called multiple times with the same input arguments but with different random states, then a single memoization file is used and gets overwritten. We could add an argument to the memoize decorator to tell the memoize decorator to use the random state when computing the hash of the input arguments to allow the use of multiple memoization files for the same function with one file for each state of the random generator.
 
 ## Limitations
 
-* does not support the property decorator in some cases as `getattr` is trying to execute the function.
-* requires all the function arguments of the memoized function to be serializable using pickle.
+* by default, requires all the function arguments of the memoized function to be serializable using pickle. This can be overridden with `argument_hasher=...` for cache-key generation.
 * may not detect all global variables dependencies.
-* will detect an added breakpoint in a function as a change in the code because the python debugger adds line in the function bytecode when using breakpoint.
-* does not detect non determinism due to use of time.
-* does not detect changes in C/C++ extension modules or external executables, unless the pyd file, dll or executable dependency is explicitly specified through the `add_data_dependency` function.
+* ambient time detection covers common direct calls such as `time.time()` and `datetime.datetime.now()`, but may still miss indirect wrappers, custom clocks, or time values obtained outside the memoized call and captured through globals or closures.
+* external-process tracking is limited to the directly resolved executable path. It does not cover full transitive native dependencies such as `.dll`/`.so` chains, plugins, registry/config driven behavior, or other runtime environment changes.
+* may miss some shell-invoked external commands when the real executable cannot be resolved from the command string. In those cases the executable should still be declared explicitly with `add_data_dependency`.
 * does not detect changes in remote dependencies fetched from the network.
-* is not thread safe. It does not support multi-threading
-* will not detect if an aliased import is modified.
-* computes argument hash using pickled object strings, which does not always produce the same string for identical objects. Could use [compute_fixed_hash](https://github.com/QUVA-Lab/artemis/blob/84d3b1daf0de363cc823d99f978e2861ed400b5b/artemis/general/hashing.py#L25).
 * has no configurable cache size.
 * will memoize only the decorated functions. 
 
@@ -258,20 +272,10 @@ Some of these failure modes can be reproduced using scripts in the [failure_mode
 
 ## TODO
 
-* add the module name of the functions in the code dependencies description.
-* filter out the lines added by the python debugger when using breakpoint before computing the hash of a function.
-* add the file modification date in the code dependencies and a test to skip checking functions hashes if the file modification date did not change.
-* add an option to save the full input arguments in the pickle file in order to be able to re-run a function directly from the pickled data
-* save the versions of module dependencies that in the lib/site-packages folder and add an option to remove function dependencies from packages under site-packages and use only module version to detect a dependency change, assuming the package in site-packages does not get modified. maybe use the file modification date for code under site-packages instead of functions ast
 * improve the detection of non-pure function so that it works when using a compiled third party module
-* allow the use of a different serialization library than pickle. It could be provided at module level to disc_memoize or as argument to the memoize decorator
-* add the possibility to provide a condition in the decorator to memoize or not
-* add a less intrusive alternative to the use of decorator by registering a function in a list of function names provided directly to disc_memoize
+* add a less intrusive alternative to the use of a decorator by registering a function in a list of function names provided directly to disk_memoize
 * implement an automatic memoization of function that are long to evaluate using similar criterion to IncPy (see references) to decide if a function should be memoize or not
-* make the tool thread-safe
-* see if we can detect compiled module loading and compiled module calling to add the compiled module as dependency.
 * publish module on [pypi.org](pypi.org)
-* see if we can make the hashing more deterministic using the method implemented in [charmonium.cache](https://pypi.org/project/charmonium.cache).
 
 ## Alternatives
 
@@ -283,3 +287,4 @@ Some of these failure modes can be reproduced using scripts in the [failure_mode
 * [Artemis.fileman.disk_memoize](https://github.com/QUVA-Lab/artemis/blob/master/artemis/fileman/disk_memoize.py) It does not detect changes in the code or data dependencies. [pdf](http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=59BEC4646686E70CFD2428EF9786B9D0?doi=10.1.1.224.164&rep=rep1&type=pdf)
 * [noWorkflow](http://gems-uff.github.io/noworkflow/). *noWorkflow: a Tool for Collecting, Analyzing, and Managing Provenance from Python Scripts* [pdf](https://par.nsf.gov/servlets/purl/10048452). Library that allows to track how data has been generated. It bears some similarity with the library as it also requires to keep track of dependencies.
 * [klepto](https://mmckerns.github.io/project/pathos/wiki/klepto.html). Allows caching of python function results to files or database archive. The detection of code change is not mentioned.
+* [exca](https://github.com/facebookresearch/exca). Developed by Facebook Research. The detection of code change is not mentioned.

{memodisk-0.0.6.dev0 → memodisk-0.1.1.dev0}/README.md

@@ -1,23 +1,23 @@
 # memodisk
 
-**memodisk** is a python module to memoize function results on disk with python code and data dependencies tracking.
+A Python package to cache function results on disk with dependency changes tracking.
 
 ![Python package](https://github.com/martinResearch/memodisk/workflows/Python%20package/badge.svg)
 
 ## Goal
 
-This package provides a python decorator to save on disk and reuse the results of functions that are long to execute. This can be referred to as *persistent memoization*.
+This package provides a Python decorator to save on disk and reuse the results of functions that are long to execute. This can be referred to as *persistent memoization*.
 
 The result of a decorated function is loaded from disk if and only if
 
 * the function has been called previously with the same input argument
-* there are no changes in the python code dependencies or data dependency files.
+* there are no changes in the Python code dependencies or data dependency files.
 
-The use of the second condition differentiates this library from most other python persistent memoization libraries. It is a useful feature when prototyping code with frequent code changes.
+The use of the second condition differentiates this library from most other Python persistent memoization libraries. It is a useful feature when prototyping code with frequent code changes.
 
 ## Warning
 
-This is a prototype with limited testing and works with python 3.10 only. There could be security risks related to the use of Pickle. Some data, code or global variables dependencies could be not detected leading the memoization to return stale results (see the limitations section). If you find failure modes that are not listed in the limitations, please create an issue with a minimal code example to reproduce the problem.
+This is a prototype with limited testing and currently targets Python 3.12+. There could be security risks related to the use of Pickle. Some data, code or global variables dependencies could be not detected leading the memoization to return stale results (see the limitations section). If you find failure modes that are not listed in the limitations, please create an issue with a minimal code example to reproduce the problem.
 
 ## Installation
 
@@ -30,9 +30,11 @@ pip install memodisk
 Using the memoization is as simple as adding the `@memoize` decorator to the function you want to use memoization with. In general you want to use memoization on a function whose execution time is longer than the time it takes to check that this function has already been called with the same argument (compute the input hashes) and load the result from disk.
 To get the largest speedups thanks to memoization, it might be necessary to refactor the code to move the parts that are long to execute into functions that take limited size inputs.
 
+The decorator also accepts options such as `mode=...`, `external_process_mode=...`, `ignore_code_changes=True`, `condition=...`, `serializer=...`, `argument_hasher=...`, and `store_call_arguments=True`. The `condition` callable receives the same arguments as the memoized function and allows caching only for selected invocations. The `serializer` option customizes how cached results are written and read back, while argument hashing remains unchanged. The `argument_hasher` option customizes cache-key generation for `args` and `kwargs`, which is useful for very large inputs or objects that are not picklable by default. The `store_call_arguments` option writes a companion pickle file containing the original `args` and `kwargs` so a cached call can be replayed later. The helper `load_cached_call_arguments(...)` reads those stored arguments back from a dependencies JSON file. The `external_process_mode` option controls subprocess and `os.system` handling: `manual` disables automatic executable tracking, `direct` records the launched executable path but marks subprocess coverage as incomplete, and `strict` raises when traced code launches external processes without complete dependency coverage. The `mode` policy also governs common ambient time APIs such as `time.time()` and `datetime.datetime.now()`: `safe` recomputes and skips cache writes, `strict` raises, and `optimistic` allows caching.
+
 ### Example 1: code dependency changes
 
-let run the code in [example_1.py](./tests/example_1.py) several times.
+let's run the code in [example_1.py](./tests/example_1.py) several times.
 
 ```python
 from memodisk import memoize
@@ -71,7 +73,7 @@ fun_b(5) = 77
 
 As you can see the function fun_b is executed only once. The second time the function is called, the result is loaded from a file on disk and the function is not executed.
 
-Let now edit the file and replace `x * x * 3` by `x * x * 2` and execute again. As expected we now get
+Let's now edit the file and replace `x * x * 3` by `x * x * 2` and execute again. As expected we now get
 
 ```
 executing fun_b
@@ -83,7 +85,7 @@ The change in the body of the function `fun_a`, that is a dependency of `fun_b`,
 
 ### Example 2: data dependency changes
 
-The example [example_2.py](./tests/example_2.py) illustrates how to keep track of data dependencies access using the build-in `open` function.
+The example [example_2.py](./tests/example_2.py) illustrates how to keep track of data dependencies access using the built-in `open` function.
 
 ```python
 from memodisk import memoize
@@ -110,13 +112,13 @@ if __name__ == "__main__":
     assert load_file() == "b"
 ```
 
-When we call `save_file("b")` we overwrite the data in `data_file.txt`. This change in the file content gets detected when calling `load_file` for the second time. This is done by automatically replacing the built-in python function *open* with a wrapper around this function that keeps track of files that are accessed for reading.
+When we call `save_file("b")` we overwrite the data in `data_file.txt`. This change in the file content gets detected when calling `load_file` for the second time. This is done by automatically replacing the built-in Python function *open* with a wrapper around this function that keeps track of files that are accessed for reading.
 
 ### Example 3: file access monkey patching
 
-The built-in `open` function is not the only way the code can access data. For example images can be loaded using opencv's `imread` function. If some data is loaded with another function than the build-in `open` function then the data dependency will not be automatically detected.
+The built-in `open` function is not the only way the code can access data. For example images can be loaded using opencv's `imread` function. If some data is loaded with another function than the built-in `open` function then the data dependency will not be automatically detected.
 
-We provide a function `add_data_dependency` that the user can call from his code next to the line of code that loads the data, with the path of the file that contains the data as input. However this can be error prone as it is very easy to forget calling `add_data_dependency` in some places.
+We provide a function `add_data_dependency` that the user can call from their code next to the line of code that loads the data, with the path of the file that contains the data as input. However this can be error prone as it is very easy to forget calling `add_data_dependency` in some places.
 
 We provide a less error-prone mechanism, through a functor called `DataLoaderWrapper`. The functor allows the user to replace any function accessing data (opencv's `imread` function for example) by a wrapper around this function so that it automatically calls the function add_data_dependency each time `imread` is used. This in-memory modification is called *monkey-patching* and is done in [example_3.py](./tests/example_3.py) using the `DataLoaderWrapper` functor.
 
@@ -130,17 +132,22 @@ cv2.imread = DataLoaderWrapper(cv2.imread)
 
 ## How this works
 
-For each function that is decorated with the `memoize` decorator we keep track of all the python functions it depends on at runtime. Similarly to what is done in the [coverage](https://coverage.readthedocs.io/en/6.0.2/), we provide through `sys.settrace` a callback function to the python interpreter that gets called each time the interpreter calls a new function or executes a new line of code. Doing run time analysis allows us to keep as dependencies only the functions that are required to evaluate the memoize function for the specific arguments. In contrast, static analysis would yield unnecessary dependencies by detecting dependencies for all possible code paths, even the one not executed for the specific set of input arguments. In order to keep the list of dependency files reasonable we exclude from this list of dependencies the functions defined in files under the python lib folder, assuming these will not get modified. The user can also provide an additional list of files he wants to exclude.
+For each function that is decorated with the `memoize` decorator we keep track of all the Python functions it depends on at runtime. Similarly to what is done in the [coverage](https://coverage.readthedocs.io/en/6.0.2/), we register a callback with Python's runtime monitoring API (`sys.monitoring` in Python 3.12+) so the interpreter notifies memodisk when Python functions start executing. Doing runtime analysis allows us to keep as dependencies only the functions that are required to evaluate the memoize function for the specific arguments. In contrast, static analysis would yield unnecessary dependencies by detecting dependencies for all possible code paths, even the one not executed for the specific set of input arguments. In order to keep the list of dependency files reasonable we exclude from this list of dependencies the functions defined in files under the Python lib folder, assuming these will not get modified. The user can also provide an additional list of files to exclude.
 
-For each function listed in the dependencies we compute a hash from its bytecode
-Using the bytecode instead of the function body text allows the user to modify the comments or the formatting a the function without invalidating the previously cached results for this function or the functions that depends on it. We could use instead the hash of the Abstract Syntax Tree of the function (see the [ast module](https://docs.python.org/3/library/ast.html)), but that would rely on the assumption that the code source is not modified during execution of the script (unless there is a way to get access to the AST from when the execution was started). One current limitation resulting from hashing the bytecode is that the python debugger modifies the bytecode when adding breakpoint, which leads to cache misses. This could be potentially resolved by filtering out the lines added by the python debugger before computing the hash of a function.
+For each function listed in the dependencies we compute a hash from its bytecode.
+Using the bytecode instead of the function body text allows the user to modify the comments or the formatting of the function without invalidating the previously cached results for this function or the functions that depend on it. We could use instead the hash of the Abstract Syntax Tree of the function (see the [ast module](https://docs.python.org/3/library/ast.html)), but that would rely on the assumption that the code source is not modified during execution of the script (unless there is a way to get access to the AST from when the execution was started). On current Python 3.12+ with `debugpy`, line breakpoints did not reproduce a bytecode hash change in our tests.
+We also store the source file modification time for each code dependency so cache validation can skip recomputing bytecode hashes when the dependency file has not changed.
+For globals or imported callables backed by packages installed in `site-packages`, we also store the distribution version so upgrading a dependency invalidates stale cache entries even when the package code itself is not traced as a user dependency.
+For compiled extension modules that are already loaded under those same top-level packages, we also store the extension file modification times so in-place `.pyd`/`.so`/`.dll` swaps invalidate the cache.
 
 We keep track of the data dependencies by storing the last modification date of the files that have been opened in read mode.
-The code dependencies hashes and data dependencies last modification dates are saved in a human readable json file in the folder `memodisk_cache` in the user's temp folder (this can be modified at the module level by changing the model variable `disk_cache_dir` ) while the result of the function is pickled in a binary file.
+When traced code launches external processes through `subprocess` or `os.system`, memodisk can also store the resolved top-level executable path as a data dependency so changing that executable invalidates stale cache entries. This is only direct executable tracking: memodisk does not attempt to discover the full runtime dependency graph of that process such as transitively loaded `.dll`/`.so` files, plugins, shell expansion, or other environment-dependent behavior. Cache entries also record whether external-process tracking was complete. In `safe` mode, incomplete external-process coverage forces recomputation instead of serving a cached result, and in `strict` mode it raises.
+When traced code calls common ambient time APIs such as `time.time()`, `time.time_ns()`, `datetime.datetime.now()`, `datetime.datetime.utcnow()`, `datetime.datetime.today()`, or `datetime.date.today()`, memodisk records that the result depended on ambient time. This tracking is intentionally treated as incomplete: in `safe` mode memodisk recomputes instead of serving or writing a cache entry, and in `strict` mode it raises. The robust way to memoize time-sensitive code is to pass the relevant timestamp or date in as an explicit function argument.
+The code dependencies hashes and data dependencies last modification dates are saved in a human readable json file in the folder `memodisk_cache` in the user's temp folder (this can be modified at the module level by changing the module variable `disk_cache_dir` ) while the result of the function is saved in a binary file using pickle by default or a custom serializer when provided.
 The names of the two generated files differ only by the extension (json and pkl) and are formatted
 by default as `{function_name}_{arguments_hash}.json` and `{function_name}_{arguments_hash}.pkl`.
 There are some subtle details to take into consideration when accessing data from a file in order to guarantee that the caching will not provide stale results.
-The data dependency "version" is obtained by recording the  modification date of the accessed file. The modification date resolution is limited and it is possible that a file gets modified during the lapse of time during which the modification date is not incremented. We guard against this by locking the file for a time that is greater than the modification date quantization step.
+The data dependency "version" is obtained by recording the modification date of the accessed file. The modification date resolution is limited and it is possible that a file gets modified during the lapse of time during which the modification date is not incremented. We guard against this by locking the file for a time that is greater than the modification date quantization step.
 
 The hash of the arguments is obtained by pickling the arguments. This can be slow if the input is large or made of many small objects.
 
@@ -156,7 +163,10 @@ Here is an example of a generated dependencies json file:
             "filename": "D:\\repos\\memodisk\\tests\\test_data_dependency_change.py",
             "bytecode_hash": "f7b971c6ea7997dc5c5222f74cec2a249e8680293511c6a8350b621643af2d07",
             "global_vars": {},
-            "closure_vars": {}
+            "closure_vars": {},
+            "package_versions": {},
+            "compiled_dependencies": {},
+            "file_last_modified_date_str": "2026-04-04 15:33:14.682488"
         }
     ],
     "data": [
@@ -165,35 +175,39 @@ Here is an example of a generated dependencies json file:
             "last_modified_date_str": "2020-03-04 15:33:14.682488"
         }
     ],
-    "random_states": null
+    "random_states": null,
+    "external_processes": [],
+    "ambient_time_sources": [],
+    "ambient_environment_sources": [],
+    "ignore_code_changes": false,
+    "result_serializer": "pickle",
+    "argument_hasher": null,
+    "store_call_arguments": false,
+    "call_arguments_file": null
 }
 ```
 
-We do not try to detect changes in the global variables but an error will be raised if any of the functions listed in the dependencies uses global variables.
+We do not try to detect changes in global variables but an error will be raised if any of the functions listed in the dependencies uses global variables.
 
 ## Using numpy's random generator
 
-A function that uses the default numpy generator is problematic from caching two reasons: 1) its output depends on the random generator state that is not provided as an explicit input to the function 2) the function modifies the state of the random generator for the functions that get called after it.
-When retrieving a cached results for such a function we want to use the state of the random generator when entering the function in the hash of the inputs and after retrieveing cached results we want to set the random state to the same state as the one we would get by running the the cached function. 
+A function that uses the default numpy generator is problematic for caching for two reasons: 1) its output depends on the random generator state that is not provided as an explicit input to the function 2) the function modifies the state of the random generator for the functions that get called after it.
+When retrieving cached results for such a function we want to use the state of the random generator when entering the function in the hash of the inputs and after retrieving cached results we want to set the random state to the same state as the one we would get by running the cached function. 
 The use of the random generator is detected by comparing the state of the random generator state before and after executing the function.
 The input state and output state of the random generator are saved in the json file and the memoized result is loaded in subsequent run only if the random state is identical to the one saved in the json file i.e. when entering the function at the first run, the result is loaded from the pickle file and the random state is modified to match the random state after execution of the function at the first run.
 
-This mechanism can fail in some cases, if a function access the random generator but restore the generator in the same state as it was when entring the function for example and thus we recommend to avoid using the default "global" numpy random generator, but instead to use instances of `numpy.random.Generator` that are passed as arguments to the functions that use the random number in order to reduce the risk of getting stale results from the memoize decorator.
+This mechanism can fail in some cases, if a function accesses the random generator but restores the generator in the same state as it was when entering the function for example and thus we recommend to avoid using the default "global" numpy random generator, but instead to use instances of `numpy.random.Generator` that are passed as arguments to the functions that use the random number in order to reduce the risk of getting stale results from the memoize decorator.
 
 If the same function is called multiple times with the same input arguments but with different random states, then a single memoization file is used and gets overwritten. We could add an argument to the memoize decorator to tell the memoize decorator to use the random state when computing the hash of the input arguments to allow the use of multiple memoization files for the same function with one file for each state of the random generator.
 
 ## Limitations
 
-* does not support the property decorator in some cases as `getattr` is trying to execute the function.
-* requires all the function arguments of the memoized function to be serializable using pickle.
+* by default, requires all the function arguments of the memoized function to be serializable using pickle. This can be overridden with `argument_hasher=...` for cache-key generation.
 * may not detect all global variables dependencies.
-* will detect an added breakpoint in a function as a change in the code because the python debugger adds line in the function bytecode when using breakpoint.
-* does not detect non determinism due to use of time.
-* does not detect changes in C/C++ extension modules or external executables, unless the pyd file, dll or executable dependency is explicitly specified through the `add_data_dependency` function.
+* ambient time detection covers common direct calls such as `time.time()` and `datetime.datetime.now()`, but may still miss indirect wrappers, custom clocks, or time values obtained outside the memoized call and captured through globals or closures.
+* external-process tracking is limited to the directly resolved executable path. It does not cover full transitive native dependencies such as `.dll`/`.so` chains, plugins, registry/config driven behavior, or other runtime environment changes.
+* may miss some shell-invoked external commands when the real executable cannot be resolved from the command string. In those cases the executable should still be declared explicitly with `add_data_dependency`.
 * does not detect changes in remote dependencies fetched from the network.
-* is not thread safe. It does not support multi-threading
-* will not detect if an aliased import is modified.
-* computes argument hash using pickled object strings, which does not always produce the same string for identical objects. Could use [compute_fixed_hash](https://github.com/QUVA-Lab/artemis/blob/84d3b1daf0de363cc823d99f978e2861ed400b5b/artemis/general/hashing.py#L25).
 * has no configurable cache size.
 * will memoize only the decorated functions. 
 
@@ -201,20 +215,10 @@ Some of these failure modes can be reproduced using scripts in the [failure_mode
 
 ## TODO
 
-* add the module name of the functions in the code dependencies description.
-* filter out the lines added by the python debugger when using breakpoint before computing the hash of a function.
-* add the file modification date in the code dependencies and a test to skip checking functions hashes if the file modification date did not change.
-* add an option to save the full input arguments in the pickle file in order to be able to re-run a function directly from the pickled data
-* save the versions of module dependencies that in the lib/site-packages folder and add an option to remove function dependencies from packages under site-packages and use only module version to detect a dependency change, assuming the package in site-packages does not get modified. maybe use the file modification date for code under site-packages instead of functions ast
 * improve the detection of non-pure function so that it works when using a compiled third party module
-* allow the use of a different serialization library than pickle. It could be provided at module level to disc_memoize or as argument to the memoize decorator
-* add the possibility to provide a condition in the decorator to memoize or not
-* add a less intrusive alternative to the use of decorator by registering a function in a list of function names provided directly to disc_memoize
+* add a less intrusive alternative to the use of a decorator by registering a function in a list of function names provided directly to disk_memoize
 * implement an automatic memoization of function that are long to evaluate using similar criterion to IncPy (see references) to decide if a function should be memoize or not
-* make the tool thread-safe
-* see if we can detect compiled module loading and compiled module calling to add the compiled module as dependency.
 * publish module on [pypi.org](pypi.org)
-* see if we can make the hashing more deterministic using the method implemented in [charmonium.cache](https://pypi.org/project/charmonium.cache).
 
 ## Alternatives
 
@@ -226,3 +230,4 @@ Some of these failure modes can be reproduced using scripts in the [failure_mode
 * [Artemis.fileman.disk_memoize](https://github.com/QUVA-Lab/artemis/blob/master/artemis/fileman/disk_memoize.py) It does not detect changes in the code or data dependencies. [pdf](http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=59BEC4646686E70CFD2428EF9786B9D0?doi=10.1.1.224.164&rep=rep1&type=pdf)
 * [noWorkflow](http://gems-uff.github.io/noworkflow/). *noWorkflow: a Tool for Collecting, Analyzing, and Managing Provenance from Python Scripts* [pdf](https://par.nsf.gov/servlets/purl/10048452). Library that allows to track how data has been generated. It bears some similarity with the library as it also requires to keep track of dependencies.
 * [klepto](https://mmckerns.github.io/project/pathos/wiki/klepto.html). Allows caching of python function results to files or database archive. The detection of code change is not mentioned.
+* [exca](https://github.com/facebookresearch/exca). Developed by Facebook Research. The detection of code change is not mentioned.

{memodisk-0.0.6.dev0 → memodisk-0.1.1.dev0}/memodisk/__init__.py

@@ -4,29 +4,34 @@ from ._version import __version__
 
 __all__ = [
     "memoize",
+    "MemoizeMode",
+    "ExternalProcessMode",
+    "ArgumentHasher",
+    "ResultSerializer",
     "add_data_dependency",
     "DataLoaderWrapper",
     "get_globals_from_code",
     "set_cache_dir",
     "open_delay",
-    "get_function_qualified_name_from_frame",
-    "get_globals_from_code",
     "get_last_cache_loading",
     "reset_last_cache_loading",
-    "get_function_from_frame",
+    "load_cached_call_arguments",
     "hashing_func_map",
     "user_ignore_files",
     "__version__",
 ]
 
 from .memodisk import (
+    ArgumentHasher,
     DataLoaderWrapper,
+    ExternalProcessMode,
+    MemoizeMode,
+    ResultSerializer,
     add_data_dependency,
-    get_function_from_frame,
-    get_function_qualified_name_from_frame,
     get_globals_from_code,
     get_last_cache_loading,
     hashing_func_map,
+    load_cached_call_arguments,
     memoize,
     open_delay,
     reset_last_cache_loading,

memodisk-0.1.1.dev0/memodisk/_version.py

@@ -0,0 +1,24 @@
+# file generated by vcs-versioning
+# don't change, don't track in version control
+from __future__ import annotations
+
+__all__ = [
+    "__version__",
+    "__version_tuple__",
+    "version",
+    "version_tuple",
+    "__commit_id__",
+    "commit_id",
+]
+
+version: str
+__version__: str
+__version_tuple__: tuple[int | str, ...]
+version_tuple: tuple[int | str, ...]
+commit_id: str | None
+__commit_id__: str | None
+
+__version__ = version = '0.1.1.dev0'
+__version_tuple__ = version_tuple = (0, 1, 1, 'dev0')
+
+__commit_id__ = commit_id = 'gd2df32186'