pivot 0.3.3__tar.gz → 0.3.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {pivot-0.3.3 → pivot-0.3.5}/PKG-INFO +1 -1
- {pivot-0.3.3 → pivot-0.3.5}/pyproject.toml +1 -1
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/checkout.py +9 -5
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/track.py +5 -6
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/exceptions.py +6 -4
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/core.py +1 -1
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/worker.py +3 -9
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/explain.py +4 -8
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/storage.py +47 -55
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/sync.py +2 -4
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/run_history.py +7 -1
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/status.py +1 -1
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/AGENTS.md +38 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/cache.py +592 -78
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/lock.py +38 -15
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/restore.py +10 -24
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/store.py +5 -9
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/track.py +23 -4
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/types.py +13 -3
- {pivot-0.3.3 → pivot-0.3.5}/LICENSE +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/README.md +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/__init__.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/ast_utils.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/AGENTS.md +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/CLAUDE.md +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/__init__.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/_run_common.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/check_ignore.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/commit.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/completion.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/config.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/console.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/dag.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/data.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/decorators.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/doctor.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/errors.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/fingerprint.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/helpers.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/history.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/import_cmd.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/init.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/list.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/metrics.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/params.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/plots.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/remote.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/repro.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/run.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/status.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/targets.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/update.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/verify.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/compose.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/config/__init__.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/config/io.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/config/models.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/dag/__init__.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/dag/render.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/decorators.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/discovery.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/__init__.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/agent_rpc.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/engine.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/graph.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/scheduler.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/sinks.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/sources.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/types.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/watch.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/worker_pool.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/AGENTS.md +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/CLAUDE.md +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/__init__.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/commit.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/fingerprint.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/git.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/ignore.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/import_artifact.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/loaders.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/metrics.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/names.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/outputs.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/parameters.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/path_policy.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/project.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/py.typed +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/registry.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/AGENTS.md +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/__init__.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/config.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/git_archive.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/github.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/__init__.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/common.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/data.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/metrics.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/params.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/plots.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/skip.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/stage_def.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/CLAUDE.md +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/__init__.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/artifact_lock.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/presentation.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/state.py +0 -0
- {pivot-0.3.3 → pivot-0.3.5}/src/pivot/yaml_config.py +0 -0
|
@@ -74,11 +74,15 @@ def _on_disk_matches_expected(
|
|
|
74
74
|
if is_dir_hash(expected_hash):
|
|
75
75
|
if not path.is_dir():
|
|
76
76
|
return False
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
77
|
+
actual = cache.hash_directory(path, cache_dir=cache_dir)
|
|
78
|
+
# A hash pivot 0.3.3 recorded lists no empty directories, and repro still honors it.
|
|
79
|
+
# When only empty directories differ, checkout would change no file, so it matches.
|
|
80
|
+
return actual["hash"] == expected_hash["hash"] or (
|
|
81
|
+
"empty_dirs" not in expected_hash and actual["manifest"] == expected_hash["manifest"]
|
|
82
|
+
)
|
|
83
|
+
if not path.is_file():
|
|
84
|
+
return False
|
|
85
|
+
actual_hash, _ = cache.hash_file(path)
|
|
82
86
|
return actual_hash == expected_hash["hash"]
|
|
83
87
|
|
|
84
88
|
|
|
@@ -154,16 +154,13 @@ def _track_single_path(
|
|
|
154
154
|
|
|
155
155
|
# Hash and cache
|
|
156
156
|
if path.is_dir():
|
|
157
|
-
|
|
157
|
+
dir_hash = cache.hash_directory(
|
|
158
158
|
path, cache_dir=cache_dir, warn_left_out_symlinks=warn_left_out_symlinks
|
|
159
159
|
)
|
|
160
|
+
manifest = dir_hash["manifest"]
|
|
160
161
|
total_size = sum(e["size"] for e in manifest)
|
|
161
162
|
num_files = len(manifest)
|
|
162
163
|
|
|
163
|
-
# copy_to_cache is idempotent and already checks existence itself; letting it see
|
|
164
|
-
# every entry (not just ones missing from the cache) is what lets it republish an
|
|
165
|
-
# object `pivot track --force` finds cached non-executable by an older pivot or a
|
|
166
|
-
# cold `pivot pull`, when this entry's bytes should be executable.
|
|
167
164
|
for entry in manifest:
|
|
168
165
|
file_path = path / entry["relpath"]
|
|
169
166
|
file_cache_path = cache.get_cache_path(cache_dir, entry["hash"])
|
|
@@ -171,11 +168,13 @@ def _track_single_path(
|
|
|
171
168
|
|
|
172
169
|
pvt_data: track_mod.PvtData = {
|
|
173
170
|
"path": path.name,
|
|
174
|
-
"hash":
|
|
171
|
+
"hash": dir_hash["hash"],
|
|
175
172
|
"size": total_size,
|
|
176
173
|
"num_files": num_files,
|
|
177
174
|
"manifest": manifest,
|
|
178
175
|
}
|
|
176
|
+
if "empty_dirs" in dir_hash:
|
|
177
|
+
pvt_data["empty_dirs"] = dir_hash["empty_dirs"]
|
|
179
178
|
else:
|
|
180
179
|
file_hash, _ = cache.hash_file(path)
|
|
181
180
|
file_size = path.stat().st_size
|
|
@@ -221,6 +221,8 @@ class CacheModeCollisionError(CacheError):
|
|
|
221
221
|
This happens when two files with identical content disagree on their executable bit (the
|
|
222
222
|
cache key is content-only) or when a cache object was populated by a path that does not
|
|
223
223
|
know a file's intended executable bit (e.g. `pivot pull`, which downloads by hash alone).
|
|
224
|
+
Pivot never changes a published cache object, so only a private copy, or the object deleted
|
|
225
|
+
and published again from bytes that carry the right bit, can give the file its bit.
|
|
224
226
|
"""
|
|
225
227
|
|
|
226
228
|
@override
|
|
@@ -228,10 +230,10 @@ class CacheModeCollisionError(CacheError):
|
|
|
228
230
|
return (
|
|
229
231
|
"Add CheckoutMode.COPY to checkout_modes (cache: checkout_mode: [..., copy] in "
|
|
230
232
|
".pivot/config.yaml) so checkout can fall back to a private, correctly-permissioned "
|
|
231
|
-
"copy instead of sharing the cache object's own permissions; or
|
|
232
|
-
"
|
|
233
|
-
"<stage>`, `pivot track --force <path>`)
|
|
234
|
-
"
|
|
233
|
+
"copy instead of sharing the cache object's own permissions; or delete the cache "
|
|
234
|
+
"object the error names, then re-run what produced the file (`pivot repro --force "
|
|
235
|
+
"<stage>`, `pivot track --force <path>`) to publish it with the right bit; until it "
|
|
236
|
+
"is published again, any workspace file symlinked to it is broken"
|
|
235
237
|
)
|
|
236
238
|
|
|
237
239
|
|
|
@@ -409,7 +409,7 @@ def verify_tracked_files(project_root: pathlib.Path, checkout_missing: bool = Fa
|
|
|
409
409
|
if path.is_file():
|
|
410
410
|
current_hash, _ = cache.hash_file(path, state_db)
|
|
411
411
|
elif path.is_dir():
|
|
412
|
-
current_hash
|
|
412
|
+
current_hash = cache.hash_directory(path, state_db, cache_dir=cache_dir)["hash"]
|
|
413
413
|
else:
|
|
414
414
|
# Path doesn't exist
|
|
415
415
|
raise FileNotFoundError(data_path)
|
|
@@ -39,7 +39,6 @@ from pivot.storage import artifact_lock, cache, lock, state
|
|
|
39
39
|
from pivot.storage import store as store_mod
|
|
40
40
|
from pivot.types import (
|
|
41
41
|
DeferredWrites,
|
|
42
|
-
DirHash,
|
|
43
42
|
FileHash,
|
|
44
43
|
HashInfo,
|
|
45
44
|
LockData,
|
|
@@ -631,8 +630,7 @@ def hash_output(
|
|
|
631
630
|
``cache_dir`` is the cache object root whose symlinks count as directory content.
|
|
632
631
|
"""
|
|
633
632
|
if path.is_dir():
|
|
634
|
-
|
|
635
|
-
return DirHash(hash=tree_hash, manifest=manifest)
|
|
633
|
+
return cache.hash_directory(path, state_db, cache_dir=cache_dir)
|
|
636
634
|
file_hash, _ = cache.hash_file(path, state_db)
|
|
637
635
|
return FileHash(hash=file_hash)
|
|
638
636
|
|
|
@@ -1063,12 +1061,9 @@ def _hash_output_paths(
|
|
|
1063
1061
|
if not path.exists():
|
|
1064
1062
|
raise exceptions.OutputMissingError(f"Stage did not produce output: {path}")
|
|
1065
1063
|
if path.is_dir():
|
|
1066
|
-
|
|
1064
|
+
output_hashes[types.identity_key(ref.identity)] = cache.hash_directory(
|
|
1067
1065
|
path, state_db, cache_dir=cache_dir, warn_left_out_symlinks=True
|
|
1068
1066
|
)
|
|
1069
|
-
output_hashes[types.identity_key(ref.identity)] = DirHash(
|
|
1070
|
-
hash=tree_hash, manifest=manifest
|
|
1071
|
-
)
|
|
1072
1067
|
else:
|
|
1073
1068
|
file_hash, _ = cache.hash_file(path, state_db)
|
|
1074
1069
|
output_hashes[types.identity_key(ref.identity)] = FileHash(hash=file_hash)
|
|
@@ -1290,8 +1285,7 @@ def hash_dependencies(
|
|
|
1290
1285
|
path = pathlib.Path(normalized_path)
|
|
1291
1286
|
try:
|
|
1292
1287
|
if path.is_dir():
|
|
1293
|
-
|
|
1294
|
-
hashes[dep] = DirHash(hash=tree_hash, manifest=manifest)
|
|
1288
|
+
hashes[dep] = cache.hash_directory(path, state_db)
|
|
1295
1289
|
else:
|
|
1296
1290
|
file_hash, fs_stat = cache.hash_file(path, state_db)
|
|
1297
1291
|
hashes[dep] = FileHash(hash=file_hash)
|
|
@@ -14,7 +14,7 @@ import pydantic
|
|
|
14
14
|
|
|
15
15
|
from pivot import parameters, project, registry, skip, types
|
|
16
16
|
from pivot.executor import worker
|
|
17
|
-
from pivot.storage import lock, state
|
|
17
|
+
from pivot.storage import lock, state, track
|
|
18
18
|
from pivot.storage import store as store_mod
|
|
19
19
|
from pivot.types import (
|
|
20
20
|
ArtifactIdentity,
|
|
@@ -30,8 +30,6 @@ if TYPE_CHECKING:
|
|
|
30
30
|
|
|
31
31
|
import pygtrie
|
|
32
32
|
|
|
33
|
-
from pivot.storage.track import PvtData
|
|
34
|
-
|
|
35
33
|
OUTPUTS_MISSING_REASON = "Outputs missing from cache"
|
|
36
34
|
"""Reason for a stage whose lock is current but whose cacheable outputs are absent locally.
|
|
37
35
|
|
|
@@ -105,7 +103,7 @@ def _find_tracked_ancestor(dep: Path, tracked_trie: pygtrie.Trie[str]) -> Path |
|
|
|
105
103
|
|
|
106
104
|
def _find_tracked_hash(
|
|
107
105
|
dep: Path,
|
|
108
|
-
tracked_files: dict[str, PvtData],
|
|
106
|
+
tracked_files: dict[str, track.PvtData],
|
|
109
107
|
tracked_trie: pygtrie.Trie[str],
|
|
110
108
|
) -> HashInfo | None:
|
|
111
109
|
"""Find hash for dep from tracked files data.
|
|
@@ -121,9 +119,7 @@ def _find_tracked_hash(
|
|
|
121
119
|
|
|
122
120
|
# Exact match - use top-level hash
|
|
123
121
|
if dep == tracked_path:
|
|
124
|
-
|
|
125
|
-
return {"hash": pvt_data["hash"], "manifest": pvt_data["manifest"]}
|
|
126
|
-
return {"hash": pvt_data["hash"]}
|
|
122
|
+
return track.pvt_to_hash_info(pvt_data)
|
|
127
123
|
|
|
128
124
|
# Nested path - find in manifest
|
|
129
125
|
if "manifest" not in pvt_data:
|
|
@@ -155,7 +151,7 @@ def get_stage_explanation(
|
|
|
155
151
|
lock_data: LockData | None,
|
|
156
152
|
force: bool = False,
|
|
157
153
|
allow_missing: bool = False,
|
|
158
|
-
tracked_files: dict[str, PvtData] | None = None,
|
|
154
|
+
tracked_files: dict[str, track.PvtData] | None = None,
|
|
159
155
|
tracked_trie: pygtrie.Trie[str] | None = None,
|
|
160
156
|
producer_hashes: dict[ArtifactIdentity, HashInfo] | None = None,
|
|
161
157
|
) -> StageExplanation:
|
|
@@ -2,7 +2,6 @@ from __future__ import annotations
|
|
|
2
2
|
|
|
3
3
|
import asyncio
|
|
4
4
|
import contextlib
|
|
5
|
-
import errno
|
|
6
5
|
import logging
|
|
7
6
|
import os
|
|
8
7
|
import re
|
|
@@ -13,6 +12,7 @@ import xxhash
|
|
|
13
12
|
|
|
14
13
|
from pivot import config, exceptions, metrics
|
|
15
14
|
from pivot.remote import config as remote_config
|
|
15
|
+
from pivot.storage import cache
|
|
16
16
|
from pivot.types import TransferResult
|
|
17
17
|
|
|
18
18
|
if TYPE_CHECKING:
|
|
@@ -38,9 +38,6 @@ BULK_EXISTS_LIST_THRESHOLD = 50
|
|
|
38
38
|
|
|
39
39
|
_HEX_PATTERN = re.compile(r"^[a-f0-9]{16}$")
|
|
40
40
|
|
|
41
|
-
# link() errors on filesystems without hard links (vfat, exFAT, many SMB and FUSE mounts).
|
|
42
|
-
_NO_HARDLINK_ERRNO = frozenset({errno.EPERM, errno.ENOTSUP, errno.EOPNOTSUPP})
|
|
43
|
-
|
|
44
41
|
# Cached S3 config to avoid re-parsing config on every call.
|
|
45
42
|
# Safe without locking - asyncio is single-threaded, worst case race just
|
|
46
43
|
# creates duplicate config (harmless).
|
|
@@ -230,66 +227,61 @@ async def _atomic_download(
|
|
|
230
227
|
authenticate remote content.
|
|
231
228
|
|
|
232
229
|
Args:
|
|
233
|
-
overwrite: Replace an existing file at local_path. When False,
|
|
234
|
-
|
|
230
|
+
overwrite: Replace an existing file at local_path. When False, local_path is a cache
|
|
231
|
+
object: an existing one is kept (another process may be hardlinking it), and the
|
|
232
|
+
download is staged so a killed one is swept (``cache.staged_cache_file``).
|
|
235
233
|
readonly: If True, set file permissions to 0o444 (for cache files).
|
|
236
234
|
"""
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
f"Remote cache file not found: {bucket}/{key}"
|
|
248
|
-
) from exc
|
|
249
|
-
if "AccessDenied" in error_str or "403" in error_str:
|
|
250
|
-
raise exceptions.RemoteError(
|
|
251
|
-
f"Access denied to {bucket}/{key}. Check AWS credentials."
|
|
252
|
-
) from exc
|
|
253
|
-
raise exceptions.RemoteError(f"Failed to download from S3: {exc}") from exc
|
|
254
|
-
actual_hash = await _stream_download_to_fd(response, fd)
|
|
255
|
-
os.close(fd)
|
|
256
|
-
fd = -1 # Mark as closed
|
|
257
|
-
if actual_hash != expected_hash:
|
|
258
|
-
raise exceptions.RemoteError(
|
|
259
|
-
f"Hash mismatch: expected {expected_hash}, got {actual_hash} "
|
|
260
|
-
+ f"for {bucket}/{key}. Delete {bucket}/{key} from the remote, then run "
|
|
261
|
-
+ "`pivot push` on a machine whose cache has a good copy and has not run "
|
|
262
|
-
+ "`pivot push`, `pivot pull`, `pivot fetch`, or `pivot status --remote` "
|
|
263
|
-
+ "against this remote for this object. Pivot records hashes those commands "
|
|
264
|
-
+ "upload, download, or find, then skips recorded hashes. If every machine "
|
|
265
|
-
+ "with a good copy has recorded it, delete `.pivot/state.lmdb/` before "
|
|
266
|
-
+ "pushing; this also clears run history and local caches."
|
|
267
|
-
)
|
|
235
|
+
with contextlib.ExitStack() as stack:
|
|
236
|
+
if overwrite:
|
|
237
|
+
local_path.parent.mkdir(parents=True, exist_ok=True)
|
|
238
|
+
fd, name = tempfile.mkstemp(dir=local_path.parent, prefix=".pivot_download_")
|
|
239
|
+
os.close(fd)
|
|
240
|
+
tmp_path = local_path.with_name(os.path.basename(name))
|
|
241
|
+
stack.callback(tmp_path.unlink, missing_ok=True)
|
|
242
|
+
else:
|
|
243
|
+
tmp_path = stack.enter_context(cache.staged_cache_file(local_path))
|
|
244
|
+
await _download_verified(s3, bucket, key, tmp_path, expected_hash)
|
|
268
245
|
if readonly:
|
|
269
246
|
os.chmod(tmp_path, 0o444)
|
|
270
247
|
if overwrite:
|
|
271
248
|
os.replace(tmp_path, local_path)
|
|
272
|
-
move_succeeded = True
|
|
273
249
|
else:
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
250
|
+
_ = cache.publish_cache_object(tmp_path, local_path)
|
|
251
|
+
|
|
252
|
+
|
|
253
|
+
async def _download_verified(
|
|
254
|
+
s3: S3Client, bucket: str, key: str, dest: Path, expected_hash: str
|
|
255
|
+
) -> None:
|
|
256
|
+
"""Stream bucket/key into the existing file ``dest`` and check its xxh64 digest."""
|
|
257
|
+
try:
|
|
258
|
+
response = await s3.get_object(Bucket=bucket, Key=key)
|
|
259
|
+
except Exception as exc:
|
|
260
|
+
error_str = str(exc)
|
|
261
|
+
if "NoSuchKey" in error_str or "404" in error_str:
|
|
262
|
+
raise exceptions.RemoteError(f"Remote cache file not found: {bucket}/{key}") from exc
|
|
263
|
+
if "AccessDenied" in error_str or "403" in error_str:
|
|
264
|
+
raise exceptions.RemoteError(
|
|
265
|
+
f"Access denied to {bucket}/{key}. Check AWS credentials."
|
|
266
|
+
) from exc
|
|
267
|
+
raise exceptions.RemoteError(f"Failed to download from S3: {exc}") from exc
|
|
268
|
+
fd = os.open(dest, os.O_WRONLY | os.O_TRUNC)
|
|
269
|
+
try:
|
|
270
|
+
actual_hash = await _stream_download_to_fd(response, fd)
|
|
288
271
|
finally:
|
|
289
|
-
|
|
272
|
+
with contextlib.suppress(OSError):
|
|
290
273
|
os.close(fd)
|
|
291
|
-
|
|
292
|
-
|
|
274
|
+
if actual_hash != expected_hash:
|
|
275
|
+
raise exceptions.RemoteError(
|
|
276
|
+
f"Hash mismatch: expected {expected_hash}, got {actual_hash} "
|
|
277
|
+
+ f"for {bucket}/{key}. Delete {bucket}/{key} from the remote, then run "
|
|
278
|
+
+ "`pivot push` on a machine whose cache has a good copy and has not run "
|
|
279
|
+
+ "`pivot push`, `pivot pull`, `pivot fetch`, or `pivot status --remote` "
|
|
280
|
+
+ "against this remote for this object. Pivot records hashes those commands "
|
|
281
|
+
+ "upload, download, or find, then skips recorded hashes. If every machine "
|
|
282
|
+
+ "with a good copy has recorded it, delete `.pivot/state.lmdb/` before "
|
|
283
|
+
+ "pushing; this also clears run history and local caches."
|
|
284
|
+
)
|
|
293
285
|
|
|
294
286
|
|
|
295
287
|
async def _stream_upload(
|
|
@@ -11,7 +11,7 @@ from pivot.remote import config as remote_config
|
|
|
11
11
|
from pivot.remote import storage as remote_mod
|
|
12
12
|
from pivot.storage import cache, lock, track
|
|
13
13
|
from pivot.storage import store as store_mod
|
|
14
|
-
from pivot.types import
|
|
14
|
+
from pivot.types import HashInfo, RemoteStatus, TransferSummary, is_dir_hash
|
|
15
15
|
|
|
16
16
|
if TYPE_CHECKING:
|
|
17
17
|
from collections.abc import Callable
|
|
@@ -175,9 +175,7 @@ def _get_file_hash_from_pvt(rel_path: str, proj_root: pathlib.Path) -> HashInfo
|
|
|
175
175
|
if track_data is None:
|
|
176
176
|
return None
|
|
177
177
|
|
|
178
|
-
|
|
179
|
-
return DirHash(hash=track_data["hash"], manifest=track_data["manifest"])
|
|
180
|
-
return FileHash(hash=track_data["hash"])
|
|
178
|
+
return track.pvt_to_hash_info(track_data)
|
|
181
179
|
|
|
182
180
|
|
|
183
181
|
def _get_tracked_hashes(proj_root: pathlib.Path) -> set[str]:
|
|
@@ -64,6 +64,7 @@ class OutputHashEntry(TypedDict):
|
|
|
64
64
|
path: str
|
|
65
65
|
hash: str
|
|
66
66
|
manifest: NotRequired[list[DirManifestEntry]]
|
|
67
|
+
empty_dirs: NotRequired[list[str]]
|
|
67
68
|
|
|
68
69
|
|
|
69
70
|
def output_hash_to_entry(path: str, oh: HashInfo) -> OutputHashEntry:
|
|
@@ -71,6 +72,8 @@ def output_hash_to_entry(path: str, oh: HashInfo) -> OutputHashEntry:
|
|
|
71
72
|
entry = OutputHashEntry(path=path, hash=oh["hash"])
|
|
72
73
|
if is_dir_hash(oh):
|
|
73
74
|
entry["manifest"] = oh["manifest"]
|
|
75
|
+
if "empty_dirs" in oh:
|
|
76
|
+
entry["empty_dirs"] = oh["empty_dirs"]
|
|
74
77
|
return entry
|
|
75
78
|
|
|
76
79
|
|
|
@@ -78,7 +81,10 @@ def entry_to_output_hash(entry: OutputHashEntry) -> FileHash | DirHash:
|
|
|
78
81
|
"""Convert serialized OutputHashEntry back to internal HashInfo."""
|
|
79
82
|
# Null check guards against corrupted JSON where manifest could be null
|
|
80
83
|
if "manifest" in entry and entry["manifest"] is not None: # pyright: ignore[reportUnnecessaryComparison]
|
|
81
|
-
|
|
84
|
+
dir_hash = DirHash(hash=entry["hash"], manifest=entry["manifest"])
|
|
85
|
+
if "empty_dirs" in entry and entry["empty_dirs"]:
|
|
86
|
+
dir_hash["empty_dirs"] = entry["empty_dirs"]
|
|
87
|
+
return dir_hash
|
|
82
88
|
return FileHash(hash=entry["hash"])
|
|
83
89
|
|
|
84
90
|
|
|
@@ -356,7 +356,7 @@ def get_tracked_files_status(
|
|
|
356
356
|
|
|
357
357
|
try:
|
|
358
358
|
if path.is_dir():
|
|
359
|
-
current_hash
|
|
359
|
+
current_hash = cache.hash_directory(path, state_db, cache_dir=cache_dir)["hash"]
|
|
360
360
|
else:
|
|
361
361
|
current_hash, _ = cache.hash_file(path, state_db)
|
|
362
362
|
except FileNotFoundError:
|
|
@@ -66,6 +66,44 @@ When multiple `pivot` processes share the same StateDB:
|
|
|
66
66
|
temporary directory and swap it. Both run under the target's restore lock, and each
|
|
67
67
|
first removes the token-tagged entries a killed restore of the same target left.
|
|
68
68
|
|
|
69
|
+
## Publishing Cache Objects
|
|
70
|
+
|
|
71
|
+
- Every writer of a file object publishes it with `cache.publish_cache_object`, which links it
|
|
72
|
+
into place and keeps an object already there. A published object never changes, mode
|
|
73
|
+
included: a hardlink checkout looks the name up, then links the inode it found, and once a
|
|
74
|
+
rename takes that inode's last name Linux fails the link with ENOENT, though the name never
|
|
75
|
+
went missing; a symlink checkout reads its executable bit through the name. A directory
|
|
76
|
+
entry whose executable bit the object does not carry is checked out as a copy.
|
|
77
|
+
- On a filesystem without hard links, `publish_cache_object` renames, and only when no object is
|
|
78
|
+
there yet. Nothing there can hardlink an object, but the check and the rename are two steps:
|
|
79
|
+
two writers that both find the name free both rename, the later object wins, and a symlink
|
|
80
|
+
checkout reads the later object's mode.
|
|
81
|
+
- Every temporary entry in a prefix directory (a file staged for an object, a download, a tree
|
|
82
|
+
being copied) is made through `cache.staged_cache_file` or `_staging_in`, named
|
|
83
|
+
`.pivot_tmp_*`, while the process holds a shared `flock` on the directory's
|
|
84
|
+
`.pivot_cache.lock`. A process that gets the exclusive lock first (it never waits for it), at
|
|
85
|
+
most once per `_CACHE_SWEEP_INTERVAL` (the lock's mtime records the last sweep), removes every
|
|
86
|
+
`.pivot_tmp_*` entry: none has a live owner. An entry it cannot remove is logged and left.
|
|
87
|
+
Never judge a temporary entry abandoned by host and pid: a process in another PID namespace,
|
|
88
|
+
or on another host sharing the cache, looks dead.
|
|
89
|
+
- Sharing a cache across hosts needs `flock` that is coherent across them (NFSv4, or NFSv3 with
|
|
90
|
+
NLM). With host-local locks (NFS `nolock` or `local_lock=flock`/`all`, Lustre `localflock`)
|
|
91
|
+
the locks here exclude nothing across hosts: one host's sweep can remove another host's live
|
|
92
|
+
temporary entry, and two hosts can repair one tree at once. A publisher checks its tree before
|
|
93
|
+
marking it, so a copy that lost files is not marked. On a filesystem that refuses `flock()`
|
|
94
|
+
(Lustre without `flock`: ENOSYS; NFS when its lock manager does not answer: ENOLCK) nothing
|
|
95
|
+
sweeps and nothing repairs a tree, and staging goes on without the shared lock.
|
|
96
|
+
- A lock file in the cache is created writable by whoever can write its directory, since NFS
|
|
97
|
+
takes an exclusive lock only on a file opened for writing; one this user cannot write is
|
|
98
|
+
opened read-only. Lock files are never deleted.
|
|
99
|
+
- A whole-directory SYMLINK tree is built under a temporary name beside its cache name, renamed
|
|
100
|
+
into place, checked, then marked (`<tree>.verified`), because a restore symlinks to a tree as
|
|
101
|
+
soon as its name exists. A restore or commit checks an unmarked tree (exact files, sizes,
|
|
102
|
+
executable bits and directories) under the tree's exclusive `<tree>.lock`, repairs it in place
|
|
103
|
+
from the files' own cache objects, then marks it. A tree it cannot repair is not used: a
|
|
104
|
+
restore goes file by file, a commit fails. `<tree>.layout-checked`, the mark pivot 0.3.4 also
|
|
105
|
+
gave partial trees, is not trusted.
|
|
106
|
+
|
|
69
107
|
## Path Storage
|
|
70
108
|
|
|
71
109
|
All paths stored in the database must be **relative** to ensure portability across machines and correct cache behavior.
|