pivot 0.3.3__tar.gz → 0.3.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (107) hide show
  1. {pivot-0.3.3 → pivot-0.3.5}/PKG-INFO +1 -1
  2. {pivot-0.3.3 → pivot-0.3.5}/pyproject.toml +1 -1
  3. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/checkout.py +9 -5
  4. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/track.py +5 -6
  5. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/exceptions.py +6 -4
  6. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/core.py +1 -1
  7. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/worker.py +3 -9
  8. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/explain.py +4 -8
  9. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/storage.py +47 -55
  10. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/sync.py +2 -4
  11. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/run_history.py +7 -1
  12. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/status.py +1 -1
  13. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/AGENTS.md +38 -0
  14. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/cache.py +592 -78
  15. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/lock.py +38 -15
  16. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/restore.py +10 -24
  17. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/store.py +5 -9
  18. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/track.py +23 -4
  19. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/types.py +13 -3
  20. {pivot-0.3.3 → pivot-0.3.5}/LICENSE +0 -0
  21. {pivot-0.3.3 → pivot-0.3.5}/README.md +0 -0
  22. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/__init__.py +0 -0
  23. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/ast_utils.py +0 -0
  24. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/AGENTS.md +0 -0
  25. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/CLAUDE.md +0 -0
  26. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/__init__.py +0 -0
  27. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/_run_common.py +0 -0
  28. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/check_ignore.py +0 -0
  29. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/commit.py +0 -0
  30. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/completion.py +0 -0
  31. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/config.py +0 -0
  32. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/console.py +0 -0
  33. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/dag.py +0 -0
  34. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/data.py +0 -0
  35. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/decorators.py +0 -0
  36. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/doctor.py +0 -0
  37. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/errors.py +0 -0
  38. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/fingerprint.py +0 -0
  39. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/helpers.py +0 -0
  40. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/history.py +0 -0
  41. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/import_cmd.py +0 -0
  42. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/init.py +0 -0
  43. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/list.py +0 -0
  44. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/metrics.py +0 -0
  45. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/params.py +0 -0
  46. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/plots.py +0 -0
  47. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/remote.py +0 -0
  48. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/repro.py +0 -0
  49. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/run.py +0 -0
  50. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/status.py +0 -0
  51. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/targets.py +0 -0
  52. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/update.py +0 -0
  53. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/cli/verify.py +0 -0
  54. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/compose.py +0 -0
  55. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/config/__init__.py +0 -0
  56. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/config/io.py +0 -0
  57. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/config/models.py +0 -0
  58. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/dag/__init__.py +0 -0
  59. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/dag/render.py +0 -0
  60. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/decorators.py +0 -0
  61. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/discovery.py +0 -0
  62. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/__init__.py +0 -0
  63. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/agent_rpc.py +0 -0
  64. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/engine.py +0 -0
  65. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/graph.py +0 -0
  66. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/scheduler.py +0 -0
  67. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/sinks.py +0 -0
  68. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/sources.py +0 -0
  69. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/types.py +0 -0
  70. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/watch.py +0 -0
  71. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/engine/worker_pool.py +0 -0
  72. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/AGENTS.md +0 -0
  73. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/CLAUDE.md +0 -0
  74. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/__init__.py +0 -0
  75. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/executor/commit.py +0 -0
  76. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/fingerprint.py +0 -0
  77. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/git.py +0 -0
  78. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/ignore.py +0 -0
  79. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/import_artifact.py +0 -0
  80. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/loaders.py +0 -0
  81. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/metrics.py +0 -0
  82. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/names.py +0 -0
  83. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/outputs.py +0 -0
  84. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/parameters.py +0 -0
  85. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/path_policy.py +0 -0
  86. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/project.py +0 -0
  87. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/py.typed +0 -0
  88. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/registry.py +0 -0
  89. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/AGENTS.md +0 -0
  90. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/__init__.py +0 -0
  91. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/config.py +0 -0
  92. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/git_archive.py +0 -0
  93. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/remote/github.py +0 -0
  94. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/__init__.py +0 -0
  95. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/common.py +0 -0
  96. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/data.py +0 -0
  97. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/metrics.py +0 -0
  98. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/params.py +0 -0
  99. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/show/plots.py +0 -0
  100. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/skip.py +0 -0
  101. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/stage_def.py +0 -0
  102. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/CLAUDE.md +0 -0
  103. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/__init__.py +0 -0
  104. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/artifact_lock.py +0 -0
  105. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/presentation.py +0 -0
  106. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/storage/state.py +0 -0
  107. {pivot-0.3.3 → pivot-0.3.5}/src/pivot/yaml_config.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: pivot
3
- Version: 0.3.3
3
+ Version: 0.3.5
4
4
  Summary: High-performance Python pipeline tool with automatic code change detection
5
5
  Author: Sami Jawhar
6
6
  Author-email: Sami Jawhar <sami@metr.org>
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "pivot"
3
- version = "0.3.3"
3
+ version = "0.3.5"
4
4
  description = "High-performance Python pipeline tool with automatic code change detection"
5
5
  requires-python = ">=3.13,<3.14"
6
6
  readme = "README.md"
@@ -74,11 +74,15 @@ def _on_disk_matches_expected(
74
74
  if is_dir_hash(expected_hash):
75
75
  if not path.is_dir():
76
76
  return False
77
- actual_hash, _ = cache.hash_directory(path, cache_dir=cache_dir)
78
- else:
79
- if not path.is_file():
80
- return False
81
- actual_hash, _ = cache.hash_file(path)
77
+ actual = cache.hash_directory(path, cache_dir=cache_dir)
78
+ # A hash pivot 0.3.3 recorded lists no empty directories, and repro still honors it.
79
+ # When only empty directories differ, checkout would change no file, so it matches.
80
+ return actual["hash"] == expected_hash["hash"] or (
81
+ "empty_dirs" not in expected_hash and actual["manifest"] == expected_hash["manifest"]
82
+ )
83
+ if not path.is_file():
84
+ return False
85
+ actual_hash, _ = cache.hash_file(path)
82
86
  return actual_hash == expected_hash["hash"]
83
87
 
84
88
 
@@ -154,16 +154,13 @@ def _track_single_path(
154
154
 
155
155
  # Hash and cache
156
156
  if path.is_dir():
157
- tree_hash, manifest = cache.hash_directory(
157
+ dir_hash = cache.hash_directory(
158
158
  path, cache_dir=cache_dir, warn_left_out_symlinks=warn_left_out_symlinks
159
159
  )
160
+ manifest = dir_hash["manifest"]
160
161
  total_size = sum(e["size"] for e in manifest)
161
162
  num_files = len(manifest)
162
163
 
163
- # copy_to_cache is idempotent and already checks existence itself; letting it see
164
- # every entry (not just ones missing from the cache) is what lets it republish an
165
- # object `pivot track --force` finds cached non-executable by an older pivot or a
166
- # cold `pivot pull`, when this entry's bytes should be executable.
167
164
  for entry in manifest:
168
165
  file_path = path / entry["relpath"]
169
166
  file_cache_path = cache.get_cache_path(cache_dir, entry["hash"])
@@ -171,11 +168,13 @@ def _track_single_path(
171
168
 
172
169
  pvt_data: track_mod.PvtData = {
173
170
  "path": path.name,
174
- "hash": tree_hash,
171
+ "hash": dir_hash["hash"],
175
172
  "size": total_size,
176
173
  "num_files": num_files,
177
174
  "manifest": manifest,
178
175
  }
176
+ if "empty_dirs" in dir_hash:
177
+ pvt_data["empty_dirs"] = dir_hash["empty_dirs"]
179
178
  else:
180
179
  file_hash, _ = cache.hash_file(path)
181
180
  file_size = path.stat().st_size
@@ -221,6 +221,8 @@ class CacheModeCollisionError(CacheError):
221
221
  This happens when two files with identical content disagree on their executable bit (the
222
222
  cache key is content-only) or when a cache object was populated by a path that does not
223
223
  know a file's intended executable bit (e.g. `pivot pull`, which downloads by hash alone).
224
+ Pivot never changes a published cache object, so only a private copy, or the object deleted
225
+ and published again from bytes that carry the right bit, can give the file its bit.
224
226
  """
225
227
 
226
228
  @override
@@ -228,10 +230,10 @@ class CacheModeCollisionError(CacheError):
228
230
  return (
229
231
  "Add CheckoutMode.COPY to checkout_modes (cache: checkout_mode: [..., copy] in "
230
232
  ".pivot/config.yaml) so checkout can fall back to a private, correctly-permissioned "
231
- "copy instead of sharing the cache object's own permissions; or, if you have the "
232
- "source bytes in hand, re-running what produced them (`pivot repro --force "
233
- "<stage>`, `pivot track --force <path>`) republishes the cache object with the "
234
- "right bit"
233
+ "copy instead of sharing the cache object's own permissions; or delete the cache "
234
+ "object the error names, then re-run what produced the file (`pivot repro --force "
235
+ "<stage>`, `pivot track --force <path>`) to publish it with the right bit; until it "
236
+ "is published again, any workspace file symlinked to it is broken"
235
237
  )
236
238
 
237
239
 
@@ -409,7 +409,7 @@ def verify_tracked_files(project_root: pathlib.Path, checkout_missing: bool = Fa
409
409
  if path.is_file():
410
410
  current_hash, _ = cache.hash_file(path, state_db)
411
411
  elif path.is_dir():
412
- current_hash, _ = cache.hash_directory(path, state_db, cache_dir=cache_dir)
412
+ current_hash = cache.hash_directory(path, state_db, cache_dir=cache_dir)["hash"]
413
413
  else:
414
414
  # Path doesn't exist
415
415
  raise FileNotFoundError(data_path)
@@ -39,7 +39,6 @@ from pivot.storage import artifact_lock, cache, lock, state
39
39
  from pivot.storage import store as store_mod
40
40
  from pivot.types import (
41
41
  DeferredWrites,
42
- DirHash,
43
42
  FileHash,
44
43
  HashInfo,
45
44
  LockData,
@@ -631,8 +630,7 @@ def hash_output(
631
630
  ``cache_dir`` is the cache object root whose symlinks count as directory content.
632
631
  """
633
632
  if path.is_dir():
634
- tree_hash, manifest = cache.hash_directory(path, state_db, cache_dir=cache_dir)
635
- return DirHash(hash=tree_hash, manifest=manifest)
633
+ return cache.hash_directory(path, state_db, cache_dir=cache_dir)
636
634
  file_hash, _ = cache.hash_file(path, state_db)
637
635
  return FileHash(hash=file_hash)
638
636
 
@@ -1063,12 +1061,9 @@ def _hash_output_paths(
1063
1061
  if not path.exists():
1064
1062
  raise exceptions.OutputMissingError(f"Stage did not produce output: {path}")
1065
1063
  if path.is_dir():
1066
- tree_hash, manifest = cache.hash_directory(
1064
+ output_hashes[types.identity_key(ref.identity)] = cache.hash_directory(
1067
1065
  path, state_db, cache_dir=cache_dir, warn_left_out_symlinks=True
1068
1066
  )
1069
- output_hashes[types.identity_key(ref.identity)] = DirHash(
1070
- hash=tree_hash, manifest=manifest
1071
- )
1072
1067
  else:
1073
1068
  file_hash, _ = cache.hash_file(path, state_db)
1074
1069
  output_hashes[types.identity_key(ref.identity)] = FileHash(hash=file_hash)
@@ -1290,8 +1285,7 @@ def hash_dependencies(
1290
1285
  path = pathlib.Path(normalized_path)
1291
1286
  try:
1292
1287
  if path.is_dir():
1293
- tree_hash, manifest = cache.hash_directory(path, state_db)
1294
- hashes[dep] = DirHash(hash=tree_hash, manifest=manifest)
1288
+ hashes[dep] = cache.hash_directory(path, state_db)
1295
1289
  else:
1296
1290
  file_hash, fs_stat = cache.hash_file(path, state_db)
1297
1291
  hashes[dep] = FileHash(hash=file_hash)
@@ -14,7 +14,7 @@ import pydantic
14
14
 
15
15
  from pivot import parameters, project, registry, skip, types
16
16
  from pivot.executor import worker
17
- from pivot.storage import lock, state
17
+ from pivot.storage import lock, state, track
18
18
  from pivot.storage import store as store_mod
19
19
  from pivot.types import (
20
20
  ArtifactIdentity,
@@ -30,8 +30,6 @@ if TYPE_CHECKING:
30
30
 
31
31
  import pygtrie
32
32
 
33
- from pivot.storage.track import PvtData
34
-
35
33
  OUTPUTS_MISSING_REASON = "Outputs missing from cache"
36
34
  """Reason for a stage whose lock is current but whose cacheable outputs are absent locally.
37
35
 
@@ -105,7 +103,7 @@ def _find_tracked_ancestor(dep: Path, tracked_trie: pygtrie.Trie[str]) -> Path |
105
103
 
106
104
  def _find_tracked_hash(
107
105
  dep: Path,
108
- tracked_files: dict[str, PvtData],
106
+ tracked_files: dict[str, track.PvtData],
109
107
  tracked_trie: pygtrie.Trie[str],
110
108
  ) -> HashInfo | None:
111
109
  """Find hash for dep from tracked files data.
@@ -121,9 +119,7 @@ def _find_tracked_hash(
121
119
 
122
120
  # Exact match - use top-level hash
123
121
  if dep == tracked_path:
124
- if "manifest" in pvt_data:
125
- return {"hash": pvt_data["hash"], "manifest": pvt_data["manifest"]}
126
- return {"hash": pvt_data["hash"]}
122
+ return track.pvt_to_hash_info(pvt_data)
127
123
 
128
124
  # Nested path - find in manifest
129
125
  if "manifest" not in pvt_data:
@@ -155,7 +151,7 @@ def get_stage_explanation(
155
151
  lock_data: LockData | None,
156
152
  force: bool = False,
157
153
  allow_missing: bool = False,
158
- tracked_files: dict[str, PvtData] | None = None,
154
+ tracked_files: dict[str, track.PvtData] | None = None,
159
155
  tracked_trie: pygtrie.Trie[str] | None = None,
160
156
  producer_hashes: dict[ArtifactIdentity, HashInfo] | None = None,
161
157
  ) -> StageExplanation:
@@ -2,7 +2,6 @@ from __future__ import annotations
2
2
 
3
3
  import asyncio
4
4
  import contextlib
5
- import errno
6
5
  import logging
7
6
  import os
8
7
  import re
@@ -13,6 +12,7 @@ import xxhash
13
12
 
14
13
  from pivot import config, exceptions, metrics
15
14
  from pivot.remote import config as remote_config
15
+ from pivot.storage import cache
16
16
  from pivot.types import TransferResult
17
17
 
18
18
  if TYPE_CHECKING:
@@ -38,9 +38,6 @@ BULK_EXISTS_LIST_THRESHOLD = 50
38
38
 
39
39
  _HEX_PATTERN = re.compile(r"^[a-f0-9]{16}$")
40
40
 
41
- # link() errors on filesystems without hard links (vfat, exFAT, many SMB and FUSE mounts).
42
- _NO_HARDLINK_ERRNO = frozenset({errno.EPERM, errno.ENOTSUP, errno.EOPNOTSUPP})
43
-
44
41
  # Cached S3 config to avoid re-parsing config on every call.
45
42
  # Safe without locking - asyncio is single-threaded, worst case race just
46
43
  # creates duplicate config (harmless).
@@ -230,66 +227,61 @@ async def _atomic_download(
230
227
  authenticate remote content.
231
228
 
232
229
  Args:
233
- overwrite: Replace an existing file at local_path. When False, an existing file is
234
- kept: a content-addressed cache entry another process may be hardlinking.
230
+ overwrite: Replace an existing file at local_path. When False, local_path is a cache
231
+ object: an existing one is kept (another process may be hardlinking it), and the
232
+ download is staged so a killed one is swept (``cache.staged_cache_file``).
235
233
  readonly: If True, set file permissions to 0o444 (for cache files).
236
234
  """
237
- local_path.parent.mkdir(parents=True, exist_ok=True)
238
- fd, tmp_path = tempfile.mkstemp(dir=local_path.parent, prefix=".pivot_download_")
239
- move_succeeded = False
240
- try:
241
- try:
242
- response = await s3.get_object(Bucket=bucket, Key=key)
243
- except Exception as exc:
244
- error_str = str(exc)
245
- if "NoSuchKey" in error_str or "404" in error_str:
246
- raise exceptions.RemoteError(
247
- f"Remote cache file not found: {bucket}/{key}"
248
- ) from exc
249
- if "AccessDenied" in error_str or "403" in error_str:
250
- raise exceptions.RemoteError(
251
- f"Access denied to {bucket}/{key}. Check AWS credentials."
252
- ) from exc
253
- raise exceptions.RemoteError(f"Failed to download from S3: {exc}") from exc
254
- actual_hash = await _stream_download_to_fd(response, fd)
255
- os.close(fd)
256
- fd = -1 # Mark as closed
257
- if actual_hash != expected_hash:
258
- raise exceptions.RemoteError(
259
- f"Hash mismatch: expected {expected_hash}, got {actual_hash} "
260
- + f"for {bucket}/{key}. Delete {bucket}/{key} from the remote, then run "
261
- + "`pivot push` on a machine whose cache has a good copy and has not run "
262
- + "`pivot push`, `pivot pull`, `pivot fetch`, or `pivot status --remote` "
263
- + "against this remote for this object. Pivot records hashes those commands "
264
- + "upload, download, or find, then skips recorded hashes. If every machine "
265
- + "with a good copy has recorded it, delete `.pivot/state.lmdb/` before "
266
- + "pushing; this also clears run history and local caches."
267
- )
235
+ with contextlib.ExitStack() as stack:
236
+ if overwrite:
237
+ local_path.parent.mkdir(parents=True, exist_ok=True)
238
+ fd, name = tempfile.mkstemp(dir=local_path.parent, prefix=".pivot_download_")
239
+ os.close(fd)
240
+ tmp_path = local_path.with_name(os.path.basename(name))
241
+ stack.callback(tmp_path.unlink, missing_ok=True)
242
+ else:
243
+ tmp_path = stack.enter_context(cache.staged_cache_file(local_path))
244
+ await _download_verified(s3, bucket, key, tmp_path, expected_hash)
268
245
  if readonly:
269
246
  os.chmod(tmp_path, 0o444)
270
247
  if overwrite:
271
248
  os.replace(tmp_path, local_path)
272
- move_succeeded = True
273
249
  else:
274
- # link() refuses an existing name, so a concurrent reader never sees the
275
- # entry disappear. The temp file is removed below either way.
276
- try:
277
- os.link(tmp_path, local_path)
278
- except FileExistsError:
279
- pass
280
- except OSError as exc:
281
- if exc.errno not in _NO_HARDLINK_ERRNO:
282
- raise
283
- # No hard links here, so nothing can be hardlinking the entry either:
284
- # rename unless another process already promoted it.
285
- if not os.path.exists(local_path):
286
- os.replace(tmp_path, local_path)
287
- move_succeeded = True
250
+ _ = cache.publish_cache_object(tmp_path, local_path)
251
+
252
+
253
+ async def _download_verified(
254
+ s3: S3Client, bucket: str, key: str, dest: Path, expected_hash: str
255
+ ) -> None:
256
+ """Stream bucket/key into the existing file ``dest`` and check its xxh64 digest."""
257
+ try:
258
+ response = await s3.get_object(Bucket=bucket, Key=key)
259
+ except Exception as exc:
260
+ error_str = str(exc)
261
+ if "NoSuchKey" in error_str or "404" in error_str:
262
+ raise exceptions.RemoteError(f"Remote cache file not found: {bucket}/{key}") from exc
263
+ if "AccessDenied" in error_str or "403" in error_str:
264
+ raise exceptions.RemoteError(
265
+ f"Access denied to {bucket}/{key}. Check AWS credentials."
266
+ ) from exc
267
+ raise exceptions.RemoteError(f"Failed to download from S3: {exc}") from exc
268
+ fd = os.open(dest, os.O_WRONLY | os.O_TRUNC)
269
+ try:
270
+ actual_hash = await _stream_download_to_fd(response, fd)
288
271
  finally:
289
- if fd >= 0:
272
+ with contextlib.suppress(OSError):
290
273
  os.close(fd)
291
- if not move_succeeded and os.path.exists(tmp_path):
292
- os.unlink(tmp_path)
274
+ if actual_hash != expected_hash:
275
+ raise exceptions.RemoteError(
276
+ f"Hash mismatch: expected {expected_hash}, got {actual_hash} "
277
+ + f"for {bucket}/{key}. Delete {bucket}/{key} from the remote, then run "
278
+ + "`pivot push` on a machine whose cache has a good copy and has not run "
279
+ + "`pivot push`, `pivot pull`, `pivot fetch`, or `pivot status --remote` "
280
+ + "against this remote for this object. Pivot records hashes those commands "
281
+ + "upload, download, or find, then skips recorded hashes. If every machine "
282
+ + "with a good copy has recorded it, delete `.pivot/state.lmdb/` before "
283
+ + "pushing; this also clears run history and local caches."
284
+ )
293
285
 
294
286
 
295
287
  async def _stream_upload(
@@ -11,7 +11,7 @@ from pivot.remote import config as remote_config
11
11
  from pivot.remote import storage as remote_mod
12
12
  from pivot.storage import cache, lock, track
13
13
  from pivot.storage import store as store_mod
14
- from pivot.types import DirHash, FileHash, HashInfo, RemoteStatus, TransferSummary, is_dir_hash
14
+ from pivot.types import HashInfo, RemoteStatus, TransferSummary, is_dir_hash
15
15
 
16
16
  if TYPE_CHECKING:
17
17
  from collections.abc import Callable
@@ -175,9 +175,7 @@ def _get_file_hash_from_pvt(rel_path: str, proj_root: pathlib.Path) -> HashInfo
175
175
  if track_data is None:
176
176
  return None
177
177
 
178
- if "manifest" in track_data:
179
- return DirHash(hash=track_data["hash"], manifest=track_data["manifest"])
180
- return FileHash(hash=track_data["hash"])
178
+ return track.pvt_to_hash_info(track_data)
181
179
 
182
180
 
183
181
  def _get_tracked_hashes(proj_root: pathlib.Path) -> set[str]:
@@ -64,6 +64,7 @@ class OutputHashEntry(TypedDict):
64
64
  path: str
65
65
  hash: str
66
66
  manifest: NotRequired[list[DirManifestEntry]]
67
+ empty_dirs: NotRequired[list[str]]
67
68
 
68
69
 
69
70
  def output_hash_to_entry(path: str, oh: HashInfo) -> OutputHashEntry:
@@ -71,6 +72,8 @@ def output_hash_to_entry(path: str, oh: HashInfo) -> OutputHashEntry:
71
72
  entry = OutputHashEntry(path=path, hash=oh["hash"])
72
73
  if is_dir_hash(oh):
73
74
  entry["manifest"] = oh["manifest"]
75
+ if "empty_dirs" in oh:
76
+ entry["empty_dirs"] = oh["empty_dirs"]
74
77
  return entry
75
78
 
76
79
 
@@ -78,7 +81,10 @@ def entry_to_output_hash(entry: OutputHashEntry) -> FileHash | DirHash:
78
81
  """Convert serialized OutputHashEntry back to internal HashInfo."""
79
82
  # Null check guards against corrupted JSON where manifest could be null
80
83
  if "manifest" in entry and entry["manifest"] is not None: # pyright: ignore[reportUnnecessaryComparison]
81
- return DirHash(hash=entry["hash"], manifest=entry["manifest"])
84
+ dir_hash = DirHash(hash=entry["hash"], manifest=entry["manifest"])
85
+ if "empty_dirs" in entry and entry["empty_dirs"]:
86
+ dir_hash["empty_dirs"] = entry["empty_dirs"]
87
+ return dir_hash
82
88
  return FileHash(hash=entry["hash"])
83
89
 
84
90
 
@@ -356,7 +356,7 @@ def get_tracked_files_status(
356
356
 
357
357
  try:
358
358
  if path.is_dir():
359
- current_hash, _ = cache.hash_directory(path, state_db, cache_dir=cache_dir)
359
+ current_hash = cache.hash_directory(path, state_db, cache_dir=cache_dir)["hash"]
360
360
  else:
361
361
  current_hash, _ = cache.hash_file(path, state_db)
362
362
  except FileNotFoundError:
@@ -66,6 +66,44 @@ When multiple `pivot` processes share the same StateDB:
66
66
  temporary directory and swap it. Both run under the target's restore lock, and each
67
67
  first removes the token-tagged entries a killed restore of the same target left.
68
68
 
69
+ ## Publishing Cache Objects
70
+
71
+ - Every writer of a file object publishes it with `cache.publish_cache_object`, which links it
72
+ into place and keeps an object already there. A published object never changes, mode
73
+ included: a hardlink checkout looks the name up, then links the inode it found, and once a
74
+ rename takes that inode's last name Linux fails the link with ENOENT, though the name never
75
+ went missing; a symlink checkout reads its executable bit through the name. A directory
76
+ entry whose executable bit the object does not carry is checked out as a copy.
77
+ - On a filesystem without hard links, `publish_cache_object` renames, and only when no object is
78
+ there yet. Nothing there can hardlink an object, but the check and the rename are two steps:
79
+ two writers that both find the name free both rename, the later object wins, and a symlink
80
+ checkout reads the later object's mode.
81
+ - Every temporary entry in a prefix directory (a file staged for an object, a download, a tree
82
+ being copied) is made through `cache.staged_cache_file` or `_staging_in`, named
83
+ `.pivot_tmp_*`, while the process holds a shared `flock` on the directory's
84
+ `.pivot_cache.lock`. A process that gets the exclusive lock first (it never waits for it), at
85
+ most once per `_CACHE_SWEEP_INTERVAL` (the lock's mtime records the last sweep), removes every
86
+ `.pivot_tmp_*` entry: none has a live owner. An entry it cannot remove is logged and left.
87
+ Never judge a temporary entry abandoned by host and pid: a process in another PID namespace,
88
+ or on another host sharing the cache, looks dead.
89
+ - Sharing a cache across hosts needs `flock` that is coherent across them (NFSv4, or NFSv3 with
90
+ NLM). With host-local locks (NFS `nolock` or `local_lock=flock`/`all`, Lustre `localflock`)
91
+ the locks here exclude nothing across hosts: one host's sweep can remove another host's live
92
+ temporary entry, and two hosts can repair one tree at once. A publisher checks its tree before
93
+ marking it, so a copy that lost files is not marked. On a filesystem that refuses `flock()`
94
+ (Lustre without `flock`: ENOSYS; NFS when its lock manager does not answer: ENOLCK) nothing
95
+ sweeps and nothing repairs a tree, and staging goes on without the shared lock.
96
+ - A lock file in the cache is created writable by whoever can write its directory, since NFS
97
+ takes an exclusive lock only on a file opened for writing; one this user cannot write is
98
+ opened read-only. Lock files are never deleted.
99
+ - A whole-directory SYMLINK tree is built under a temporary name beside its cache name, renamed
100
+ into place, checked, then marked (`<tree>.verified`), because a restore symlinks to a tree as
101
+ soon as its name exists. A restore or commit checks an unmarked tree (exact files, sizes,
102
+ executable bits and directories) under the tree's exclusive `<tree>.lock`, repairs it in place
103
+ from the files' own cache objects, then marks it. A tree it cannot repair is not used: a
104
+ restore goes file by file, a commit fails. `<tree>.layout-checked`, the mark pivot 0.3.4 also
105
+ gave partial trees, is not trusted.
106
+
69
107
  ## Path Storage
70
108
 
71
109
  All paths stored in the database must be **relative** to ensure portability across machines and correct cache behavior.