pivot 0.3.4__tar.gz → 0.3.5__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (107) hide show
  1. {pivot-0.3.4 → pivot-0.3.5}/PKG-INFO +1 -1
  2. {pivot-0.3.4 → pivot-0.3.5}/pyproject.toml +1 -1
  3. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/track.py +0 -4
  4. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/exceptions.py +6 -4
  5. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/remote/storage.py +47 -55
  6. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/AGENTS.md +38 -0
  7. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/cache.py +430 -80
  8. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/restore.py +6 -7
  9. {pivot-0.3.4 → pivot-0.3.5}/LICENSE +0 -0
  10. {pivot-0.3.4 → pivot-0.3.5}/README.md +0 -0
  11. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/__init__.py +0 -0
  12. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/ast_utils.py +0 -0
  13. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/AGENTS.md +0 -0
  14. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/CLAUDE.md +0 -0
  15. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/__init__.py +0 -0
  16. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/_run_common.py +0 -0
  17. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/check_ignore.py +0 -0
  18. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/checkout.py +0 -0
  19. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/commit.py +0 -0
  20. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/completion.py +0 -0
  21. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/config.py +0 -0
  22. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/console.py +0 -0
  23. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/dag.py +0 -0
  24. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/data.py +0 -0
  25. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/decorators.py +0 -0
  26. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/doctor.py +0 -0
  27. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/errors.py +0 -0
  28. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/fingerprint.py +0 -0
  29. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/helpers.py +0 -0
  30. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/history.py +0 -0
  31. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/import_cmd.py +0 -0
  32. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/init.py +0 -0
  33. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/list.py +0 -0
  34. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/metrics.py +0 -0
  35. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/params.py +0 -0
  36. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/plots.py +0 -0
  37. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/remote.py +0 -0
  38. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/repro.py +0 -0
  39. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/run.py +0 -0
  40. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/status.py +0 -0
  41. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/targets.py +0 -0
  42. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/update.py +0 -0
  43. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/cli/verify.py +0 -0
  44. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/compose.py +0 -0
  45. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/config/__init__.py +0 -0
  46. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/config/io.py +0 -0
  47. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/config/models.py +0 -0
  48. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/dag/__init__.py +0 -0
  49. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/dag/render.py +0 -0
  50. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/decorators.py +0 -0
  51. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/discovery.py +0 -0
  52. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/engine/__init__.py +0 -0
  53. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/engine/agent_rpc.py +0 -0
  54. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/engine/engine.py +0 -0
  55. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/engine/graph.py +0 -0
  56. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/engine/scheduler.py +0 -0
  57. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/engine/sinks.py +0 -0
  58. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/engine/sources.py +0 -0
  59. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/engine/types.py +0 -0
  60. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/engine/watch.py +0 -0
  61. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/engine/worker_pool.py +0 -0
  62. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/executor/AGENTS.md +0 -0
  63. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/executor/CLAUDE.md +0 -0
  64. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/executor/__init__.py +0 -0
  65. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/executor/commit.py +0 -0
  66. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/executor/core.py +0 -0
  67. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/executor/worker.py +0 -0
  68. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/explain.py +0 -0
  69. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/fingerprint.py +0 -0
  70. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/git.py +0 -0
  71. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/ignore.py +0 -0
  72. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/import_artifact.py +0 -0
  73. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/loaders.py +0 -0
  74. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/metrics.py +0 -0
  75. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/names.py +0 -0
  76. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/outputs.py +0 -0
  77. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/parameters.py +0 -0
  78. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/path_policy.py +0 -0
  79. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/project.py +0 -0
  80. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/py.typed +0 -0
  81. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/registry.py +0 -0
  82. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/remote/AGENTS.md +0 -0
  83. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/remote/__init__.py +0 -0
  84. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/remote/config.py +0 -0
  85. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/remote/git_archive.py +0 -0
  86. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/remote/github.py +0 -0
  87. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/remote/sync.py +0 -0
  88. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/run_history.py +0 -0
  89. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/show/__init__.py +0 -0
  90. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/show/common.py +0 -0
  91. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/show/data.py +0 -0
  92. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/show/metrics.py +0 -0
  93. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/show/params.py +0 -0
  94. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/show/plots.py +0 -0
  95. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/skip.py +0 -0
  96. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/stage_def.py +0 -0
  97. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/status.py +0 -0
  98. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/CLAUDE.md +0 -0
  99. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/__init__.py +0 -0
  100. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/artifact_lock.py +0 -0
  101. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/lock.py +0 -0
  102. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/presentation.py +0 -0
  103. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/state.py +0 -0
  104. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/store.py +0 -0
  105. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/storage/track.py +0 -0
  106. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/types.py +0 -0
  107. {pivot-0.3.4 → pivot-0.3.5}/src/pivot/yaml_config.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: pivot
3
- Version: 0.3.4
3
+ Version: 0.3.5
4
4
  Summary: High-performance Python pipeline tool with automatic code change detection
5
5
  Author: Sami Jawhar
6
6
  Author-email: Sami Jawhar <sami@metr.org>
@@ -1,6 +1,6 @@
1
1
  [project]
2
2
  name = "pivot"
3
- version = "0.3.4"
3
+ version = "0.3.5"
4
4
  description = "High-performance Python pipeline tool with automatic code change detection"
5
5
  requires-python = ">=3.13,<3.14"
6
6
  readme = "README.md"
@@ -161,10 +161,6 @@ def _track_single_path(
161
161
  total_size = sum(e["size"] for e in manifest)
162
162
  num_files = len(manifest)
163
163
 
164
- # copy_to_cache is idempotent and already checks existence itself; letting it see
165
- # every entry (not just ones missing from the cache) is what lets it republish an
166
- # object `pivot track --force` finds cached non-executable by an older pivot or a
167
- # cold `pivot pull`, when this entry's bytes should be executable.
168
164
  for entry in manifest:
169
165
  file_path = path / entry["relpath"]
170
166
  file_cache_path = cache.get_cache_path(cache_dir, entry["hash"])
@@ -221,6 +221,8 @@ class CacheModeCollisionError(CacheError):
221
221
  This happens when two files with identical content disagree on their executable bit (the
222
222
  cache key is content-only) or when a cache object was populated by a path that does not
223
223
  know a file's intended executable bit (e.g. `pivot pull`, which downloads by hash alone).
224
+ Pivot never changes a published cache object, so only a private copy, or the object deleted
225
+ and published again from bytes that carry the right bit, can give the file its bit.
224
226
  """
225
227
 
226
228
  @override
@@ -228,10 +230,10 @@ class CacheModeCollisionError(CacheError):
228
230
  return (
229
231
  "Add CheckoutMode.COPY to checkout_modes (cache: checkout_mode: [..., copy] in "
230
232
  ".pivot/config.yaml) so checkout can fall back to a private, correctly-permissioned "
231
- "copy instead of sharing the cache object's own permissions; or, if you have the "
232
- "source bytes in hand, re-running what produced them (`pivot repro --force "
233
- "<stage>`, `pivot track --force <path>`) republishes the cache object with the "
234
- "right bit"
233
+ "copy instead of sharing the cache object's own permissions; or delete the cache "
234
+ "object the error names, then re-run what produced the file (`pivot repro --force "
235
+ "<stage>`, `pivot track --force <path>`) to publish it with the right bit; until it "
236
+ "is published again, any workspace file symlinked to it is broken"
235
237
  )
236
238
 
237
239
 
@@ -2,7 +2,6 @@ from __future__ import annotations
2
2
 
3
3
  import asyncio
4
4
  import contextlib
5
- import errno
6
5
  import logging
7
6
  import os
8
7
  import re
@@ -13,6 +12,7 @@ import xxhash
13
12
 
14
13
  from pivot import config, exceptions, metrics
15
14
  from pivot.remote import config as remote_config
15
+ from pivot.storage import cache
16
16
  from pivot.types import TransferResult
17
17
 
18
18
  if TYPE_CHECKING:
@@ -38,9 +38,6 @@ BULK_EXISTS_LIST_THRESHOLD = 50
38
38
 
39
39
  _HEX_PATTERN = re.compile(r"^[a-f0-9]{16}$")
40
40
 
41
- # link() errors on filesystems without hard links (vfat, exFAT, many SMB and FUSE mounts).
42
- _NO_HARDLINK_ERRNO = frozenset({errno.EPERM, errno.ENOTSUP, errno.EOPNOTSUPP})
43
-
44
41
  # Cached S3 config to avoid re-parsing config on every call.
45
42
  # Safe without locking - asyncio is single-threaded, worst case race just
46
43
  # creates duplicate config (harmless).
@@ -230,66 +227,61 @@ async def _atomic_download(
230
227
  authenticate remote content.
231
228
 
232
229
  Args:
233
- overwrite: Replace an existing file at local_path. When False, an existing file is
234
- kept: a content-addressed cache entry another process may be hardlinking.
230
+ overwrite: Replace an existing file at local_path. When False, local_path is a cache
231
+ object: an existing one is kept (another process may be hardlinking it), and the
232
+ download is staged so a killed one is swept (``cache.staged_cache_file``).
235
233
  readonly: If True, set file permissions to 0o444 (for cache files).
236
234
  """
237
- local_path.parent.mkdir(parents=True, exist_ok=True)
238
- fd, tmp_path = tempfile.mkstemp(dir=local_path.parent, prefix=".pivot_download_")
239
- move_succeeded = False
240
- try:
241
- try:
242
- response = await s3.get_object(Bucket=bucket, Key=key)
243
- except Exception as exc:
244
- error_str = str(exc)
245
- if "NoSuchKey" in error_str or "404" in error_str:
246
- raise exceptions.RemoteError(
247
- f"Remote cache file not found: {bucket}/{key}"
248
- ) from exc
249
- if "AccessDenied" in error_str or "403" in error_str:
250
- raise exceptions.RemoteError(
251
- f"Access denied to {bucket}/{key}. Check AWS credentials."
252
- ) from exc
253
- raise exceptions.RemoteError(f"Failed to download from S3: {exc}") from exc
254
- actual_hash = await _stream_download_to_fd(response, fd)
255
- os.close(fd)
256
- fd = -1 # Mark as closed
257
- if actual_hash != expected_hash:
258
- raise exceptions.RemoteError(
259
- f"Hash mismatch: expected {expected_hash}, got {actual_hash} "
260
- + f"for {bucket}/{key}. Delete {bucket}/{key} from the remote, then run "
261
- + "`pivot push` on a machine whose cache has a good copy and has not run "
262
- + "`pivot push`, `pivot pull`, `pivot fetch`, or `pivot status --remote` "
263
- + "against this remote for this object. Pivot records hashes those commands "
264
- + "upload, download, or find, then skips recorded hashes. If every machine "
265
- + "with a good copy has recorded it, delete `.pivot/state.lmdb/` before "
266
- + "pushing; this also clears run history and local caches."
267
- )
235
+ with contextlib.ExitStack() as stack:
236
+ if overwrite:
237
+ local_path.parent.mkdir(parents=True, exist_ok=True)
238
+ fd, name = tempfile.mkstemp(dir=local_path.parent, prefix=".pivot_download_")
239
+ os.close(fd)
240
+ tmp_path = local_path.with_name(os.path.basename(name))
241
+ stack.callback(tmp_path.unlink, missing_ok=True)
242
+ else:
243
+ tmp_path = stack.enter_context(cache.staged_cache_file(local_path))
244
+ await _download_verified(s3, bucket, key, tmp_path, expected_hash)
268
245
  if readonly:
269
246
  os.chmod(tmp_path, 0o444)
270
247
  if overwrite:
271
248
  os.replace(tmp_path, local_path)
272
- move_succeeded = True
273
249
  else:
274
- # link() refuses an existing name, so a concurrent reader never sees the
275
- # entry disappear. The temp file is removed below either way.
276
- try:
277
- os.link(tmp_path, local_path)
278
- except FileExistsError:
279
- pass
280
- except OSError as exc:
281
- if exc.errno not in _NO_HARDLINK_ERRNO:
282
- raise
283
- # No hard links here, so nothing can be hardlinking the entry either:
284
- # rename unless another process already promoted it.
285
- if not os.path.exists(local_path):
286
- os.replace(tmp_path, local_path)
287
- move_succeeded = True
250
+ _ = cache.publish_cache_object(tmp_path, local_path)
251
+
252
+
253
+ async def _download_verified(
254
+ s3: S3Client, bucket: str, key: str, dest: Path, expected_hash: str
255
+ ) -> None:
256
+ """Stream bucket/key into the existing file ``dest`` and check its xxh64 digest."""
257
+ try:
258
+ response = await s3.get_object(Bucket=bucket, Key=key)
259
+ except Exception as exc:
260
+ error_str = str(exc)
261
+ if "NoSuchKey" in error_str or "404" in error_str:
262
+ raise exceptions.RemoteError(f"Remote cache file not found: {bucket}/{key}") from exc
263
+ if "AccessDenied" in error_str or "403" in error_str:
264
+ raise exceptions.RemoteError(
265
+ f"Access denied to {bucket}/{key}. Check AWS credentials."
266
+ ) from exc
267
+ raise exceptions.RemoteError(f"Failed to download from S3: {exc}") from exc
268
+ fd = os.open(dest, os.O_WRONLY | os.O_TRUNC)
269
+ try:
270
+ actual_hash = await _stream_download_to_fd(response, fd)
288
271
  finally:
289
- if fd >= 0:
272
+ with contextlib.suppress(OSError):
290
273
  os.close(fd)
291
- if not move_succeeded and os.path.exists(tmp_path):
292
- os.unlink(tmp_path)
274
+ if actual_hash != expected_hash:
275
+ raise exceptions.RemoteError(
276
+ f"Hash mismatch: expected {expected_hash}, got {actual_hash} "
277
+ + f"for {bucket}/{key}. Delete {bucket}/{key} from the remote, then run "
278
+ + "`pivot push` on a machine whose cache has a good copy and has not run "
279
+ + "`pivot push`, `pivot pull`, `pivot fetch`, or `pivot status --remote` "
280
+ + "against this remote for this object. Pivot records hashes those commands "
281
+ + "upload, download, or find, then skips recorded hashes. If every machine "
282
+ + "with a good copy has recorded it, delete `.pivot/state.lmdb/` before "
283
+ + "pushing; this also clears run history and local caches."
284
+ )
293
285
 
294
286
 
295
287
  async def _stream_upload(
@@ -66,6 +66,44 @@ When multiple `pivot` processes share the same StateDB:
66
66
  temporary directory and swap it. Both run under the target's restore lock, and each
67
67
  first removes the token-tagged entries a killed restore of the same target left.
68
68
 
69
+ ## Publishing Cache Objects
70
+
71
+ - Every writer of a file object publishes it with `cache.publish_cache_object`, which links it
72
+ into place and keeps an object already there. A published object never changes, mode
73
+ included: a hardlink checkout looks the name up, then links the inode it found, and once a
74
+ rename takes that inode's last name Linux fails the link with ENOENT, though the name never
75
+ went missing; a symlink checkout reads its executable bit through the name. A directory
76
+ entry whose executable bit the object does not carry is checked out as a copy.
77
+ - On a filesystem without hard links, `publish_cache_object` renames, and only when no object is
78
+ there yet. Nothing there can hardlink an object, but the check and the rename are two steps:
79
+ two writers that both find the name free both rename, the later object wins, and a symlink
80
+ checkout reads the later object's mode.
81
+ - Every temporary entry in a prefix directory (a file staged for an object, a download, a tree
82
+ being copied) is made through `cache.staged_cache_file` or `_staging_in`, named
83
+ `.pivot_tmp_*`, while the process holds a shared `flock` on the directory's
84
+ `.pivot_cache.lock`. A process that gets the exclusive lock first (it never waits for it), at
85
+ most once per `_CACHE_SWEEP_INTERVAL` (the lock's mtime records the last sweep), removes every
86
+ `.pivot_tmp_*` entry: none has a live owner. An entry it cannot remove is logged and left.
87
+ Never judge a temporary entry abandoned by host and pid: a process in another PID namespace,
88
+ or on another host sharing the cache, looks dead.
89
+ - Sharing a cache across hosts needs `flock` that is coherent across them (NFSv4, or NFSv3 with
90
+ NLM). With host-local locks (NFS `nolock` or `local_lock=flock`/`all`, Lustre `localflock`)
91
+ the locks here exclude nothing across hosts: one host's sweep can remove another host's live
92
+ temporary entry, and two hosts can repair one tree at once. A publisher checks its tree before
93
+ marking it, so a copy that lost files is not marked. On a filesystem that refuses `flock()`
94
+ (Lustre without `flock`: ENOSYS; NFS when its lock manager does not answer: ENOLCK) nothing
95
+ sweeps and nothing repairs a tree, and staging goes on without the shared lock.
96
+ - A lock file in the cache is created writable by whoever can write its directory, since NFS
97
+ takes an exclusive lock only on a file opened for writing; one this user cannot write is
98
+ opened read-only. Lock files are never deleted.
99
+ - A whole-directory SYMLINK tree is built under a temporary name beside its cache name, renamed
100
+ into place, checked, then marked (`<tree>.verified`), because a restore symlinks to a tree as
101
+ soon as its name exists. A restore or commit checks an unmarked tree (exact files, sizes,
102
+ executable bits and directories) under the tree's exclusive `<tree>.lock`, repairs it in place
103
+ from the files' own cache objects, then marks it. A tree it cannot repair is not used: a
104
+ restore goes file by file, a commit fails. `<tree>.layout-checked`, the mark pivot 0.3.4 also
105
+ gave partial trees, is not trusted.
106
+
69
107
  ## Path Storage
70
108
 
71
109
  All paths stored in the database must be **relative** to ensure portability across machines and correct cache behavior.