@smthrs/artifacts 0.0.0-stage → 1.0.0-rc.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (171) hide show
  1. package/CHANGELOG.md +77 -0
  2. package/LICENSE +21 -0
  3. package/README.md +175 -2
  4. package/dist/cjs/ArtifactBackupLease.d.ts +34 -0
  5. package/dist/cjs/ArtifactBackupLease.d.ts.map +1 -0
  6. package/dist/cjs/ArtifactBackupLease.js +183 -0
  7. package/dist/cjs/ArtifactBackupLease.js.map +7 -0
  8. package/dist/cjs/ArtifactStore.d.ts +129 -0
  9. package/dist/cjs/ArtifactStore.d.ts.map +1 -0
  10. package/dist/cjs/ArtifactStore.js +73 -0
  11. package/dist/cjs/ArtifactStore.js.map +7 -0
  12. package/dist/cjs/ArtifactStoreError.d.ts +82 -0
  13. package/dist/cjs/ArtifactStoreError.d.ts.map +1 -0
  14. package/dist/cjs/ArtifactStoreError.js +72 -0
  15. package/dist/cjs/ArtifactStoreError.js.map +7 -0
  16. package/dist/cjs/ArtifactStoreMetrics.d.ts +70 -0
  17. package/dist/cjs/ArtifactStoreMetrics.d.ts.map +1 -0
  18. package/dist/cjs/ArtifactStoreMetrics.js +51 -0
  19. package/dist/cjs/ArtifactStoreMetrics.js.map +7 -0
  20. package/dist/cjs/ArtifactSweep.d.ts +185 -0
  21. package/dist/cjs/ArtifactSweep.d.ts.map +1 -0
  22. package/dist/cjs/ArtifactSweep.js +171 -0
  23. package/dist/cjs/ArtifactSweep.js.map +7 -0
  24. package/dist/cjs/CombinedArtifacts.d.ts +103 -0
  25. package/dist/cjs/CombinedArtifacts.d.ts.map +1 -0
  26. package/dist/cjs/CombinedArtifacts.js +189 -0
  27. package/dist/cjs/CombinedArtifacts.js.map +7 -0
  28. package/dist/cjs/FileLease.d.ts +117 -0
  29. package/dist/cjs/FileLease.d.ts.map +1 -0
  30. package/dist/cjs/FileLease.js +208 -0
  31. package/dist/cjs/FileLease.js.map +7 -0
  32. package/dist/cjs/FileSystemArtifactStore.d.ts +93 -0
  33. package/dist/cjs/FileSystemArtifactStore.d.ts.map +1 -0
  34. package/dist/cjs/FileSystemArtifactStore.js +276 -0
  35. package/dist/cjs/FileSystemArtifactStore.js.map +7 -0
  36. package/dist/cjs/MemoryArtifactStore.d.ts +25 -0
  37. package/dist/cjs/MemoryArtifactStore.d.ts.map +1 -0
  38. package/dist/cjs/MemoryArtifactStore.js +84 -0
  39. package/dist/cjs/MemoryArtifactStore.js.map +7 -0
  40. package/dist/cjs/NoopArtifactStore.d.ts +17 -0
  41. package/dist/cjs/NoopArtifactStore.d.ts.map +1 -0
  42. package/dist/cjs/NoopArtifactStore.js +46 -0
  43. package/dist/cjs/NoopArtifactStore.js.map +7 -0
  44. package/dist/cjs/RemoteArtifacts.d.ts +237 -0
  45. package/dist/cjs/RemoteArtifacts.d.ts.map +1 -0
  46. package/dist/cjs/RemoteArtifacts.js +405 -0
  47. package/dist/cjs/RemoteArtifacts.js.map +7 -0
  48. package/dist/cjs/index.d.ts +65 -0
  49. package/dist/cjs/index.d.ts.map +1 -0
  50. package/dist/cjs/index.js +47 -0
  51. package/dist/cjs/index.js.map +7 -0
  52. package/dist/cjs/internal/ArtifactLocks.d.ts +30 -0
  53. package/dist/cjs/internal/ArtifactLocks.d.ts.map +1 -0
  54. package/dist/cjs/internal/ArtifactLocks.js +93 -0
  55. package/dist/cjs/internal/ArtifactLocks.js.map +7 -0
  56. package/dist/cjs/internal/ArtifactPath.d.ts +19 -0
  57. package/dist/cjs/internal/ArtifactPath.d.ts.map +1 -0
  58. package/dist/cjs/internal/ArtifactPath.js +74 -0
  59. package/dist/cjs/internal/ArtifactPath.js.map +7 -0
  60. package/dist/cjs/measureBytes.d.ts +18 -0
  61. package/dist/cjs/measureBytes.d.ts.map +1 -0
  62. package/dist/cjs/measureBytes.js +43 -0
  63. package/dist/cjs/measureBytes.js.map +7 -0
  64. package/dist/cjs/package.json +1 -0
  65. package/dist/cjs/snapshotBytes.d.ts +19 -0
  66. package/dist/cjs/snapshotBytes.d.ts.map +1 -0
  67. package/dist/cjs/snapshotBytes.js +41 -0
  68. package/dist/cjs/snapshotBytes.js.map +7 -0
  69. package/dist/cjs/validateDigest.d.ts +22 -0
  70. package/dist/cjs/validateDigest.d.ts.map +1 -0
  71. package/dist/cjs/validateDigest.js +42 -0
  72. package/dist/cjs/validateDigest.js.map +7 -0
  73. package/dist/esm/ArtifactBackupLease.d.ts +34 -0
  74. package/dist/esm/ArtifactBackupLease.d.ts.map +1 -0
  75. package/dist/esm/ArtifactBackupLease.js +126 -0
  76. package/dist/esm/ArtifactBackupLease.js.map +1 -0
  77. package/dist/esm/ArtifactStore.d.ts +129 -0
  78. package/dist/esm/ArtifactStore.d.ts.map +1 -0
  79. package/dist/esm/ArtifactStore.js +83 -0
  80. package/dist/esm/ArtifactStore.js.map +1 -0
  81. package/dist/esm/ArtifactStoreError.d.ts +82 -0
  82. package/dist/esm/ArtifactStoreError.d.ts.map +1 -0
  83. package/dist/esm/ArtifactStoreError.js +77 -0
  84. package/dist/esm/ArtifactStoreError.js.map +1 -0
  85. package/dist/esm/ArtifactStoreMetrics.d.ts +70 -0
  86. package/dist/esm/ArtifactStoreMetrics.d.ts.map +1 -0
  87. package/dist/esm/ArtifactStoreMetrics.js +77 -0
  88. package/dist/esm/ArtifactStoreMetrics.js.map +1 -0
  89. package/dist/esm/ArtifactSweep.d.ts +185 -0
  90. package/dist/esm/ArtifactSweep.d.ts.map +1 -0
  91. package/dist/esm/ArtifactSweep.js +210 -0
  92. package/dist/esm/ArtifactSweep.js.map +1 -0
  93. package/dist/esm/CombinedArtifacts.d.ts +103 -0
  94. package/dist/esm/CombinedArtifacts.d.ts.map +1 -0
  95. package/dist/esm/CombinedArtifacts.js +214 -0
  96. package/dist/esm/CombinedArtifacts.js.map +1 -0
  97. package/dist/esm/FileLease.d.ts +117 -0
  98. package/dist/esm/FileLease.d.ts.map +1 -0
  99. package/dist/esm/FileLease.js +249 -0
  100. package/dist/esm/FileLease.js.map +1 -0
  101. package/dist/esm/FileSystemArtifactStore.d.ts +93 -0
  102. package/dist/esm/FileSystemArtifactStore.d.ts.map +1 -0
  103. package/dist/esm/FileSystemArtifactStore.js +336 -0
  104. package/dist/esm/FileSystemArtifactStore.js.map +1 -0
  105. package/dist/esm/MemoryArtifactStore.d.ts +25 -0
  106. package/dist/esm/MemoryArtifactStore.d.ts.map +1 -0
  107. package/dist/esm/MemoryArtifactStore.js +67 -0
  108. package/dist/esm/MemoryArtifactStore.js.map +1 -0
  109. package/dist/esm/NoopArtifactStore.d.ts +17 -0
  110. package/dist/esm/NoopArtifactStore.d.ts.map +1 -0
  111. package/dist/esm/NoopArtifactStore.js +27 -0
  112. package/dist/esm/NoopArtifactStore.js.map +1 -0
  113. package/dist/esm/RemoteArtifacts.d.ts +237 -0
  114. package/dist/esm/RemoteArtifacts.d.ts.map +1 -0
  115. package/dist/esm/RemoteArtifacts.js +557 -0
  116. package/dist/esm/RemoteArtifacts.js.map +1 -0
  117. package/dist/esm/index.d.ts +65 -0
  118. package/dist/esm/index.d.ts.map +1 -0
  119. package/dist/esm/index.js +65 -0
  120. package/dist/esm/index.js.map +1 -0
  121. package/dist/esm/internal/ArtifactLocks.d.ts +30 -0
  122. package/dist/esm/internal/ArtifactLocks.d.ts.map +1 -0
  123. package/dist/esm/internal/ArtifactLocks.js +86 -0
  124. package/dist/esm/internal/ArtifactLocks.js.map +1 -0
  125. package/dist/esm/internal/ArtifactPath.d.ts +19 -0
  126. package/dist/esm/internal/ArtifactPath.d.ts.map +1 -0
  127. package/dist/esm/internal/ArtifactPath.js +59 -0
  128. package/dist/esm/internal/ArtifactPath.js.map +1 -0
  129. package/dist/esm/measureBytes.d.ts +18 -0
  130. package/dist/esm/measureBytes.d.ts.map +1 -0
  131. package/dist/esm/measureBytes.js +22 -0
  132. package/dist/esm/measureBytes.js.map +1 -0
  133. package/dist/esm/snapshotBytes.d.ts +19 -0
  134. package/dist/esm/snapshotBytes.d.ts.map +1 -0
  135. package/dist/esm/snapshotBytes.js +30 -0
  136. package/dist/esm/snapshotBytes.js.map +1 -0
  137. package/dist/esm/validateDigest.d.ts +22 -0
  138. package/dist/esm/validateDigest.d.ts.map +1 -0
  139. package/dist/esm/validateDigest.js +26 -0
  140. package/dist/esm/validateDigest.js.map +1 -0
  141. package/docs/README.md +202 -0
  142. package/docs/api.md +484 -0
  143. package/docs/concepts/content-addressing.md +174 -0
  144. package/docs/concepts/coordination.md +141 -0
  145. package/docs/concepts/tiers.md +147 -0
  146. package/docs/guides/fence-a-backup.md +123 -0
  147. package/docs/guides/reclaim-disk-space.md +143 -0
  148. package/docs/guides/serve-the-artifact-protocol.md +140 -0
  149. package/docs/guides/share-artifacts-across-machines.md +180 -0
  150. package/docs/guides/test-against-an-artifact-store.md +158 -0
  151. package/docs/installation.md +102 -0
  152. package/docs/quickstart.md +183 -0
  153. package/docs/troubleshooting.md +232 -0
  154. package/package.json +179 -3
  155. package/src/ArtifactBackupLease.ts +223 -0
  156. package/src/ArtifactStore.ts +152 -0
  157. package/src/ArtifactStoreError.ts +96 -0
  158. package/src/ArtifactStoreMetrics.ts +83 -0
  159. package/src/ArtifactSweep.ts +346 -0
  160. package/src/CombinedArtifacts.ts +369 -0
  161. package/src/FileLease.ts +397 -0
  162. package/src/FileSystemArtifactStore.ts +459 -0
  163. package/src/MemoryArtifactStore.ts +83 -0
  164. package/src/NoopArtifactStore.ts +30 -0
  165. package/src/RemoteArtifacts.ts +814 -0
  166. package/src/index.ts +71 -0
  167. package/src/internal/ArtifactLocks.ts +110 -0
  168. package/src/internal/ArtifactPath.ts +72 -0
  169. package/src/measureBytes.ts +28 -0
  170. package/src/snapshotBytes.ts +34 -0
  171. package/src/validateDigest.ts +31 -0
package/docs/api.md ADDED
@@ -0,0 +1,484 @@
1
+ ---
2
+ title: "API reference"
3
+ description: "Every export of @smthrs/artifacts: the store contract and its errors, the filesystem, memory, no-op, HTTP, and combined implementations, the sweep, backup lease, and file lease, and the two metrics."
4
+ ---
5
+
6
+ `@smthrs/artifacts` exports seven modules. Each is available from the root as a
7
+ namespace and from its own subpath:
8
+
9
+ ```ts
10
+ import { ArtifactStore } from "@smthrs/artifacts"
11
+ import * as ArtifactStoreSubpath from "@smthrs/artifacts/ArtifactStore"
12
+ ```
13
+
14
+ Addresses are the SHA-256 digests [`@smthrs/crypto`](/api/crypto) computes
15
+ through the injected `Crypto` service, which with `effect` is the whole
16
+ dependency list.
17
+
18
+ ## Example
19
+
20
+ ```ts
21
+ import { ArtifactStore } from "@smthrs/artifacts"
22
+
23
+ const program = Effect.gen(function*() {
24
+ const store = yield* ArtifactStore.ArtifactStore
25
+ const address = yield* store.put(new TextEncoder().encode("build output"))
26
+ return yield* store.get(address)
27
+ })
28
+ ```
29
+
30
+ ## ArtifactStore
31
+
32
+ `@smthrs/artifacts/ArtifactStore`. The contract, its errors, and the three
33
+ local implementations.
34
+
35
+ ### The contract
36
+
37
+ ```ts
38
+ interface Service {
39
+ readonly put: (bytes: Uint8Array) => Effect<Digest, ArtifactStoreError, Crypto>
40
+ readonly get: (
41
+ digest: string
42
+ ) => Effect<Uint8Array, ArtifactMissing | ArtifactCorruption | ArtifactStoreError, Crypto>
43
+ readonly has: (digest: string) => Effect<boolean, ArtifactStoreError>
44
+ readonly findMissing: (digests: Iterable<string>) => Effect<Array<string>, ArtifactStoreError>
45
+ }
46
+ ```
47
+
48
+ | Member | Behavior |
49
+ | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
50
+ | `put` | Measures `bytes`, publishes them under that address, and returns it. Idempotent: storing the same bytes twice returns the same address and stores one copy. Every implementation snapshots the caller's buffer when the Effect begins and never retains the caller's reference. |
51
+ | `get` | Returns the bytes at `digest`, having verified that they still hash to it. Every successful read returns a new array. Fails with `ArtifactMissing` for a valid address with no bytes, `ArtifactCorruption` for a mismatch, and `ArtifactStoreError` for a host or transport refusal. |
52
+ | `has` | Whether this tier holds an artifact at `digest`. Does not read or hash the bytes, so it needs no `Crypto`. |
53
+ | `findMissing` | Which of `digests` this tier does not hold. The result is guaranteed to be a subset of the input, deduplicated in first-seen order. |
54
+
55
+ `ArtifactStore` is the service tag, with identity
56
+ `@smthrs/artifacts/ArtifactStore`.
57
+
58
+ Read operations accept a plain `string` rather than the `Digest` brand on
59
+ purpose: an address read back out of a durable row is untrusted input, so the
60
+ store validates it instead of asking every caller to re-brand a persisted
61
+ column. `put` returns the brand, because it measured the bytes itself.
62
+
63
+ ### Digests
64
+
65
+ | Export | Meaning |
66
+ | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
67
+ | `Digest` | Schema and branded type for exactly 64 lowercase hexadecimal SHA-256 characters, re-exported from `@smthrs/crypto`. |
68
+ | `validateDigest(digest: string)` | `Effect<Digest, ArtifactStoreError>`. Refuses anything other than the canonical representation with code `invalid_digest`. The failure text is a bounded constant, so a hostile value cannot copy itself into a log or a durable error. |
69
+ | `measureBytes(bytes: Uint8Array)` | `Effect<Digest, ArtifactStoreError, Crypto>`. Computes an address with the injected `Crypto` service, never retaining the bytes in an error. |
70
+ | `snapshotBytes(bytes: Uint8Array)` | `Effect<Uint8Array, ArtifactStoreError>`. Copies a caller-owned buffer when the Effect begins. A refused copy fails as `unavailable`, the code for a host refusal, because `Crypto` has not been consulted yet. |
71
+
72
+ Every implementation validates an address before logging it or interpolating it
73
+ into a path or a URL.
74
+
75
+ ### Errors
76
+
77
+ | Export | Shape |
78
+ | ------------------------ | --------------------------------------------------------------------------------------------------------------------- |
79
+ | `ArtifactStoreError` | Tag `@smthrs/artifacts/ArtifactStoreError`, fields `code`, `message`, optional `cause`. |
80
+ | `ArtifactMissing` | Tag `@smthrs/artifacts/ArtifactMissing`, fields `code: "artifact_missing"` and `digest`. |
81
+ | `ArtifactCorruption` | Tag `@smthrs/artifacts/ArtifactCorruption`, fields `code: "artifact_corruption"`, `recordedDigest`, `measuredDigest`. |
82
+ | `ArtifactStoreErrorCode` | Schema and type for the five stable codes. |
83
+
84
+ A miss and a corruption are separate tags rather than codes because a
85
+ read-through composition acts on them differently: a miss is an ordinary
86
+ outcome a second tier may satisfy, and a corruption is a violation of the
87
+ store's strongest invariant.
88
+
89
+ | Code | Retryable | Meaning |
90
+ | ----------------------- | --------- | ----------------------------------------------------------- |
91
+ | `invalid_configuration` | No | The caller supplied an option the store refuses. Permanent. |
92
+ | `invalid_digest` | No | The value is not a canonical SHA-256 address. Permanent. |
93
+ | `digest_failed` | Depends | The `Crypto` service failed to compute a digest. |
94
+ | `unavailable` | Depends | A host refused, or a no-op store has no implementation. |
95
+ | `transport_failed` | Depends | The shared tier refused, answered unusably, or timed out. |
96
+
97
+ ### The filesystem store
98
+
99
+ ```ts
100
+ declare const makeFileSystem: (fs: FileSystem, path: Path, options?: FileSystemOptions) => Service
101
+ declare const layerFileSystem: (
102
+ options?: FileSystemOptions
103
+ ) => Layer<ArtifactStore, never, FileSystem | Path>
104
+ ```
105
+
106
+ `path` is the host's `Path` service (`NodePath.layer` on Node). The directory
107
+ sync walks the objects directory's ancestors with it, so a Windows drive path
108
+ syncs its real parents.
109
+
110
+ Publishes at `${directory}/${digest.slice(0, 2)}/${digest}`. Bytes land at a
111
+ unique temp path in the destination fanout directory, are synced when required,
112
+ and are renamed into place. Every deduplicated `put` verifies the existing blob
113
+ and rewrites an unreadable or mismatched address atomically, then freshens the
114
+ blob's modification time so a retention sweep reads it as recently referenced.
115
+
116
+ `findMissing` deduplicates and validates the full input before filesystem
117
+ probes, then checks at most 16 addresses concurrently. Missing digests retain
118
+ first-request order; a failed probe fails the batch as `unavailable`.
119
+
120
+ | `FileSystemOptions` field | Default | Meaning |
121
+ | ------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
122
+ | `directory` | `defaultDirectory` | Where blobs are stored. Workspace-relative, so a workspace can be moved or copied whole. An `ArtifactSweep` over the same store must name the same directory. |
123
+ | `fileMode` | `0o600` | Creation mode for new payload files, restricted by the host umask. Existing blobs are unchanged. |
124
+ | `directoryMode` | `0o700` | Creation mode for objects and fanout directories, restricted by the host umask. Lock directories remain `0700`. |
125
+ | `durability` | `"required"` | `required` syncs the blob, fanout directory, objects directory, and its ancestors and propagates any refusal. `best-effort` tolerates sync refusals. Both modes require exclusive writable handles and symlink inspection. |
126
+ | `coordination` | `"required"` | `required` combines an in-process semaphore with heartbeat-backed lock files shared by every cooperating process. `process` keeps only the semaphore, giving up cross-process exclusion between writers and sweep deletion. |
127
+
128
+ `defaultDirectory` is `.flows/objects`. It is exported because the store and
129
+ its sweep must name the same directory, and a second private copy of the
130
+ literal is exactly how they would drift apart.
131
+
132
+ `required` bounds the writer-versus-sweeper race rather than eliminating it. A
133
+ lock is heartbeated every 10 seconds, reclaimed once it is 60 seconds stale, and
134
+ acquisition gives up after 2 minutes. Contenders that find the same stale lock
135
+ race for a claim file named after its owner, so the lock is moved at most once
136
+ and a fresh replacement is never displaced. A holder whose host stalls past 60
137
+ seconds is still reaped while running. Neither
138
+ `ArtifactSweep.RemoveOptions.ifUnmodifiedSinceMs` nor the backup lease protects
139
+ against that, because both depend on the same lock. Build a
140
+ store and its sweep with the same directory and coordination mode; no runtime
141
+ check can detect a mismatch. See
142
+ [Coordination between processes](./concepts/coordination.md).
143
+
144
+ The store also sweeps its own crash orphans, `.tmp-*` payloads and `.locks/`
145
+ entries older than one hour, once per store instance on its first publication.
146
+ That is scratch reclamation, never garbage collection of published blobs.
147
+
148
+ ### The memory store
149
+
150
+ ```ts
151
+ declare const makeMemory: () => Service
152
+ declare const layerMemory: Layer<ArtifactStore>
153
+ ```
154
+
155
+ For tests and for a browser host with no durable filesystem. It copies at both
156
+ boundaries around a private map, so no reference a caller can still mutate
157
+ aliases the stored content. It does not rehash a read, because its address
158
+ space is private and cannot disagree with itself.
159
+
160
+ `layerMemory` is a value, not a function.
161
+
162
+ ### The no-op store
163
+
164
+ ```ts
165
+ declare const makeNoop: (overrides?: Partial<Service>) => Service
166
+ declare const layerNoop: (overrides?: Partial<Service>) => Layer<ArtifactStore>
167
+ ```
168
+
169
+ Every operation fails with an `unavailable` `ArtifactStoreError` whose message
170
+ is `<method> is unavailable`. Overrides replace individual methods, which is
171
+ how a test states the one behavior it cares about.
172
+
173
+ ## RemoteArtifacts
174
+
175
+ `@smthrs/artifacts/RemoteArtifacts`. The shared tier, spoken over Effect's
176
+ `HttpClient`.
177
+
178
+ ```ts
179
+ declare const make: (options: Options) => Effect<Service, ArtifactStoreError, HttpClient>
180
+ declare const layer: (
181
+ options: Options
182
+ ) => Layer<ArtifactStore.ArtifactStore, ArtifactStoreError, HttpClient>
183
+ ```
184
+
185
+ `Service` extends `ArtifactStore.Service` with one field, `downloadPolicy`.
186
+ Composing `layer` alone makes every read and write a network round trip; the
187
+ intended production shape is `CombinedArtifacts.layer` with this as its remote
188
+ tier.
189
+
190
+ The wire protocol is `GET`, `PUT`, and `HEAD` at `/cas/{digest}` and `POST` at
191
+ `/cas/findMissing`, resolved beneath the endpoint. Every download is bounded and
192
+ digest-verified before it is returned. Redirects are never followed: a fetch
193
+ transport runs with `redirect: "manual"`, and any `3xx` other than the
194
+ resumable-upload `308` fails as an unexpected status, so credential headers
195
+ never reach another origin. The complete server-side contract is in
196
+ [Serve the artifact protocol](./guides/serve-the-artifact-protocol.md).
197
+
198
+ Effect's HTTP client opens one span per request, named for the request method.
199
+ It records no request or response header, so a trace carries no credential. A
200
+ host that sets `HttpClient.TracerHeaderFilter` to record headers gets every
201
+ name configured in `headers` as `<redacted>`. A `4xx` or `5xx` answer ends the
202
+ span as failed with the status in `error.type`, so a `HEAD` for an absent
203
+ artifact shows a failed span although `has` returned `false`.
204
+
205
+ ### Options
206
+
207
+ | Field | Default | Meaning |
208
+ | ----------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
209
+ | `endpoint` | required | The cache root, for example `https://cache.example.com`. A trailing slash is ignored. Refused at construction as `invalid_configuration` when it is not a string, when no `URL` parser accepts it, when the scheme is not `https:` (plain `http:` is allowed only for the literal loopback hosts `localhost`, `127.0.0.1`, and `[::1]`), and when it carries userinfo, a query, or a fragment. The refusal names only the violated rule and never echoes the endpoint. |
210
+ | `headers` | none | Headers sent with every request: this is the credential seam. Construction-time on purpose, because a credential arriving as a step input would be hashed into a step key and persisted everywhere the journal goes. Applied before the protocol's own headers, so no configuration can strip a `content-type`, `content-length`, or `content-range`. |
211
+ | `downloadTimeout` | 60 seconds | One download, including the complete response body. |
212
+ | `uploadTimeout` | 60 seconds | One upload, including every resume probe and chunk. |
213
+ | `requestTimeout` | 60 seconds | One `HEAD` probe, or one `findMissing` batch and its response. |
214
+ | `maxDownloadBytes` | 256 MiB | The largest download buffered. A `Content-Length` past the bound is refused before a body byte is read; an incremental read stops at most one chunk past it. |
215
+ | `maxFindMissingResponseBytes` | 256 KiB | The largest `findMissing` response accepted. May only lower the protocol bound. |
216
+ | `chunkBytes` | absent | Above this many bytes an upload travels as `Content-Range` requests, resuming from the prefix the tier kept. |
217
+ | `downloadPolicy` | `"all"` | How eagerly a composition reading through this tier materializes blobs locally. |
218
+
219
+ `findMissing` validates its input before sending anything, sends at most 1,000
220
+ digests per batch, and filters the response back to requested digests, so a
221
+ tier answering with an unrequested digest cannot make a caller upload bytes it
222
+ never asked about.
223
+
224
+ ### Download policy
225
+
226
+ | Export | Meaning |
227
+ | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
228
+ | `DownloadPolicy` | Schema and type for `"all"`, `"toplevel"`, `"minimal"`. |
229
+ | `downloadPolicies` | A frozen array of all three, in materialization order. |
230
+ | `downloadPolicyOf(store)` | Reads the policy a store declares, or `undefined` for one that declares none, which is every local store and any foreign implementation. |
231
+
232
+ `all` prefetches every referenced blob when a replay is admitted. `toplevel`
233
+ prefetches nothing and materializes a blob on the first read that needs it.
234
+ `minimal` prefetches nothing and materializes nothing.
235
+ `CombinedArtifacts.get` honors the last two;
236
+ [`@smthrs/engine-store`](/api/engine-store)'s `ArtifactSync.hydrate` honors the
237
+ first.
238
+
239
+ ### Chunked uploads
240
+
241
+ With `chunkBytes` set, the client sends a `HEAD` existence probe, then an empty
242
+ `Content-Range: bytes */{total}` probe to discover a retained prefix, then
243
+ ranged chunks:
244
+
245
+ | Response | Client action |
246
+ | ----------------------------- | ----------------------------------------------------------- |
247
+ | `308` | Continue after the reported `Range: bytes=0-{last}` prefix. |
248
+ | `2xx` on the completing chunk | Confirm the complete length with `HEAD`. |
249
+ | `2xx` before completion | Treat the tier as range-unaware and send the whole blob. |
250
+ | `400`, `411`, or `416` | Send the whole blob. |
251
+ | Any other status | Fail the upload as a transport error. |
252
+
253
+ The offset never moves backward. If a server ignores ranges, omits a confirming
254
+ length, or stores a partial body, the whole-blob `PUT` overwrites the partial
255
+ result. Both Smithers cache services answer a ranged `PUT` with `400` and cap
256
+ one request body at 16 MiB, so `chunkBytes` against them degrades
257
+ to one whole-blob `PUT` and a larger artifact is refused with `413`.
258
+
259
+ ## CombinedArtifacts
260
+
261
+ `@smthrs/artifacts/CombinedArtifacts`. Local first, shared second, with
262
+ write-back.
263
+
264
+ ```ts
265
+ declare const make: (options: Options) => Effect<RemoteArtifacts.Service, ArtifactStoreError>
266
+ declare const layer: <EL, RL, ER, RR>(
267
+ options: LayerOptions<EL, RL, ER, RR>
268
+ ) => Layer<ArtifactStore.ArtifactStore, EL | ER | ArtifactStoreError, RL | RR>
269
+ ```
270
+
271
+ | `Options` field | Default | Meaning |
272
+ | ------------------ | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
273
+ | `local` | required | The fast, machine-local tier, as a `Service`. Every read tries this one first. |
274
+ | `remote` | required | The shared tier, as a `Service`. Consulted only on a local miss or corruption. |
275
+ | `uploadTimeout` | 60 seconds | How long a `put` waits for its opportunistic upload before abandoning it. An abandoned upload is dropped exactly like a refused one. |
276
+ | `writeBackTimeout` | 60 seconds | How long a `get` waits for opportunistic local write-back before interrupting it and returning the verified remote bytes. |
277
+ | `downloadPolicy` | the remote tier's, else `"all"` | Overrides the shared tier's declared policy for this composition. |
278
+
279
+ `LayerOptions` is the same shape with `local` and `remote` as `Effect`s. Both
280
+ tiers are supplied as effects rather than layers because they inhabit the same
281
+ tag: composing two `Layer<ArtifactStore>` would shadow one with the other. Pair
282
+ `Effect.map(Effect.all([FileSystem.FileSystem, Path.Path]), ([fs, path]) => ArtifactStore.makeFileSystem(fs, path))`
283
+ with `RemoteArtifacts.make`.
284
+
285
+ Both timeout options must be finite, positive durations. Invalid values fail
286
+ construction with `ArtifactStoreError` code `invalid_configuration`.
287
+
288
+ **Reads.** The local tier answers first. A hit returns. `ArtifactMissing` and
289
+ `ArtifactCorruption` both fall through to the shared tier, and they are kept
290
+ apart because only corruption earns a write-back that `minimal` would otherwise
291
+ skip. A local `ArtifactStoreError` fails the read rather than falling through:
292
+ a host that refused has not answered the question, and paying for the network
293
+ instead would hide a broken local tier behind a working shared one.
294
+ Local write-back after a remote read is opportunistic. A typed failure or
295
+ `writeBackTimeout` expiry returns the verified remote bytes. Timeout interrupts
296
+ the write-back and waits for its interruption cleanup before returning.
297
+
298
+ **Writes.** `put` publishes locally first and returns the local digest. The
299
+ remote upload is opportunistic, deduplicated in flight by digest, and bounded by
300
+ `uploadTimeout`. A remote refusal or timeout does not fail the operation that
301
+ produced the bytes. The shared-cache publication protocol uses `findMissing`,
302
+ upload, and confirmation before it publishes a cache entry, so an abandoned
303
+ upload costs another transfer rather than correctness.
304
+
305
+ **Probes.** `has` asks local, then remote. `findMissing` asks the remote tier
306
+ only about what the local tier could not answer, so the result stays a subset of
307
+ the input because each stage filters the previous stage's output.
308
+
309
+ `minimal` still writes back after local corruption, because replacing an
310
+ address the local tier already claims is repair rather than growth.
311
+
312
+ ## ArtifactSweep
313
+
314
+ `@smthrs/artifacts/ArtifactSweep`. Host-local enumeration and fenced deletion.
315
+ This surface is deliberately not part of `ArtifactStore.Service`: a remote tier
316
+ can neither enumerate its address space nor accept a delete, and only the
317
+ filesystem store implements it.
318
+
319
+ ```ts
320
+ interface Service {
321
+ readonly inventory: Effect<Array<BlobStat>, ArtifactStoreError>
322
+ readonly remove: (digest: string, options?: RemoveOptions) => Effect<boolean, ArtifactStoreError>
323
+ }
324
+
325
+ declare const makeFileSystem: (fs: FileSystem, options?: SweepOptions) => Service
326
+ declare const layerFileSystem: (options?: SweepOptions) => Layer<ArtifactSweep, never, FileSystem>
327
+ declare const makeNoop: (overrides?: Partial<Service>) => Service
328
+ declare const layerNoop: (overrides?: Partial<Service>) => Layer<ArtifactSweep>
329
+ ```
330
+
331
+ `ArtifactSweep` is the service tag, with identity
332
+ `@smthrs/artifacts/ArtifactSweep`.
333
+
334
+ | Export | Fields |
335
+ | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
336
+ | `BlobStat` | `digest`, `modifiedAtMs`, `sizeBytes`. |
337
+ | `RemoveOptions` | `ifUnmodifiedSinceMs`: delete only while the blob's mtime is at or before this bound. |
338
+ | `SweepOptions` | `directory` (defaults to `ArtifactStore.defaultDirectory`) and `coordination` (defaults to `required`). No `durability`, because a sweep never writes a blob. |
339
+
340
+ `inventory` returns a `BlobStat` only for canonical fanout files whose
341
+ modification time is measurable. It skips temp files, lock files, foreign
342
+ paths, nested paths, directories sitting at blob addresses, and entries that
343
+ disappear during the scan. A directory that was never created is an empty
344
+ inventory, not a failure.
345
+
346
+ Candidate metadata reads run with concurrency 16 and retain directory-listing
347
+ order. Each candidate keeps its path checks; a failed metadata read skips only
348
+ that candidate.
349
+
350
+ `remove` returns `false` for three outcomes a caller cannot tell apart, all of
351
+ which mean nothing was reclaimed and retrying later is safe: the blob was
352
+ already gone, it failed the `ifUnmodifiedSinceMs` fence, or a live backup lease
353
+ fenced the deletion. The last one is not progress, so a collector counting
354
+ reclaimed bytes must not count it.
355
+
356
+ Under `required`, one deletion costs two lock acquisitions, the per-digest lock
357
+ and the workspace-global backup-lease gate, which is roughly ten filesystem
358
+ operations and two forked heartbeat fibers per blob. The gate is one file for
359
+ the whole workspace, so concurrent deletions serialize through it.
360
+
361
+ The policy this mechanism serves, which digests are live and how long a dead
362
+ blob is kept, belongs to [`@smthrs/engine-store`](/api/engine-store)'s
363
+ `ArtifactGc`. See [Reclaim disk space](./guides/reclaim-disk-space.md).
364
+
365
+ ## ArtifactBackupLease
366
+
367
+ `@smthrs/artifacts/ArtifactBackupLease`. Cross-process exclusion between a
368
+ filesystem backup and sweep deletion.
369
+
370
+ ```ts
371
+ declare const withLease: <A, E, R, E2>(
372
+ fs: FileSystem,
373
+ directory: string,
374
+ effect: Effect<A, E, R>,
375
+ failure: (cause: unknown) => E2
376
+ ) => Effect<A, E | E2, R>
377
+
378
+ declare const unlessActive: <A, E, R, E2>(
379
+ fs: FileSystem,
380
+ directory: string,
381
+ effect: Effect<A, E, R>,
382
+ failure: (cause: unknown) => E2
383
+ ) => Effect<Option<A>, E | E2, R>
384
+ ```
385
+
386
+ `withLease` holds a heartbeat-backed marker while `effect` runs, so no sweep in
387
+ any process using `directory` can delete a blob. Publication stays
388
+ unconstrained: new blobs may appear during a backup, but a blob already
389
+ referenced by the frozen database cannot disappear underneath it. Acquisition
390
+ gives up after 2 minutes and fails through `failure`. A crashed lease becomes
391
+ reclaimable once its heartbeat is 60 seconds stale, and finding no marker to
392
+ release is the ordinary end of a slow lease rather than a fault.
393
+
394
+ `unlessActive` checks the marker and runs `effect` under the same
395
+ workspace-global gate, so the check and the operation cannot be separated. It
396
+ returns `None` when a live backup deliberately fenced the operation, which
397
+ `ArtifactSweep.remove` reports as `false`.
398
+
399
+ `failure` maps an unknown host cause into the caller's error type, which is how
400
+ these combinators stay usable from a tool with its own error channel. See
401
+ [Fence a backup against the sweep](./guides/fence-a-backup.md).
402
+
403
+ ## FileLease
404
+
405
+ `@smthrs/artifacts/FileLease`. The cross-process lock-file lease under the
406
+ per-digest locks and `ArtifactBackupLease`, exported for other packages that
407
+ need the same fence; `@smthrs/engine-store` uses it for the workspace commit
408
+ lock.
409
+
410
+ ```ts
411
+ declare const withLease: <A, E, R, E2>(
412
+ fs: FileSystem,
413
+ lockPath: string,
414
+ effect: Effect<A, E, R>,
415
+ failure: (cause: unknown) => E2,
416
+ options?: Options
417
+ ) => Effect<A, E | E2, R>
418
+
419
+ declare const hold: <A, E, R, E2>(
420
+ fs: FileSystem,
421
+ lockPath: string,
422
+ effect: Effect<A, E, R>,
423
+ failure: (cause: unknown) => E2,
424
+ options?: HoldOptions
425
+ ) => Effect<A, E | E2, R>
426
+ ```
427
+
428
+ `withLease` creates `lockPath` with `wx`, writes a unique owner token, and
429
+ refreshes its mtime while `effect` runs. It reclaims a lock whose mtime is
430
+ older than the stale bound, at most once per lock generation, and fails with
431
+ `failure(TimeoutError)` when acquisition misses its deadline. Every other host
432
+ refusal reaches `failure` as the `PlatformError`. A directory at `lockPath` is
433
+ treated as a lock with no readable owner and reclaimed on the same bound.
434
+ `hold` is the same lease without a deadline, for a caller that bounds a wider
435
+ wait itself; `onAcquired` runs once the lease is held. `guard`, when given,
436
+ runs before every create, heartbeat, rename, and remove of the lock, claim, or
437
+ tombstone, and its failure refuses that step through `failure`. Both fail
438
+ through `failure` with a `RangeError`, before touching the lock, unless
439
+ `0 < heartbeatEvery < staleAfterMs` and `staleAfterMs` is finite: a shorter
440
+ stale bound would let a contender reclaim a live holder's lock.
441
+
442
+ | Option | Default | Meaning |
443
+ | ---------------- | -------------- | ------------------------------------------------ |
444
+ | `heartbeatEvery` | `"10 seconds"` | How often the holder refreshes the lock's mtime. |
445
+ | `staleAfterMs` | `60_000` | Age past which a lock or reclaim claim is stale. |
446
+ | `acquireWithin` | `"2 minutes"` | Acquisition deadline (`withLease` only). |
447
+ | `retryEvery` | `"25 millis"` | Sleep between attempts while contended. |
448
+ | `label` | `"File lease"` | Names the lock in the reclaimed-holder warning. |
449
+ | `annotations` | none | Extra fields logged with that warning. |
450
+
451
+ The host needs exclusive `wx` file creation, `readFileString`, `stat` with an
452
+ mtime, `utimes`, `rename`, and `remove`. Reclaim writes siblings named
453
+ `<lockPath>.reclaim-<token>` and `<lockPath>.stale-<token>`. The fence is
454
+ bounded: see [Coordination](./concepts/coordination.md).
455
+
456
+ ## ArtifactStoreMetrics
457
+
458
+ `@smthrs/artifacts/ArtifactStoreMetrics`. Metric handles only; no exporter
459
+ ships in this package.
460
+
461
+ | Export | Counter | Updated by |
462
+ | ---------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------- |
463
+ | `puts` | `flows_artifact_puts` | Successful filesystem and memory puts, including deduplicated ones. |
464
+ | `gets` | `flows_artifact_gets` | Successful filesystem and memory gets. |
465
+ | `remoteFailures` | `flows_artifact_remote_failures` | Shared uploads and local write-backs `CombinedArtifacts` dropped, by `operation` (`put`, `write_back`). |
466
+
467
+ `remoteFailure.put` and `remoteFailure.write_back` are the attributed views.
468
+ Drops also log a warning naming the operation and error code, at most once a
469
+ minute per operation.
470
+
471
+ `RemoteArtifacts` is deliberately uninstrumented and the counters carry no tier
472
+ attribute, so read them as local artifact store traffic. A combined read the
473
+ shared tier serves increments no get; the write-back that materializes it
474
+ increments a put indistinguishable from a producer publishing new bytes.
475
+ Missing and corrupt reads are counted nowhere: they are error evidence, not
476
+ throughput.
477
+
478
+ ## Reclaiming published blobs
479
+
480
+ Reclaiming published blobs never happens as a side effect of a store call. The
481
+ filesystem store sweeps only stale scratch and lock files. The mark policy and
482
+ grace period for published blobs belong to
483
+ [`@smthrs/engine-store`](/api/engine-store), which also owns artifact
484
+ publication ordering.