@smthrs/artifacts 0.0.0-stage → 1.0.0-rc.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (171) hide show
  1. package/CHANGELOG.md +77 -0
  2. package/LICENSE +21 -0
  3. package/README.md +173 -2
  4. package/dist/cjs/ArtifactBackupLease.d.ts +34 -0
  5. package/dist/cjs/ArtifactBackupLease.d.ts.map +1 -0
  6. package/dist/cjs/ArtifactBackupLease.js +175 -0
  7. package/dist/cjs/ArtifactBackupLease.js.map +7 -0
  8. package/dist/cjs/ArtifactStore.d.ts +129 -0
  9. package/dist/cjs/ArtifactStore.d.ts.map +1 -0
  10. package/dist/cjs/ArtifactStore.js +73 -0
  11. package/dist/cjs/ArtifactStore.js.map +7 -0
  12. package/dist/cjs/ArtifactStoreError.d.ts +82 -0
  13. package/dist/cjs/ArtifactStoreError.d.ts.map +1 -0
  14. package/dist/cjs/ArtifactStoreError.js +72 -0
  15. package/dist/cjs/ArtifactStoreError.js.map +7 -0
  16. package/dist/cjs/ArtifactStoreMetrics.d.ts +70 -0
  17. package/dist/cjs/ArtifactStoreMetrics.d.ts.map +1 -0
  18. package/dist/cjs/ArtifactStoreMetrics.js +51 -0
  19. package/dist/cjs/ArtifactStoreMetrics.js.map +7 -0
  20. package/dist/cjs/ArtifactSweep.d.ts +185 -0
  21. package/dist/cjs/ArtifactSweep.d.ts.map +1 -0
  22. package/dist/cjs/ArtifactSweep.js +163 -0
  23. package/dist/cjs/ArtifactSweep.js.map +7 -0
  24. package/dist/cjs/CombinedArtifacts.d.ts +103 -0
  25. package/dist/cjs/CombinedArtifacts.d.ts.map +1 -0
  26. package/dist/cjs/CombinedArtifacts.js +189 -0
  27. package/dist/cjs/CombinedArtifacts.js.map +7 -0
  28. package/dist/cjs/FileLease.d.ts +113 -0
  29. package/dist/cjs/FileLease.d.ts.map +1 -0
  30. package/dist/cjs/FileLease.js +201 -0
  31. package/dist/cjs/FileLease.js.map +7 -0
  32. package/dist/cjs/FileSystemArtifactStore.d.ts +93 -0
  33. package/dist/cjs/FileSystemArtifactStore.d.ts.map +1 -0
  34. package/dist/cjs/FileSystemArtifactStore.js +276 -0
  35. package/dist/cjs/FileSystemArtifactStore.js.map +7 -0
  36. package/dist/cjs/MemoryArtifactStore.d.ts +25 -0
  37. package/dist/cjs/MemoryArtifactStore.d.ts.map +1 -0
  38. package/dist/cjs/MemoryArtifactStore.js +84 -0
  39. package/dist/cjs/MemoryArtifactStore.js.map +7 -0
  40. package/dist/cjs/NoopArtifactStore.d.ts +17 -0
  41. package/dist/cjs/NoopArtifactStore.d.ts.map +1 -0
  42. package/dist/cjs/NoopArtifactStore.js +46 -0
  43. package/dist/cjs/NoopArtifactStore.js.map +7 -0
  44. package/dist/cjs/RemoteArtifacts.d.ts +237 -0
  45. package/dist/cjs/RemoteArtifacts.d.ts.map +1 -0
  46. package/dist/cjs/RemoteArtifacts.js +405 -0
  47. package/dist/cjs/RemoteArtifacts.js.map +7 -0
  48. package/dist/cjs/index.d.ts +65 -0
  49. package/dist/cjs/index.d.ts.map +1 -0
  50. package/dist/cjs/index.js +47 -0
  51. package/dist/cjs/index.js.map +7 -0
  52. package/dist/cjs/internal/ArtifactLocks.d.ts +30 -0
  53. package/dist/cjs/internal/ArtifactLocks.d.ts.map +1 -0
  54. package/dist/cjs/internal/ArtifactLocks.js +93 -0
  55. package/dist/cjs/internal/ArtifactLocks.js.map +7 -0
  56. package/dist/cjs/internal/ArtifactPath.d.ts +19 -0
  57. package/dist/cjs/internal/ArtifactPath.d.ts.map +1 -0
  58. package/dist/cjs/internal/ArtifactPath.js +74 -0
  59. package/dist/cjs/internal/ArtifactPath.js.map +7 -0
  60. package/dist/cjs/measureBytes.d.ts +18 -0
  61. package/dist/cjs/measureBytes.d.ts.map +1 -0
  62. package/dist/cjs/measureBytes.js +43 -0
  63. package/dist/cjs/measureBytes.js.map +7 -0
  64. package/dist/cjs/package.json +1 -0
  65. package/dist/cjs/snapshotBytes.d.ts +19 -0
  66. package/dist/cjs/snapshotBytes.d.ts.map +1 -0
  67. package/dist/cjs/snapshotBytes.js +41 -0
  68. package/dist/cjs/snapshotBytes.js.map +7 -0
  69. package/dist/cjs/validateDigest.d.ts +22 -0
  70. package/dist/cjs/validateDigest.d.ts.map +1 -0
  71. package/dist/cjs/validateDigest.js +42 -0
  72. package/dist/cjs/validateDigest.js.map +7 -0
  73. package/dist/esm/ArtifactBackupLease.d.ts +34 -0
  74. package/dist/esm/ArtifactBackupLease.d.ts.map +1 -0
  75. package/dist/esm/ArtifactBackupLease.js +117 -0
  76. package/dist/esm/ArtifactBackupLease.js.map +1 -0
  77. package/dist/esm/ArtifactStore.d.ts +129 -0
  78. package/dist/esm/ArtifactStore.d.ts.map +1 -0
  79. package/dist/esm/ArtifactStore.js +83 -0
  80. package/dist/esm/ArtifactStore.js.map +1 -0
  81. package/dist/esm/ArtifactStoreError.d.ts +82 -0
  82. package/dist/esm/ArtifactStoreError.d.ts.map +1 -0
  83. package/dist/esm/ArtifactStoreError.js +77 -0
  84. package/dist/esm/ArtifactStoreError.js.map +1 -0
  85. package/dist/esm/ArtifactStoreMetrics.d.ts +70 -0
  86. package/dist/esm/ArtifactStoreMetrics.d.ts.map +1 -0
  87. package/dist/esm/ArtifactStoreMetrics.js +77 -0
  88. package/dist/esm/ArtifactStoreMetrics.js.map +1 -0
  89. package/dist/esm/ArtifactSweep.d.ts +185 -0
  90. package/dist/esm/ArtifactSweep.d.ts.map +1 -0
  91. package/dist/esm/ArtifactSweep.js +202 -0
  92. package/dist/esm/ArtifactSweep.js.map +1 -0
  93. package/dist/esm/CombinedArtifacts.d.ts +103 -0
  94. package/dist/esm/CombinedArtifacts.d.ts.map +1 -0
  95. package/dist/esm/CombinedArtifacts.js +214 -0
  96. package/dist/esm/CombinedArtifacts.js.map +1 -0
  97. package/dist/esm/FileLease.d.ts +113 -0
  98. package/dist/esm/FileLease.d.ts.map +1 -0
  99. package/dist/esm/FileLease.js +239 -0
  100. package/dist/esm/FileLease.js.map +1 -0
  101. package/dist/esm/FileSystemArtifactStore.d.ts +93 -0
  102. package/dist/esm/FileSystemArtifactStore.d.ts.map +1 -0
  103. package/dist/esm/FileSystemArtifactStore.js +336 -0
  104. package/dist/esm/FileSystemArtifactStore.js.map +1 -0
  105. package/dist/esm/MemoryArtifactStore.d.ts +25 -0
  106. package/dist/esm/MemoryArtifactStore.d.ts.map +1 -0
  107. package/dist/esm/MemoryArtifactStore.js +67 -0
  108. package/dist/esm/MemoryArtifactStore.js.map +1 -0
  109. package/dist/esm/NoopArtifactStore.d.ts +17 -0
  110. package/dist/esm/NoopArtifactStore.d.ts.map +1 -0
  111. package/dist/esm/NoopArtifactStore.js +27 -0
  112. package/dist/esm/NoopArtifactStore.js.map +1 -0
  113. package/dist/esm/RemoteArtifacts.d.ts +237 -0
  114. package/dist/esm/RemoteArtifacts.d.ts.map +1 -0
  115. package/dist/esm/RemoteArtifacts.js +554 -0
  116. package/dist/esm/RemoteArtifacts.js.map +1 -0
  117. package/dist/esm/index.d.ts +65 -0
  118. package/dist/esm/index.d.ts.map +1 -0
  119. package/dist/esm/index.js +65 -0
  120. package/dist/esm/index.js.map +1 -0
  121. package/dist/esm/internal/ArtifactLocks.d.ts +30 -0
  122. package/dist/esm/internal/ArtifactLocks.d.ts.map +1 -0
  123. package/dist/esm/internal/ArtifactLocks.js +86 -0
  124. package/dist/esm/internal/ArtifactLocks.js.map +1 -0
  125. package/dist/esm/internal/ArtifactPath.d.ts +19 -0
  126. package/dist/esm/internal/ArtifactPath.d.ts.map +1 -0
  127. package/dist/esm/internal/ArtifactPath.js +59 -0
  128. package/dist/esm/internal/ArtifactPath.js.map +1 -0
  129. package/dist/esm/measureBytes.d.ts +18 -0
  130. package/dist/esm/measureBytes.d.ts.map +1 -0
  131. package/dist/esm/measureBytes.js +22 -0
  132. package/dist/esm/measureBytes.js.map +1 -0
  133. package/dist/esm/snapshotBytes.d.ts +19 -0
  134. package/dist/esm/snapshotBytes.d.ts.map +1 -0
  135. package/dist/esm/snapshotBytes.js +30 -0
  136. package/dist/esm/snapshotBytes.js.map +1 -0
  137. package/dist/esm/validateDigest.d.ts +22 -0
  138. package/dist/esm/validateDigest.d.ts.map +1 -0
  139. package/dist/esm/validateDigest.js +26 -0
  140. package/dist/esm/validateDigest.js.map +1 -0
  141. package/docs/README.md +200 -0
  142. package/docs/api.md +474 -0
  143. package/docs/concepts/content-addressing.md +174 -0
  144. package/docs/concepts/coordination.md +141 -0
  145. package/docs/concepts/tiers.md +147 -0
  146. package/docs/guides/fence-a-backup.md +123 -0
  147. package/docs/guides/reclaim-disk-space.md +143 -0
  148. package/docs/guides/serve-the-artifact-protocol.md +140 -0
  149. package/docs/guides/share-artifacts-across-machines.md +180 -0
  150. package/docs/guides/test-against-an-artifact-store.md +158 -0
  151. package/docs/installation.md +98 -0
  152. package/docs/quickstart.md +182 -0
  153. package/docs/troubleshooting.md +232 -0
  154. package/package.json +179 -3
  155. package/src/ArtifactBackupLease.ts +213 -0
  156. package/src/ArtifactStore.ts +152 -0
  157. package/src/ArtifactStoreError.ts +96 -0
  158. package/src/ArtifactStoreMetrics.ts +83 -0
  159. package/src/ArtifactSweep.ts +336 -0
  160. package/src/CombinedArtifacts.ts +369 -0
  161. package/src/FileLease.ts +380 -0
  162. package/src/FileSystemArtifactStore.ts +459 -0
  163. package/src/MemoryArtifactStore.ts +83 -0
  164. package/src/NoopArtifactStore.ts +30 -0
  165. package/src/RemoteArtifacts.ts +809 -0
  166. package/src/index.ts +71 -0
  167. package/src/internal/ArtifactLocks.ts +110 -0
  168. package/src/internal/ArtifactPath.ts +72 -0
  169. package/src/measureBytes.ts +28 -0
  170. package/src/snapshotBytes.ts +34 -0
  171. package/src/validateDigest.ts +31 -0
@@ -0,0 +1,140 @@
1
+ ---
2
+ title: "Serve the artifact protocol"
3
+ description: "The HTTP surface a shared artifact tier owes RemoteArtifacts: four requests, the status codes that mean hit, miss, and failure, the body bounds, and the optional resumable upload sequence."
4
+ sidebar:
5
+ order: 2
6
+ ---
7
+
8
+ `RemoteArtifacts` speaks a dumb-HTTP content-addressed cache protocol, the one
9
+ Bazel's `HttpCacheClient` documents, with two additions. If you are writing the
10
+ service on the other end, this page is the contract.
11
+
12
+ Every digest in a path or a body is 64 lowercase hexadecimal characters. The
13
+ client validates that before a request leaves, so a malformed address never
14
+ reaches your service.
15
+
16
+ ## The four requests
17
+
18
+ All paths are resolved beneath the configured endpoint. A trailing slash on the
19
+ endpoint is ignored.
20
+
21
+ | Request | Conforming answer | Anything else |
22
+ | ----------------------- | ---------------------------------------------------- | -------------------------------------- |
23
+ | `GET /cas/{digest}` | `2xx` with the blob bytes, or `404` for a miss | Fails the read as `transport_failed` |
24
+ | `PUT /cas/{digest}` | Any `2xx` once the blob is durable | Fails the upload as `transport_failed` |
25
+ | `HEAD /cas/{digest}` | `2xx` when the blob is present, `404` when it is not | Fails the probe as `transport_failed` |
26
+ | `POST /cas/findMissing` | `200` with `{"missing":[...]}` | Fails the batch as `transport_failed` |
27
+
28
+ `404` is the only status that means "miss". Every other non-2xx is a failure of
29
+ the tier, which is the only classification a dumb-HTTP cache can support,
30
+ because there is no richer error envelope on the wire.
31
+
32
+ ## What the client does with your answers
33
+
34
+ Knowing this saves you from defending against problems the client already
35
+ handles, and from assuming defenses it does not have.
36
+
37
+ - **Every download is digest verified.** A `GET` body that does not hash to the
38
+ requested address is rejected with `ArtifactCorruption`, never returned. Your
39
+ service cannot substitute content, and it does not need to prove it did not.
40
+ - **An absent body is zero bytes, not an error.** The digest check decides
41
+ whether empty content is the requested artifact.
42
+ - **`Content-Length` is read strictly.** It must be plain decimal digits naming
43
+ a safe integer. A malformed one fails the read before the body is buffered.
44
+ - **Downloads are bounded.** A declared length past `maxDownloadBytes`, 256 MiB
45
+ by default, is refused before a body byte is read, and an incremental read
46
+ stops one chunk past the bound.
47
+ - **`findMissing` answers are filtered.** The client keeps only digests it
48
+ asked about, in request order. A digest you return that nobody asked for
49
+ makes the client upload nothing.
50
+
51
+ ## The findMissing body
52
+
53
+ Request, sent as `application/json`:
54
+
55
+ ```json
56
+ { "digests": ["077303668cf56af8d162bb5ccccd7127f2e8baff448bf5b649530a98e9c943da"] }
57
+ ```
58
+
59
+ Response, also JSON:
60
+
61
+ ```json
62
+ { "missing": ["077303668cf56af8d162bb5ccccd7127f2e8baff448bf5b649530a98e9c943da"] }
63
+ ```
64
+
65
+ The client enforces three bounds on its side, and a conforming service can rely
66
+ on them:
67
+
68
+ | Bound | Value |
69
+ | ------------------- | ------------------------------------------------------------------------------------------------------------- |
70
+ | Digests per request | At most 1,000. A larger input is split into several batches. |
71
+ | Request body | At most 256 KiB, which the 1,000-digest cap already guarantees. |
72
+ | Response body | At most `maxFindMissingResponseBytes`, 256 KiB by default, and that option may only lower the protocol bound. |
73
+
74
+ An empty input never leaves the process, so your service never sees a
75
+ zero-digest batch.
76
+
77
+ ## The optional resumable upload
78
+
79
+ A client with `RemoteArtifacts.Options.chunkBytes` set sends a large blob as a
80
+ sequence of ranged `PUT` requests. Supporting this is optional, and refusing it
81
+ correctly costs the client one round trip.
82
+
83
+ The sequence is:
84
+
85
+ 1. `HEAD /cas/{digest}`. Answer `2xx` with a `Content-Length` equal to the
86
+ whole blob if you already hold it, and the client sends nothing more.
87
+ 2. `PUT /cas/{digest}` with `Content-Range: bytes */{total}` and an empty body.
88
+ Answer `308` with `Range: bytes=0-{last}` naming the prefix you hold, or
89
+ omit the header if you hold none.
90
+ 3. `PUT /cas/{digest}` with `Content-Range: bytes {a}-{b}/{total}` per chunk.
91
+ Answer `308` to continue, and a `2xx` on the chunk that completes the blob.
92
+ 4. `HEAD /cas/{digest}` again. The client confirms the stored length before it
93
+ reports the digest as published.
94
+
95
+ Only `308` continues the sequence. A `2xx` to the empty probe, or to a chunk
96
+ that does not complete the blob, reads as a tier that ignored `Content-Range`
97
+ and stored the body it was handed, which is what plain WebDAV `PUT` does.
98
+
99
+ **To refuse ranged uploads,** answer `400`, `411`, or `416`. `400` is RFC 9110
100
+ section 14.5's answer from a resource that does not support partial `PUT`, and
101
+ it is what the [reference implementation](#the-reference-implementation) below
102
+ sends. The client then sends one whole-blob `PUT`, which overwrites any partial
103
+ body the sequence left behind, so the blob always lands whole.
104
+
105
+ Falling back on `400` cannot mask a genuine `400` from your service: the
106
+ whole-blob `PUT` that follows presents the same URL, digest, and credential, so
107
+ a real refusal comes straight back on that request.
108
+
109
+ ## The reference implementation
110
+
111
+ Smithers ships a conforming service, deployed as a Cloudflare Worker over R2
112
+ and D1, and a self-hosted build of the same protocol. Its behavior is a useful
113
+ default when you are deciding what your own should do:
114
+
115
+ | Request | What it does |
116
+ | ----------------------- | ---------------------------------------------------------------------------------------- |
117
+ | `GET /cas/{digest}` | Streams the object as `application/octet-stream`; missing objects return `404` |
118
+ | `PUT /cas/{digest}` | Hashes the complete upload before an atomic publication; a digest mismatch returns `400` |
119
+ | `HEAD /cas/{digest}` | Checks for the object without returning a body; missing objects return `404` |
120
+ | `POST /cas/findMissing` | Accepts `{"digests":[...]}` and returns unique missing digests in request order |
121
+
122
+ Its bounds: a `/cas` upload is capped at 16 MiB and must be
123
+ `application/octet-stream`; a `findMissing` request is capped at 256 KiB, 1,000
124
+ digests, and `application/json`. Reads accept either credential; `PUT` and
125
+ `DELETE` require the write credential and answer `403` to the read one, before
126
+ the request body is read.
127
+
128
+ Because it caps a request body at 16 MiB and refuses ranged `PUT` with `400`, a
129
+ blob larger than 16 MiB is refused with `413` whether or not `chunkBytes` is
130
+ set. See [`@smthrs/build`](https://github.com/smithersai/smithers/tree/main/packages/smithers/build) for how that service is
131
+ deployed and configured.
132
+
133
+ ## Related
134
+
135
+ - [Share artifacts across machines](./share-artifacts-across-machines.md): the
136
+ client side of this protocol.
137
+ - [The three tiers](../concepts/tiers.md): what the combined store does with a
138
+ miss, a failure, and a corrupt address.
139
+ - [Troubleshooting](../troubleshooting.md): the transport failures a
140
+ nonconforming answer produces.
@@ -0,0 +1,180 @@
1
+ ---
2
+ title: "Share artifacts across machines"
3
+ description: "Compose a local filesystem store behind an HTTP cache with CombinedArtifacts, choose a download policy, and set the deadlines and chunk size the shared tier needs."
4
+ sidebar:
5
+ order: 1
6
+ ---
7
+
8
+ A machine that reads an artifact another machine produced needs two tiers: the
9
+ fast local one it owns, and a shared HTTP one it does not. `CombinedArtifacts`
10
+ composes them so a read tries local first and a write is durable locally before
11
+ the network is involved at all.
12
+
13
+ ## Before you start
14
+
15
+ - A shared cache endpoint that speaks the artifact protocol. It must be HTTPS,
16
+ unless its host is loopback, and it may carry no userinfo, query, or fragment. If you are standing one up,
17
+ see [Serve the artifact protocol](./serve-the-artifact-protocol.md).
18
+ - The credential the endpoint expects, as a header value.
19
+ - An `HttpClient` in scope. `FetchHttpClient.layer` from `effect` is enough.
20
+
21
+ ## 1. Build the shared tier
22
+
23
+ `RemoteArtifacts.make` validates its options and returns a store. The endpoint
24
+ and its headers are construction options, never step inputs: a credential that
25
+ arrived as an input would be hashed into a step key and persisted everywhere
26
+ the journal goes.
27
+
28
+ ```ts
29
+ import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
30
+ import * as CombinedArtifacts from "@smthrs/artifacts/CombinedArtifacts"
31
+ import * as RemoteArtifacts from "@smthrs/artifacts/RemoteArtifacts"
32
+ import * as Effect from "effect/Effect"
33
+ import * as FileSystem from "effect/FileSystem"
34
+ import * as Path from "effect/Path"
35
+
36
+ declare const token: string
37
+
38
+ const combined = Effect.gen(function*() {
39
+ const fileSystem = yield* FileSystem.FileSystem
40
+ const local = ArtifactStore.makeFileSystem(fileSystem, yield* Path.Path)
41
+ const remote = yield* RemoteArtifacts.make({
42
+ endpoint: "https://cas.example.com",
43
+ headers: { authorization: `Bearer ${token}` }
44
+ })
45
+ return yield* CombinedArtifacts.make({ local, remote })
46
+ })
47
+ ```
48
+
49
+ Construction fails with `invalid_configuration` for a value that is not a
50
+ string, one no `URL` parser accepts, a scheme other than `https:`, and an
51
+ endpoint carrying userinfo, a query, or a fragment. The refusal names only the
52
+ violated rule and never echoes the endpoint, so a rejected
53
+ `https://user:secret@host` cannot leak its credential into a log line.
54
+
55
+ Your own headers are applied first and the protocol's are applied after, so no
56
+ configuration can strip the `content-type`, `content-length`, or
57
+ `content-range` the tier needs to interpret the body.
58
+
59
+ ## 2. Provide the composition as a layer
60
+
61
+ `CombinedArtifacts.layer` takes both tiers as effects rather than layers,
62
+ because both inhabit the `ArtifactStore` tag and merging two layers of one tag
63
+ would shadow one with the other:
64
+
65
+ ```ts
66
+ import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
67
+ import * as CombinedArtifacts from "@smthrs/artifacts/CombinedArtifacts"
68
+ import * as RemoteArtifacts from "@smthrs/artifacts/RemoteArtifacts"
69
+ import * as Effect from "effect/Effect"
70
+ import * as FileSystem from "effect/FileSystem"
71
+ import * as Path from "effect/Path"
72
+
73
+ declare const token: string
74
+
75
+ export const sharedArtifacts = CombinedArtifacts.layer({
76
+ local: Effect.map(
77
+ Effect.all([FileSystem.FileSystem, Path.Path]),
78
+ ([fs, path]) => ArtifactStore.makeFileSystem(fs, path, { directory: ".flows/objects" })
79
+ ),
80
+ remote: RemoteArtifacts.make({
81
+ endpoint: "https://cas.example.com",
82
+ headers: { authorization: `Bearer ${token}` },
83
+ downloadPolicy: "toplevel",
84
+ chunkBytes: 8 * 1024 * 1024
85
+ })
86
+ })
87
+ ```
88
+
89
+ The layer requires whatever its two effects require, here a `FileSystem`, a
90
+ `Path`, and an `HttpClient`, and it provides `ArtifactStore` to everything above it.
91
+
92
+ ## 3. Choose a download policy
93
+
94
+ The policy answers one question: how eagerly does this machine copy other
95
+ machines' artifacts onto its own disk?
96
+
97
+ | Choose | When |
98
+ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
99
+ | `all` | The default. A long-lived machine that will read most of what it fetches, and wants replay to read local bytes. |
100
+ | `toplevel` | A machine that should not prefetch a whole run's artifacts, but should keep the ones it actually reads. |
101
+ | `minimal` | An ephemeral or disk-constrained host: a CI container, a sandbox. Reads are served straight from the shared tier and the local store never grows. |
102
+
103
+ Declare it once on the shared tier, as above, and every composition reading
104
+ through that tier inherits it. `CombinedArtifacts.Options.downloadPolicy`
105
+ overrides it for one composition when a single host needs to differ.
106
+
107
+ `minimal` still writes back after local corruption. That is repair of an
108
+ address the local tier already claims, not growth.
109
+
110
+ ## 4. Set the deadlines
111
+
112
+ Every remote exchange is bounded, and every bound defaults to 60 seconds,
113
+ Bazel's `--remote_timeout` default for the same protocol:
114
+
115
+ | Option | Covers |
116
+ | ----------------------------------------- | --------------------------------------------------------------------------------- |
117
+ | `RemoteArtifacts.Options.downloadTimeout` | One download, including the complete response body |
118
+ | `RemoteArtifacts.Options.uploadTimeout` | One upload, including every resume probe and chunk |
119
+ | `RemoteArtifacts.Options.requestTimeout` | One `HEAD` probe, or one `findMissing` batch and its response |
120
+ | `CombinedArtifacts.Options.uploadTimeout` | How long a combined `put` waits for its opportunistic upload before abandoning it |
121
+
122
+ Two more bounds protect this process from a mis-serving tier.
123
+ `maxDownloadBytes` defaults to 256 MiB: a `Content-Length` past the bound is
124
+ refused before a body byte is read, and an incremental read stops the moment it
125
+ crosses. `maxFindMissingResponseBytes` defaults to the protocol's 256 KiB and
126
+ may only lower that bound.
127
+
128
+ ## 5. Turn on chunked uploads, if your tier supports them
129
+
130
+ With `chunkBytes` set, a blob past that size travels as a sequence of
131
+ `Content-Range` `PUT` requests, preceded by a `HEAD` existence probe and an
132
+ empty `bytes */{total}` probe that asks what prefix the tier already holds.
133
+
134
+ | Tier's answer | Client action |
135
+ | ----------------------------- | ---------------------------------------------------------- |
136
+ | `308` | Continue after the reported `Range: bytes=0-{last}` prefix |
137
+ | `2xx` on the completing chunk | Confirm the stored length with `HEAD` |
138
+ | `2xx` before completion | Treat the tier as range-unaware and send the whole blob |
139
+ | `400`, `411`, or `416` | Send the whole blob |
140
+ | Anything else | Fail the upload as a transport error |
141
+
142
+ The offset never moves backward, and a tier that ignores ranges, omits a
143
+ confirming length, or keeps only part of the body gets one whole-blob `PUT`
144
+ that overwrites whatever the sequence left behind. So the cost of turning the
145
+ dial on against a server that never learned about it is round trips, never a
146
+ digest published over bytes the server does not hold.
147
+
148
+ Set it when a proxy caps request bodies, or when artifacts are large enough
149
+ that losing a transfer partway is expensive. It is absent by default.
150
+
151
+ The hosted and self-hosted Smithers cache services do not implement the
152
+ resumable sequence: both answer a ranged `PUT` with `400` and cap one request
153
+ body at 16 MiB. Against them `chunkBytes` costs the probe round trip
154
+ and the blob travels whole, and a blob past 16 MiB is refused with `413` in
155
+ either mode.
156
+
157
+ ## What you get
158
+
159
+ A read that misses locally reaches the shared tier, and the fetched bytes are
160
+ digest verified before they are returned, because the shared tier is the least
161
+ trusted store there is: it is written by machines this one has never met. A
162
+ mis-serving or compromised cache can waste a round trip; it can never
163
+ substitute content.
164
+
165
+ A write is durable locally before the upload starts, and the upload's failure
166
+ is dropped rather than propagated. Ordering between the artifact tier and the
167
+ step cache is not this package's job:
168
+ [`@smthrs/engine-store`](/api/engine-store)'s `ArtifactSync` enforces that a
169
+ cache entry is never observable in the shared tier while an artifact it
170
+ references is missing from the shared artifact tier.
171
+
172
+ ## Related
173
+
174
+ - [The three tiers](../concepts/tiers.md): what falls through, what does not,
175
+ and why.
176
+ - [Serve the artifact protocol](./serve-the-artifact-protocol.md): the HTTP
177
+ surface a conforming shared tier owes.
178
+ - [Share results with artifacts and the step cache](/docs/guides/artifacts-cache/):
179
+ the end-to-end composition on smithers.sh, including the step cache's own
180
+ shared tier.
@@ -0,0 +1,158 @@
1
+ ---
2
+ title: "Test against an artifact store"
3
+ description: "Use the in-memory store as the default test tier, the no-op stores with per-method overrides to script a refusing host, and a real loopback server to exercise the HTTP tier."
4
+ sidebar:
5
+ order: 5
6
+ ---
7
+
8
+ Every tier in this package is a value you construct, so a test swaps the tier
9
+ rather than mocking a module. Three substitutions cover almost everything:
10
+ memory for a real directory, a no-op with overrides for a refusing host, and a
11
+ loopback server for the shared tier.
12
+
13
+ ## Use the memory store as the default
14
+
15
+ `ArtifactStore.makeMemory()` needs no directory, no temporary files, and no
16
+ cleanup, and it computes real digests through the same `Crypto` service the
17
+ filesystem store uses:
18
+
19
+ ```ts
20
+ import * as NodeCrypto from "@effect/platform-node/NodeCrypto"
21
+ import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
22
+ import * as Effect from "effect/Effect"
23
+
24
+ const payload = new TextEncoder().encode("a step's output")
25
+
26
+ const roundTrip = Effect.gen(function*() {
27
+ const store = ArtifactStore.makeMemory()
28
+ const digest = yield* store.put(payload)
29
+ return new TextDecoder().decode(yield* store.get(digest))
30
+ })
31
+
32
+ const result = await Effect.runPromise(roundTrip.pipe(Effect.provide(NodeCrypto.layer)))
33
+ ```
34
+
35
+ It copies on both boundaries, so a test that mutates the array it put still
36
+ reads the original bytes back. What it does not do is verify a read, because
37
+ its address space is private and cannot disagree with itself. A test that needs
38
+ a corrupt address needs the filesystem store and a directory to tamper with, as
39
+ in the [Quickstart](../quickstart.md#corrupt-a-blob-then-heal-it).
40
+
41
+ To assert on the address without hard-coding a digest, measure it the same way
42
+ the store does:
43
+
44
+ ```ts
45
+ import * as NodeCrypto from "@effect/platform-node/NodeCrypto"
46
+ import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
47
+ import * as Effect from "effect/Effect"
48
+
49
+ const expectedAddress = await Effect.runPromise(
50
+ ArtifactStore.measureBytes(new TextEncoder().encode("payload")).pipe(Effect.provide(NodeCrypto.layer))
51
+ )
52
+ ```
53
+
54
+ ## Script a refusing host with the no-op stores
55
+
56
+ `makeNoop` fails every operation with an `unavailable` `ArtifactStoreError`,
57
+ and takes per-method overrides, so a test states exactly the one behavior it
58
+ cares about:
59
+
60
+ ```ts
61
+ import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
62
+ import * as Effect from "effect/Effect"
63
+
64
+ /** A store whose reads always refuse, to prove a caller does not fall through. */
65
+ export const offline = ArtifactStore.makeNoop({
66
+ get: () =>
67
+ Effect.fail(
68
+ new ArtifactStore.ArtifactStoreError({ code: "unavailable", message: "the host filesystem is offline" })
69
+ )
70
+ })
71
+ ```
72
+
73
+ `ArtifactSweep.makeNoop` and `ArtifactSweep.layerNoop` work the same way:
74
+
75
+ ```ts
76
+ import * as ArtifactSweep from "@smthrs/artifacts/ArtifactSweep"
77
+ import * as Effect from "effect/Effect"
78
+
79
+ /** A sweep that enumerates nothing, so a collector's empty path is exercised. */
80
+ export const sweepless = ArtifactSweep.layerNoop({
81
+ inventory: Effect.succeed([])
82
+ })
83
+ ```
84
+
85
+ Use these to pin the distinctions the combined store makes. A local tier whose
86
+ `get` fails with `ArtifactStoreError` must not fall through to the shared tier;
87
+ one that fails with `ArtifactMissing` must.
88
+
89
+ ## Compose two memory tiers
90
+
91
+ `CombinedArtifacts` does not care what its tiers are, so the whole read-through
92
+ and write-back contract is testable with two maps:
93
+
94
+ ```ts
95
+ import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
96
+ import * as CombinedArtifacts from "@smthrs/artifacts/CombinedArtifacts"
97
+ import * as Effect from "effect/Effect"
98
+
99
+ const shared = Effect.gen(function*() {
100
+ const local = ArtifactStore.makeMemory()
101
+ const remote = ArtifactStore.makeMemory()
102
+ return yield* CombinedArtifacts.make({ local, remote })
103
+ })
104
+ ```
105
+
106
+ Seed the remote tier, read through the combination, and then probe the local
107
+ tier's `has` to prove whether the download policy wrote back.
108
+
109
+ ## Exercise the HTTP tier over a real socket
110
+
111
+ `RemoteArtifacts` accepts plain HTTP only for a loopback host. To exercise the
112
+ HTTPS authority a deployment declares, run a loopback HTTP server and rewrite
113
+ that authority onto its port, so the request, body, status code, and
114
+ connection are all real and only TLS termination is missing:
115
+
116
+ ```ts
117
+ import * as Layer from "effect/Layer"
118
+ import * as FetchHttpClient from "effect/unstable/http/FetchHttpClient"
119
+
120
+ /** The real fetch transport, with a declared HTTPS authority mapped to `port`. */
121
+ const transport = (port: number) =>
122
+ Layer.provide(
123
+ FetchHttpClient.layer,
124
+ Layer.succeed(
125
+ FetchHttpClient.Fetch,
126
+ ((input, init) => {
127
+ const href = input instanceof URL ? input.href : typeof input === "string" ? input : (input as Request).url
128
+ const url = new URL(href)
129
+ url.protocol = "http:"
130
+ url.host = `127.0.0.1:${port}`
131
+ return globalThis.fetch(url, init)
132
+ }) as typeof globalThis.fetch
133
+ )
134
+ )
135
+ ```
136
+
137
+ Then build the store against `https://cas.test` and provide `transport(port)`.
138
+ A stub `HttpClient` is the right tool for header-by-header assertions about a
139
+ single request; a real socket is the only way to assert that an interrupted
140
+ transfer resumes from the prefix a real server kept.
141
+
142
+ ## What not to stub
143
+
144
+ Do not replace the digest check. It is the invariant the whole package exists
145
+ to hold, and a test that stubs it proves nothing about the code that ships. To
146
+ test the corruption path, corrupt a real blob and let the real `get` refuse it.
147
+
148
+ Do not stub the `Crypto` service to return a constant either. Real digests are
149
+ cheap, and a constant digest makes deduplication, `findMissing`, and the
150
+ corruption check all indistinguishable from each other.
151
+
152
+ ## Related
153
+
154
+ - [Quickstart](../quickstart.md): the corruption and repair path against a real
155
+ directory.
156
+ - [The three tiers](../concepts/tiers.md): the distinctions your assertions
157
+ should pin.
158
+ - [API reference](../api.md): every constructor and its overrides.
@@ -0,0 +1,98 @@
1
+ ---
2
+ title: "Installation"
3
+ description: "Install @smthrs/artifacts, the runtime it requires, the Effect services each tier needs in scope, and the public import forms."
4
+ sidebar:
5
+ order: 1
6
+ ---
7
+
8
+ ## Install the package
9
+
10
+ Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
11
+
12
+ The package requires Node.js 26.4.0 or later and ships as both ESM and
13
+ CommonJS with TypeScript declarations. It has exactly two runtime dependencies,
14
+ [`effect`](https://effect.website) and [`@smthrs/crypto`](/api/crypto), and both
15
+ install with it.
16
+
17
+ ## What each tier needs in scope
18
+
19
+ Nothing in this package opens a file or a socket by itself. Every host effect
20
+ goes through an Effect service, which is what lets the same code run in Node,
21
+ in Bun, in a browser tab, and inside a sandbox:
22
+
23
+ | Service | Needed by | Node layer |
24
+ | -------------------------------------------------------------- | ------------------------------------------------- | ----------------------- |
25
+ | `Crypto.Crypto` from `effect/Crypto` | `put` and `get` on every store | `NodeCrypto.layer` |
26
+ | `FileSystem.FileSystem` from `effect/FileSystem` | the filesystem store, the sweep, the backup lease | `NodeFileSystem.layer` |
27
+ | `Path.Path` from `effect/Path` | the filesystem store | `NodePath.layer` |
28
+ | `HttpClient.HttpClient` from `effect/unstable/http/HttpClient` | `RemoteArtifacts` | `FetchHttpClient.layer` |
29
+
30
+ `has` and `findMissing` never hash anything, so they do not require `Crypto`.
31
+ The memory and no-op stores require no `FileSystem` or `Path` at all, which is what makes
32
+ them usable in a browser and in a test with no temporary directory.
33
+
34
+ The Node implementations live in `@effect/platform-node`:
35
+
36
+ ```bash
37
+ pnpm add @effect/platform-node@4.0.0-rc.115 @effect/platform-node-shared@4.0.0-rc.115
38
+ ```
39
+
40
+ ```ts
41
+ import * as NodeCrypto from "@effect/platform-node/NodeCrypto"
42
+ import * as NodeFileSystem from "@effect/platform-node/NodeFileSystem"
43
+ import * as NodePath from "@effect/platform-node/NodePath"
44
+ import * as Layer from "effect/Layer"
45
+ import * as FetchHttpClient from "effect/unstable/http/FetchHttpClient"
46
+
47
+ /** Everything every tier in this package can ask for. */
48
+ export const host = Layer.mergeAll(NodeCrypto.layer, NodeFileSystem.layer, NodePath.layer, FetchHttpClient.layer)
49
+ ```
50
+
51
+ `FetchHttpClient` ships inside `effect`, so the shared tier adds no dependency
52
+ of its own.
53
+
54
+ ## Import forms
55
+
56
+ The root entry point re-exports every module as a namespace:
57
+
58
+ ```ts
59
+ import {
60
+ ArtifactBackupLease,
61
+ ArtifactStore,
62
+ ArtifactStoreMetrics,
63
+ ArtifactSweep,
64
+ CombinedArtifacts,
65
+ RemoteArtifacts
66
+ } from "@smthrs/artifacts"
67
+ ```
68
+
69
+ Each module is also importable from its own subpath, which is the form the
70
+ guides and the [API reference](./api.md) use:
71
+
72
+ ```ts
73
+ import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
74
+ import * as ArtifactSweep from "@smthrs/artifacts/ArtifactSweep"
75
+ ```
76
+
77
+ Two subpath forms are blocked in the export map and are not public API:
78
+ `@smthrs/artifacts/internal/*` and `@smthrs/artifacts/*/index`.
79
+ `@smthrs/artifacts/package.json` is exported.
80
+
81
+ ## What a full composition adds
82
+
83
+ A host that spills step outputs into this store composes it with the engine
84
+ packages.
85
+
86
+ - [`@smthrs/engine-store`](/api/engine-store) owns `StepBoundary`, which decides
87
+ when a step output is inline and when it is spilled here by digest. It also
88
+ owns `ArtifactGc`, the mark half of collection that drives `ArtifactSweep`,
89
+ and `ArtifactSync`, which honors the download policy on replay.
90
+ - [`@smthrs/step-cache`](/api/step-cache) maps a step key to the recorded result
91
+ that references those digests.
92
+
93
+ The end-to-end composition, from a flow to a shared cache, is on smithers.sh in
94
+ [Share results with artifacts and the step cache](/docs/guides/artifacts-cache/).
95
+
96
+ ## Next step
97
+
98
+ Publish and read your first artifact in the [Quickstart](./quickstart.md).