@smthrs/artifacts 0.0.0-stage → 1.0.0-rc.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +77 -0
- package/LICENSE +21 -0
- package/README.md +175 -2
- package/dist/cjs/ArtifactBackupLease.d.ts +34 -0
- package/dist/cjs/ArtifactBackupLease.d.ts.map +1 -0
- package/dist/cjs/ArtifactBackupLease.js +183 -0
- package/dist/cjs/ArtifactBackupLease.js.map +7 -0
- package/dist/cjs/ArtifactStore.d.ts +129 -0
- package/dist/cjs/ArtifactStore.d.ts.map +1 -0
- package/dist/cjs/ArtifactStore.js +73 -0
- package/dist/cjs/ArtifactStore.js.map +7 -0
- package/dist/cjs/ArtifactStoreError.d.ts +82 -0
- package/dist/cjs/ArtifactStoreError.d.ts.map +1 -0
- package/dist/cjs/ArtifactStoreError.js +72 -0
- package/dist/cjs/ArtifactStoreError.js.map +7 -0
- package/dist/cjs/ArtifactStoreMetrics.d.ts +70 -0
- package/dist/cjs/ArtifactStoreMetrics.d.ts.map +1 -0
- package/dist/cjs/ArtifactStoreMetrics.js +51 -0
- package/dist/cjs/ArtifactStoreMetrics.js.map +7 -0
- package/dist/cjs/ArtifactSweep.d.ts +185 -0
- package/dist/cjs/ArtifactSweep.d.ts.map +1 -0
- package/dist/cjs/ArtifactSweep.js +171 -0
- package/dist/cjs/ArtifactSweep.js.map +7 -0
- package/dist/cjs/CombinedArtifacts.d.ts +103 -0
- package/dist/cjs/CombinedArtifacts.d.ts.map +1 -0
- package/dist/cjs/CombinedArtifacts.js +189 -0
- package/dist/cjs/CombinedArtifacts.js.map +7 -0
- package/dist/cjs/FileLease.d.ts +117 -0
- package/dist/cjs/FileLease.d.ts.map +1 -0
- package/dist/cjs/FileLease.js +208 -0
- package/dist/cjs/FileLease.js.map +7 -0
- package/dist/cjs/FileSystemArtifactStore.d.ts +93 -0
- package/dist/cjs/FileSystemArtifactStore.d.ts.map +1 -0
- package/dist/cjs/FileSystemArtifactStore.js +276 -0
- package/dist/cjs/FileSystemArtifactStore.js.map +7 -0
- package/dist/cjs/MemoryArtifactStore.d.ts +25 -0
- package/dist/cjs/MemoryArtifactStore.d.ts.map +1 -0
- package/dist/cjs/MemoryArtifactStore.js +84 -0
- package/dist/cjs/MemoryArtifactStore.js.map +7 -0
- package/dist/cjs/NoopArtifactStore.d.ts +17 -0
- package/dist/cjs/NoopArtifactStore.d.ts.map +1 -0
- package/dist/cjs/NoopArtifactStore.js +46 -0
- package/dist/cjs/NoopArtifactStore.js.map +7 -0
- package/dist/cjs/RemoteArtifacts.d.ts +237 -0
- package/dist/cjs/RemoteArtifacts.d.ts.map +1 -0
- package/dist/cjs/RemoteArtifacts.js +405 -0
- package/dist/cjs/RemoteArtifacts.js.map +7 -0
- package/dist/cjs/index.d.ts +65 -0
- package/dist/cjs/index.d.ts.map +1 -0
- package/dist/cjs/index.js +47 -0
- package/dist/cjs/index.js.map +7 -0
- package/dist/cjs/internal/ArtifactLocks.d.ts +30 -0
- package/dist/cjs/internal/ArtifactLocks.d.ts.map +1 -0
- package/dist/cjs/internal/ArtifactLocks.js +93 -0
- package/dist/cjs/internal/ArtifactLocks.js.map +7 -0
- package/dist/cjs/internal/ArtifactPath.d.ts +19 -0
- package/dist/cjs/internal/ArtifactPath.d.ts.map +1 -0
- package/dist/cjs/internal/ArtifactPath.js +74 -0
- package/dist/cjs/internal/ArtifactPath.js.map +7 -0
- package/dist/cjs/measureBytes.d.ts +18 -0
- package/dist/cjs/measureBytes.d.ts.map +1 -0
- package/dist/cjs/measureBytes.js +43 -0
- package/dist/cjs/measureBytes.js.map +7 -0
- package/dist/cjs/package.json +1 -0
- package/dist/cjs/snapshotBytes.d.ts +19 -0
- package/dist/cjs/snapshotBytes.d.ts.map +1 -0
- package/dist/cjs/snapshotBytes.js +41 -0
- package/dist/cjs/snapshotBytes.js.map +7 -0
- package/dist/cjs/validateDigest.d.ts +22 -0
- package/dist/cjs/validateDigest.d.ts.map +1 -0
- package/dist/cjs/validateDigest.js +42 -0
- package/dist/cjs/validateDigest.js.map +7 -0
- package/dist/esm/ArtifactBackupLease.d.ts +34 -0
- package/dist/esm/ArtifactBackupLease.d.ts.map +1 -0
- package/dist/esm/ArtifactBackupLease.js +126 -0
- package/dist/esm/ArtifactBackupLease.js.map +1 -0
- package/dist/esm/ArtifactStore.d.ts +129 -0
- package/dist/esm/ArtifactStore.d.ts.map +1 -0
- package/dist/esm/ArtifactStore.js +83 -0
- package/dist/esm/ArtifactStore.js.map +1 -0
- package/dist/esm/ArtifactStoreError.d.ts +82 -0
- package/dist/esm/ArtifactStoreError.d.ts.map +1 -0
- package/dist/esm/ArtifactStoreError.js +77 -0
- package/dist/esm/ArtifactStoreError.js.map +1 -0
- package/dist/esm/ArtifactStoreMetrics.d.ts +70 -0
- package/dist/esm/ArtifactStoreMetrics.d.ts.map +1 -0
- package/dist/esm/ArtifactStoreMetrics.js +77 -0
- package/dist/esm/ArtifactStoreMetrics.js.map +1 -0
- package/dist/esm/ArtifactSweep.d.ts +185 -0
- package/dist/esm/ArtifactSweep.d.ts.map +1 -0
- package/dist/esm/ArtifactSweep.js +210 -0
- package/dist/esm/ArtifactSweep.js.map +1 -0
- package/dist/esm/CombinedArtifacts.d.ts +103 -0
- package/dist/esm/CombinedArtifacts.d.ts.map +1 -0
- package/dist/esm/CombinedArtifacts.js +214 -0
- package/dist/esm/CombinedArtifacts.js.map +1 -0
- package/dist/esm/FileLease.d.ts +117 -0
- package/dist/esm/FileLease.d.ts.map +1 -0
- package/dist/esm/FileLease.js +249 -0
- package/dist/esm/FileLease.js.map +1 -0
- package/dist/esm/FileSystemArtifactStore.d.ts +93 -0
- package/dist/esm/FileSystemArtifactStore.d.ts.map +1 -0
- package/dist/esm/FileSystemArtifactStore.js +336 -0
- package/dist/esm/FileSystemArtifactStore.js.map +1 -0
- package/dist/esm/MemoryArtifactStore.d.ts +25 -0
- package/dist/esm/MemoryArtifactStore.d.ts.map +1 -0
- package/dist/esm/MemoryArtifactStore.js +67 -0
- package/dist/esm/MemoryArtifactStore.js.map +1 -0
- package/dist/esm/NoopArtifactStore.d.ts +17 -0
- package/dist/esm/NoopArtifactStore.d.ts.map +1 -0
- package/dist/esm/NoopArtifactStore.js +27 -0
- package/dist/esm/NoopArtifactStore.js.map +1 -0
- package/dist/esm/RemoteArtifacts.d.ts +237 -0
- package/dist/esm/RemoteArtifacts.d.ts.map +1 -0
- package/dist/esm/RemoteArtifacts.js +557 -0
- package/dist/esm/RemoteArtifacts.js.map +1 -0
- package/dist/esm/index.d.ts +65 -0
- package/dist/esm/index.d.ts.map +1 -0
- package/dist/esm/index.js +65 -0
- package/dist/esm/index.js.map +1 -0
- package/dist/esm/internal/ArtifactLocks.d.ts +30 -0
- package/dist/esm/internal/ArtifactLocks.d.ts.map +1 -0
- package/dist/esm/internal/ArtifactLocks.js +86 -0
- package/dist/esm/internal/ArtifactLocks.js.map +1 -0
- package/dist/esm/internal/ArtifactPath.d.ts +19 -0
- package/dist/esm/internal/ArtifactPath.d.ts.map +1 -0
- package/dist/esm/internal/ArtifactPath.js +59 -0
- package/dist/esm/internal/ArtifactPath.js.map +1 -0
- package/dist/esm/measureBytes.d.ts +18 -0
- package/dist/esm/measureBytes.d.ts.map +1 -0
- package/dist/esm/measureBytes.js +22 -0
- package/dist/esm/measureBytes.js.map +1 -0
- package/dist/esm/snapshotBytes.d.ts +19 -0
- package/dist/esm/snapshotBytes.d.ts.map +1 -0
- package/dist/esm/snapshotBytes.js +30 -0
- package/dist/esm/snapshotBytes.js.map +1 -0
- package/dist/esm/validateDigest.d.ts +22 -0
- package/dist/esm/validateDigest.d.ts.map +1 -0
- package/dist/esm/validateDigest.js +26 -0
- package/dist/esm/validateDigest.js.map +1 -0
- package/docs/README.md +202 -0
- package/docs/api.md +484 -0
- package/docs/concepts/content-addressing.md +174 -0
- package/docs/concepts/coordination.md +141 -0
- package/docs/concepts/tiers.md +147 -0
- package/docs/guides/fence-a-backup.md +123 -0
- package/docs/guides/reclaim-disk-space.md +143 -0
- package/docs/guides/serve-the-artifact-protocol.md +140 -0
- package/docs/guides/share-artifacts-across-machines.md +180 -0
- package/docs/guides/test-against-an-artifact-store.md +158 -0
- package/docs/installation.md +102 -0
- package/docs/quickstart.md +183 -0
- package/docs/troubleshooting.md +232 -0
- package/package.json +179 -3
- package/src/ArtifactBackupLease.ts +223 -0
- package/src/ArtifactStore.ts +152 -0
- package/src/ArtifactStoreError.ts +96 -0
- package/src/ArtifactStoreMetrics.ts +83 -0
- package/src/ArtifactSweep.ts +346 -0
- package/src/CombinedArtifacts.ts +369 -0
- package/src/FileLease.ts +397 -0
- package/src/FileSystemArtifactStore.ts +459 -0
- package/src/MemoryArtifactStore.ts +83 -0
- package/src/NoopArtifactStore.ts +30 -0
- package/src/RemoteArtifacts.ts +814 -0
- package/src/index.ts +71 -0
- package/src/internal/ArtifactLocks.ts +110 -0
- package/src/internal/ArtifactPath.ts +72 -0
- package/src/measureBytes.ts +28 -0
- package/src/snapshotBytes.ts +34 -0
- package/src/validateDigest.ts +31 -0
package/docs/api.md
ADDED
|
@@ -0,0 +1,484 @@
|
|
|
1
|
+
---
|
|
2
|
+
title: "API reference"
|
|
3
|
+
description: "Every export of @smthrs/artifacts: the store contract and its errors, the filesystem, memory, no-op, HTTP, and combined implementations, the sweep, backup lease, and file lease, and the two metrics."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
`@smthrs/artifacts` exports seven modules. Each is available from the root as a
|
|
7
|
+
namespace and from its own subpath:
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
import { ArtifactStore } from "@smthrs/artifacts"
|
|
11
|
+
import * as ArtifactStoreSubpath from "@smthrs/artifacts/ArtifactStore"
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Addresses are the SHA-256 digests [`@smthrs/crypto`](/api/crypto) computes
|
|
15
|
+
through the injected `Crypto` service, which with `effect` is the whole
|
|
16
|
+
dependency list.
|
|
17
|
+
|
|
18
|
+
## Example
|
|
19
|
+
|
|
20
|
+
```ts
|
|
21
|
+
import { ArtifactStore } from "@smthrs/artifacts"
|
|
22
|
+
|
|
23
|
+
const program = Effect.gen(function*() {
|
|
24
|
+
const store = yield* ArtifactStore.ArtifactStore
|
|
25
|
+
const address = yield* store.put(new TextEncoder().encode("build output"))
|
|
26
|
+
return yield* store.get(address)
|
|
27
|
+
})
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
## ArtifactStore
|
|
31
|
+
|
|
32
|
+
`@smthrs/artifacts/ArtifactStore`. The contract, its errors, and the three
|
|
33
|
+
local implementations.
|
|
34
|
+
|
|
35
|
+
### The contract
|
|
36
|
+
|
|
37
|
+
```ts
|
|
38
|
+
interface Service {
|
|
39
|
+
readonly put: (bytes: Uint8Array) => Effect<Digest, ArtifactStoreError, Crypto>
|
|
40
|
+
readonly get: (
|
|
41
|
+
digest: string
|
|
42
|
+
) => Effect<Uint8Array, ArtifactMissing | ArtifactCorruption | ArtifactStoreError, Crypto>
|
|
43
|
+
readonly has: (digest: string) => Effect<boolean, ArtifactStoreError>
|
|
44
|
+
readonly findMissing: (digests: Iterable<string>) => Effect<Array<string>, ArtifactStoreError>
|
|
45
|
+
}
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
| Member | Behavior |
|
|
49
|
+
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
50
|
+
| `put` | Measures `bytes`, publishes them under that address, and returns it. Idempotent: storing the same bytes twice returns the same address and stores one copy. Every implementation snapshots the caller's buffer when the Effect begins and never retains the caller's reference. |
|
|
51
|
+
| `get` | Returns the bytes at `digest`, having verified that they still hash to it. Every successful read returns a new array. Fails with `ArtifactMissing` for a valid address with no bytes, `ArtifactCorruption` for a mismatch, and `ArtifactStoreError` for a host or transport refusal. |
|
|
52
|
+
| `has` | Whether this tier holds an artifact at `digest`. Does not read or hash the bytes, so it needs no `Crypto`. |
|
|
53
|
+
| `findMissing` | Which of `digests` this tier does not hold. The result is guaranteed to be a subset of the input, deduplicated in first-seen order. |
|
|
54
|
+
|
|
55
|
+
`ArtifactStore` is the service tag, with identity
|
|
56
|
+
`@smthrs/artifacts/ArtifactStore`.
|
|
57
|
+
|
|
58
|
+
Read operations accept a plain `string` rather than the `Digest` brand on
|
|
59
|
+
purpose: an address read back out of a durable row is untrusted input, so the
|
|
60
|
+
store validates it instead of asking every caller to re-brand a persisted
|
|
61
|
+
column. `put` returns the brand, because it measured the bytes itself.
|
|
62
|
+
|
|
63
|
+
### Digests
|
|
64
|
+
|
|
65
|
+
| Export | Meaning |
|
|
66
|
+
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
67
|
+
| `Digest` | Schema and branded type for exactly 64 lowercase hexadecimal SHA-256 characters, re-exported from `@smthrs/crypto`. |
|
|
68
|
+
| `validateDigest(digest: string)` | `Effect<Digest, ArtifactStoreError>`. Refuses anything other than the canonical representation with code `invalid_digest`. The failure text is a bounded constant, so a hostile value cannot copy itself into a log or a durable error. |
|
|
69
|
+
| `measureBytes(bytes: Uint8Array)` | `Effect<Digest, ArtifactStoreError, Crypto>`. Computes an address with the injected `Crypto` service, never retaining the bytes in an error. |
|
|
70
|
+
| `snapshotBytes(bytes: Uint8Array)` | `Effect<Uint8Array, ArtifactStoreError>`. Copies a caller-owned buffer when the Effect begins. A refused copy fails as `unavailable`, the code for a host refusal, because `Crypto` has not been consulted yet. |
|
|
71
|
+
|
|
72
|
+
Every implementation validates an address before logging it or interpolating it
|
|
73
|
+
into a path or a URL.
|
|
74
|
+
|
|
75
|
+
### Errors
|
|
76
|
+
|
|
77
|
+
| Export | Shape |
|
|
78
|
+
| ------------------------ | --------------------------------------------------------------------------------------------------------------------- |
|
|
79
|
+
| `ArtifactStoreError` | Tag `@smthrs/artifacts/ArtifactStoreError`, fields `code`, `message`, optional `cause`. |
|
|
80
|
+
| `ArtifactMissing` | Tag `@smthrs/artifacts/ArtifactMissing`, fields `code: "artifact_missing"` and `digest`. |
|
|
81
|
+
| `ArtifactCorruption` | Tag `@smthrs/artifacts/ArtifactCorruption`, fields `code: "artifact_corruption"`, `recordedDigest`, `measuredDigest`. |
|
|
82
|
+
| `ArtifactStoreErrorCode` | Schema and type for the five stable codes. |
|
|
83
|
+
|
|
84
|
+
A miss and a corruption are separate tags rather than codes because a
|
|
85
|
+
read-through composition acts on them differently: a miss is an ordinary
|
|
86
|
+
outcome a second tier may satisfy, and a corruption is a violation of the
|
|
87
|
+
store's strongest invariant.
|
|
88
|
+
|
|
89
|
+
| Code | Retryable | Meaning |
|
|
90
|
+
| ----------------------- | --------- | ----------------------------------------------------------- |
|
|
91
|
+
| `invalid_configuration` | No | The caller supplied an option the store refuses. Permanent. |
|
|
92
|
+
| `invalid_digest` | No | The value is not a canonical SHA-256 address. Permanent. |
|
|
93
|
+
| `digest_failed` | Depends | The `Crypto` service failed to compute a digest. |
|
|
94
|
+
| `unavailable` | Depends | A host refused, or a no-op store has no implementation. |
|
|
95
|
+
| `transport_failed` | Depends | The shared tier refused, answered unusably, or timed out. |
|
|
96
|
+
|
|
97
|
+
### The filesystem store
|
|
98
|
+
|
|
99
|
+
```ts
|
|
100
|
+
declare const makeFileSystem: (fs: FileSystem, path: Path, options?: FileSystemOptions) => Service
|
|
101
|
+
declare const layerFileSystem: (
|
|
102
|
+
options?: FileSystemOptions
|
|
103
|
+
) => Layer<ArtifactStore, never, FileSystem | Path>
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
`path` is the host's `Path` service (`NodePath.layer` on Node). The directory
|
|
107
|
+
sync walks the objects directory's ancestors with it, so a Windows drive path
|
|
108
|
+
syncs its real parents.
|
|
109
|
+
|
|
110
|
+
Publishes at `${directory}/${digest.slice(0, 2)}/${digest}`. Bytes land at a
|
|
111
|
+
unique temp path in the destination fanout directory, are synced when required,
|
|
112
|
+
and are renamed into place. Every deduplicated `put` verifies the existing blob
|
|
113
|
+
and rewrites an unreadable or mismatched address atomically, then freshens the
|
|
114
|
+
blob's modification time so a retention sweep reads it as recently referenced.
|
|
115
|
+
|
|
116
|
+
`findMissing` deduplicates and validates the full input before filesystem
|
|
117
|
+
probes, then checks at most 16 addresses concurrently. Missing digests retain
|
|
118
|
+
first-request order; a failed probe fails the batch as `unavailable`.
|
|
119
|
+
|
|
120
|
+
| `FileSystemOptions` field | Default | Meaning |
|
|
121
|
+
| ------------------------- | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
122
|
+
| `directory` | `defaultDirectory` | Where blobs are stored. Workspace-relative, so a workspace can be moved or copied whole. An `ArtifactSweep` over the same store must name the same directory. |
|
|
123
|
+
| `fileMode` | `0o600` | Creation mode for new payload files, restricted by the host umask. Existing blobs are unchanged. |
|
|
124
|
+
| `directoryMode` | `0o700` | Creation mode for objects and fanout directories, restricted by the host umask. Lock directories remain `0700`. |
|
|
125
|
+
| `durability` | `"required"` | `required` syncs the blob, fanout directory, objects directory, and its ancestors and propagates any refusal. `best-effort` tolerates sync refusals. Both modes require exclusive writable handles and symlink inspection. |
|
|
126
|
+
| `coordination` | `"required"` | `required` combines an in-process semaphore with heartbeat-backed lock files shared by every cooperating process. `process` keeps only the semaphore, giving up cross-process exclusion between writers and sweep deletion. |
|
|
127
|
+
|
|
128
|
+
`defaultDirectory` is `.flows/objects`. It is exported because the store and
|
|
129
|
+
its sweep must name the same directory, and a second private copy of the
|
|
130
|
+
literal is exactly how they would drift apart.
|
|
131
|
+
|
|
132
|
+
`required` bounds the writer-versus-sweeper race rather than eliminating it. A
|
|
133
|
+
lock is heartbeated every 10 seconds, reclaimed once it is 60 seconds stale, and
|
|
134
|
+
acquisition gives up after 2 minutes. Contenders that find the same stale lock
|
|
135
|
+
race for a claim file named after its owner, so the lock is moved at most once
|
|
136
|
+
and a fresh replacement is never displaced. A holder whose host stalls past 60
|
|
137
|
+
seconds is still reaped while running. Neither
|
|
138
|
+
`ArtifactSweep.RemoveOptions.ifUnmodifiedSinceMs` nor the backup lease protects
|
|
139
|
+
against that, because both depend on the same lock. Build a
|
|
140
|
+
store and its sweep with the same directory and coordination mode; no runtime
|
|
141
|
+
check can detect a mismatch. See
|
|
142
|
+
[Coordination between processes](./concepts/coordination.md).
|
|
143
|
+
|
|
144
|
+
The store also sweeps its own crash orphans, `.tmp-*` payloads and `.locks/`
|
|
145
|
+
entries older than one hour, once per store instance on its first publication.
|
|
146
|
+
That is scratch reclamation, never garbage collection of published blobs.
|
|
147
|
+
|
|
148
|
+
### The memory store
|
|
149
|
+
|
|
150
|
+
```ts
|
|
151
|
+
declare const makeMemory: () => Service
|
|
152
|
+
declare const layerMemory: Layer<ArtifactStore>
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
For tests and for a browser host with no durable filesystem. It copies at both
|
|
156
|
+
boundaries around a private map, so no reference a caller can still mutate
|
|
157
|
+
aliases the stored content. It does not rehash a read, because its address
|
|
158
|
+
space is private and cannot disagree with itself.
|
|
159
|
+
|
|
160
|
+
`layerMemory` is a value, not a function.
|
|
161
|
+
|
|
162
|
+
### The no-op store
|
|
163
|
+
|
|
164
|
+
```ts
|
|
165
|
+
declare const makeNoop: (overrides?: Partial<Service>) => Service
|
|
166
|
+
declare const layerNoop: (overrides?: Partial<Service>) => Layer<ArtifactStore>
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Every operation fails with an `unavailable` `ArtifactStoreError` whose message
|
|
170
|
+
is `<method> is unavailable`. Overrides replace individual methods, which is
|
|
171
|
+
how a test states the one behavior it cares about.
|
|
172
|
+
|
|
173
|
+
## RemoteArtifacts
|
|
174
|
+
|
|
175
|
+
`@smthrs/artifacts/RemoteArtifacts`. The shared tier, spoken over Effect's
|
|
176
|
+
`HttpClient`.
|
|
177
|
+
|
|
178
|
+
```ts
|
|
179
|
+
declare const make: (options: Options) => Effect<Service, ArtifactStoreError, HttpClient>
|
|
180
|
+
declare const layer: (
|
|
181
|
+
options: Options
|
|
182
|
+
) => Layer<ArtifactStore.ArtifactStore, ArtifactStoreError, HttpClient>
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
`Service` extends `ArtifactStore.Service` with one field, `downloadPolicy`.
|
|
186
|
+
Composing `layer` alone makes every read and write a network round trip; the
|
|
187
|
+
intended production shape is `CombinedArtifacts.layer` with this as its remote
|
|
188
|
+
tier.
|
|
189
|
+
|
|
190
|
+
The wire protocol is `GET`, `PUT`, and `HEAD` at `/cas/{digest}` and `POST` at
|
|
191
|
+
`/cas/findMissing`, resolved beneath the endpoint. Every download is bounded and
|
|
192
|
+
digest-verified before it is returned. Redirects are never followed: a fetch
|
|
193
|
+
transport runs with `redirect: "manual"`, and any `3xx` other than the
|
|
194
|
+
resumable-upload `308` fails as an unexpected status, so credential headers
|
|
195
|
+
never reach another origin. The complete server-side contract is in
|
|
196
|
+
[Serve the artifact protocol](./guides/serve-the-artifact-protocol.md).
|
|
197
|
+
|
|
198
|
+
Effect's HTTP client opens one span per request, named for the request method.
|
|
199
|
+
It records no request or response header, so a trace carries no credential. A
|
|
200
|
+
host that sets `HttpClient.TracerHeaderFilter` to record headers gets every
|
|
201
|
+
name configured in `headers` as `<redacted>`. A `4xx` or `5xx` answer ends the
|
|
202
|
+
span as failed with the status in `error.type`, so a `HEAD` for an absent
|
|
203
|
+
artifact shows a failed span although `has` returned `false`.
|
|
204
|
+
|
|
205
|
+
### Options
|
|
206
|
+
|
|
207
|
+
| Field | Default | Meaning |
|
|
208
|
+
| ----------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
209
|
+
| `endpoint` | required | The cache root, for example `https://cache.example.com`. A trailing slash is ignored. Refused at construction as `invalid_configuration` when it is not a string, when no `URL` parser accepts it, when the scheme is not `https:` (plain `http:` is allowed only for the literal loopback hosts `localhost`, `127.0.0.1`, and `[::1]`), and when it carries userinfo, a query, or a fragment. The refusal names only the violated rule and never echoes the endpoint. |
|
|
210
|
+
| `headers` | none | Headers sent with every request: this is the credential seam. Construction-time on purpose, because a credential arriving as a step input would be hashed into a step key and persisted everywhere the journal goes. Applied before the protocol's own headers, so no configuration can strip a `content-type`, `content-length`, or `content-range`. |
|
|
211
|
+
| `downloadTimeout` | 60 seconds | One download, including the complete response body. |
|
|
212
|
+
| `uploadTimeout` | 60 seconds | One upload, including every resume probe and chunk. |
|
|
213
|
+
| `requestTimeout` | 60 seconds | One `HEAD` probe, or one `findMissing` batch and its response. |
|
|
214
|
+
| `maxDownloadBytes` | 256 MiB | The largest download buffered. A `Content-Length` past the bound is refused before a body byte is read; an incremental read stops at most one chunk past it. |
|
|
215
|
+
| `maxFindMissingResponseBytes` | 256 KiB | The largest `findMissing` response accepted. May only lower the protocol bound. |
|
|
216
|
+
| `chunkBytes` | absent | Above this many bytes an upload travels as `Content-Range` requests, resuming from the prefix the tier kept. |
|
|
217
|
+
| `downloadPolicy` | `"all"` | How eagerly a composition reading through this tier materializes blobs locally. |
|
|
218
|
+
|
|
219
|
+
`findMissing` validates its input before sending anything, sends at most 1,000
|
|
220
|
+
digests per batch, and filters the response back to requested digests, so a
|
|
221
|
+
tier answering with an unrequested digest cannot make a caller upload bytes it
|
|
222
|
+
never asked about.
|
|
223
|
+
|
|
224
|
+
### Download policy
|
|
225
|
+
|
|
226
|
+
| Export | Meaning |
|
|
227
|
+
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
|
|
228
|
+
| `DownloadPolicy` | Schema and type for `"all"`, `"toplevel"`, `"minimal"`. |
|
|
229
|
+
| `downloadPolicies` | A frozen array of all three, in materialization order. |
|
|
230
|
+
| `downloadPolicyOf(store)` | Reads the policy a store declares, or `undefined` for one that declares none, which is every local store and any foreign implementation. |
|
|
231
|
+
|
|
232
|
+
`all` prefetches every referenced blob when a replay is admitted. `toplevel`
|
|
233
|
+
prefetches nothing and materializes a blob on the first read that needs it.
|
|
234
|
+
`minimal` prefetches nothing and materializes nothing.
|
|
235
|
+
`CombinedArtifacts.get` honors the last two;
|
|
236
|
+
[`@smthrs/engine-store`](/api/engine-store)'s `ArtifactSync.hydrate` honors the
|
|
237
|
+
first.
|
|
238
|
+
|
|
239
|
+
### Chunked uploads
|
|
240
|
+
|
|
241
|
+
With `chunkBytes` set, the client sends a `HEAD` existence probe, then an empty
|
|
242
|
+
`Content-Range: bytes */{total}` probe to discover a retained prefix, then
|
|
243
|
+
ranged chunks:
|
|
244
|
+
|
|
245
|
+
| Response | Client action |
|
|
246
|
+
| ----------------------------- | ----------------------------------------------------------- |
|
|
247
|
+
| `308` | Continue after the reported `Range: bytes=0-{last}` prefix. |
|
|
248
|
+
| `2xx` on the completing chunk | Confirm the complete length with `HEAD`. |
|
|
249
|
+
| `2xx` before completion | Treat the tier as range-unaware and send the whole blob. |
|
|
250
|
+
| `400`, `411`, or `416` | Send the whole blob. |
|
|
251
|
+
| Any other status | Fail the upload as a transport error. |
|
|
252
|
+
|
|
253
|
+
The offset never moves backward. If a server ignores ranges, omits a confirming
|
|
254
|
+
length, or stores a partial body, the whole-blob `PUT` overwrites the partial
|
|
255
|
+
result. Both Smithers cache services answer a ranged `PUT` with `400` and cap
|
|
256
|
+
one request body at 16 MiB, so `chunkBytes` against them degrades
|
|
257
|
+
to one whole-blob `PUT` and a larger artifact is refused with `413`.
|
|
258
|
+
|
|
259
|
+
## CombinedArtifacts
|
|
260
|
+
|
|
261
|
+
`@smthrs/artifacts/CombinedArtifacts`. Local first, shared second, with
|
|
262
|
+
write-back.
|
|
263
|
+
|
|
264
|
+
```ts
|
|
265
|
+
declare const make: (options: Options) => Effect<RemoteArtifacts.Service, ArtifactStoreError>
|
|
266
|
+
declare const layer: <EL, RL, ER, RR>(
|
|
267
|
+
options: LayerOptions<EL, RL, ER, RR>
|
|
268
|
+
) => Layer<ArtifactStore.ArtifactStore, EL | ER | ArtifactStoreError, RL | RR>
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
| `Options` field | Default | Meaning |
|
|
272
|
+
| ------------------ | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
|
|
273
|
+
| `local` | required | The fast, machine-local tier, as a `Service`. Every read tries this one first. |
|
|
274
|
+
| `remote` | required | The shared tier, as a `Service`. Consulted only on a local miss or corruption. |
|
|
275
|
+
| `uploadTimeout` | 60 seconds | How long a `put` waits for its opportunistic upload before abandoning it. An abandoned upload is dropped exactly like a refused one. |
|
|
276
|
+
| `writeBackTimeout` | 60 seconds | How long a `get` waits for opportunistic local write-back before interrupting it and returning the verified remote bytes. |
|
|
277
|
+
| `downloadPolicy` | the remote tier's, else `"all"` | Overrides the shared tier's declared policy for this composition. |
|
|
278
|
+
|
|
279
|
+
`LayerOptions` is the same shape with `local` and `remote` as `Effect`s. Both
|
|
280
|
+
tiers are supplied as effects rather than layers because they inhabit the same
|
|
281
|
+
tag: composing two `Layer<ArtifactStore>` would shadow one with the other. Pair
|
|
282
|
+
`Effect.map(Effect.all([FileSystem.FileSystem, Path.Path]), ([fs, path]) => ArtifactStore.makeFileSystem(fs, path))`
|
|
283
|
+
with `RemoteArtifacts.make`.
|
|
284
|
+
|
|
285
|
+
Both timeout options must be finite, positive durations. Invalid values fail
|
|
286
|
+
construction with `ArtifactStoreError` code `invalid_configuration`.
|
|
287
|
+
|
|
288
|
+
**Reads.** The local tier answers first. A hit returns. `ArtifactMissing` and
|
|
289
|
+
`ArtifactCorruption` both fall through to the shared tier, and they are kept
|
|
290
|
+
apart because only corruption earns a write-back that `minimal` would otherwise
|
|
291
|
+
skip. A local `ArtifactStoreError` fails the read rather than falling through:
|
|
292
|
+
a host that refused has not answered the question, and paying for the network
|
|
293
|
+
instead would hide a broken local tier behind a working shared one.
|
|
294
|
+
Local write-back after a remote read is opportunistic. A typed failure or
|
|
295
|
+
`writeBackTimeout` expiry returns the verified remote bytes. Timeout interrupts
|
|
296
|
+
the write-back and waits for its interruption cleanup before returning.
|
|
297
|
+
|
|
298
|
+
**Writes.** `put` publishes locally first and returns the local digest. The
|
|
299
|
+
remote upload is opportunistic, deduplicated in flight by digest, and bounded by
|
|
300
|
+
`uploadTimeout`. A remote refusal or timeout does not fail the operation that
|
|
301
|
+
produced the bytes. The shared-cache publication protocol uses `findMissing`,
|
|
302
|
+
upload, and confirmation before it publishes a cache entry, so an abandoned
|
|
303
|
+
upload costs another transfer rather than correctness.
|
|
304
|
+
|
|
305
|
+
**Probes.** `has` asks local, then remote. `findMissing` asks the remote tier
|
|
306
|
+
only about what the local tier could not answer, so the result stays a subset of
|
|
307
|
+
the input because each stage filters the previous stage's output.
|
|
308
|
+
|
|
309
|
+
`minimal` still writes back after local corruption, because replacing an
|
|
310
|
+
address the local tier already claims is repair rather than growth.
|
|
311
|
+
|
|
312
|
+
## ArtifactSweep
|
|
313
|
+
|
|
314
|
+
`@smthrs/artifacts/ArtifactSweep`. Host-local enumeration and fenced deletion.
|
|
315
|
+
This surface is deliberately not part of `ArtifactStore.Service`: a remote tier
|
|
316
|
+
can neither enumerate its address space nor accept a delete, and only the
|
|
317
|
+
filesystem store implements it.
|
|
318
|
+
|
|
319
|
+
```ts
|
|
320
|
+
interface Service {
|
|
321
|
+
readonly inventory: Effect<Array<BlobStat>, ArtifactStoreError>
|
|
322
|
+
readonly remove: (digest: string, options?: RemoveOptions) => Effect<boolean, ArtifactStoreError>
|
|
323
|
+
}
|
|
324
|
+
|
|
325
|
+
declare const makeFileSystem: (fs: FileSystem, options?: SweepOptions) => Service
|
|
326
|
+
declare const layerFileSystem: (options?: SweepOptions) => Layer<ArtifactSweep, never, FileSystem>
|
|
327
|
+
declare const makeNoop: (overrides?: Partial<Service>) => Service
|
|
328
|
+
declare const layerNoop: (overrides?: Partial<Service>) => Layer<ArtifactSweep>
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
`ArtifactSweep` is the service tag, with identity
|
|
332
|
+
`@smthrs/artifacts/ArtifactSweep`.
|
|
333
|
+
|
|
334
|
+
| Export | Fields |
|
|
335
|
+
| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
336
|
+
| `BlobStat` | `digest`, `modifiedAtMs`, `sizeBytes`. |
|
|
337
|
+
| `RemoveOptions` | `ifUnmodifiedSinceMs`: delete only while the blob's mtime is at or before this bound. |
|
|
338
|
+
| `SweepOptions` | `directory` (defaults to `ArtifactStore.defaultDirectory`) and `coordination` (defaults to `required`). No `durability`, because a sweep never writes a blob. |
|
|
339
|
+
|
|
340
|
+
`inventory` returns a `BlobStat` only for canonical fanout files whose
|
|
341
|
+
modification time is measurable. It skips temp files, lock files, foreign
|
|
342
|
+
paths, nested paths, directories sitting at blob addresses, and entries that
|
|
343
|
+
disappear during the scan. A directory that was never created is an empty
|
|
344
|
+
inventory, not a failure.
|
|
345
|
+
|
|
346
|
+
Candidate metadata reads run with concurrency 16 and retain directory-listing
|
|
347
|
+
order. Each candidate keeps its path checks; a failed metadata read skips only
|
|
348
|
+
that candidate.
|
|
349
|
+
|
|
350
|
+
`remove` returns `false` for three outcomes a caller cannot tell apart, all of
|
|
351
|
+
which mean nothing was reclaimed and retrying later is safe: the blob was
|
|
352
|
+
already gone, it failed the `ifUnmodifiedSinceMs` fence, or a live backup lease
|
|
353
|
+
fenced the deletion. The last one is not progress, so a collector counting
|
|
354
|
+
reclaimed bytes must not count it.
|
|
355
|
+
|
|
356
|
+
Under `required`, one deletion costs two lock acquisitions, the per-digest lock
|
|
357
|
+
and the workspace-global backup-lease gate, which is roughly ten filesystem
|
|
358
|
+
operations and two forked heartbeat fibers per blob. The gate is one file for
|
|
359
|
+
the whole workspace, so concurrent deletions serialize through it.
|
|
360
|
+
|
|
361
|
+
The policy this mechanism serves, which digests are live and how long a dead
|
|
362
|
+
blob is kept, belongs to [`@smthrs/engine-store`](/api/engine-store)'s
|
|
363
|
+
`ArtifactGc`. See [Reclaim disk space](./guides/reclaim-disk-space.md).
|
|
364
|
+
|
|
365
|
+
## ArtifactBackupLease
|
|
366
|
+
|
|
367
|
+
`@smthrs/artifacts/ArtifactBackupLease`. Cross-process exclusion between a
|
|
368
|
+
filesystem backup and sweep deletion.
|
|
369
|
+
|
|
370
|
+
```ts
|
|
371
|
+
declare const withLease: <A, E, R, E2>(
|
|
372
|
+
fs: FileSystem,
|
|
373
|
+
directory: string,
|
|
374
|
+
effect: Effect<A, E, R>,
|
|
375
|
+
failure: (cause: unknown) => E2
|
|
376
|
+
) => Effect<A, E | E2, R>
|
|
377
|
+
|
|
378
|
+
declare const unlessActive: <A, E, R, E2>(
|
|
379
|
+
fs: FileSystem,
|
|
380
|
+
directory: string,
|
|
381
|
+
effect: Effect<A, E, R>,
|
|
382
|
+
failure: (cause: unknown) => E2
|
|
383
|
+
) => Effect<Option<A>, E | E2, R>
|
|
384
|
+
```
|
|
385
|
+
|
|
386
|
+
`withLease` holds a heartbeat-backed marker while `effect` runs, so no sweep in
|
|
387
|
+
any process using `directory` can delete a blob. Publication stays
|
|
388
|
+
unconstrained: new blobs may appear during a backup, but a blob already
|
|
389
|
+
referenced by the frozen database cannot disappear underneath it. Acquisition
|
|
390
|
+
gives up after 2 minutes and fails through `failure`. A crashed lease becomes
|
|
391
|
+
reclaimable once its heartbeat is 60 seconds stale, and finding no marker to
|
|
392
|
+
release is the ordinary end of a slow lease rather than a fault.
|
|
393
|
+
|
|
394
|
+
`unlessActive` checks the marker and runs `effect` under the same
|
|
395
|
+
workspace-global gate, so the check and the operation cannot be separated. It
|
|
396
|
+
returns `None` when a live backup deliberately fenced the operation, which
|
|
397
|
+
`ArtifactSweep.remove` reports as `false`.
|
|
398
|
+
|
|
399
|
+
`failure` maps an unknown host cause into the caller's error type, which is how
|
|
400
|
+
these combinators stay usable from a tool with its own error channel. See
|
|
401
|
+
[Fence a backup against the sweep](./guides/fence-a-backup.md).
|
|
402
|
+
|
|
403
|
+
## FileLease
|
|
404
|
+
|
|
405
|
+
`@smthrs/artifacts/FileLease`. The cross-process lock-file lease under the
|
|
406
|
+
per-digest locks and `ArtifactBackupLease`, exported for other packages that
|
|
407
|
+
need the same fence; `@smthrs/engine-store` uses it for the workspace commit
|
|
408
|
+
lock.
|
|
409
|
+
|
|
410
|
+
```ts
|
|
411
|
+
declare const withLease: <A, E, R, E2>(
|
|
412
|
+
fs: FileSystem,
|
|
413
|
+
lockPath: string,
|
|
414
|
+
effect: Effect<A, E, R>,
|
|
415
|
+
failure: (cause: unknown) => E2,
|
|
416
|
+
options?: Options
|
|
417
|
+
) => Effect<A, E | E2, R>
|
|
418
|
+
|
|
419
|
+
declare const hold: <A, E, R, E2>(
|
|
420
|
+
fs: FileSystem,
|
|
421
|
+
lockPath: string,
|
|
422
|
+
effect: Effect<A, E, R>,
|
|
423
|
+
failure: (cause: unknown) => E2,
|
|
424
|
+
options?: HoldOptions
|
|
425
|
+
) => Effect<A, E | E2, R>
|
|
426
|
+
```
|
|
427
|
+
|
|
428
|
+
`withLease` creates `lockPath` with `wx`, writes a unique owner token, and
|
|
429
|
+
refreshes its mtime while `effect` runs. It reclaims a lock whose mtime is
|
|
430
|
+
older than the stale bound, at most once per lock generation, and fails with
|
|
431
|
+
`failure(TimeoutError)` when acquisition misses its deadline. Every other host
|
|
432
|
+
refusal reaches `failure` as the `PlatformError`. A directory at `lockPath` is
|
|
433
|
+
treated as a lock with no readable owner and reclaimed on the same bound.
|
|
434
|
+
`hold` is the same lease without a deadline, for a caller that bounds a wider
|
|
435
|
+
wait itself; `onAcquired` runs once the lease is held. `guard`, when given,
|
|
436
|
+
runs before every create, heartbeat, rename, and remove of the lock, claim, or
|
|
437
|
+
tombstone, and its failure refuses that step through `failure`. Both fail
|
|
438
|
+
through `failure` with a `RangeError`, before touching the lock, unless
|
|
439
|
+
`0 < heartbeatEvery < staleAfterMs` and `staleAfterMs` is finite: a shorter
|
|
440
|
+
stale bound would let a contender reclaim a live holder's lock.
|
|
441
|
+
|
|
442
|
+
| Option | Default | Meaning |
|
|
443
|
+
| ---------------- | -------------- | ------------------------------------------------ |
|
|
444
|
+
| `heartbeatEvery` | `"10 seconds"` | How often the holder refreshes the lock's mtime. |
|
|
445
|
+
| `staleAfterMs` | `60_000` | Age past which a lock or reclaim claim is stale. |
|
|
446
|
+
| `acquireWithin` | `"2 minutes"` | Acquisition deadline (`withLease` only). |
|
|
447
|
+
| `retryEvery` | `"25 millis"` | Sleep between attempts while contended. |
|
|
448
|
+
| `label` | `"File lease"` | Names the lock in the reclaimed-holder warning. |
|
|
449
|
+
| `annotations` | none | Extra fields logged with that warning. |
|
|
450
|
+
|
|
451
|
+
The host needs exclusive `wx` file creation, `readFileString`, `stat` with an
|
|
452
|
+
mtime, `utimes`, `rename`, and `remove`. Reclaim writes siblings named
|
|
453
|
+
`<lockPath>.reclaim-<token>` and `<lockPath>.stale-<token>`. The fence is
|
|
454
|
+
bounded: see [Coordination](./concepts/coordination.md).
|
|
455
|
+
|
|
456
|
+
## ArtifactStoreMetrics
|
|
457
|
+
|
|
458
|
+
`@smthrs/artifacts/ArtifactStoreMetrics`. Metric handles only; no exporter
|
|
459
|
+
ships in this package.
|
|
460
|
+
|
|
461
|
+
| Export | Counter | Updated by |
|
|
462
|
+
| ---------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------- |
|
|
463
|
+
| `puts` | `flows_artifact_puts` | Successful filesystem and memory puts, including deduplicated ones. |
|
|
464
|
+
| `gets` | `flows_artifact_gets` | Successful filesystem and memory gets. |
|
|
465
|
+
| `remoteFailures` | `flows_artifact_remote_failures` | Shared uploads and local write-backs `CombinedArtifacts` dropped, by `operation` (`put`, `write_back`). |
|
|
466
|
+
|
|
467
|
+
`remoteFailure.put` and `remoteFailure.write_back` are the attributed views.
|
|
468
|
+
Drops also log a warning naming the operation and error code, at most once a
|
|
469
|
+
minute per operation.
|
|
470
|
+
|
|
471
|
+
`RemoteArtifacts` is deliberately uninstrumented and the counters carry no tier
|
|
472
|
+
attribute, so read them as local artifact store traffic. A combined read the
|
|
473
|
+
shared tier serves increments no get; the write-back that materializes it
|
|
474
|
+
increments a put indistinguishable from a producer publishing new bytes.
|
|
475
|
+
Missing and corrupt reads are counted nowhere: they are error evidence, not
|
|
476
|
+
throughput.
|
|
477
|
+
|
|
478
|
+
## Reclaiming published blobs
|
|
479
|
+
|
|
480
|
+
Reclaiming published blobs never happens as a side effect of a store call. The
|
|
481
|
+
filesystem store sweeps only stale scratch and lock files. The mark policy and
|
|
482
|
+
grace period for published blobs belong to
|
|
483
|
+
[`@smthrs/engine-store`](/api/engine-store), which also owns artifact
|
|
484
|
+
publication ordering.
|