@smthrs/artifacts 0.0.0-stage → 1.0.0-rc.4

package/docs/concepts/coordination.md

@@ -0,0 +1,141 @@
+---
+title: "Coordination between processes"
+description: "How one objects directory stays safe when several processes write and sweep it: per-digest lock files, heartbeats, the stale-reclaim window, the backup lease, and their limits."
+sidebar:
+  order: 3
+---
+
+An objects directory belongs to a workspace, not to a process. A CLI run, a
+gateway, a sandboxed step, and a garbage collector can all be holding the same
+`.flows/objects` open at once. Publication is already atomic, so two writers
+racing on one digest is harmless. The dangerous overlap is a writer and a
+sweeper: one is deduplicating against a blob the other is deleting.
+
+`FileSystemOptions.coordination` and `SweepOptions.coordination` are the dial
+for that overlap.
+
+| Mode                 | What it does                                                                                                   |
+| -------------------- | -------------------------------------------------------------------------------------------------------------- |
+| `required` (default) | An in-process semaphore plus heartbeat-backed lock files under `.locks/`, shared by every cooperating process. |
+| `process`            | The in-process semaphore only. The explicit weaker browser and test mode.                                      |
+
+## The lock protocol
+
+Under `required`, a put or a sweep deletion takes the lock file for its digest
+before touching the blob:
+
+| Bound                          | Value                          |
+| ------------------------------ | ------------------------------ |
+| Heartbeat interval             | 10 seconds                     |
+| Stale after                    | 60 seconds without a heartbeat |
+| Acquisition deadline           | 2 minutes                      |
+| Retry interval while contended | 25 milliseconds                |
+
+A holder that crashes stops heartbeating, and the next contender reclaims its
+lock a minute later. A contender that cannot acquire within two minutes fails
+rather than waiting forever. This deadline includes the in-process semaphore
+wait, directory creation, and lockfile acquisition. It ends before the protected
+operation starts. Uninterruptible host completion and release finalizers can
+extend cancellation latency. Semaphores are scoped by filesystem service,
+objects directory, and digest; unrelated objects roots do not share permits.
+
+## The fence is bounded, not absolute
+
+Reclaiming a stale lock is serialized per lock generation. A contender reads
+the owner token, then measures the age. If the lock is stale, the contender
+races for a `wx` claim file named after that token. Only the claim winner
+renames the lock away, and only after re-reading that the path still holds the
+same stale token. Two contenders that measured the same lock as stale
+therefore move it at most once, and neither moves the fresh lock the other
+just took.
+
+The fence still assumes a holder that stops heartbeating is dead. A holder
+whose host stalls past 60 seconds is reaped while it is still running, and its
+heartbeat only logs the loss. Neither the mtime check nor the backup gate
+independently protects against that loss of mutual exclusion. The mtime check
+is followed by a separate delete, and the backup gate uses the same lock
+protocol.
+
+## Both sides must agree
+
+A lock fences only the parties that take it. Build a store and its sweep with
+the same `directory` and the same `coordination`:
+
+```ts
+import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
+import * as ArtifactSweep from "@smthrs/artifacts/ArtifactSweep"
+import * as Layer from "effect/Layer"
+
+const directory = ".flows/objects"
+
+export const artifacts = Layer.merge(
+  ArtifactStore.layerFileSystem({ directory, coordination: "required" }),
+  ArtifactSweep.layerFileSystem({ directory, coordination: "required" })
+)
+```
+
+Nothing checks the pairing, and a mismatch is silent. A sweep on `required`
+beside a store on `process` takes lock files no writer ever observes: the fence
+reads as armed and protects nothing.
+
+A store on `process` also gives up cross-process exclusion between writers and
+sweep deletion, and a sweep on `process` skips the backup lease entirely.
+
+## The backup lease
+
+A filesystem backup freezes a database and then copies the blobs that database
+references. Publication during that window is fine, because a new blob cannot
+already be referenced by a frozen snapshot. Deletion is not: a sweep that
+removes a blob the frozen database names leaves a backup that cannot be
+restored.
+
+`ArtifactBackupLease` is the exclusion for exactly that. The backup holds one
+heartbeat-backed marker, `.backup-lease`, for the whole copy, with the same 10
+second heartbeat and 60 second staleness bound as the digest locks. Sweep
+deletion checks that marker while holding a short-lived workspace-global gate,
+`.backup-lease-gate`, so the check and the delete cannot be separated. A
+crashed lease becomes reclaimable once its heartbeat goes stale. Marker cleanup
+is registered before acquisition; interruption during marker creation or gate
+release still removes the marker owned by that acquisition. Failed gate release
+also runs marker cleanup, which may wait for the gate to become reclaimable.
+
+`ArtifactBackupLease.unlessActive` answers `None` when a live backup fenced the
+operation, and `ArtifactSweep.remove` turns that into `false`. See
+[Fence a backup against the sweep](../guides/fence-a-backup.md).
+
+## What coordination costs
+
+Under `required`, one sweep deletion takes two locks, the per-digest lock and
+then the workspace-global backup-lease gate. That is roughly ten filesystem
+operations and two forked heartbeat fibers per blob. The gate is one file for
+the whole workspace, so concurrent deletions serialize through it.
+
+That is the price of never deleting a blob a running backup already recorded.
+Size a collection window with it in mind: a sweep of ten thousand blobs is
+tens of thousands of filesystem operations, not ten thousand unlinks.
+
+## Crash orphans
+
+Lock files and temp payloads outlive a hard-killed process. Nothing else
+observes either, because reads resolve only canonical paths and a lock is
+reclaimed on contention alone, so a digest nobody publishes again would keep
+its lock file forever.
+
+Each filesystem store therefore runs one conservative sweep of its own scratch
+on its first publication: `.tmp-*` payloads and `.locks/` entries whose
+modification time is more than an hour old. An hour is far beyond any live
+writer's publication window and sixty times the lock staleness bound, so a file
+that old belongs to no living holder. A file whose age the host cannot report
+says nothing about its owner and survives.
+
+This is a sweep of scratch, not garbage collection. Reclaiming published
+artifacts is always an explicit `ArtifactSweep` call.
+
+## Related
+
+- [Reclaim disk space](../guides/reclaim-disk-space.md): the sweep, its
+  inventory, and the mtime fence.
+- [Fence a backup against the sweep](../guides/fence-a-backup.md): holding the
+  lease from a backup tool.
+- [Troubleshooting](../troubleshooting.md): what a lock timeout looks like and
+  what to change.

package/docs/concepts/tiers.md

@@ -0,0 +1,147 @@
+---
+title: "The three tiers"
+description: "One store contract with a filesystem, memory, and HTTP implementation, how CombinedArtifacts composes a local and a shared tier, and what the download policy changes about a read."
+sidebar:
+  order: 2
+---
+
+`ArtifactStore.Service` is four operations over one address space:
+
+| Operation              | Meaning                                                                                  |
+| ---------------------- | ---------------------------------------------------------------------------------------- |
+| `put(bytes)`           | Store the bytes under their own digest and return that address. Idempotent.              |
+| `get(digest)`          | Return the bytes at the address, verified against it.                                    |
+| `has(digest)`          | Whether this tier holds an artifact at the address.                                      |
+| `findMissing(digests)` | Which of these addresses this tier does not hold, as a deduplicated subset of the input. |
+
+Every implementation in this package is that same shape, which is the whole
+reason a composition can swap one for another without touching a caller.
+
+## The implementations
+
+| Constructor                                       | Backing                                             | Notes                                                                                         |
+| ------------------------------------------------- | --------------------------------------------------- | --------------------------------------------------------------------------------------------- |
+| `ArtifactStore.makeFileSystem(fs, path, options)` | A directory reached through Effect's `FileSystem`   | The durable host tier. Atomic publication, digest-verified reads, cross-process coordination. |
+| `ArtifactStore.makeMemory()`                      | A private `Map`                                     | Tests and browser hosts with no durable filesystem. Copies on both boundaries.                |
+| `ArtifactStore.makeNoop(overrides)`               | Nothing                                             | Every operation fails as `unavailable`, with per-method overrides. The honest absence.        |
+| `RemoteArtifacts.make(options)`                   | An HTTP cache reached through Effect's `HttpClient` | The shared tier. Bounded, digest-verified downloads.                                          |
+| `CombinedArtifacts.make(options)`                 | Two of the above                                    | Local first, shared second, with write-back.                                                  |
+
+`RemoteArtifacts.Service` extends the store contract with one field,
+`downloadPolicy`, so a composition can read the shared tier's own preference
+instead of being told it twice. `RemoteArtifacts.downloadPolicyOf` reads that
+field off any store and answers `undefined` for one that declares none, which
+every local store does.
+
+## What a combined read does
+
+`CombinedArtifacts.get` asks the local tier first and treats its three possible
+answers differently:
+
+| Local answer         | What happens                                                                                 |
+| -------------------- | -------------------------------------------------------------------------------------------- |
+| Bytes                | Returned. The shared tier is not contacted.                                                  |
+| `ArtifactMissing`    | Falls through to the shared tier.                                                            |
+| `ArtifactCorruption` | Falls through to the shared tier, and the fetched bytes are written back under every policy. |
+| `ArtifactStoreError` | The read fails.                                                                              |
+
+That last row is the one worth pausing on. A host that refused the read has not
+answered the question, and quietly paying for a network round trip instead
+would hide a broken local tier behind a working shared one. A miss is an
+answer; a refusal is not.
+
+The write-back is opportunistic in the same way the upload is. The bytes are
+already in hand, so a full disk, a read-only mount, or a refused sync costs the
+next read a round trip rather than failing this one. `writeBackTimeout` bounds
+the write-back, defaulting to 60 seconds. At expiry the write-back is interrupted;
+the read returns the verified remote bytes after interruption cleanup finishes.
+
+## What a combined write does
+
+`put` publishes to the local tier first and returns the local digest. The local
+tier is the one this machine's replays resolve against, so an unreachable
+shared tier must not stop an artifact from being recorded.
+
+The upload that follows is opportunistic, and its refusal is dropped rather
+than propagated. Failing here would fail whatever produced the bytes because a
+cache was unreachable, which is the opposite of the line above. Two properties
+keep it contained:
+
+- **Deduplicated in flight by digest.** Two callers in one process that publish
+  the same artifact join one upload instead of both pushing the same bytes.
+  The entry is removed before its deferred is completed, so a later put starts
+  a fresh upload rather than replaying a stale outcome, and an interrupted
+  upload resolves waiters with a typed failure instead of tearing them down.
+- **Bounded by `uploadTimeout`,** 60 seconds by default. A shared tier that
+  stalls instead of refusing must not hold the local answer hostage.
+
+Nothing depends on that upload. What actually guarantees a shared cache entry's
+blobs are durable is the publication protocol that runs before the entry is
+published: `findMissing`, then upload, then confirm. A dropped upload here
+costs that protocol one re-upload, never correctness.
+
+`has` asks local, then shared. `findMissing` asks the shared tier only about
+what the local tier could not answer, which is one network round trip over the
+smallest possible set, and the result stays a subset of the input because each
+stage filters the previous stage's output.
+
+## The download policy
+
+The dial is Bazel's `--remote_download_{all,toplevel,minimal}`, and it lives on
+the shared tier because the shared tier is the thing being conserved. Set it
+with `RemoteArtifacts.Options.downloadPolicy`, or override it for one
+composition with `CombinedArtifacts.Options.downloadPolicy`. With neither, the
+policy is `all`.
+
+| Policy     | Prefetch on replay admission | Write-back on a read                       |
+| ---------- | ---------------------------- | ------------------------------------------ |
+| `all`      | Every referenced blob        | Yes                                        |
+| `toplevel` | Nothing                      | Yes, on the first read that needs the blob |
+| `minimal`  | Nothing                      | No                                         |
+
+Two different modules honor different halves. `CombinedArtifacts.get` honors
+the write-back column. [`@smthrs/engine-store`](/api/engine-store)'s
+`ArtifactSync.hydrate` honors the prefetch column.
+
+`minimal` still writes back after local corruption. An address the local tier
+already claims, one that `has` and `findMissing` both report as present, must
+be one it can serve, or the publication protocol has been told a lie no later
+read can correct. Replacing such an address is repair, not growth.
+
+Choosing between them is a job, not a concept:
+[Share artifacts across machines](../guides/share-artifacts-across-machines.md)
+walks the composition.
+
+## What the metrics count
+
+`ArtifactStoreMetrics` defines two counters, `flows_artifact_puts` and
+`flows_artifact_gets`. Only the filesystem and memory implementations update
+them; `RemoteArtifacts` is deliberately uninstrumented, and the counters carry
+no tier attribute. Read them as local artifact store traffic:
+
+- A combined read the local tier serves counts one get.
+- A combined read the shared tier serves counts no get, because no local store
+  answered it.
+- The write-back that materializes such a read counts a put, indistinguishable
+  from a producer publishing new bytes.
+- A deduplicated put still counts: the caller stored bytes and received an
+  address either way.
+- Missing and corrupt reads count nothing. They are error evidence, not
+  throughput.
+
+A third counter, `flows_artifact_remote_failures`, counts the shared uploads
+(`operation: put`) and local write-backs (`operation: write_back`) a combined
+store dropped. Neither fails the caller, so this counter and its warning are
+the only evidence that the shared tier stopped filling.
+
+No exporter ships in this package; provide one, for example
+[`@smthrs/observability`](/api/observability), and the counters appear in it.
+
+## Related
+
+- [Content addressing](./content-addressing.md): why a read can verify rather
+  than trust.
+- [Coordination between processes](./coordination.md): what happens when two
+  processes share one objects directory.
+- [Serve the artifact protocol](../guides/serve-the-artifact-protocol.md): the
+  HTTP surface a shared tier owes.

package/docs/guides/fence-a-backup.md

@@ -0,0 +1,123 @@
+---
+title: "Fence a backup against the sweep"
+description: "Hold ArtifactBackupLease.withLease around a filesystem backup so a concurrent sweep cannot delete a blob the frozen database references, and use unlessActive when you are the one deleting."
+sidebar:
+  order: 4
+---
+
+A filesystem backup freezes a database and then copies the blobs that database
+references. Between the freeze and the last copy, a garbage collector running
+in another process can delete a blob the frozen snapshot names, leaving a
+backup that cannot be restored.
+
+`ArtifactBackupLease` is the cross-process exclusion for exactly that window.
+
+## Before you start
+
+- The objects directory the store publishes into. The lease marker lives inside
+  it, so a backup and a sweep pointed at different directories fence nothing.
+- Both sides on `coordination: "required"`. A sweep built with `process` skips
+  the lease entirely.
+
+## Hold the lease around the copy
+
+`withLease` acquires the marker, heartbeats it while your effect runs, and
+releases it afterwards. Pass a `failure` function that maps an unknown host
+cause into your own error type:
+
+```ts
+import * as ArtifactBackupLease from "@smthrs/artifacts/ArtifactBackupLease"
+import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
+import * as Effect from "effect/Effect"
+import * as FileSystem from "effect/FileSystem"
+
+const leaseFailure = (cause: unknown): ArtifactStore.ArtifactStoreError =>
+  new ArtifactStore.ArtifactStoreError({
+    code: "unavailable",
+    message: "the artifact backup lease could not be held",
+    cause
+  })
+
+/** Runs `copy` while no sweep in any process can delete a referenced blob. */
+export const underLease = <A, E, R>(copy: Effect.Effect<A, E, R>) =>
+  Effect.gen(function*() {
+    const fs = yield* FileSystem.FileSystem
+    return yield* ArtifactBackupLease.withLease(fs, ArtifactStore.defaultDirectory, copy, leaseFailure)
+  })
+```
+
+Wrap the whole backup, from the database freeze through the last blob copy. A
+lease held only around the copy leaves the freeze unprotected, which is where
+the reference list came from.
+
+## What the lease stops, and what it does not
+
+It stops deletion. It does not stop publication: new blobs may appear
+throughout a backup, and that is harmless, because a blob published after the
+freeze cannot be referenced by the frozen database.
+
+The bounds are the same ones the per-digest locks use:
+
+| Bound                | Value                          |
+| -------------------- | ------------------------------ |
+| Heartbeat interval   | 10 seconds                     |
+| Stale after          | 60 seconds without a heartbeat |
+| Acquisition deadline | 2 minutes                      |
+
+A crashed backup stops heartbeating, and its marker becomes reclaimable a
+minute later, so a hard-killed process cannot fence collection forever. The
+flip side is that a backup whose host stalls past 60 seconds can have its lease
+reaped while it is still copying.
+
+Releasing is tolerant of that. A lease whose heartbeat lapsed has its marker
+reaped by whoever noticed, so finding nothing to release is the ordinary end of
+a slow lease rather than a fault.
+
+Acquisition is bounded rather than infinite: if another backup holds the lease
+for more than two minutes, `withLease` fails through your `failure` function
+instead of waiting.
+
+## When you are the one deleting
+
+`ArtifactSweep.remove` already consults the lease under
+`coordination: "required"`, so a collector built on that surface needs nothing
+here. Use `ArtifactBackupLease.unlessActive` only when you are writing a
+different deletion path over the same objects directory:
+
+`unlessActive` runs your effect only when no live lease exists, and it holds
+the workspace-global gate across both the check and the effect, so the two
+cannot be separated by a backup starting in between. It answers `Option.none()`
+when a backup deliberately fenced the operation, which is what
+`ArtifactSweep.remove` turns into `false`:
+
+```ts
+import * as ArtifactBackupLease from "@smthrs/artifacts/ArtifactBackupLease"
+import * as Effect from "effect/Effect"
+import * as FileSystem from "effect/FileSystem"
+import * as Option from "effect/Option"
+
+/** Deletes one path unless a live backup fenced the deletion. */
+export const removeUnlessBackingUp = (directory: string, path: string) =>
+  Effect.gen(function*() {
+    const fs = yield* FileSystem.FileSystem
+    const outcome = yield* ArtifactBackupLease.unlessActive(
+      fs,
+      directory,
+      fs.remove(path).pipe(Effect.as(true), Effect.mapError(leaseFailure)),
+      leaseFailure
+    )
+    return Option.getOrElse(outcome, () => false)
+  })
+```
+
+Treat `none` as "nothing was reclaimed, retry after the backup finishes", never
+as "the blob is gone".
+
+## Related
+
+- [Reclaim disk space](./reclaim-disk-space.md): the collector side, and why
+  its `false` is ambiguous on purpose.
+- [Coordination between processes](../concepts/coordination.md): the gate, the
+  markers, and the stale-reclaim window they share.
+- [Retention and cleanup](/docs/guides/retention/): backing up a project before
+  collecting it, on smithers.sh.

package/docs/guides/reclaim-disk-space.md

@@ -0,0 +1,143 @@
+---
+title: "Reclaim disk space"
+description: "Enumerate an objects directory with ArtifactSweep, delete a blob behind the mtime fence, count only what you actually reclaimed, and hand the policy to the engine collector that owns it."
+sidebar:
+  order: 3
+---
+
+An objects directory grows forever unless something removes from it. This
+package ships the mechanics: enumerate what is there, and delete one blob
+safely. It ships no policy, because deciding which digests are still live needs
+the durable roots, and only the engine composition can see those.
+
+Most projects never call this API directly.
+[`@smthrs/engine-store`](/api/engine-store)'s `ArtifactGc` is the collector
+built on it: its mark phase walks the durable roots, and `gc()` is an explicit
+call that nothing schedules, because deletion is the irreversible direction.
+Note that [`smthrs gc`](/cli/gc) collects terminal runs and does not touch the
+artifact tier at all.
+[Retention and cleanup](/docs/guides/retention/#4-sweep-unreferenced-artifacts)
+is the operator's version of this page. Read on if you are writing a collector
+of your own.
+
+## Before you start
+
+- The objects directory the store publishes into. Get it wrong and you
+  enumerate somewhere nothing was ever written.
+- A live set: the digests something still references. Producing one is your
+  collector's job, not this package's.
+
+## 1. Provide the sweep beside the store
+
+Build both with the same `directory` and the same `coordination`. Nothing
+checks the pairing, and a mismatch is silent:
+
+```ts
+import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
+import * as ArtifactSweep from "@smthrs/artifacts/ArtifactSweep"
+import * as Layer from "effect/Layer"
+
+const directory = ".flows/objects"
+
+export const artifacts = Layer.merge(
+  ArtifactStore.layerFileSystem({ directory, coordination: "required" }),
+  ArtifactSweep.layerFileSystem({ directory, coordination: "required" })
+)
+```
+
+`SweepOptions` has no `durability` field, deliberately: a sweep never writes a
+blob, so accepting the option and ignoring it would be a lie.
+
+## 2. Enumerate what is on disk
+
+`inventory` returns a `BlobStat` for each canonical blob: its digest, its
+modification time in milliseconds, and its size in bytes.
+
+It is conservative by construction. Only paths in the store's canonical fanout
+shape are blobs, so temp files, lock files, foreign files, nested paths, and a
+directory sitting at a blob address are all skipped rather than deleted. A blob
+that vanishes between the listing and the stat was removed by someone else and
+is skipped too. So is one whose modification time the host cannot report: what
+the sweep cannot judge, it must not touch.
+
+A store that never published anything has no directory at all. That is an empty
+inventory, not a failure.
+
+## 3. Delete behind the fence
+
+`RemoveOptions.ifUnmodifiedSinceMs` rides inside the deletion rather than in a
+prior read. A blob freshened after you computed your live set, by a concurrent
+put re-referencing the same bytes, fails the fence and survives:
+
+```ts
+import * as ArtifactSweep from "@smthrs/artifacts/ArtifactSweep"
+import * as Effect from "effect/Effect"
+
+/** Removes every blob outside `live` that has not been touched since `keepAfterMs`. */
+export const collect = (live: ReadonlySet<string>, keepAfterMs: number) =>
+  Effect.gen(function*() {
+    const sweep = yield* ArtifactSweep.ArtifactSweep
+    let reclaimedBytes = 0
+    for (const blob of yield* sweep.inventory) {
+      if (live.has(blob.digest) || blob.modifiedAtMs > keepAfterMs) continue
+      const removed = yield* sweep.remove(blob.digest, { ifUnmodifiedSinceMs: keepAfterMs })
+      if (removed) reclaimedBytes += blob.sizeBytes
+    }
+    return reclaimedBytes
+  })
+```
+
+The inventory check is the cheap filter; the fence in `remove` is the one that
+actually holds, because the world can change between the two.
+
+`keepAfterMs` is where a grace period lives. Pick an instant far enough in the
+past that a blob published or freshened while your mark phase was running
+survives this pass: `ArtifactGc` defaults to two weeks.
+
+## 4. Read the boolean correctly
+
+`remove` reports whether bytes were removed. A missing blob reports `false`
+rather than failing, so a crashed and re-run sweep converges instead of erroring
+on its own progress.
+
+`false` covers three outcomes a caller cannot tell apart:
+
+- The blob was already gone.
+- It failed the `ifUnmodifiedSinceMs` fence.
+- A live backup lease fenced the deletion.
+
+All three mean nothing was reclaimed and retrying later is safe. The third is
+not progress: the blob is still there, so a collector counting reclaimed bytes
+must not count it, and the next pass after the backup finishes removes it. The
+loop above already gets this right by adding `blob.sizeBytes` only on `true`.
+
+## 5. Budget for the coordination cost
+
+Under `coordination: "required"`, one deletion takes the per-digest lock and
+then the workspace-global backup-lease gate: roughly ten filesystem operations
+and two forked heartbeat fibers per blob. The gate is one file for the whole
+workspace, so concurrent deletions serialize through it.
+
+That is the price of never deleting a blob a running backup already recorded.
+Size a collection window with it in mind rather than assuming one unlink per
+blob.
+
+## Verify it worked
+
+Count the blobs before and after:
+
+```bash
+find .flows/objects -type f -name '????????????????????????????????????????????????????????????????' | wc -l
+```
+
+The name pattern matches a 64-character digest, so temp and lock files stay out
+of the count.
+
+## Related
+
+- [Fence a backup against the sweep](./fence-a-backup.md): the other side of
+  the lease this sweep consults.
+- [Coordination between processes](../concepts/coordination.md): the lock
+  protocol, its bounds, and what still holds when the lock does not.
+- [Retention and cleanup](/docs/guides/retention/): the operator's runbook on
+  smithers.sh.