@smthrs/artifacts 0.0.0-stage → 1.0.0-rc.3

package/docs/quickstart.md

@@ -0,0 +1,182 @@
+---
+title: "Quickstart"
+description: "Publish bytes to a real objects directory, read them back, watch the address deduplicate a second put, and watch a corrupted blob get refused and then healed."
+sidebar:
+  order: 2
+---
+
+This quickstart runs the filesystem store against a real directory. By the end
+you will have published an artifact, read it back, seen the address absorb a
+second identical put, and seen the store refuse and then repair a blob you
+corrupted on purpose.
+
+## Prerequisites
+
+- Node.js 26.4.0 or later.
+- An empty directory to work in. The store writes under `.flows/objects`
+  relative to the process working directory.
+- The package and the Node host layers. Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
+
+## Compose the store
+
+Create `quickstart.ts`. The filesystem store needs Effect's `FileSystem` to
+reach the disk and `Crypto` to measure bytes, so the composition provides both
+and keeps them visible for the rest of the walkthrough:
+
+```ts
+import * as NodeCrypto from "@effect/platform-node/NodeCrypto"
+import * as NodeFileSystem from "@effect/platform-node/NodeFileSystem"
+import * as NodePath from "@effect/platform-node/NodePath"
+import * as ArtifactStore from "@smthrs/artifacts/ArtifactStore"
+import * as Effect from "effect/Effect"
+import * as FileSystem from "effect/FileSystem"
+import * as Layer from "effect/Layer"
+
+const directory = ".flows/objects"
+const payload = new TextEncoder().encode("the bytes a step produced")
+
+const host = Layer.mergeAll(NodeFileSystem.layer, NodePath.layer, NodeCrypto.layer)
+const layer = ArtifactStore.layerFileSystem({ directory }).pipe(Layer.provideMerge(host))
+```
+
+`Layer.provideMerge` feeds the host services into the store and keeps them in
+the resulting layer, so the snippets below can use `FileSystem` directly to
+tamper with a blob.
+
+## Publish an artifact and read it back
+
+`put` measures the bytes, publishes them, and returns their address. `get`
+verifies that the stored bytes still hash to the address before returning
+them:
+
+```ts
+const digest = await Effect.runPromise(
+  Effect.gen(function*() {
+    const store = yield* ArtifactStore.ArtifactStore
+    const address = yield* store.put(payload)
+
+    console.log("address:", address)
+    console.log("read back:", new TextDecoder().decode(yield* store.get(address)))
+    return address
+  }).pipe(Effect.provide(layer))
+)
+```
+
+Each snippet below is another statement in the same file. They all provide the
+same `layer`, so they all reach the same directory.
+
+## Put the same bytes twice
+
+There is no second copy and no second address. The store verifies the blob
+already at the address, freshens its modification time so a retention sweep
+reads it as recently referenced, and returns the same digest:
+
+```ts
+const again = await Effect.runPromise(
+  Effect.gen(function*() {
+    const store = yield* ArtifactStore.ArtifactStore
+    return yield* store.put(payload)
+  }).pipe(Effect.provide(layer))
+)
+
+console.log("same address:", again === digest)
+```
+
+## Read an address nothing was published under
+
+A well-formed address with no bytes behind it is `ArtifactMissing`, a distinct
+tag rather than a generic failure, because a composition with a second tier
+acts on it:
+
+```ts
+const miss = await Effect.runPromise(
+  Effect.gen(function*() {
+    const store = yield* ArtifactStore.ArtifactStore
+    return yield* store.get("0".repeat(64))
+  }).pipe(
+    Effect.catchTag("@smthrs/artifacts/ArtifactMissing", (failure) => Effect.succeed(failure.code)),
+    Effect.provide(layer)
+  )
+)
+
+console.log("absent address:", miss)
+```
+
+An address that is not 64 lowercase hexadecimal characters never reaches the
+disk at all: it fails with an `ArtifactStoreError` whose code is
+`invalid_digest`.
+
+## Corrupt a blob, then heal it
+
+Overwrite the published blob with different bytes. The next read measures what
+it found, sees that it no longer matches the address, and refuses. The repair
+is an ordinary put: because the store verifies the blob already at an address
+rather than trusting that the path exists, the mismatch falls through to the
+atomic rewrite:
+
+```ts
+await Effect.runPromise(
+  Effect.gen(function*() {
+    const store = yield* ArtifactStore.ArtifactStore
+    const fs = yield* FileSystem.FileSystem
+    const blob = `${directory}/${digest.slice(0, 2)}/${digest}`
+
+    yield* fs.writeFile(blob, new TextEncoder().encode("tampered"))
+
+    const refusal = yield* store.get(digest).pipe(
+      Effect.catchTag("@smthrs/artifacts/ArtifactCorruption", (failure) => Effect.succeed(failure.measuredDigest))
+    )
+    console.log("measured instead:", refusal)
+
+    yield* store.put(payload)
+    console.log("healed:", new TextDecoder().decode(yield* store.get(digest)))
+  }).pipe(Effect.provide(layer))
+)
+```
+
+## Run it
+
+Run the file with your TypeScript runner. The output is:
+
+```text
+address: 077303668cf56af8d162bb5ccccd7127f2e8baff448bf5b649530a98e9c943da
+read back: the bytes a step produced
+same address: true
+absent address: artifact_missing
+measured instead: d121be3103007b41edf96f8262925f8c7d61894afe9a041843b631f69445bc57
+healed: the bytes a step produced
+```
+
+Look at what landed on disk:
+
+```bash
+find .flows/objects -type f
+```
+
+```text
+.flows/objects/07/077303668cf56af8d162bb5ccccd7127f2e8baff448bf5b649530a98e9c943da
+```
+
+The two-hex directory is the digest's own prefix. It exists so a workspace that
+publishes hundreds of thousands of artifacts never puts them all in one
+directory.
+
+## What just happened
+
+Every operation was addressed by content, so nothing above needed a file name,
+a bucket, or an identifier of its own. The second put stored no second copy
+because the address already existed and verified, but it was not free: it
+hashed its input, rehashed the stored blob, took the digest's lock, freshened
+the blob, and synced it. The corrupted read failed loudly instead of returning
+bytes that were not the recorded artifact, and the repair was the same call
+that published it the first time.
+
+## Next steps
+
+- [Content addressing](./concepts/content-addressing.md): why each of those
+  outcomes is the only one the store can give.
+- [Share artifacts across machines](./guides/share-artifacts-across-machines.md):
+  put the same store behind an HTTP cache so a second machine reads what this
+  one produced.
+- [Test against an artifact store](./guides/test-against-an-artifact-store.md):
+  the in-memory store, which needs no directory and no temporary files.

package/docs/troubleshooting.md

@@ -0,0 +1,232 @@
+---
+title: "Troubleshooting"
+description: "Every failure @smthrs/artifacts reports, plus the silent problems that produce no failure at all: what happened, why, and what to change."
+---
+
+Every failure here is typed. Read the tag first, then the `code` inside
+`ArtifactStoreError`, then the message. The full schemas are on the
+[API reference](./api.md).
+
+Three tags exist, and they mean different things on purpose:
+
+| Tag                                    | Meaning                                                           |
+| -------------------------------------- | ----------------------------------------------------------------- |
+| `@smthrs/artifacts/ArtifactMissing`    | An ordinary miss. Another tier may still satisfy it.              |
+| `@smthrs/artifacts/ArtifactCorruption` | An integrity violation. Bytes at an address no longer hash to it. |
+| `@smthrs/artifacts/ArtifactStoreError` | The store, host, or transport refused. Carries a `code`.          |
+
+## ArtifactMissing
+
+**What happened.** The address is well formed, and this tier holds no bytes
+under it. The error carries the validated `digest`.
+
+**What to change.** Usually nothing: a read-through composition acts on this
+rather than failing. If a blob you expect really is gone, check in this order:
+
+- The objects directory. A store pointed at a different `directory` than the
+  one that published is the most common cause, and it looks exactly like a
+  cold cache.
+- A sweep. `ArtifactSweep.remove` reclaimed it, which means the mark phase did
+  not consider it live.
+- The layout. Addresses published under the pre-1.0 flat
+  `<directory>/<digest>` layout are misses that re-publish. There is no
+  compatibility shim.
+- The shared tier. A combined `put` uploads opportunistically, so a blob local
+  to the producing machine may never have reached the cache. See
+  [The three tiers](./concepts/tiers.md).
+
+## ArtifactCorruption
+
+**What happened.** The bytes found at an address do not hash to it. The error
+carries both `recordedDigest` and `measuredDigest`.
+
+**What to change.** Publish the correct bytes again. Because every put verifies
+the existing blob rather than trusting the path, the mismatch falls through to
+an atomic rewrite that heals the address. A combined store does this for you:
+it fetches from the shared tier and writes back under every download policy,
+including `minimal`.
+
+If it recurs on one host, suspect the disk or another process writing into the
+objects directory. If it recurs from one shared tier, suspect the tier: a
+mis-serving cache is exactly what this check exists to catch.
+
+## invalid_digest
+
+**Message.** `artifact digest must be exactly 64 lowercase hexadecimal characters`
+
+**What happened.** A value that is not a canonical SHA-256 address reached a
+store or sweep operation. The message is a constant, so it never echoes the
+value.
+
+**What to change.** Look at what produced the string. Uppercase hexadecimal, a
+truncated digest, a filesystem path, and a `key1_`-prefixed step key are all
+common. A step key is not an artifact address: keys come from
+[`@smthrs/keys`](/api/keys) and address cache entries, while artifact addresses
+come from `put`.
+
+## invalid_configuration
+
+**What happened.** An option was refused at construction, before any request
+left the process. Every message names the violated rule and nothing else.
+
+| Message                                                                         | Cause                                                                                                              |
+| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
+| `invalid remote artifact option: endpoint`                                      | The endpoint is not a string.                                                                                      |
+| `invalid remote artifact endpoint`                                              | No `URL` parser accepts it.                                                                                        |
+| `remote artifact endpoint must use HTTPS`                                       | The scheme is not `https:`, and the host is not literal loopback (`localhost`, `127.0.0.1`, `[::1]`).              |
+| `remote artifact endpoint must not contain credentials, a query, or a fragment` | The endpoint carries userinfo, `?`, or `#`.                                                                        |
+| `invalid remote artifact option: options`                                       | A header name is not an HTTP token, or a header value is not a string, exceeds 16 KiB, or has a control character. |
+| `invalid remote artifact option: <name>`                                        | A deadline, `maxDownloadBytes`, `maxFindMissingResponseBytes`, `chunkBytes`, or `downloadPolicy` is out of range.  |
+| `invalid combined artifact option: uploadTimeout`                               | The combined upload deadline is not a positive finite duration.                                                    |
+| `invalid combined artifact option: downloadPolicy`                              | The policy is not `all`, `toplevel`, or `minimal`.                                                                 |
+
+**What to change.** Move a credential out of the URL and into
+`RemoteArtifacts.Options.headers`, which is the credential seam. Note that
+`maxFindMissingResponseBytes` may only lower the protocol's 256 KiB bound, so a
+larger value is refused.
+
+## unavailable
+
+**What happened.** A host refused, or the store has no implementation.
+
+| Message                                                  | Cause                                                                                                       |
+| -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
+| `the host filesystem refused an artifact operation: ...` | The `FileSystem` service refused a read, write, rename, sync, or lock.                                      |
+| `the host filesystem refused a sweep operation: ...`     | The same, from `ArtifactSweep`.                                                                             |
+| `the host could not copy the artifact bytes`             | The snapshot of the caller's buffer failed: a detached buffer, or an allocation the runtime would not give. |
+| `<method> is unavailable`                                | A `makeNoop` store or sweep, with no override for that method.                                              |
+
+**What to change.** The cause carries the underlying host error. Two cases have
+non-obvious fixes:
+
+- A `TimeoutError` in the cause of a filesystem operation means lock
+  acquisition gave up after two minutes. Another process is holding the digest
+  lock, or the workspace-global backup-lease gate, and is still heartbeating.
+  Find it, or drop the store to `coordination: "process"` if cross-process
+  exclusion is not something this host needs.
+- A sync refusal can use `durability: "best-effort"` when weaker durability is
+  acceptable. Both modes still require exclusive writable handles and symlink
+  inspection. If the host lacks either, use a memory or remote tier until the
+  host supplies them. A detected symlink or directory replacement also fails
+  as `unavailable`; audit the store before retrying.
+
+`the shared upload was interrupted before it settled` is a special case: it is
+the typed answer given to callers waiting on an in-flight combined upload that
+was interrupted. A combined `put` already drops it, and the next put of that
+digest starts a fresh upload.
+
+## digest_failed
+
+**Message.** `the Crypto service failed to compute an artifact digest`
+
+**What happened.** The injected `Crypto` service refused. Nothing about the
+bytes or the store is implicated.
+
+**What to change.** Check the `Crypto` layer the composition provides.
+`NodeCrypto.layer` from `@effect/platform-node` is the Node implementation. A
+composition with no `Crypto` at all does not compile, so this is a runtime
+refusal, not a missing layer.
+
+## transport_failed
+
+**What happened.** The shared tier answered in a way the protocol cannot use.
+
+| Message                                                                                 | Cause                                                                                                         |
+| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
+| `the remote artifact tier refused <operation>`                                          | The HTTP client could not complete the exchange, or the body could not be read.                               |
+| `the remote artifact tier answered <operation> with HTTP <status>`                      | Any non-2xx other than the `404` that means a miss.                                                           |
+| `the remote artifact tier answered <operation> with <n> bytes, past the <m>-byte bound` | A body past `maxDownloadBytes` or `maxFindMissingResponseBytes`, refused from `Content-Length` or mid-stream. |
+| `the remote artifact tier did not finish <operation> within its configured deadline`    | The exchange outlived `downloadTimeout`, `uploadTimeout`, or `requestTimeout`.                                |
+
+**What to change.** Match the operation to the request in
+[Serve the artifact protocol](./guides/serve-the-artifact-protocol.md). A
+`403` on a `PUT` while `GET` works means a read credential where a write one is
+needed. A `413` means the tier's request-body cap: the Smithers cache
+services cap one body at 16 MiB, and no `chunkBytes` setting works around that,
+because they refuse ranged `PUT` with `400`.
+
+Raise a deadline only after checking that the tier is actually slow rather than
+unreachable. `CombinedArtifacts.get` propagates a remote `ArtifactStoreError`,
+including download timeouts. `CombinedArtifacts.put` ignores failed opportunistic
+uploads. `ArtifactSync.hydrate` separately converts remote failure into replay
+rejection.
+
+## An upload succeeded here but a second machine cannot read it
+
+**What happened.** A combined `put` returns the local digest and uploads
+opportunistically. If the shared tier refused or stalled, the upload was
+dropped and the operation still succeeded, by design: a cache being unreachable
+must not fail the step that produced the bytes.
+
+**What to change.** Nothing, if you are relying on the publication protocol.
+[`@smthrs/engine-store`](/api/engine-store)'s `ArtifactSync` runs `findMissing`,
+uploads, and confirms before a cache entry becomes observable, so a dropped
+upload costs one re-upload rather than correctness. If you are calling `put`
+directly and need the shared copy, probe the shared tier with `has` or
+`findMissing` afterwards.
+
+## The sweep enumerates nothing, or deletes nothing
+
+**What happened.** `inventory` returned an empty array, or `remove` keeps
+answering `false`.
+
+**What to change.**
+
+- **Empty inventory.** The sweep is pointed at a directory the store never
+  published into. A directory that does not exist is an empty inventory, not a
+  failure. Build the store and the sweep with the same `directory`.
+- **`remove` answers `false`.** Three outcomes share that answer: the blob was
+  already gone, it failed the `ifUnmodifiedSinceMs` fence, or a live backup
+  lease fenced the deletion. The third is not progress, so do not count those
+  bytes as reclaimed; the next pass after the backup finishes removes them. See
+  [Reclaim disk space](./guides/reclaim-disk-space.md).
+- **Foreign files survive.** Only paths in the canonical `xx/<digest>` fanout
+  shape are blobs. Temp files, lock files, nested paths, and anything else are
+  skipped rather than deleted, and so is a blob whose modification time the
+  host cannot report.
+
+## A sweep deleted a blob a writer was publishing
+
+**What happened.** The store and the sweep disagree about `coordination`. A
+sweep on `required` beside a store on `process` takes lock files no writer ever
+observes: the fence reads as armed and protects nothing.
+
+**What to change.** Build both with the same `coordination` and the same
+`directory`. Nothing checks the pairing at runtime, and no error is raised.
+See [Coordination between processes](./concepts/coordination.md).
+
+## The get counter looks too low
+
+**What happened.** `flows_artifact_gets` and `flows_artifact_puts` count local
+artifact store traffic only. `RemoteArtifacts` is uninstrumented, so a combined
+read the shared tier served increments no get counter, and the write-back that
+materializes it increments a put indistinguishable from a producer publishing
+new bytes.
+
+**What to change.** Read the counters as local traffic rather than as artifact
+operations. There is no tier attribute; adding one would change the published
+counter shape. The full accounting is in
+[The three tiers](./concepts/tiers.md#what-the-metrics-count).
+
+## Chunked uploads are not chunking
+
+**What happened.** `chunkBytes` is set, and every blob still travels as one
+whole-blob `PUT`.
+
+**What to change.** Probably nothing: that is the designed degradation. A tier
+that answers the empty `Content-Range: bytes */{total}` probe with `400`,
+`411`, or `416`, or that answers `2xx` to a chunk that does not complete the
+blob, is treated as range-unaware, and the client sends the blob whole so it
+always lands. Both Smithers cache services answer `400`.
+
+If you control the tier and want resumable transfers, implement the `308`
+sequence in
+[Serve the artifact protocol](./guides/serve-the-artifact-protocol.md).
+
+## Related
+
+- [API reference](./api.md): the exact error schemas and codes.
+- [Content addressing](./concepts/content-addressing.md): why corruption is a
+  refusal rather than a returned value.
+- [Coordination between processes](./concepts/coordination.md): the lock
+  bounds behind the timeout failures.

package/package.json

@@ -1,6 +1,182 @@
 {
   "name": "@smthrs/artifacts",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
+  "type": "module",
+  "version": "1.0.0-rc.3",
+  "license": "MIT",
+  "smthrs": {
+    "group": "engine"
+  },
+  "description": "Effect service for the Smithers content-addressed artifact store, local and remote",
+  "homepage": "https://github.com/smithersai/smithers",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/smithersai/smithers.git",
+    "directory": "packages/smithers/flows/artifacts"
+  },
+  "bugs": {
+    "url": "https://github.com/smithersai/smithers/issues"
+  },
+  "tags": [
+    "typescript",
+    "effect",
+    "cache",
+    "content-addressed"
+  ],
+  "keywords": [
+    "typescript",
+    "effect",
+    "cache",
+    "content-addressed"
+  ],
+  "engines": {
+    "node": ">=26.4.0"
+  },
+  "sideEffects": [],
+  "exports": {
+    "./package.json": "./package.json",
+    ".": {
+      "import": {
+        "types": "./dist/esm/index.d.ts",
+        "default": "./dist/esm/index.js"
+      },
+      "require": {
+        "types": "./dist/cjs/index.d.ts",
+        "default": "./dist/cjs/index.js"
+      }
+    },
+    "./internal/*": null,
+    "./*/index": null,
+    "./ArtifactBackupLease": {
+      "import": {
+        "types": "./dist/esm/ArtifactBackupLease.d.ts",
+        "default": "./dist/esm/ArtifactBackupLease.js"
+      },
+      "require": {
+        "types": "./dist/cjs/ArtifactBackupLease.d.ts",
+        "default": "./dist/cjs/ArtifactBackupLease.js"
+      }
+    },
+    "./ArtifactStore": {
+      "import": {
+        "types": "./dist/esm/ArtifactStore.d.ts",
+        "default": "./dist/esm/ArtifactStore.js"
+      },
+      "require": {
+        "types": "./dist/cjs/ArtifactStore.d.ts",
+        "default": "./dist/cjs/ArtifactStore.js"
+      }
+    },
+    "./ArtifactStoreMetrics": {
+      "import": {
+        "types": "./dist/esm/ArtifactStoreMetrics.d.ts",
+        "default": "./dist/esm/ArtifactStoreMetrics.js"
+      },
+      "require": {
+        "types": "./dist/cjs/ArtifactStoreMetrics.d.ts",
+        "default": "./dist/cjs/ArtifactStoreMetrics.js"
+      }
+    },
+    "./ArtifactSweep": {
+      "import": {
+        "types": "./dist/esm/ArtifactSweep.d.ts",
+        "default": "./dist/esm/ArtifactSweep.js"
+      },
+      "require": {
+        "types": "./dist/cjs/ArtifactSweep.d.ts",
+        "default": "./dist/cjs/ArtifactSweep.js"
+      }
+    },
+    "./CombinedArtifacts": {
+      "import": {
+        "types": "./dist/esm/CombinedArtifacts.d.ts",
+        "default": "./dist/esm/CombinedArtifacts.js"
+      },
+      "require": {
+        "types": "./dist/cjs/CombinedArtifacts.d.ts",
+        "default": "./dist/cjs/CombinedArtifacts.js"
+      }
+    },
+    "./FileLease": {
+      "import": {
+        "types": "./dist/esm/FileLease.d.ts",
+        "default": "./dist/esm/FileLease.js"
+      },
+      "require": {
+        "types": "./dist/cjs/FileLease.d.ts",
+        "default": "./dist/cjs/FileLease.js"
+      }
+    },
+    "./RemoteArtifacts": {
+      "import": {
+        "types": "./dist/esm/RemoteArtifacts.d.ts",
+        "default": "./dist/esm/RemoteArtifacts.js"
+      },
+      "require": {
+        "types": "./dist/cjs/RemoteArtifacts.d.ts",
+        "default": "./dist/cjs/RemoteArtifacts.js"
+      }
+    },
+    "./index": {
+      "import": {
+        "types": "./dist/esm/index.d.ts",
+        "default": "./dist/esm/index.js"
+      },
+      "require": {
+        "types": "./dist/cjs/index.d.ts",
+        "default": "./dist/cjs/index.js"
+      }
+    }
+  },
+  "files": [
+    "src/**/*.ts",
+    "dist/**/*.js",
+    "dist/**/*.js.map",
+    "dist/**/*.d.ts",
+    "dist/**/*.d.ts.map",
+    "dist/**/package.json",
+    "LICENSE",
+    "README.md",
+    "CHANGELOG.md",
+    "docs/"
+  ],
+  "publishConfig": {
+    "access": "public",
+    "provenance": true,
+    "tag": "next"
+  },
+  "dependencies": {
+    "@smthrs/crypto": "1.0.0-rc.3"
+  },
+  "devDependencies": {
+    "@effect/language-service": "0.87.2",
+    "@effect/platform-node": "4.0.0-rc.115",
+    "@effect/vitest": "4.0.0-rc.115",
+    "@eslint/js": "9.39.5",
+    "@types/node": "26.4.1",
+    "@vitest/coverage-v8": "5.0.0",
+    "dprint": "0.57.1",
+    "effect": "4.0.0-rc.115",
+    "esbuild": "0.28.2",
+    "eslint": "9.39.5",
+    "eslint-import-resolver-typescript": "4.4.5",
+    "eslint-plugin-import": "2.32.0",
+    "eslint-plugin-jsdoc": "64.3.4",
+    "eslint-plugin-unicorn": "65.0.0",
+    "madge": "8.0.0",
+    "typescript": "5.9.3",
+    "typescript-eslint": "8.69.0",
+    "vitest": "5.0.0"
+  },
+  "peerDependencies": {
+    "effect": "4.0.0-rc.115"
+  },
+  "scripts": {
+    "lint": "eslint src --max-warnings=0 && dprint check",
+    "format": "dprint fmt",
+    "build": "node scripts/build.mjs",
+    "check": "tsc -b tsconfig.json && tsc -p tsconfig.test.json --noEmit",
+    "circular": "node scripts/circular.mjs",
+    "test": "vitest",
+    "coverage": "vitest --coverage"
+  }
 }