@smthrs/journal 0.0.0-stage → 1.0.0-rc.3

package/docs/quickstart.md

@@ -0,0 +1,143 @@
+---
+title: "Quickstart"
+description: "Compose a SQLite journal, write on both channels, flush the lossy one, and read the run back: one file, no other Smithers package."
+sidebar:
+  order: 2
+---
+
+This quickstart writes one run's history and reads it back. It uses the
+production `SqlJournal` layer over a real SQLite file, so nothing here is a
+test double, and it needs no other Smithers package.
+
+By the end you will have seen a durable receipt, a lossy receipt, the flush
+barrier that makes the lossy entry readable, and a payload that arrives back
+redacted.
+
+## Prerequisites
+
+- Node.js 26.4.0 or later, where the built-in SQLite module lives.
+- An ESM package (`"type": "module"`, because the program ends in a top-level
+  `await`) with the journal, the database it writes through, and `effect`
+  installed. See [Installation](./installation.md) for why `effect` is pinned.
+
+Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
+
+## Compose the layer
+
+Create `quickstart.ts`. `SqlJournal.layer` needs a SQL client and a durable
+writer, and `Migrations.layer` creates the journal's tables before the journal
+is exposed:
+
+```ts
+import * as DurableWriter from "@smthrs/database/DurableWriter"
+import * as NodeDatabase from "@smthrs/database/node/NodeDatabase"
+import { Journal, JournalEvent, Migrations, SqlJournal } from "@smthrs/journal"
+import * as Effect from "effect/Effect"
+import * as Layer from "effect/Layer"
+
+const database = Layer.provideMerge(
+  DurableWriter.layer(),
+  NodeDatabase.layer({ filename: "quickstart.db" })
+)
+
+const journalLayer = SqlJournal.layer({ capacity: 1024, overflow: "reject" }).pipe(
+  Layer.provide(Layer.provideMerge(Migrations.layer, database))
+)
+```
+
+`capacity` bounds two things at once: how many entries the lossy admission
+queue holds, and how many the `changes` buffer keeps for a slow subscriber.
+`overflow: "reject"` makes a full queue a typed `queue_overflow` failure rather
+than a silent drop.
+
+## Write on both channels
+
+Identifiers are branded, so cast the strings once at the edge:
+
+```ts
+const runId = "run-1" as JournalEvent.RunId
+const sourceId = "engine" as JournalEvent.SourceId
+
+const program = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+
+  const created = yield* journal.emitDurableUnfenced({
+    runId,
+    sourceId,
+    eventType: "run.created",
+    payload: { flow: "build", token: "ghp_0123456789abcdefghijklmnopqrstuvwxyz" }
+  })
+  console.log("durable:", created._tag, "seq", created.seq)
+
+  yield* journal.emitLossy({
+    runId,
+    sourceId,
+    eventType: "step.progress",
+    payload: { step: "compile", percent: 40 }
+  })
+  yield* journal.flush
+
+  const page = yield* journal.entries({ runId, limit: 100 })
+  for (const entry of page.entries) {
+    console.log(entry.seq, entry.eventType, JSON.stringify(entry.payload))
+  }
+  console.log("hasMore:", page.hasMore)
+})
+```
+
+Two choices in that block are worth naming:
+
+- `emitDurableUnfenced` is the durable write with no ownership fence. It is
+  correct here because this program owns no run. The fenced `emitDurable` is
+  what an engine calls, and it needs the run claimed and activated through the
+  journal's `Consensus` strategy, which
+  [`@smthrs/run-store`](/api/run-store) drives; see
+  [Write a fenced lifecycle event](./guides/write-lifecycle-events.md).
+- `flush` is the barrier for the lossy channel only. The durable write was
+  already on disk when its receipt arrived; the lossy one is still in the queue
+  until `flush` returns.
+
+## Run it
+
+```ts
+await Effect.runPromise(program.pipe(Effect.provide(journalLayer), Effect.orDie))
+```
+
+Run it with Node, which strips the types itself on 26.4.0 and later:
+
+```bash
+node quickstart.ts
+```
+
+The output is:
+
+```text
+durable: Accepted seq 0
+0 run.created {"flow":"build","token":"[REDACTED]"}
+1 step.progress {"percent":40,"step":"compile"}
+hasMore: false
+```
+
+## Read the output
+
+Three things in those four lines are the journal's contract, not incidental:
+
+- **`token` came back as `[REDACTED]`.** The value was scrubbed before the row
+  was encoded, so the credential is not in the file and never was. See
+  [Redaction](./concepts/redaction.md).
+- **The `step.progress` keys came back sorted.** The journal persists canonical
+  JSON with object keys sorted, because those bytes are what an idempotency
+  check compares. See
+  [Producer identity and idempotency](./concepts/idempotency.md).
+- **Both entries share one `seq` line.** `seq` is the run's canonical replay
+  order across both channels, so the durable and lossy entries interleave in
+  one history rather than in two.
+
+## Next steps
+
+- [The two channels](./concepts/two-channels.md): which write to reach for, and
+  what each receipt means.
+- [Read and follow a run](./guides/read-a-run.md): paging, the lossless
+  follower, and the local tail.
+- [Test against a real journal](./guides/testing.md): the same layer over an
+  in-memory database, in one call.

package/docs/troubleshooting.md

@@ -0,0 +1,214 @@
+---
+title: "Troubleshooting"
+description: "Every JournalError code, the three failures that are not errors at all, what causes each, and what to change."
+---
+
+Every journal operation fails with a `JournalError` carrying a stable `code`.
+Find your code below. Three common problems produce no error at all, and they
+are at the end.
+
+The full error schema is in the [API reference](./api.md). Every Smithers error
+code across every package, this one's included, is listed under
+[`@smthrs/journal` error codes](/docs/reference/errors/#smthrsjournal).
+
+## sink_failed with "consensus guard failed"
+
+**What happened.** A fenced write, meaning `emitDurable`, `checkpoint`, or
+`compact`, asked the consensus strategy whether the owner still holds the run
+and the strategy could not answer: with `SqlConsensus`, its
+`flows_consensus_leases` table is missing or the database refused the read.
+
+**What to change.** Run the journal's migration set before building the
+journal, or take `@smthrs/engine-store/Migrations`'s `sets`, which is the
+whole durable schema in dependency order. A strategy failure is never
+reported as `fence_lost`. See
+[Installation](./installation.md#what-a-fenced-write-needs).
+
+This is a composition problem, not a race. It is `sink_failed` rather than
+`fence_lost` precisely so the two cannot be confused.
+
+## fence_lost
+
+**What happened.** The run no longer records the owner you supplied. Either
+another process reclaimed it, or its status is no longer `running`.
+
+**What to change.** Stop writing. `fence_lost` is ownership news, not a
+transient failure: retrying with the same token cannot succeed, and reaching for
+`emitDurableUnfenced` to get past it writes exactly the zombie entry the fence
+exists to reject. See [The owner fence](./concepts/owner-fence.md).
+
+## invalid_event
+
+**What happened.** The input violates the journal's contract before anything
+was allocated. The common causes are:
+
+- An identifier that is empty, longer than 1,024 UTF-16 code units, carrying an
+  unpaired surrogate, or carrying a NUL.
+- An owner that is missing, null, or not an `OwnerId`, most often a fractional
+  or negative `pid`.
+- A payload nested past `Redaction.maxDepth` (256 container edges).
+- An encoded `payload` plus `meta` larger than the layer's `maxEntryBytes`, when
+  that option is set.
+
+**What to change.** Fix the argument. A refused event costs no sequence and
+leaves no gap, so nothing needs cleaning up. On a fenced call, note that
+`invalid_event` rather than `fence_lost` means the problem is the token you
+passed, not who owns the run.
+
+## idempotency_conflict
+
+**What happened.** A producer identity `(runId, sourceId, sourceSeq)` was
+re-emitted with different content, under the default `dedupe: "content"`.
+
+**What to change.** Either stop reusing the sequence for a different event, or
+declare `dedupe: "identity"` if the sequence is derived from the event's own
+content and a collision genuinely is the same event observed twice.
+
+If the failure surfaced from `flush` rather than from the emit, the original
+entry had been evicted from the in-process index, so admission had no resident
+entry to compare against. That changes where you see the failure, not whether
+it happens. See
+[Producer identity and idempotency](./concepts/idempotency.md).
+
+## Sequence exhaustion
+
+**What happened.** `invalid_event` reports an exhausted canonical sequence
+range. A queued admission can encounter this at `flush` if another writer
+reaches the limit before the batch commits.
+
+**What to change.** Start a new run. Ordinary overlapping writers do not cause
+sequence conflicts: the queued writer advances an overtaken reservation above
+the committed tail. See [Commit ordering](./concepts/two-channels.md#commit-ordering).
+
+## queue_overflow
+
+**What happened.** The lossy admission queue is full and the layer's `overflow`
+is `reject`.
+
+**What to change.** Raise `capacity`, or choose a dropping policy if losing
+telemetry is acceptable: `drop-newest` returns a `Dropped` receipt, and
+`drop-oldest` returns `Accepted` with an `evicted` count. If the queue is full
+because the writer is not draining, look for a `flush` or a long transaction
+holding the write lock.
+
+## journal_closed
+
+**What happened.** The operation ran against a closed journal. Either the
+layer's scope was released and the admission queue is gone, or the composition
+provided `Journal.layerNoop()`.
+
+**What to change.** Keep the journal's scope open for as long as callers hold
+the service, or provide a real layer where one is required.
+
+## sink_failed
+
+**What happened.** The database failed on a write path. The cause carries the
+underlying `DatabaseError`.
+
+**What to change.** Read the cause first: a missing table is a composition
+problem, and a locked or unreadable file is an operational one. A transient
+outage is not permanent for the journal. The writer fiber survives a failed
+batch, and `emitDurable` opens its own transaction inline, so the durable
+channel works again as soon as the database is healthy.
+
+## read_failed
+
+**What happened.** The database failed on a read path: `entries`, `stream`, or
+the floor read behind them.
+
+**What to change.** Treat it as a retryable storage condition rather than a
+journal defect. It is a separate code from `unknown` for exactly that reason.
+
+## decode_failed
+
+**What happened.** A persisted row no longer matches the schema that reads it.
+
+**What to change.** This is schema drift, not a caller mistake. Check that the
+migration set the database was built with matches the package version reading
+it.
+
+## projection_failed
+
+**What happened.** A projection's `reduce` failed, which ends the projection
+stream.
+
+**What to change.** Fix the reducer. The journal is unaffected: later emissions
+still succeed, and a fresh `project` call folds the same history again from the
+start. Make a reducer that meets an entry it does not recognize return the state
+unchanged.
+
+## checkpoint_invalid
+
+**What happened.** One of two things:
+
+- `checkpoint` was given a `seq` that names no committed entry of the run, or
+  one at or below the run's compaction floor. In the second case the failure
+  carries the floor in `checkpointSeq`.
+- `compact` found no checkpoint to truncate below, either at all or at the
+  `upTo` you named.
+
+**What to change.** Checkpoint a committed sequence above the floor before
+compacting to it. See [Compact a long-running run](./guides/compact-a-run.md).
+
+## reader_behind
+
+**What happened.** `compact` refused because a live in-process `stream` still
+needs a sequence the truncation would delete. `checkpointSeq` is the checkpoint
+it declined to truncate below.
+
+**What to change.** Let the reader catch up, or drop it, then compact again.
+Readers in other processes cannot be seen from here, so they are protected by
+the read-side `compacted` guard rather than by this refusal.
+
+## compacted
+
+**What happened.** A read started below the run's compaction floor.
+`checkpointSeq` carries the floor.
+
+**What to change.** Resync from the checkpoint: read `latestCheckpoint(runId)`,
+apply its `state`, and continue from
+`stream({ runId, afterSequence: checkpoint.seq })`. The worked version is in
+[Compact a long-running run](./guides/compact-a-run.md#resync-a-reader).
+
+## unknown
+
+**What happened.** A journal defect that no other code classifies.
+
+**What to change.** This code is reserved for genuinely unclassified failures.
+Report it with the message and cause.
+
+## A stream or a projection never finishes
+
+**What happened.** `stream` and `project` replay a run's history and then follow
+its tail. Following is the point, so neither completes on its own, and a
+`Stream.runCollect`, `Stream.runDrain`, or `Stream.runLast` with no bound hangs
+forever.
+
+**What to change.** Bound the stream: `Stream.take(n)` for a snapshot,
+`Stream.takeUntil` for a condition, or a forked fiber you interrupt when the
+view is torn down. See [Fold a run into a projection](./guides/fold-a-projection.md#bound-the-stream).
+
+## A write hangs inside transact
+
+**What happened.** The body of a `transact` called `flush`, or did work that
+waits on something outside the transaction.
+
+**What to change.** Keep a `transact` body to storage work only: no flow
+bodies, no host calls, no waits. `flush` in particular waits on the queued
+lossy writer, which is waiting on the transaction you are holding. See
+[Commit state and its entry together](./guides/commit-state-and-entry.md).
+
+## Entries are missing from a live tail
+
+**What happened.** `changes` is a bounded sliding buffer sized by the layer's
+`capacity`. A slow subscriber loses entries with no error and no gap signal.
+`changes` also carries only the entries this process commits; another writer on
+the same database publishes into its own buffer.
+
+**What to change.** Use `stream` wherever a missing entry would be a
+correctness bug. It is the lossless follower, it reports sink losses, and it
+rechecks the durable tail rather than trusting one process's buffer. See
+[Read and follow a run](./guides/read-a-run.md).
+
+If a lossy entry is missing from a read rather than from a tail, `flush` first:
+`emitLossy` returns before its entry is committed.

package/package.json

@@ -1,6 +1,266 @@
 {
   "name": "@smthrs/journal",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
+  "type": "module",
+  "version": "1.0.0-rc.3",
+  "license": "MIT",
+  "smthrs": {
+    "group": "engine"
+  },
+  "description": "Effect services for the Smithers event journal: the immutable run history and the owner fence on its durable channel",
+  "homepage": "https://github.com/smithersai/smithers",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/smithersai/smithers.git",
+    "directory": "packages/smithers/flows/journal"
+  },
+  "bugs": {
+    "url": "https://github.com/smithersai/smithers/issues"
+  },
+  "tags": [
+    "typescript",
+    "effect",
+    "journal",
+    "durability"
+  ],
+  "keywords": [
+    "typescript",
+    "effect",
+    "journal",
+    "durability"
+  ],
+  "engines": {
+    "node": ">=26.4.0"
+  },
+  "sideEffects": [],
+  "exports": {
+    "./package.json": "./package.json",
+    ".": {
+      "import": {
+        "types": "./dist/esm/index.d.ts",
+        "default": "./dist/esm/index.js"
+      },
+      "require": {
+        "types": "./dist/cjs/index.d.ts",
+        "default": "./dist/cjs/index.js"
+      }
+    },
+    "./internal/*": null,
+    "./migrations/*": null,
+    "./*/index": null,
+    "./Journal": {
+      "import": {
+        "types": "./dist/esm/Journal.d.ts",
+        "default": "./dist/esm/Journal.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Journal.d.ts",
+        "default": "./dist/cjs/Journal.js"
+      }
+    },
+    "./JournalEvent": {
+      "import": {
+        "types": "./dist/esm/JournalEvent.d.ts",
+        "default": "./dist/esm/JournalEvent.js"
+      },
+      "require": {
+        "types": "./dist/cjs/JournalEvent.d.ts",
+        "default": "./dist/cjs/JournalEvent.js"
+      }
+    },
+    "./EngineEvent": {
+      "import": {
+        "types": "./dist/esm/EngineEvent.d.ts",
+        "default": "./dist/esm/EngineEvent.js"
+      },
+      "require": {
+        "types": "./dist/cjs/EngineEvent.d.ts",
+        "default": "./dist/cjs/EngineEvent.js"
+      }
+    },
+    "./JournalMetrics": {
+      "import": {
+        "types": "./dist/esm/JournalMetrics.d.ts",
+        "default": "./dist/esm/JournalMetrics.js"
+      },
+      "require": {
+        "types": "./dist/cjs/JournalMetrics.d.ts",
+        "default": "./dist/cjs/JournalMetrics.js"
+      }
+    },
+    "./Migrations": {
+      "import": {
+        "types": "./dist/esm/Migrations.d.ts",
+        "default": "./dist/esm/Migrations.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Migrations.d.ts",
+        "default": "./dist/cjs/Migrations.js"
+      }
+    },
+    "./OwnerId": {
+      "import": {
+        "types": "./dist/esm/OwnerId.d.ts",
+        "default": "./dist/esm/OwnerId.js"
+      },
+      "require": {
+        "types": "./dist/cjs/OwnerId.d.ts",
+        "default": "./dist/cjs/OwnerId.js"
+      }
+    },
+    "./Projection": {
+      "import": {
+        "types": "./dist/esm/Projection.d.ts",
+        "default": "./dist/esm/Projection.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Projection.d.ts",
+        "default": "./dist/cjs/Projection.js"
+      }
+    },
+    "./RedactedLogger": {
+      "import": {
+        "types": "./dist/esm/RedactedLogger.d.ts",
+        "default": "./dist/esm/RedactedLogger.js"
+      },
+      "require": {
+        "types": "./dist/cjs/RedactedLogger.d.ts",
+        "default": "./dist/cjs/RedactedLogger.js"
+      }
+    },
+    "./Redaction": {
+      "import": {
+        "types": "./dist/esm/Redaction.d.ts",
+        "default": "./dist/esm/Redaction.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Redaction.d.ts",
+        "default": "./dist/cjs/Redaction.js"
+      }
+    },
+    "./SqlJournal": {
+      "import": {
+        "types": "./dist/esm/SqlJournal.d.ts",
+        "default": "./dist/esm/SqlJournal.js"
+      },
+      "require": {
+        "types": "./dist/cjs/SqlJournal.d.ts",
+        "default": "./dist/cjs/SqlJournal.js"
+      }
+    },
+    "./index": {
+      "import": {
+        "types": "./dist/esm/index.d.ts",
+        "default": "./dist/esm/index.js"
+      },
+      "require": {
+        "types": "./dist/cjs/index.d.ts",
+        "default": "./dist/cjs/index.js"
+      }
+    },
+    "./test/Notifying": {
+      "import": {
+        "types": "./dist/esm/test/Notifying.d.ts",
+        "default": "./dist/esm/test/Notifying.js"
+      },
+      "require": {
+        "types": "./dist/cjs/test/Notifying.d.ts",
+        "default": "./dist/cjs/test/Notifying.js"
+      }
+    },
+    "./test/TestJournal": {
+      "import": {
+        "types": "./dist/esm/test/TestJournal.d.ts",
+        "default": "./dist/esm/test/TestJournal.js"
+      },
+      "require": {
+        "types": "./dist/cjs/test/TestJournal.d.ts",
+        "default": "./dist/cjs/test/TestJournal.js"
+      }
+    },
+    "./JournalGeneration": {
+      "import": {
+        "types": "./dist/esm/JournalGeneration.d.ts",
+        "default": "./dist/esm/JournalGeneration.js"
+      },
+      "require": {
+        "types": "./dist/cjs/JournalGeneration.d.ts",
+        "default": "./dist/cjs/JournalGeneration.js"
+      }
+    },
+    "./Consensus": {
+      "import": {
+        "types": "./dist/esm/Consensus.d.ts",
+        "default": "./dist/esm/Consensus.js"
+      },
+      "require": {
+        "types": "./dist/cjs/Consensus.d.ts",
+        "default": "./dist/cjs/Consensus.js"
+      }
+    },
+    "./SqlConsensus": {
+      "import": {
+        "types": "./dist/esm/SqlConsensus.d.ts",
+        "default": "./dist/esm/SqlConsensus.js"
+      },
+      "require": {
+        "types": "./dist/cjs/SqlConsensus.d.ts",
+        "default": "./dist/cjs/SqlConsensus.js"
+      }
+    }
+  },
+  "files": [
+    "src/**/*.ts",
+    "dist/**/*.js",
+    "dist/**/*.js.map",
+    "dist/**/*.d.ts",
+    "dist/**/*.d.ts.map",
+    "dist/**/package.json",
+    "LICENSE",
+    "README.md",
+    "CHANGELOG.md",
+    "docs/"
+  ],
+  "publishConfig": {
+    "access": "public",
+    "provenance": true,
+    "tag": "next"
+  },
+  "dependencies": {
+    "@smthrs/database": "1.0.0-rc.3"
+  },
+  "devDependencies": {
+    "@effect/language-service": "0.87.2",
+    "@effect/sql-sqlite-node": "4.0.0-rc.115",
+    "@effect/vitest": "4.0.0-rc.115",
+    "@eslint/js": "9.39.5",
+    "@smthrs/flow": "1.0.0-rc.3",
+    "@types/node": "26.4.1",
+    "@vitest/coverage-v8": "5.0.0",
+    "dprint": "0.57.1",
+    "effect": "4.0.0-rc.115",
+    "esbuild": "0.28.2",
+    "eslint": "9.39.5",
+    "eslint-import-resolver-typescript": "4.4.5",
+    "eslint-plugin-import": "2.32.0",
+    "eslint-plugin-jsdoc": "64.3.4",
+    "eslint-plugin-unicorn": "65.0.0",
+    "madge": "8.0.0",
+    "typescript": "5.9.3",
+    "typescript-eslint": "8.69.0",
+    "vitest": "5.0.0",
+    "fast-check": "4.9.0"
+  },
+  "peerDependencies": {
+    "effect": "4.0.0-rc.115"
+  },
+  "scripts": {
+    "lint": "eslint src --max-warnings=0 && dprint check",
+    "format": "dprint fmt",
+    "build": "node scripts/build.mjs",
+    "check": "tsc -b tsconfig.json && tsc -p tsconfig.test.json --noEmit",
+    "circular": "node scripts/circular.mjs",
+    "test": "vitest",
+    "coverage": "node ../database/scripts/test-matrix.mjs",
+    "test:matrix": "node ../database/scripts/test-matrix.mjs"
+  }
 }