@smthrs/journal 0.0.0-stage → 1.0.0-rc.3

package/docs/guides/read-a-run.md

@@ -0,0 +1,139 @@
+---
+title: "Read and follow a run"
+description: "Choose between entries, stream, and changes; page a long history; and know which reader reports a loss and which one drops entries silently."
+sidebar:
+  order: 3
+---
+
+The journal has three reads, and they differ in exactly one dimension: what
+happens when the reader cannot keep up. Pick by that.
+
+| Read      | Shape                                     | Behind a fast writer          |
+| --------- | ----------------------------------------- | ----------------------------- |
+| `entries` | one page, in sequence order               | nothing is lost; you page     |
+| `stream`  | replay then follow, per run               | lossless; reports a sink loss |
+| `changes` | live tail, every run this process commits | silently drops entries        |
+
+## Page a history with entries
+
+`entries` reads one page in sequence order and reports whether another follows.
+Use it for a poller, an export, or any bounded read:
+
+```ts
+import { Journal, JournalEvent } from "@smthrs/journal"
+import * as Effect from "effect/Effect"
+
+const readAll = (runId: JournalEvent.RunId) =>
+  Effect.gen(function*() {
+    const journal = yield* Journal.Journal
+    let after: JournalEvent.Seq | undefined = undefined
+
+    for (;;) {
+      const page: Journal.EntriesPage = after === undefined
+        ? yield* journal.entries({ runId, limit: 500 })
+        : yield* journal.entries({ runId, after, limit: 500 })
+
+      for (const entry of page.entries) console.log(entry.seq, entry.eventType)
+
+      const last = page.entries.at(-1)
+      if (!page.hasMore || last === undefined) break
+      after = last.seq
+    }
+  })
+```
+
+`limit` is capped at `Journal.maxEntriesLimit` (10,000). A page is decoded into
+memory in full before the caller sees its first entry, so an unbounded limit
+would let one call materialize a whole run.
+
+For a nonempty list of at most 64 exact event types, use the optional `eventTypes` field on `entries`:
+
+```ts
+const finished = yield * journal.entries({ runId, eventTypes: ["build.finished", "build.failed"], limit: 100 })
+```
+
+This is a new public query option. Filtering happens in SQL before the page
+limit, using the journal's `(run_id, event_type, seq)` index. `SqlJournal` is the
+existing SQLite-specific adapter and explicitly selects that index: SQLite can
+otherwise choose the run/sequence index for an `IN` query and scan unrelated
+history. Multiple types may sort matching rows by sequence. The dialect-specific
+hint stays inside `SqlJournal`; `Journal.Service` remains platform-independent. `after` remains the
+canonical run sequence, and `hasMore` counts only matching events. Gaps between
+matching sequences are expected. Duplicate types do not duplicate entries. No match returns an empty final page. Omitting
+the option preserves the existing full-run read. The event type uses the same
+bounded identifier schema as journal emission.
+
+Filtering does not restore compacted evidence: a cursor below the run's
+compaction floor still fails with `compacted`, even when no matching row remains.
+Apply journal migration `0005_run_event_type` before using the updated service.
+Custom `Journal.Service` adapters must implement the filter before a caller
+relies on it; an older adapter may silently ignore an unknown option.
+
+## Follow a run with stream
+
+`stream` replays the run's durable history from `afterSequence`, then follows
+its committed tail. It is the lossless reader, and the one to use wherever a
+missing entry would be a correctness bug:
+
+```ts
+import * as Stream from "effect/Stream"
+
+const follow = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  yield* journal.stream({ runId }).pipe(
+    Stream.runForEach((entry) => Effect.log(`${entry.seq} ${entry.eventType}`))
+  )
+})
+```
+
+Three properties are worth knowing before you use it:
+
+- It pages the history rather than materializing it, so the first entry arrives
+  after one page rather than after the whole run.
+- It reports a sink loss that happens while it is following, to every consumer
+  that was following at the time.
+- **It never completes.** Following is the point, so bound the stream yourself:
+  `Stream.take`, `Stream.takeUntil`, or a forked fiber you interrupt.
+
+A cursor below the run's compaction floor fails with `compacted` and the
+sequence to resync from, never a silently shortened history. See
+[Checkpoints and compaction](../concepts/compaction.md).
+
+## Tail this process with changes
+
+`changes` is a scoped subscription to the entries this process commits, across
+every run:
+
+```ts
+import * as PubSub from "effect/PubSub"
+
+const tail = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  const subscription = yield* journal.changes
+  const entry = yield* PubSub.take(subscription)
+  console.log(entry.runId, entry.seq)
+}).pipe(Effect.scoped)
+```
+
+Two limits make it a local convenience rather than a reader of record:
+
+- It is a bounded sliding buffer sized by the layer's `capacity`. A slow
+  subscriber loses entries with no error and no gap signal.
+- Only this process's commits arrive. Another writer on the same database
+  publishes into its own buffer, which is the second reason `stream` exists: it
+  rechecks the durable tail.
+
+Entries published to `changes` are deeply frozen, so one subscriber cannot
+mutate the value another subscriber is about to read.
+
+## Make a lossy write readable first
+
+A `flush` before a read makes queued lossy entries visible. A durable entry
+needs no flush; it was committed when its receipt arrived.
+
+## Next steps
+
+- [Fold a run into a projection](./fold-a-projection.md) to turn entries into a
+  served view.
+- [Compact a long-running run](./compact-a-run.md) for what a reader below the
+  floor should do.

package/docs/guides/redact-log-output.md

@@ -0,0 +1,94 @@
+---
+title: "Keep credentials out of log output"
+description: "Install RedactedLogger so the journal's redaction rules also cover the terminal, the JSON log document, and the spans a tracer logger exports."
+sidebar:
+  order: 6
+---
+
+The journal redacts on the write path, so a credential an action logs or
+returns never reaches a committed row. The operator's terminal is the other
+place the same credential surfaces: every built-in Effect logger renders through
+the fiber's `Console`, so an action that hands a token to `Effect.logInfo`
+writes it to stderr in full, and it reaches whatever collects that stream.
+
+`RedactedLogger` closes that half with the same rules rather than a second rule
+set. A log line is a diagnostic, so it takes `Redaction.diagnosticRules`: the
+default rules plus the spellings too broad for a durable row.
+
+## Install the layer
+
+```ts
+import { RedactedLogger } from "@smthrs/journal"
+import * as Layer from "effect/Layer"
+
+const logging = RedactedLogger.layer()
+```
+
+The layer composes over whatever loggers are already installed rather than
+choosing one, so an operator keeps the format they had: their own logger, their
+own layout. A host that installed its own logger keeps it too.
+
+Pass `Redaction.Options` to extend or replace the rules, exactly as you would
+for the journal write path. Spread `Redaction.diagnosticRules` to add to them.
+
+## What it covers
+
+Two halves carry load, and both are necessary:
+
+- **The log event itself.** The message, the cause, and the log annotations are
+  redacted before the wrapped logger is handed them. This matters because a
+  logger is free to read the event instead of rendering it, and one in Effect's
+  default set does: `Logger.tracerLogger` publishes the message as a span event
+  name and the pretty cause as the span's `effect.cause` attribute, and never
+  touches `Console`. Redacting only the console would hide a credential on
+  stderr and export it in clear to whatever OTLP collector is configured.
+  Cause stack annotations are redacted too: span names, lazy stack text, parent
+  frames, and interruptor frames. The original cause is unchanged; the delegate
+  receives rebuilt reasons with redacted frame chains.
+- **The fiber's console.** The wrapped logger is handed a view of the fiber
+  whose `Console` is a redacting console, so whatever the logger renders for
+  itself, a pretty line or a JSON document, passes through the rules on its way
+  to the stream.
+
+Cost is bounded by the rules: each rule is linear in the length of the line,
+with no catastrophic backtracking.
+
+## Two differences from the write path
+
+**A too-deep value is named, not refused.** The journal throws on a payload
+past `Redaction.maxDepth` so the write fails with a typed `invalid_event`. A
+logger cannot: a throw there is caught one frame up and replaces every argument
+on the line with a placeholder, so one deep member would cost the operator the
+whole line. `wrap` therefore defaults `onTooDeep` to `"name"`. A caller passing
+its own `onTooDeep` still wins.
+
+**An `Error` is rebuilt as a plain `Error`.** Its `message` and `stack` live on
+non-enumerable properties, so the own-key rebuild the write path uses would
+print `{}` where the operator expected a failure. The copy keeps the name,
+message, stack, and own members with the rules applied, but not the class. A
+copy built on the original's prototype would inherit everything the prototype
+defines, and `name`, `cause`, `toJSON`, `Symbol.toStringTag`, and
+`nodejs.util.inspect.custom` each carried a credential out that way.
+
+## Wrap a single logger
+
+Use `wrap` when you are constructing a logger rather than composing a layer:
+
+```ts
+import { RedactedLogger } from "@smthrs/journal"
+import * as Logger from "effect/Logger"
+
+const json = RedactedLogger.wrap(Logger.formatJson)
+```
+
+`wrap` is idempotent. Both the CLI and the durable runtime install the layer,
+and a detached run is both at once, so a logger this module already wrapped is
+returned unchanged rather than paying the rules twice per line.
+`RedactedLogger.isRedacted` reports whether a logger is already wrapped.
+
+## Next steps
+
+- [Redaction](../concepts/redaction.md) for the rules, the bounds, and where
+  redaction deliberately stops.
+- [Observability](/docs/guides/observability/) for exporting the spans this
+  logger writes into.

package/docs/guides/testing.md

@@ -0,0 +1,124 @@
+---
+title: "Test against a real journal"
+description: "Use TestJournal for the production journal over in-memory SQLite, Notifying to inject a crash at an exact transition, and the noop layer for a composition that must not write."
+sidebar:
+  order: 7
+---
+
+The journal is deterministic to test because its only nondeterministic input is
+the database, and the database is a layer. This package ships two doubles and
+one stub, and none of them reimplements the journal.
+
+Neither double is exported from the root entry point, so import both by
+subpath. `TestJournal` binds a Node SQLite database and runs on Node only;
+`Notifying` wraps a service and is platform independent.
+
+## Run the real journal in memory
+
+`TestJournal.layer` provides the production `SqlJournal` over a fresh in-memory
+database, with migrations already run:
+
+```ts
+import * as TestJournal from "@smthrs/journal/test/TestJournal"
+
+const layer = TestJournal.layer()
+```
+
+`TestJournalOptions` is `SqlJournalOptions` with `capacity` and `overflow` made
+optional, so every option forwards to `SqlJournal.layer` unchanged and a suite
+can exercise index-bound eviction, the `maxEntryBytes` entry bound, a custom or
+noop redactor, and the compaction policy through the bundle instead of
+hand-assembling the layer stack the bundle exists to hide:
+
+```ts
+const strict = TestJournal.layer({
+  capacity: 8,
+  overflow: "drop-oldest",
+  sourceEventCache: 2,
+  redact: Redaction.makeNoop()
+})
+```
+
+The defaults are `capacity: 1024` and `overflow: "reject"`.
+
+Fenced writes need the run claimed and activated through the journal's
+`Consensus` strategy. This bundle fences through `SqlConsensus`, so a suite
+that exercises `emitDurable`, `checkpoint`, or `compact` takes the run through
+`Consensus.claim` and `activate` first, or takes the whole engine bundle
+instead.
+
+## Get every store over one database
+
+`@smthrs/journal/test/TestJournal` provides the journal and nothing else. For
+the journal, run, attempt, and cache services over one database, take
+`@smthrs/engine-store/test/TestStores` from
+[`@smthrs/engine-store`](/api/engine-store). That bundle includes the run-store
+migrations, so the fence has a table to read.
+
+## Inject a crash at an exact transition
+
+`Notifying` wraps a record-of-Effect-methods service so a hook fires around
+every operation. The hook runs in the calling fiber, so a hook that dies,
+interrupts, or awaits a `Latch` injects crashes, fence loss, and exact
+sequencing at any durable transition point:
+
+```ts
+import { Journal } from "@smthrs/journal"
+import * as Notifying from "@smthrs/journal/test/Notifying"
+import * as Effect from "effect/Effect"
+
+const crashAfterCommit = Notifying.layer(
+  Journal.Journal,
+  (op, order) => op === "emitDurable" && order === "after" ? Effect.die("crash") : Effect.void
+)
+```
+
+Three details shape how you write a hook:
+
+- `op` is the service method name and `args` are the call arguments, so one
+  hook can select the exact call it cares about.
+- The `after` firing only happens when the operation succeeds. A failed
+  operation is already the interesting event.
+- Effect-returning methods and plain `Effect` properties are hooked;
+  non-Effect members, such as the `Stream`-returning `stream` and `project`,
+  delegate untouched.
+
+`Notifying.layer` reads the real service from the environment and re-provides
+the wrapper under the same key, so an existing stack gains a listener without
+touching its construction. `Notifying.wrap` is the direct form, and it is
+generic over any such service, not just the journal.
+
+## Assert that nothing wrote
+
+`Journal.layerNoop()` provides a closed journal whose every operation fails
+`journal_closed`. Use it for a composition that must not write, or to prove a
+code path never reaches the journal. Override individual methods to record
+calls:
+
+```ts
+import { Journal } from "@smthrs/journal"
+import * as Effect from "effect/Effect"
+
+const recorded: Array<string> = []
+
+const layer = Journal.layerNoop({
+  emitLossy: (input) => {
+    recorded.push(input.eventType)
+    return Effect.succeed({ _tag: "Accepted", seq: 0, sourceSeq: 0 } as Journal.EmitReceipt)
+  }
+})
+```
+
+The closed stub's `transact` runs its effect directly, because a stub with no
+sink has no transaction to open and no crash window to close. A double that
+models rollback overrides it.
+
+## Remember the flush barrier
+
+A test that writes on the lossy channel and then reads must `flush` in between.
+A durable write needs no flush; it was committed when its receipt arrived.
+
+## Next steps
+
+- [The two channels](../concepts/two-channels.md) for what each receipt proves.
+- [Testing flows](/docs/guides/testing-flows/) for the layer above this one.

package/docs/guides/write-lifecycle-events.md

@@ -0,0 +1,110 @@
+---
+title: "Write a fenced lifecycle event"
+description: "Emit a durable lifecycle entry with an OwnerId, handle fence_lost as ownership news rather than an error, and know when the unfenced channel is correct."
+sidebar:
+  order: 1
+---
+
+A lifecycle event is the evidence that a run moved: created, started,
+finished, failed. It belongs on the durable channel, and the durable channel is
+fenced, so the write also proves the process still owns the run.
+
+## Before you start
+
+The fence is the journal's `Consensus` strategy: a fenced write commits only
+while the strategy records the owner as holding the run, so the run must have
+been claimed and activated first. [`@smthrs/run-store`](/api/run-store) drives
+that for a durable run. See
+[Installation](../installation.md#what-a-fenced-write-needs).
+
+## Emit the entry
+
+Pass the owner your process holds for the run:
+
+```ts
+import { Journal, JournalEvent } from "@smthrs/journal"
+import type * as OwnerId from "@smthrs/journal/OwnerId"
+import * as Effect from "effect/Effect"
+
+const runId = "run-1" as JournalEvent.RunId
+const sourceId = "engine" as JournalEvent.SourceId
+
+const owner: OwnerId.OwnerId = {
+  hostId: "host-1",
+  pid: process.pid,
+  nonce: "run-1-claim"
+}
+
+const started = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  return yield* journal.emitDurable({
+    runId,
+    sourceId,
+    eventType: "run.started",
+    payload: { attempt: 1 }
+  }, owner)
+})
+```
+
+The receipt arrives after the transaction commits, so the `seq` it carries is
+already on disk. Do not advance the run or expose its result before this
+returns: that ordering is the whole reason the channel exists.
+
+## Handle a lost fence
+
+`fence_lost` is not an error to retry. It is news: another process owns this
+run now, and this one must stop writing.
+
+```ts
+const guarded = started.pipe(
+  Effect.catchTag("@smthrs/journal/JournalError", (failure) =>
+    failure.code === "fence_lost"
+      ? Effect.logWarning("run reclaimed by another owner; standing down").pipe(Effect.as(undefined))
+      : Effect.fail(failure))
+)
+```
+
+Retrying with the same token cannot succeed, and reaching for
+`emitDurableUnfenced` to get past it writes exactly the zombie entry the fence
+exists to reject.
+
+A neighbouring code needs a different response. `invalid_event` on a fenced
+call means the owner you passed is not an `OwnerId`, most often a fractional or
+negative `pid`. Fix the argument.
+
+## Give the producer a stable identity
+
+Supply `sourceSeq` when the caller can derive a stable number for the event.
+The journal then dedupes a retry against the committed row instead of writing a
+second entry:
+
+```ts
+const finished = Effect.gen(function*() {
+  const journal = yield* Journal.Journal
+  return yield* journal.emitDurable({
+    runId,
+    sourceId,
+    sourceSeq: 7 as JournalEvent.SourceSeq,
+    eventType: "run.finished",
+    payload: { outcome: "succeeded" }
+  }, owner)
+})
+```
+
+An exact retry returns `Duplicate` with the original `seq`. A reused sequence
+carrying different content fails `idempotency_conflict`, unless the input
+declares `dedupe: "identity"`. See
+[Producer identity and idempotency](../concepts/idempotency.md).
+
+## When to use the unfenced channel
+
+`emitDurableUnfenced` is correct for an admission that owns no run and is
+first-writer-wins by design: an external trigger delivered by a sweeper, an
+import, a repair tool. It is the wrong answer to a `fence_lost`.
+
+## Next steps
+
+- [Commit state and its entry together](./commit-state-and-entry.md) when the
+  entry describes a state transition that must not disagree with it.
+- [The owner fence](../concepts/owner-fence.md) for what the fence checks and
+  why the token lives in this package.

package/docs/installation.md

@@ -0,0 +1,133 @@
+---
+title: "Installation"
+description: "Install @smthrs/journal, choose a database layer, install the migration sets a journal needs, and learn which extra table the fenced write path reads."
+sidebar:
+  order: 1
+---
+
+## Install the package
+
+Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
+
+The package requires Node.js 26.4.0 or later and ships as both ESM and
+CommonJS with TypeScript declarations. Its only runtime dependencies are
+[`effect`](https://effect.website) and [`@smthrs/database`](/api/database).
+Composing a journal layer imports from both of them by name, so install them as
+direct dependencies rather than relying on them being hoisted.
+
+`effect` is a peer dependency pinned at `4.0.0-rc.115`. Install exactly that
+version: two copies of `effect` in one program are two sets of service tags, so
+a journal layer built against one copy cannot be provided to a program holding
+the other, and the mismatch shows up as a missing service rather than as a
+version error.
+
+## Add a database
+
+`SqlJournal.layer` needs two services from `@smthrs/database`: a
+`SqlClient.SqlClient` to read through and a `DurableWriter` to write through.
+On Node, `NodeDatabase.layer` provides the client and `DurableWriter.layer()`
+provides the writer. This composition selects the optional Node SQLite driver:
+
+```bash
+pnpm add @effect/sql-sqlite-node@4.0.0-rc.115
+```
+
+```ts
+import * as DurableWriter from "@smthrs/database/DurableWriter"
+import * as NodeDatabase from "@smthrs/database/node/NodeDatabase"
+import * as Layer from "effect/Layer"
+
+const database = Layer.provideMerge(
+  DurableWriter.layer(),
+  NodeDatabase.layer({ filename: "flows.db" })
+)
+```
+
+SQLite is the supported backend at `1.0.0-rc.0`. PostgreSQL and PGlite are
+not; see [storage compatibility](/docs/migration/compatibility#storage).
+
+## Install the migrations
+
+`Migrations.layer` creates this package's two tables and nothing else:
+`flows_journal_events` with its event-type index, and
+`flows_journal_checkpoints`.
+
+```ts
+import { Migrations, SqlJournal } from "@smthrs/journal"
+import * as Layer from "effect/Layer"
+
+const journal = SqlJournal.layer({ capacity: 1024, overflow: "reject" }).pipe(
+  Layer.provide(Layer.provideMerge(Migrations.layer, database))
+)
+```
+
+An application that also needs run, cache, or engine tables composes migration
+sets rather than stacking layers. `Migrations.set` is the journal's namespaced
+set, and `@smthrs/database`'s `Migrations.layer` runs several sets over one
+`flows_migrations` table, giving each package a reserved id block so two
+packages' `0001_initial` cannot collide:
+
+```ts
+import * as DatabaseMigrations from "@smthrs/database/Migrations"
+import * as JournalMigrations from "@smthrs/journal/Migrations"
+import * as RunStoreMigrations from "@smthrs/run-store/Migrations"
+
+const migrations = DatabaseMigrations.layer([
+  JournalMigrations.set,
+  RunStoreMigrations.set
+])
+```
+
+`@smthrs/engine-store/Migrations` already exports `sets`, the whole durable
+schema a durable engine needs, in dependency order. Take that list if you are
+composing an engine rather than a standalone journal.
+
+## What a fenced write needs
+
+`emitDurable`, `checkpoint`, and `compact` gate their write on the `Consensus`
+strategy the journal was built with. `SqlJournal.layer` fences through
+`SqlConsensus`, whose lease table the journal's own migrations create, so the
+fenced channel needs nothing beyond this package's migration set; an owner
+takes the run through the strategy's `claim` and `activate`, which
+[`@smthrs/run-store`](/api/run-store) drives for a durable run.
+
+`emitLossy`, `emitDurableUnfenced`,
+`entries`, `stream`, `project`, and `flush` read no table but the journal's
+own, so a standalone journal serves them without run-store.
+
+For why the fence exists and when the unfenced channel is the correct one, see
+[the owner fence](./concepts/owner-fence.md).
+
+## Import forms
+
+The root entry point re-exports every module as a namespace:
+
+```ts
+import { Journal, JournalEvent, Migrations, Redaction, SqlJournal } from "@smthrs/journal"
+```
+
+Each module is also importable from its own subpath, which is the form the API
+reference uses:
+
+```ts
+import * as Journal from "@smthrs/journal/Journal"
+import * as SqlJournal from "@smthrs/journal/SqlJournal"
+```
+
+The root is written against the driver-neutral `@smthrs/database` contract and
+bundles for the browser. Neither test double is exported from the root, so both
+are imported by subpath. `TestJournal` binds a Node SQLite database and runs on
+Node only; `Notifying` wraps a service and is platform independent:
+
+```ts
+import * as Notifying from "@smthrs/journal/test/Notifying"
+import * as TestJournal from "@smthrs/journal/test/TestJournal"
+```
+
+Three subpath forms are not public and are blocked in the export map:
+`@smthrs/journal/internal/*`, `@smthrs/journal/migrations/*`, and
+`@smthrs/journal/*/index`. `@smthrs/journal/package.json` is exported.
+
+## Next step
+
+Write and read one run in the [Quickstart](./quickstart.md).